I’m currently investigating a memory leak that I believe to be linked to Hints. Looking across our fleet of installs, the common pattern is that installs Sideboards that monitor extension state + Hints without endpoints. Exists in several versions, testing below is 16.2
Orange > 1000 handsets, no subscriptions. Blue < 200 endpoints, a number of hints without endpoints and a lot of subscriptions. End of graph is removal of hints without endpoints.
Running under valgrind + don’t optimise set my lab environment with several hundred hints and approximately 90 subscriptions to hints without endpoints and a couple with, results in the following after roughly 24 hours.
Memory start:
89ca75f1acd9 1.66% 613.5MiB / 31.26GiB 1.92% 0B / 0B 174MB / 0B 69
Memory finish:
89ca75f1acd9 1.60% 958.9MiB / 31.26GiB 3.00% 0B / 0B 176MB / 0B 64
Example of watcher growth:
6010@default : PJSIP/805EC00F6D1F State:Idle Presence:not_set Watchers 0
6011@default : PJSIP/001565F655C7 State:Idle Presence:not_set Watchers 1
db6b57d1-84e3-47c3-a: State:Unavailable Presence:not_set Watchers 0
7002@default : State:Unavailable Presence:not_set Watchers 70
7003@default : State:Unavailable Presence:not_set Watchers 70
7000@default : State:Unavailable Presence:not_set Watchers 17
7001@default : State:Unavailable Presence:not_set Watchers 70
7006@default : State:Unavailable Presence:not_set Watchers 70
7007@default : State:Unavailable Presence:not_set Watchers 70
7004@default : State:Unavailable Presence:not_set Watchers 70
6fa36fe5-2d79-4c1d-9: State:Unavailable Presence:not_set Watchers 0
7005@default : State:Unavailable Presence:not_set Watchers 70
268d8d5f-2def-4714-8: State:Unavailable Presence:not_set Watchers 0
7008@default : State:Unavailable Presence:not_set Watchers 69
7009@default : State:Unavailable Presence:not_set Watchers 70
The attached valgrind.txt
doesn’t appear to contain anything relevant. The current workaround is not populating hints that don’t have endpoints. However while the hardware could be configured not to subscribe to extensions without endpoints, our softphone solution doesn’t appear as flexible.
Doing a core restart gracefully
frees the memory + resets the watcher count.