We are using Asterisk hints to track endpoints’ states using custom hints. There are around 7000 active subscriptions at a time.
When the watchers unsubscribe from the hints, after ~5000 unsubscribes Asterisk hangs. There are tcptls.c: FILE * open failed!
WARNINGs in logs, calls in progress are not dropped, but new ones can’t be established. Most of the registrations become invalid - clients are dropped with error 503.
There are no other warnings or messages provided by asterisk neither in it’s own logs nor in syslog.
Asterisk’s version is 11.23.1. I know that is an old unsupported version, migrating to ast18 is in progress, but we have to keep this version running in production for a while.
That sounds like a file descriptor leak. Run “ls /proc/ppppp/fd” where ppppp is the process ID of the main Asterisk process, and at a shell prompt. If that is continually growing you have a leak, Using ls -l, rather than ls may give you a hint as to what represents leaked descriptors and what reflects normal usage.
However, your only choices, are periodic restarts, discontinuing the use of the feature that is causing the leak, or increasing the ulimit, assuming that you haven’t hit a global limit. The ulimit for Asterisk will be set during the startup process, so you will need to restart it at least once.
If there is a leak, any increased limit will, eventually, be exceeded.
Thanks @david551 for the tip, and sorry for the late answer. We checked file descriptor leak, seems like that is not the root of the issue as it’s number is not growing near/at hang. Cheked descriptor limits of OS and Asterisk itself - it wasn’t reached. Our last tip is about submaxexpiry. It’s value was set much higher than maxexpiry at the time the erorrs showed up. I disabled it’s explicit set so the default value is used (maxexpiry's value). Hope for the best
