Too many open files error

I know this has come up many times before, but I’d like to see if there’s some root cause here that can be solved beyond “restart Asterisk”.

[2022-07-29 10:08:17] WARNING[3632][C-00000280]: alertpipe.c:48 ast_alertpipe_init: Failed to create alert pipe with eventfd(), falling back to pipe(): Too many open files
[2022-07-29 10:08:17] WARNING[3632][C-00000280]: alertpipe.c:57 ast_alertpipe_init: Failed to create alert pipe: Too many open files
[2022-07-29 10:08:17] WARNING[3632][C-00000280]: core_unreal.c:1263 ast_unreal_new_channels: Unable to allocate chan channel structure
[2022-07-29 10:08:17] ERROR[3632][C-00000280]: app_confbridge.c:1882 join_conference_bridge: Could not allocate announcer channel for conference 'oe1'

I noticed that I am now receiving errors about too many open files when trying to do certain things.

Running a few lsof commands, I get the following:

root@voip:~# lsof -p 23042 | grep "CLOSE_WAIT" | wc -l
306
root@voip:~# lsof -p 23042 | grep "=STREAM" | wc -l
622
root@voip:~# lsof -p 23042 | wc -l
1418

The CLOSE_WAIT are all TCP, not UDP, e.g.

asterisk 23042 asterisk 96u IPv4 75923799 0t0 TCP REDACTED:53084->REDACTED:https (CLOSE_WAIT)
asterisk 23042 asterisk 97u unix 0x00000000080a3ac0 0t0 73800835 type=STREAM

To me, those first 2 numbers are suspiciously high. 1418 is greater than the ulimit of 1024, which explains why we get this message, but this has almost never happened before and the system was only restarted a few days ago, so it’s like file descriptors are getting leaked somehow. I’m assuming there’s no way to view fd usage by module?

I still say 1418 is high because there are FOUR channels on the system right now, so something is definitely off.

Would anyone know what STREAM and CLOSE_WAIT correspond to, and why they might be so high? Could this indicate something wrong with a particular network module, say, PJSIP and incoming SIP connections timing out, something like that?