Asterisk service frozen

[Aug 21 15:00:18] WARNING[2479] pbx.c: Failed to create new channel thread
[Aug 21 15:00:18] WARNING[2479] chan_sip.c: Failed to start PBX :frowning:

Received a bunch of these notifications within the LOGS after we had to restart the asterisk service. During the outage we could not even console ā€œasterisk -rā€. Where else can I look to determine what caused this error?

main/utils.c

It probably means you are out of memory. It could mean you are non-root and have exceeded your process limit.

Memory is currently at 15GB out of 16GB being used. Didnā€™t have a chance to check the memory prior to killing asterisk service.

main/utils.c ? What am I supposed to be looking into within that configuration file?

Itā€™s not a configuration file.

Which measure did you use for used memory?

Basically, the error means that pthread_create system library routine failed. For details of what pthread_create does, try typing ā€œman pthread_createā€ at a shell prompt. (PTHREAD_THREADS_MAX will not be user configurable, and is 16384 on my system. If you have that many threads, you have worse problems than I thought, so I have to assume you are out of memory. Even exceeding ulimit -u is likely to represent too many calls on a single system.)

Mem: 16440104k total, 16232532k used, 207572k free, 186516k buffers
Swap: 18481144k total, 632k used, 18480512k free, 14618192k cached

I am using top to get the memory stats.

I notice we are down to about 200MB of memory free as of this morning. But shouldnā€™t the swap file kick in when the memory is FULL?

The large cached figure indicates that you are not tight on memory. I has to be some other resource limit that is being exceeded. How many concurrent calls are you trying to handle?

(The output of the free command on CentOS 5.3 is more helpful, as the middle line shows memory that is immediately available. Both Linux and, for that matter, modern Windows, deliberately run with a high total memory used.:

total used free shared buffers cached Mem: 515444 464528 50916 0 6644 52296 -/+ buffers/cache: 405588 109856 Swap: 2031608 844092 1187516
This is a test and development machine, not a production one.)

total used free shared buffers cached
Mem: 16440104 16234248 205856 0 186516 14618704
-/+ buffers/cache: 1429028 15011076
Swap: 18481144 632 18480512

Upwards of 150 concurrent calls, internal trunks and SIP providers.