We have this problem in our Asterisk server. When number of active calls reaches around 140 we can no more connect to asterisk console. asterisk -r only shows the welcome message and then exits to shell automatically.
We have Intel E3-1245 4 core processor with 32 GB RAM. CPU usages goes around 40-42% when this kind of situation arises. We also found that the server doesnt actually drop any calls but we simply cant monitor our server when this kind of situation arises.
Please give us some clue like what can be the cause of this problem and how can we fix it. Let us know if you need any further details.
I assume you have run out of some system resource. I’d check you don’t have any effective limit on open file descriptors, but otherwise you will need to determine if asterisk -r is crashing, in which case you will need a backtrace, or you will need to do a system call trace to get an idea of where it is failing.
Thanks for your reply. We have updated the ulimit. Initially it was set to 1024. We updated it to 65536. But still we are facing the same issue. We got the below error from the log during the time of occurrence.
[Jan 10 18:59:59] ERROR[6394] acl.c: Cannot create socket
[Jan 10 18:59:59] WARNING[6394] res_rtp_asterisk.c: Unable to allocate RTP socket: Too many open files
ulimit -a gives
# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 256523
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 256523
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
We are really in trouble with this issue. Please help us. Thanks in advance.
If you are leaking file descriptors, doing do a bash ls on /proc/nnn/fd, where nnn is the PID of Asterisk, should not result in a continual increase in the number of pseudo files listed. If you are leaking some other resource, you will have to do an appropriate measurement (ps -l will tell you if you are leaking memory).
A resource leak would indicate a bug, in which case you may have to upgrade Asterisk or submit a but report and wait for it to be fixed, depending on whether it is a known or unknown bug.
[root@localhost fd]# ls -l /proc/6346/fd | wc -l
537
[root@localhost fd]# ls -l /proc/6346/fd | wc -l
531
[root@localhost fd]# ls -l /proc/6346/fd | wc -l
523
[root@localhost fd]# ls -l /proc/6346/fd | wc -l
507
[root@localhost fd]# ls -l /proc/6346/fd | wc -l
501
[root@localhost fd]# ls -l /proc/6346/fd | wc -l
521
[root@localhost fd]# ls -l /proc/6346/fd | wc -l
535
[root@localhost fd]# ls -l /proc/6346/fd | wc -l
539
[root@localhost fd]# ls -l /proc/6346/fd | wc -l
547
[root@localhost fd]# ls -l /proc/6346/fd | wc -l
547
[root@localhost fd]# ls -l /proc/6346/fd | wc -l
553
The time we took this we had around 60 calls in our server. I can see the number of open files for asterisk is decreasing and increasing (not continually increasing I guess). Will you call it a resource leak? Please let me know.
Hi,
Though the overall ulimit is set to 65536 I found Max open file limit particularly for asterisk process is set to 1024 soft limit and 4096 hard limit. I think this is what causing the problem.
]# cat /proc/6346/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 256523 256523 processes
Max open files 1024 4096 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 256523 256523 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Can you please tell me how to increase ulimit of specific process (asterisk in my case).
Increase it in the parent process, i.e. the startup script. If that doesn’t work, and no-one else answer, treat this as a Linux issue, not an Asterisk one.
(If you had a file descriptor resource leak, the result of the ls | wc would have continually grown until it reached the 1023, even without large numbers of calls.)