Memory allocation failure on 11.25.1

Asterisk Asterisk Support
1

After an update from 11.20.1 to 11.25.1 after a few days asterisk crashes and then restarts itself.

In the full log I see this:
[2017-02-03 17:27:11] ERROR[15329] /usr/src/asterisk-11.25.1/include/asterisk/utils.h: Memory Allocation Failure in function ast_str_create at line 435 of /usr/src/aste
risk-11.25.1/include/asterisk/strings.h
[2017-02-03 17:27:11] WARNING[15329] chan_sip.c: sip_xmit of 0x7f49dc3e8360 (len 139955200328584) to 141.135.151.77:12384 returned -2: Cannot allocate memory
[2017-02-03 17:27:11] ERROR[15329] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2017-02-03 17:27:15] Asterisk 11.25.1 built by root @ voip01.lentrix.eu on a x86_64 running Linux on 2017-01-26 18:32:33 UTC
[Feb 3 17:27:15] VERBOSE[8444] config.c: == Parsing ‘/etc/asterisk/logger_logfiles_custom.conf’: Found
[2017-02-03 17:27:15] VERBOSE[8444] config.c: == Parsing ‘/etc/asterisk/asterisk.conf’: Found

Memory usage is constantly monitored on our server and there seems to be enough free memory all the time.

Any ideas to troubleshoot this?

Thanks,

2

Run ps -le and find out which process is hogging the memory.

Also check that you allocated a reasonable amount of memory for the workload the machine has to handle.

If it proves to be a memory leak in Asterisk, watch it growing and try and work out when it grows, and by how much.

3

ps aux --sort -rss

4

Thanks for the replies.

However, we’re monitoring the memory by nagios and here it shows the free memory never drops, it’s always about 3G (with 4G RAM installed).

Also, we’re clearing memory in crontab:
*/5 * * * * sync; echo 3 > /proc/sys/vm/drop_caches

This night, I got this error in asterisk, had to restart it completely:
[2017-02-04 07:22:06] WARNING[40243] asterisk.c: No more connections allowed

Never saw this before.

5

The I/O failures suggested that the kernel was out of memory, but that is not consistent with the strangely high free memory (free memory should drop until it reaches significantly less than 10%).

If it is a limit on Asterisk itself, I would look at your ulimit settings.

6

This server is running a few years without problems. This behavior started after an asterisk update from 11.20 to 11.25.

Hereby the ulimit settings:

ulimit -a 11:13
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) 10240
-c: core file size (blocks) 0
-m: resident set size (kbytes) unlimited
-u: processes 15209
-n: file descriptors 1024
-l: locked-in-memory size (kb) 64
-v: address space (kb) unlimited
-x: file locks unlimited
-i: pending signals 15209
-q: bytes in POSIX msg queues 819200
-e: max nice 0
-r: max rt priority 0

7

Asterisk 11 is only being maintained for security issues.

8

System just crashes again with the same error:

[2017-02-07 14:26:15] WARNING[11650] chan_sip.c: sip_xmit of 0x7f270bb0d310 (len 139805614670888) to 141.135.151.77:12154 returned -2: Cannot allocate memory

Nothing unusuals to see at ps aux --sort -rss.
Isn’t Asterisk just trying to allocate an huge buffer (len 139805614670888) which causes the crash?

Also, after restart I noticed those warnings.

pbx.c:3244 pbx_find_extension: Maximum PBX stack exceeded

9

Looks like the open files is a little low (-n). I Increased mine to 40000 (Ubuntu). I made it part of my install. I’m also using 13.12.

#To allow more open files.
$ nano /etc/security/limits.conf
#Add
root soft  nofile 40000
root hard  nofile 40000

$ nano /etc/pam.d/common-session
#Add
session required pam_limits.so

root@asterisk-whconfig01:/home/whasterisk# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15726
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 40000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 15726
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
1 Like