Hello,
I have been fighting a huge memory leak in Asterisk the past 2 weeks. My system (Asterisk 13, with FreePBX 13 on CentOS 6) was running quite well for more then a year. Suddenly this memory leak started, and I can’t figure out what triggered it. It consumes %100 of my 16GB in less then 24 hours, and I must restart Asterisk daily,disconnecting many ongoing calls.
I have updated my Asterisk to version 13.17.0, but that did not help.
Please help me figure out what is causing this.
I uploaded the last day and month memory usage chart.
Thanks,
start by monitoring your system resources using top and ps commands
I did that - no doubt it is asterisk eating all memory.
Top:
htop:
Here are a few things you can try to help narrow it down:
You said it started 2 weeks ago, so what changed around then? Did you upgrade Asterisk at that time? FreePBX? Add any new phones? Changed the dialplan, or any AMI/AGI/ARI scripts/apps? Something else?
Can you observe the increase? Is it steady or does it spike at times? If it spikes what’s happening on the system at that time?
Look in the log. Are there any noticeable warnings or errors that started showing up in the last couple of weeks that were not there before?
If you are able turn on malloc debug in Asterisk and you might be able to see what subsystem is eating memory.
If you suspect there is a memory leak in Asterisk please file an issue. However before doing so please follow the instructions on the Asterisk Issue Guideilnes and Memory Leak Debugging pages as it will help in diagnosing the problem.
The only thing changed, is an update to the ARI application in use. I went a few times over the changes and couldn’t find anything related. However, I am going to let the system run a few days with the old application to see if it will make the problem disappear.
If indeed it relates to the ARI application, can you give me any ideas what on the application can cause this?
Run ‘core show taskprocessors’ in the CLI. Check for anything queued.
Can you see anything queued here:
Processor Processed In Queue Max Depth Low water High water
250 taskprocessors
tgross
September 13, 2017, 5:02pm
8
We experienced the same issue about a month ago. It began occurring after we upgraded from FreePBX version 10.13.66-15 to 10.13.66-20.
I noticed that swap space was filling up so I doubled the memory on the server.
I was then able to diagnose that we had over 9000 tftp sessions showing up in the process list (ps-faux | grep tftp) after a couple of days.
After I killed those and turned off tftp I was able to get the system to stabilize.
My final solution was to move the tftp provisioning off onto another server. I mounted that servers /tftpboot directory onto the /tftpboot directory on the pbx and that allows me to make the changes in Endpoint Manager.
So far that has worked.
I have other FreePBX installations running the same version without any issues. I suspect that it has to do with one particular phone we are using, the Cisco CP7911, that is causing the issue but I can’t prove that.
wa4zlw
September 13, 2017, 7:42pm
10
I have similar with Asterisk 14/ Fpbx 14. I opened https://issues.asterisk.org/jira/browse/ASTERISK-27258
Can not recompile.
II’m getting memory messages in the CLI from PJSIP along with other weird errors from time-to-time
here is my taskprocessors:
pbx*CLI> core show taskprocessors
Processor Processed In Queue Max Depth Low water High water
144 taskprocessors
Leon
Thanks for the suggestion, but, we do not have any problem with the tftp service. For us, it is clearly the asterisk service that eats all memory. Any idea or suggestion how to narrow our research to figure what in asterisk is doing that?
If indeed it relates to the ARI application, can you give me any ideas what on the application can cause this?
Could be a number of things. For instance channels being created, but not properly hungup/destroyed, or bridges not being destroyed when done, etc…
From the Asterisk CLI what does “core show channels” look like? If it lists a bunch of channels that you know are no longer in use then perhaps they have not been hung up properly. Same for bridging. You can check if any bridges are still hanging around with “bridge show all”.
these seems to be an issue when using realtime configuration I guess.I have the same issue with 2 different servers using Asterisk 13.17.2, and without any system load Asterisk start using all the system memory, and I need to keep restarting the Asterisks service to fix it.
a2billing*CLI> core show taskprocessors
Processor Processed In Queue Max Depth Low water High water
app_voicemail 0 0 0 450 500
ast_msg_queue 0 0 0 450 500
CCSS_core 0 0 0 450 500
hep_queue_tp 0 0 0 450 500
iax2_transmit 0 0 0 450 500
Sorcery 0 0 0 450 500
Sorcery-control 1 0 1 450 500
sorcery/bucket-00000000 0 0 0 450 500
sorcery/file-00000001 0 0 0 450 500
stasis-core 306 0 2 450 500
stasis-core-control 763 0 2 450 500
subm:ast_bridge_topic_all-cached-00000012 1 0 1 450 500
subm:ast_channel_topic_all-00000017 3911 0 69 450 500
subm:ast_channel_topic_all-cached-00000013 6040 0 69 450 500
subm:ast_device_state_topic-00000002 792 0 2 450 500
subm:ast_device_state_topic-00000004 791 0 3 450 500
subm:ast_device_state_topic-00000016 784 0 3 450 500
subm:ast_presence_state_topic_all-00000005 1 0 1 450 500
subm:ast_security-00000014 57995 0 2 450 500
subm:ast_system-00000009 34 0 31 450 500
subm:ast_system-0000000a 33 0 20 450 500
subm:ast_system-00000011 32 0 20 450 500
subm:cdr_engine-00000003 6206 0 69 4500 5000
subm:cel_aggregation_topic-00000006 6042 0 69 2700 3000
subm:endpoint_topic_all-cached-00000008 883 0 3 450 500
subm:manager_topic-00000007 65148 0 69 2700 3000
subm:stasis_mwi_topic-00000015 1 0 1 450 500
subp:SIP/0978228585-0000000c 2 0 1 450 500
subp:SIP/109-0000000d 2 0 1 450 500
subp:SIP/2936943285-0000000b 292 0 2 450 500
subp:SIP/freepbx1-0000000e 4 0 2 450 500
subp:SIP/OXILOG_92.222.202.150-0000000f 4 0 2 450 500
subp:SIP/voxbeam-00000010 4 0 2 450 500
33 taskprocessors
How can I check if my server is using realtime configuration? My server is 13.17.0 and this problem is driving me crazy. If you have any workaround I will be happy to try it.
@kharwell @jcolp any thought about this issue guys ?
jcolp
October 28, 2017, 10:28pm
16
I’m not aware of any changes that have happened to chan_sip realtime that would have caused such a thing. For memory leaks in general we need more information[1].
[1] https://wiki.asterisk.org/wiki/display/AST/Memory+Leak+Debugging
If it may help someone, we have found, the cause of the problem. It looks now that using the jitter buffer is causing it, maybe in combination of using a holding bridge with many participants.
Hi,
I was working on it for more than 6 months, this is not a memory cache issue and you CANNOT fix it by “sync; echo 3 > /proc/sys/vm/drop_caches”. It’s only the kernel issue that fixed in kernel 4.x lt. So you can use this instruction for upgrade your kernel: https://www.tecmint.com/install-upgrade-kernel-version-in-centos-7/
I hope it will be helpful everyone has this issue.
2 Likes
