Memory leak consumes 16GB in less then 24 hours


#1

Hello,

I have been fighting a huge memory leak in Asterisk the past 2 weeks. My system (Asterisk 13, with FreePBX 13 on CentOS 6) was running quite well for more then a year. Suddenly this memory leak started, and I can’t figure out what triggered it. It consumes %100 of my 16GB in less then 24 hours, and I must restart Asterisk daily,disconnecting many ongoing calls.
The system (apart from a light use of office extensions with the FreePBX), is a very big IVR system implemented with ARI. We get about 35,000 calls daily, with about 700 concurrent calls on peak time.

I have updated my Asterisk to version 13.17.0, but that did not help.

Please help me figure out what is causing this.

I uploaded the last day and month memory usage chart.

Month
Day

Thanks,
Eyal Hasson.


#2

start by monitoring your system resources using top and ps commands


#3

I did that - no doubt it is asterisk eating all memory.

Top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4921 asterisk 20 0 19.2g 2.3g 16m S 574.5 15.1 24169:07 asterisk

htop:
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
4921 asterisk 20 0 19.2G 2403M 17048 S 583. 15.2 402h /usr/sbin/asterisk -f -U asterisk -G asterisk -g -vvvg -c


#4

Here are a few things you can try to help narrow it down:

You said it started 2 weeks ago, so what changed around then? Did you upgrade Asterisk at that time? FreePBX? Add any new phones? Changed the dialplan, or any AMI/AGI/ARI scripts/apps? Something else?

Can you observe the increase? Is it steady or does it spike at times? If it spikes what’s happening on the system at that time?

Look in the log. Are there any noticeable warnings or errors that started showing up in the last couple of weeks that were not there before?

If you are able turn on malloc debug in Asterisk and you might be able to see what subsystem is eating memory.

If you suspect there is a memory leak in Asterisk please file an issue. However before doing so please follow the instructions on the Asterisk Issue Guideilnes and Memory Leak Debugging pages as it will help in diagnosing the problem.


#5

The only thing changed, is an update to the ARI application in use. I went a few times over the changes and couldn’t find anything related. However, I am going to let the system run a few days with the old application to see if it will make the problem disappear.
The increase is steady of about 5% per hour.
Nothing special on the log.
I use FreePBX, and I do not know if I have control on the compiling options to use malloc debug.

If indeed it relates to the ARI application, can you give me any ideas what on the application can cause this?


#6

Run ‘core show taskprocessors’ in the CLI. Check for anything queued.


#7

Can you see anything queued here:
Connected to Asterisk 13.17.0 currently running on PhoneLinuxSrv (pid = 4931)
PhoneLinuxSrv*CLI> core show taskprocessors

Processor Processed In Queue Max Depth Low water High water
app_voicemail 93 0 57 450 500
ast_msg_queue 0 0 0 450 500
CCSS_core 0 0 0 450 500
hep_queue_tp 0 0 0 450 500
iax2_transmit 0 0 0 450 500
pjsip/default-0000000c 4 0 1 450 500
pjsip/default-0000000d 4 0 1 450 500
pjsip/default-0000000e 3 0 1 450 500
pjsip/default-0000000f 3 0 1 450 500
pjsip/default-00000010 3 0 1 450 500
pjsip/default-00000011 3 0 1 450 500
pjsip/default-00000012 3 0 1 450 500
pjsip/default-00000013 3 0 1 450 500
pjsip/distributor-0000001d 15 0 1 450 500
pjsip/distributor-0000001e 21 0 1 450 500
pjsip/distributor-0000001f 20 0 1 450 500
pjsip/distributor-00000020 21 0 1 450 500
pjsip/distributor-00000021 21 0 1 450 500
pjsip/distributor-00000022 24 0 1 450 500
pjsip/distributor-00000023 26 0 1 450 500
pjsip/distributor-00000024 26 0 1 450 500
pjsip/distributor-00000025 21 0 1 450 500
pjsip/distributor-00000026 22 0 1 450 500
pjsip/distributor-00000027 19 0 1 450 500
pjsip/distributor-00000028 23 0 1 450 500
pjsip/distributor-00000029 21 0 1 450 500
pjsip/distributor-0000002a 19 0 1 450 500
pjsip/distributor-0000002b 16 0 1 450 500
pjsip/distributor-0000002c 18 0 1 450 500
pjsip/distributor-0000002d 19 0 1 450 500
pjsip/distributor-0000002e 17 0 1 450 500
pjsip/distributor-0000002f 22 0 1 450 500
pjsip/distributor-00000030 24 0 1 450 500
pjsip/distributor-00000031 18 0 1 450 500
pjsip/distributor-00000032 15 0 1 450 500
pjsip/distributor-00000033 16 0 1 450 500
pjsip/distributor-00000034 18 0 1 450 500
pjsip/distributor-00000035 9 0 1 450 500
pjsip/distributor-00000036 18 0 1 450 500
pjsip/distributor-00000037 27 0 1 450 500
pjsip/distributor-00000038 22 0 1 450 500
pjsip/distributor-00000039 30 0 1 450 500
pjsip/distributor-0000003a 20 0 1 450 500
pjsip/distributor-0000003b 19 0 1 450 500
pjsip/messaging 0 0 0 450 500
pjsip/mwi-00000044 0 0 0 450 500
pjsip/mwi-00000045 0 0 0 450 500
pjsip/mwi-00000046 0 0 0 450 500
pjsip/mwi-00000047 0 0 0 450 500
pjsip/mwi-00000048 0 0 0 450 500
pjsip/mwi-00000049 0 0 0 450 500
pjsip/mwi-0000004a 0 0 0 450 500
pjsip/mwi-0000004b 0 0 0 450 500
SIP 653 0 2 450 500
SIP-control 4337 0 4 450 500
Sorcery 19 0 2 450 500
Sorcery-control 43 0 2 450 500
sorcery/acl-000000dc 0 0 0 450 500
sorcery/aor-00000019 2 0 1 450 500
sorcery/asterisk-publication-00000043 0 0 0 450 500
sorcery/auth-00000014 1 0 1 450 500
sorcery/bucket-00000000 0 0 0 450 500
sorcery/contact-00000018 2 0 1 1350 1500
sorcery/contact_status-0000001a 2 0 1 1350 1500
sorcery/domain_alias-0000001b 0 0 0 450 500
sorcery/endpoint-00000015 2 0 1 450 500
sorcery/file-00000001 0 0 0 450 500
sorcery/global-0000001c 7 0 1 450 500
sorcery/identify-00000042 0 0 0 450 500
sorcery/inbound-publication-0000003e 0 0 0 450 500
sorcery/log_mappings-0000000a 0 0 0 450 500
sorcery/nat_hook-00000016 0 0 0 450 500
sorcery/outbound-publish-00000040 0 0 0 450 500
sorcery/phoneprov-000000d9 0 0 0 450 500
sorcery/registration-000000da 3 0 1 450 500
sorcery/resource_list-0000003d 0 0 0 450 500
sorcery/subscription_persistence-0000003c 0 0 0 450 500
sorcery/system-0000000b 0 0 0 450 500
sorcery/transport-00000017 0 0 0 450 500
stasis-core 12154186 0 5 450 500
stasis-core-control 24308511 0 8 450 500
subm:ast_bridge_topic_all-000000e8 62674 0 4 450 500
subm:ast_bridge_topic_all-000000ee 62654 0 4 450 500
subm:ast_bridge_topic_all-000000f0 62653 0 4 450 500
subm:ast_bridge_topic_all-000000f2 56144 0 4 450 500
subm:ast_bridge_topic_all-cached-000000dd 106468 0 6 450 500
subm:ast_channel_topic_all-000000e3 8862069 0 23 450 500
subm:ast_channel_topic_all-000000e7 8862068 0 22 450 500
subm:ast_channel_topic_all-cached-000000de 12671928 0 67 450 500
subm:ast_channel_topic_all-cached-000000df 12671927 0 61 450 500
subm:ast_channel_topic_all-cached-000000e2 12671926 0 40 450 500
subm:ast_device_state_topic-00000002 130030 0 4 450 500
subm:ast_device_state_topic-00000004 130029 0 9 450 500
subm:ast_device_state_topic-000000e6 129783 0 3 450 500
subm:ast_parking-00000041 1 0 1 450 500
subm:ast_presence_state_topic_all-00000005 1 0 1 450 500
subm:ast_security-000000e5 58226 0 2 450 500
subm:ast_system-00000009 12304 0 253 450 500
subm:ast_system-0000004d 12303 0 6 450 500
subm:ast_system-000000d7 12302 0 2 450 500
subm:ast_system-000000db 12278 0 1 450 500
subm:cdr_engine-00000003 12779853 0 2215 4500 5000
subm:cel_aggregation_topic-00000006 12778398 0 105 2700 3000
subm:endpoint_topic_all-cached-00000008 12664456 0 48 450 500
subm:endpoint_topic_all-cached-000000e4 12664113 0 59 450 500
subm:KhlSimulationApp-000000f3 339 0 5 450 500
subm:KhlSystemApp-000000f1 1 0 1 450 500
subm:KhlSystemApp2-000000e9 183034 0 8 450 500
subm:KhlSystemAppGood-000000ef 10938375 0 59 450 500
subm:manager_topic-00000007 19417050 0 491 2700 3000
subm:stasis_mwi_topic-000000e0 163 0 2 450 500
subm:stasis_mwi_topic-000000e1 156 0 2 450 500
subp:0176@default-000000f4 2 0 1 450 500
subp:0919@default-000000f5 2 0 1 450 500
subp:1039@default-000000f6 3 0 2 450 500
subp:1728@default-000000f7 1 0 1 450 500
subp:2501@default-000000f8 2 0 1 450 500
subp:2799@default-000000f9 3 0 1 450 500
subp:9102@default-000000fa 3 0 1 450 500
subp:9103@default-000000fb 3 0 1 450 500
subp:9104@default-000000fc 2 0 1 450 500
subp:9105@default-000000fd 2 0 1 450 500
subp:9106@default-000000fe 1 0 1 450 500
subp:9107@default-000000ff 2 0 1 450 500
subp:9108@default-00000100 2 0 1 450 500
subp:9109@default-00000101 2 0 1 450 500
subp:9110@default-00000102 2 0 1 450 500
subp:9111@default-00000103 1 0 1 450 500
subp:9112@default-00000104 1 0 1 450 500
subp:9113@default-00000105 2 0 1 450 500
subp:9114@default-00000106 2 0 1 450 500
subp:9115@default-00000107 2 0 1 450 500
subp:9116@default-00000108 2 0 1 450 500
subp:9117@default-00000109 1 0 1 450 500
subp:9118@default-0000010a 2 0 1 450 500
subp:9119@default-0000010b 1 0 1 450 500
subp:9120@default-0000010d 1 0 1 450 500
subp:9121@default-0000010f 1 0 1 450 500
subp:9134@device-00000110 1 0 1 450 500
subp:9210@default-00000111 1 0 1 450 500
subp:9810@default-00000112 2 0 1 450 500
subp:9811@default-00000113 1 0 1 450 500
subp:9999@default-00000114 2 0 1 450 500
subp:IAX2/9805-000000d8 2 0 2 450 500
subp:SIP/001101726-0000004e 46046 0 13 450 500
subp:SIP/0017-0000004f 187 0 5 450 500
subp:SIP/0131-00000050 245 0 4 450 500
subp:SIP/0142-00000051 2142 0 4 450 500
subp:SIP/0176-00000052 2 0 1 450 500
subp:SIP/0217-00000054 2 0 1 450 500
subp:SIP/0286-00000055 2 0 2 450 500
subp:SIP/0310-00000056 2 0 2 450 500
subp:SIP/0381-00000057 2 0 1 450 500
subp:SIP/0469-00000058 2 0 1 450 500
subp:SIP/0484-00000059 2 0 1 450 500
subp:SIP/0490-0000005a 2 0 1 450 500
subp:SIP/0530-0000005b 2 0 1 450 500
subp:SIP/0674-0000005c 2 0 1 450 500
subp:SIP/072trunk-0000005d 805527 0 27 450 500
subp:SIP/0784-0000005e 2 0 1 450 500
subp:SIP/0866-0000005f 2 0 1 450 500
subp:SIP/0898-00000060 1101 0 2 450 500
subp:SIP/0918-00000061 1549 0 3 450 500
subp:SIP/0919-00000062 2 0 1 450 500
subp:SIP/0939-00000064 2 0 2 450 500
subp:SIP/1039-00000065 2 0 1 450 500
subp:SIP/1210-00000067 2 0 1 450 500
subp:SIP/1353-00000068 2 0 1 450 500
subp:SIP/1530-00000069 2 0 1 450 500
subp:SIP/1601-0000006a 2 0 1 450 500
subp:SIP/1608-0000006b 716 0 4 450 500
subp:SIP/1654801-0000006c 226559 0 17 450 500
subp:SIP/1654801_copy_4-0000006d 2 0 1 450 500
subp:SIP/1654901-0000006e 2 0 1 450 500
subp:SIP/1728-0000006f 2 0 1 450 500
subp:SIP/1788-00000071 2 0 1 450 500
subp:SIP/1820-00000072 2 0 1 450 500
subp:SIP/1953-00000073 2 0 1 450 500
subp:SIP/1997-00000074 2 0 2 450 500
subp:SIP/2072-00000075 2 0 1 450 500
subp:SIP/2113-00000076 2 0 1 450 500
subp:SIP/2193-00000077 2 0 1 450 500
subp:SIP/21D6DF6E61-00000078 2 0 1 450 500
subp:SIP/2341-00000079 2 0 2 450 500
subp:SIP/2499-0000007a 2 0 1 450 500
subp:SIP/2501-0000007b 2 0 1 450 500
subp:SIP/2799-0000007d 2 0 1 450 500
subp:SIP/5001-0000007f 42442 0 5 450 500
subp:SIP/5002-00000080 2 0 1 450 500
subp:SIP/5003-00000081 2 0 1 450 500
subp:SIP/9102-00000082 25163 0 10 450 500
subp:SIP/9103-00000084 5128 0 7 450 500
subp:SIP/9104-00000086 3843 0 6 450 500
subp:SIP/9105-00000088 12005 0 11 450 500
subp:SIP/9106-0000008a 2 0 1 450 500
subp:SIP/9107-0000008c 589 0 4 450 500
subp:SIP/9108-0000008e 2 0 1 450 500
subp:SIP/9109-00000090 330 0 3 450 500
subp:SIP/9110-00000092 971 0 2 450 500
subp:SIP/9111-00000094 605 0 6 450 500
subp:SIP/9112-00000096 300 0 5 450 500
subp:SIP/9113-00000098 88 0 2 450 500
subp:SIP/9114-0000009a 2 0 1 450 500
subp:SIP/9115-0000009c 2767 0 4 450 500
subp:SIP/9116-0000009e 1366 0 3 450 500
subp:SIP/9117-000000a0 8367 0 5 450 500
subp:SIP/9118-000000a2 1487 0 4 450 500
subp:SIP/9119-000000a4 818 0 4 450 500
subp:SIP/9120-0000010c 1771 0 4 450 500
subp:SIP/9121-0000010e 1510 0 3 450 500
subp:SIP/9134-000000a6 2600 0 2 450 500
subp:SIP/9135-000000a8 2494 0 2 450 500
subp:SIP/9136-000000a9 134 0 2 450 500
subp:SIP/9210-000000aa 2 0 1 450 500
subp:SIP/9801-000000b0 196 0 2 450 500
subp:SIP/9802-000000b1 88 0 2 450 500
subp:SIP/9803-000000b2 88 0 2 450 500
subp:SIP/9804-000000b3 88 0 2 450 500
subp:SIP/9806-000000b4 46 0 2 450 500
subp:SIP/9810-000000b5 2 0 1 450 500
subp:SIP/9811-000000b7 2 0 1 450 500
subp:SIP/99051000136006-000000b9 17629 0 16 450 500
subp:SIP/9999-000000ba 2 0 1 450 500
subp:SIP/AsrSrvTrunk-000000bc 4 0 1 450 500
subp:SIP/BackupSystemTrunk-000000bd 2 0 1 450 500
subp:SIP/BackupSystemTrunk_copy_2-000000be 2 0 1 450 500
subp:SIP/BursaSIP-000000bf 37979 0 7 450 500
subp:SIP/BursaUser-000000c0 2 0 1 450 500
subp:SIP/CellcomSIP1-000000c1 4232918 0 32 450 500
subp:SIP/CellcomSIP1_copy_5-000000c2 164 0 2 450 500
subp:SIP/CellcomSIP1_copy_5_copy_12-000000c3 61752 0 9 450 500
subp:SIP/CellcomSIP2-000000c4 7035788 0 42 450 500
subp:SIP/CellcomUser-000000c5 2 0 1 450 500
subp:SIP/CellcomUser_copy_5-000000c6 2 0 1 450 500
subp:SIP/CellcomUser_copy_5_copy_12-000000c7 2 0 1 450 500
subp:SIP/DiallerUser-000000c8 2 0 1 450 500
subp:SIP/Dollar-Pashevorks-000000c9 1561 0 3 450 500
subp:SIP/ITC_Trunk-000000ca 2 0 1 450 500
subp:SIP/ITC_User-000000cb 2 0 1 450 500
subp:SIP/MehadrinNetTrunk-000000cc 2 0 1 450 500
subp:SIP/o72user-000000cd 2 0 1 450 500
subp:SIP/Orbtalk60Channels-000000ce 474 0 3 450 500
subp:SIP/Orbtalk_10_Rabbanim-000000d0 2 0 1 450 500
subp:SIP/OrbtalkTestTrunk-000000cf 2 0 1 450 500
subp:SIP/Skype-SIP-Register-000000d1 4 0 2 450 500
subp:SIP/SkypeTrunk-000000d2 2 0 2 450 500
subp:SIP/SkypeUser-000000d3 2 0 1 450 500
subp:SIP/Voipfone-SIP-000000d4 4 0 2 450 500
subp:SIP/Voipfone-SIP-Rabbanim-000000d5 4 0 2 450 500
subp:SIP/Voipfone-SIP-UK-Office-000000d6 4 0 2 450 500

250 taskprocessors


#8

We experienced the same issue about a month ago. It began occurring after we upgraded from FreePBX version 10.13.66-15 to 10.13.66-20.

I noticed that swap space was filling up so I doubled the memory on the server.

I was then able to diagnose that we had over 9000 tftp sessions showing up in the process list (ps-faux | grep tftp) after a couple of days.

After I killed those and turned off tftp I was able to get the system to stabilize.

My final solution was to move the tftp provisioning off onto another server. I mounted that servers /tftpboot directory onto the /tftpboot directory on the pbx and that allows me to make the changes in Endpoint Manager.

So far that has worked.

I have other FreePBX installations running the same version without any issues. I suspect that it has to do with one particular phone we are using, the Cisco CP7911, that is causing the issue but I can’t prove that.


#10

I have similar with Asterisk 14/ Fpbx 14. I opened https://issues.asterisk.org/jira/browse/ASTERISK-27258

Can not recompile.

II’m getting memory messages in the CLI from PJSIP along with other weird errors from time-to-time

here is my taskprocessors:

pbx*CLI> core show taskprocessors

Processor Processed In Queue Max Depth Low water High water
app_voicemail 186 0 15 450 500
ast_msg_queue 0 0 0 450 500
CCSS_core 0 0 0 450 500
dns_system_resolver_tp 0 0 0 450 500
hep_queue_tp 0 0 0 450 500
iax2_transmit 0 0 0 450 500
pjsip/default-0000000d 508 0 1 450 500
pjsip/default-0000000e 502 0 1 450 500
pjsip/default-0000000f 521 0 1 450 500
pjsip/default-00000010 513 0 1 450 500
pjsip/default-00000011 509 0 1 450 500
pjsip/default-00000012 504 0 1 450 500
pjsip/default-00000013 517 0 1 450 500
pjsip/default-00000014 507 0 1 450 500
pjsip/distributor-00000022 140 0 1 450 500
pjsip/distributor-00000023 161 0 2 450 500
pjsip/distributor-00000024 130 0 2 450 500
pjsip/distributor-00000025 114 0 2 450 500
pjsip/distributor-00000026 100 0 1 450 500
pjsip/distributor-00000027 114 0 1 450 500
pjsip/distributor-00000028 137 0 2 450 500
pjsip/distributor-00000029 128 0 2 450 500
pjsip/distributor-0000002a 220 0 2 450 500
pjsip/distributor-0000002b 135 0 2 450 500
pjsip/distributor-0000002c 96 0 1 450 500
pjsip/distributor-0000002d 110 0 1 450 500
pjsip/distributor-0000002e 112 0 1 450 500
pjsip/distributor-0000002f 119 0 2 450 500
pjsip/distributor-00000030 90 0 1 450 500
pjsip/distributor-00000031 120 0 1 450 500
pjsip/distributor-00000032 114 0 1 450 500
pjsip/distributor-00000033 126 0 2 450 500
pjsip/distributor-00000034 128 0 1 450 500
pjsip/distributor-00000035 106 0 1 450 500
pjsip/distributor-00000036 145 0 2 450 500
pjsip/distributor-00000037 122 0 1 450 500
pjsip/distributor-00000038 98 0 1 450 500
pjsip/distributor-00000039 110 0 1 450 500
pjsip/distributor-0000003a 225 0 2 450 500
pjsip/distributor-0000003b 114 0 2 450 500
pjsip/distributor-0000003c 118 0 1 450 500
pjsip/distributor-0000003d 112 0 1 450 500
pjsip/distributor-0000003e 132 0 1 450 500
pjsip/distributor-0000003f 151 0 3 450 500
pjsip/distributor-00000040 124 0 1 450 500
pjsip/exten_state 0 0 0 450 500
pjsip/messaging 0 0 0 450 500
pjsip/mwi-00000048 1038 0 7 450 500
pjsip/mwi-00000049 1001 0 7 450 500
pjsip/mwi-0000004a 965 0 7 450 500
pjsip/mwi-0000004b 926 0 7 450 500
pjsip/mwi-0000004c 886 0 7 450 500
pjsip/mwi-0000004d 845 0 6 450 500
pjsip/mwi-0000004e 807 0 6 450 500
pjsip/mwi-0000004f 769 0 6 450 500
SIP 12188 0 21 450 500
SIP-control 24833 0 20 450 500
Sorcery 4358 0 5 450 500
Sorcery-control 8754 0 4 450 500
sorcery/acl-0000005f 0 0 0 450 500
sorcery/aor-0000001a 1 0 1 450 500
sorcery/asterisk-publication-00000054 0 0 0 450 500
sorcery/auth-00000015 0 0 0 450 500
sorcery/bucket-00000000 0 0 0 450 500
sorcery/contact-00000019 84 0 2 1350 1500
sorcery/contact_status-0000001b 4272 0 3 1350 1500
sorcery/domain_alias-0000001c 0 0 0 450 500
sorcery/endpoint-00000016 1 0 1 450 500
sorcery/file-00000001 0 0 0 450 500
sorcery/global-0000001d 5 0 1 450 500
sorcery/identify-00000047 0 0 0 450 500
sorcery/inbound-publication-00000044 0 0 0 450 500
sorcery/log_mappings-0000000b 0 0 0 450 500
sorcery/nat_hook-00000017 0 0 0 450 500
sorcery/opus-0000000a 0 0 0 450 500
sorcery/outbound-publish-00000041 0 0 0 450 500
sorcery/registration-00000060 1 0 1 450 500
sorcery/resource_list-00000043 0 0 0 450 500
sorcery/subscription_persistence-00000042 0 0 0 450 500
sorcery/system-0000000c 0 0 0 450 500
sorcery/transport-00000018 0 0 0 450 500
stasis-core 2125 0 21 450 500
stasis-core-control 5596 0 31 450 500
subm:ast_bridge_topic_all-cached-00000063 1 0 1 450 500
subm:ast_channel_topic_all-00000069 1849 0 350 450 500
subm:ast_channel_topic_all-0000006c 1848 0 350 450 500
subm:ast_channel_topic_all-cached-00000062 2452 0 471 450 500
subm:ast_channel_topic_all-cached-00000064 2451 0 471 450 500
subm:ast_channel_topic_all-cached-00000068 2450 0 471 450 500
subm:ast_device_state_topic-00000002 95 0 18 450 500
subm:ast_device_state_topic-00000004 94 0 18 450 500
subm:ast_device_state_topic-0000006b 47 0 5 450 500
subm:ast_parking-00000046 1 0 1 450 500
subm:ast_presence_state_topic_all-00000005 1 0 1 450 500
subm:ast_security-00000065 5933 0 1 450 500
subm:ast_system-00000009 623 0 36 450 500
subm:ast_system-00000055 622 0 12 450 500
subm:ast_system-00000056 621 0 12 450 500
subm:ast_system-00000061 586 0 2 450 500
subm:cdr_engine-00000003 2473 0 471 4500 5000
subm:cel_aggregation_topic-00000006 2455 0 471 2700 3000
subm:endpoint_topic_all-cached-00000008 2673 0 471 450 500
subm:endpoint_topic_all-cached-00000067 2622 0 471 450 500
subm:manager_topic-00000007 9375 0 471 2700 3000
subm:stasis_mwi_topic-00000066 183 0 15 450 500
subm:stasis_mwi_topic-0000006a 182 0 15 450 500
subp:701@default-00000058 2 0 1 450 500
subp:701@device-0000006d 63 0 12 450 500
subp:701@device-0000006f 62 0 12 450 500
subp:701@device-00000070 61 0 11 450 500
subp:701@device-00000073 60 0 10 450 500
subp:701@device-00000074 59 0 9 450 500
subp:701@device-00000075 58 0 8 450 500
subp:701@device-00000077 57 0 7 450 500
subp:701@device-00000079 56 0 6 450 500
subp:701@device-0000007a 55 0 5 450 500
subp:701@device-0000007b 54 0 4 450 500
subp:701@device-0000007c 53 0 3 450 500
subp:701@device-0000007f 52 0 2 450 500
subp:701@device-00000080 51 0 1 450 500
subp:701@device-00000094 39 0 1 450 500
subp:701@device-000000d8 1 0 1 450 500
subp:702@device-00000071 115 0 7 450 500
subp:702@device-00000072 114 0 6 450 500
subp:702@device-00000076 113 0 5 450 500
subp:702@device-00000078 112 0 4 450 500
subp:702@device-0000007d 111 0 3 450 500
subp:702@device-0000007e 110 0 2 450 500
subp:702@device-00000081 109 0 1 450 500
subp:702@device-00000092 87 0 1 450 500
subp:702@device-00000096 84 0 1 450 500
subp:702@device-000000d9 1 0 1 450 500
subp:703@default-0000005a 2 0 1 450 500
subp:703@device-00000052 1 0 1 450 500
subp:PJSIP/701-0000001e 253 0 90 450 500
subp:PJSIP/702-0000001f 231 0 75 450 500
subp:PJSIP/703-00000020 3 0 3 450 500
subp:PJSIP/dpma_endpoint-00000021 3 0 3 450 500
subp:SIP/99701-00000057 2 0 2 450 500
subp:SIP/99703-00000059 2 0 2 450 500
subp:SIP/TollFreeProxy-0000005b 4 0 2 450 500
subp:SIP/vitel-inbound-0000005c 4 0 2 450 500
subp:SIP/Vitelity-0000005d 4 0 2 450 500
subp:SIP/VoIPms-0000005e 2117 0 471 450 500

144 taskprocessors

Leon


#11

Thanks for the suggestion, but, we do not have any problem with the tftp service. For us, it is clearly the asterisk service that eats all memory. Any idea or suggestion how to narrow our research to figure what in asterisk is doing that?


#12

If indeed it relates to the ARI application, can you give me any ideas what on the application can cause this?

Could be a number of things. For instance channels being created, but not properly hungup/destroyed, or bridges not being destroyed when done, etc…

From the Asterisk CLI what does “core show channels” look like? If it lists a bunch of channels that you know are no longer in use then perhaps they have not been hung up properly. Same for bridging. You can check if any bridges are still hanging around with “bridge show all”.


#13

these seems to be an issue when using realtime configuration I guess.I have the same issue with 2 different servers using Asterisk 13.17.2, and without any system load Asterisk start using all the system memory, and I need to keep restarting the Asterisks service to fix it.

a2billing*CLI> core show taskprocessors

Processor                                      Processed   In Queue  Max Depth  Low water High water
app_voicemail                                          0          0          0        450        500
ast_msg_queue                                          0          0          0        450        500
CCSS_core                                              0          0          0        450        500
hep_queue_tp                                           0          0          0        450        500
iax2_transmit                                          0          0          0        450        500
Sorcery                                                0          0          0        450        500
Sorcery-control                                        1          0          1        450        500
sorcery/bucket-00000000                                0          0          0        450        500
sorcery/file-00000001                                  0          0          0        450        500
stasis-core                                          306          0          2        450        500
stasis-core-control                                  763          0          2        450        500
subm:ast_bridge_topic_all-cached-00000012              1          0          1        450        500
subm:ast_channel_topic_all-00000017                 3911          0         69        450        500
subm:ast_channel_topic_all-cached-00000013          6040          0         69        450        500
subm:ast_device_state_topic-00000002                 792          0          2        450        500
subm:ast_device_state_topic-00000004                 791          0          3        450        500
subm:ast_device_state_topic-00000016                 784          0          3        450        500
subm:ast_presence_state_topic_all-00000005             1          0          1        450        500
subm:ast_security-00000014                         57995          0          2        450        500
subm:ast_system-00000009                              34          0         31        450        500
subm:ast_system-0000000a                              33          0         20        450        500
subm:ast_system-00000011                              32          0         20        450        500
subm:cdr_engine-00000003                            6206          0         69       4500       5000
subm:cel_aggregation_topic-00000006                 6042          0         69       2700       3000
subm:endpoint_topic_all-cached-00000008              883          0          3        450        500
subm:manager_topic-00000007                        65148          0         69       2700       3000
subm:stasis_mwi_topic-00000015                         1          0          1        450        500
subp:SIP/0978228585-0000000c                           2          0          1        450        500
subp:SIP/109-0000000d                                  2          0          1        450        500
subp:SIP/2936943285-0000000b                         292          0          2        450        500
subp:SIP/freepbx1-0000000e                             4          0          2        450        500
subp:SIP/OXILOG_92.222.202.150-0000000f                4          0          2        450        500
subp:SIP/voxbeam-00000010                              4          0          2        450        500

33 taskprocessors


#14

How can I check if my server is using realtime configuration? My server is 13.17.0 and this problem is driving me crazy. If you have any workaround I will be happy to try it.


#15

@kharwell @jcolp any thought about this issue guys ?


#16

I’m not aware of any changes that have happened to chan_sip realtime that would have caused such a thing. For memory leaks in general we need more information[1].

[1] https://wiki.asterisk.org/wiki/display/AST/Memory+Leak+Debugging