Memory consumption rising with time

[issam]

Hi everyone,

I have Asterisk 13.11 installed in a debian 8 machine. The memory consumption of asterisk rises with time until it is killed by the system with an Out of memory error, or i restart the service.

I had the same problem with 13.5. I do a lot of outbound dialing using AMI Originate, and i pass some variables with the originate action. And i use chan_sip. I disabled most of the modules that i don't use.

Here is the output of memory show summary. I see here that a lot is allocated to stasis_channels.c. Is it related to ARI? because i don't use it.

Thank you

memory show summary 
    386741 bytes in        513 allocations in file /usr/src/asterisk-13.11.0/include/asterisk/strings.h
    376848 bytes in       1355 allocations in file /usr/src/asterisk-13.11.0/include/asterisk/threadstorage.h
      1616 bytes in          7 allocations in file acl.c
       736 bytes in          1 allocations in file app_bridgewait.c
       992 bytes in          1 allocations in file app_confbridge.c
      3231 bytes in         18 allocations in file app_dial.c
      1520 bytes in         38 allocations in file app_macro.c
    161650 bytes in        160 allocations in file app_mixmonitor.c
    392552 bytes in       1037 allocations in file app_queue.c
      3248 bytes in         58 allocations in file app_stack.c
     13642 bytes in        587 allocations in file asterisk.c
   6354576 bytes in      79429 allocations in file astobj2.c
      2400 bytes in         20 allocations in file audiohook.c
       320 bytes in         20 allocations in file autochan.c
        96 bytes in          2 allocations in file autoservice.c
    196912 bytes in        610 allocations in file bridge.c
      7936 bytes in        160 allocations in file bridge_basic.c
     23040 bytes in         64 allocations in file bridge_channel.c
      1016 bytes in          1 allocations in file bucket.c
       896 bytes in          6 allocations in file ccss.c
    211994 bytes in        829 allocations in file cdr.c
      6968 bytes in          7 allocations in file cel.c
   1506812 bytes in       1716 allocations in file chan_sip.c
    469203 bytes in       1679 allocations in file channel.c
    136786 bytes in        381 allocations in file channel_internal_api.c
      8839 bytes in        452 allocations in file cli.c
      6176 bytes in         44 allocations in file codec.c
    155272 bytes in         73 allocations in file confbridge/conf_config_parser.c
     87537 bytes in        632 allocations in file config.c
     40568 bytes in        191 allocations in file config_options.c
       160 bytes in          1 allocations in file core_local.c
     17978 bytes in        156 allocations in file core_unreal.c
     16884 bytes in         27 allocations in file data.c
       306 bytes in         22 allocations in file datastore.c
   3761627 bytes in      45498 allocations in file devicestate.c
      4906 bytes in         55 allocations in file dial.c
      2880 bytes in          2 allocations in file dsp.c
    471728 bytes in       2183 allocations in file endpoints.c
      3072 bytes in        128 allocations in file features.c
    105235 bytes in        817 allocations in file features_config.c
    675406 bytes in         72 allocations in file file.c
      9554 bytes in        137 allocations in file format.c
      1016 bytes in          1 allocations in file format_cache.c
   2057912 bytes in      19016 allocations in file format_cap.c
    230879 bytes (    230879 cache) in        601 allocations in file frame.c
        48 bytes in          2 allocations in file func_curl.c
       736 bytes in          1 allocations in file func_dialgroup.c
     14245 bytes in        124 allocations in file func_lock.c
    133653 bytes in        411 allocations in file func_odbc.c
     12696 bytes in         41 allocations in file hashtab.c
     78554 bytes in        548 allocations in file http.c
     66444 bytes in       1191 allocations in file indications.c
      8228 bytes in          4 allocations in file io.c
 694852603 bytes in    5841291 allocations in file json.c
      2296 bytes in          1 allocations in file libasteriskssl.c
     24467 bytes in        206 allocations in file loader.c
     10398 bytes in         62 allocations in file logger.c
  56076693 bytes in      97726 allocations in file manager.c
      1208 bytes in         99 allocations in file max_forwards.c
   3567568 bytes in      20437 allocations in file media_index.c
        96 bytes in          2 allocations in file message.c
       832 bytes in          2 allocations in file named_acl.c
      1760 bytes in          1 allocations in file named_locks.c
    301938 bytes in       3159 allocations in file pbx.c
    335879 bytes in        427 allocations in file pbx_app.c
     30066 bytes in        859 allocations in file pbx_config.c
    348432 bytes in        349 allocations in file pbx_functions.c
       725 bytes in         29 allocations in file pbx_hangup_handler.c
     52155 bytes in       1234 allocations in file pbx_variables.c
        56 bytes in          1 allocations in file presencestate.c
       448 bytes in          1 allocations in file res_calendar.c
      2669 bytes in          6 allocations in file res_clialiases.c
      1118 bytes in         23 allocations in file res_config_sqlite.c
       256 bytes in          1 allocations in file res_config_sqlite3.c
       320 bytes in          2 allocations in file res_fax.c
     36263 bytes in        100 allocations in file res_http_websocket.c
     27225 bytes in         49 allocations in file res_musiconhold.c
       752 bytes in          5 allocations in file res_odbc.c
   1801455 bytes in      39373 allocations in file res_rtp_asterisk.c
    619632 bytes in        216 allocations in file res_srtp.c
     13192 bytes in         97 allocations in file res_timing_timerfd.c
  11903866 bytes in      92944 allocations in file rtp_engine.c
    270232 bytes in       1035 allocations in file sched.c
       288 bytes in         18 allocations in file sdp_srtp.c
      3143 bytes in         52 allocations in file sip/route.c
     32528 bytes in          2 allocations in file smoother.c
      7981 bytes in         34 allocations in file sorcery.c
   4495009 bytes in      48335 allocations in file stasis.c
  14101952 bytes in      70908 allocations in file stasis_bridges.c
  16985391 bytes in     302065 allocations in file stasis_cache.c
     46680 bytes in        389 allocations in file stasis_cache_pattern.c
1399673157 bytes in    1589412 allocations in file stasis_channels.c
     22256 bytes in        214 allocations in file stasis_endpoints.c
 185585653 bytes in    1364699 allocations in file stasis_message.c
     59688 bytes in        536 allocations in file stasis_message_router.c
     37104 bytes in          2 allocations in file stdtime/localtime.c
  56542320 bytes in     332585 allocations in file stringfields.c
   4981983 bytes in      40242 allocations in file strings.c
  43394553 bytes in    1354311 allocations in file taskprocessor.c
     21106 bytes in        130 allocations in file tcptls.c
     27104 bytes in        288 allocations in file threadpool.c
      1712 bytes in        100 allocations in file timing.c
   1754032 bytes in        148 allocations in file translate.c
       208 bytes in          2 allocations in file udptl.c
     15698 bytes in        207 allocations in file utils.c
   1029575 bytes in       5492 allocations in file xmldoc.c
2517337991 bytes allocated (230879 in caches) in 11371037 selected allocations

2517337991 bytes in all allocations
    544312 bytes in deferred free large allocations
     45892 bytes in deferred free small allocations
    590204 bytes in deferred free allocations
2517928195 bytes in all allocations and deferred free allocations

[mkozusnik]

Working on the same problem over here...

Help finding a memory leak

New 13.5 installation on an i7 with 16 GB of RAM and 2x SSDs. Low usage, but memory grows about 8% per day under load. This site is closed after hours, so the usage stops growing at night. I can't seem to track down where the memory is going. I read that at 13.8, the taskprocessor shows individual modules. If I can figure it out on 13.5, I'll leave it there, otherwise, I'll have to upgrade. Recompiled with malloc_debug. Never used Valgrind, so I'm trying to avoid it for now. Any suggestions wo…

[issam]

Hello again.

So, yesterday at this time, asterisk was taking 51% of memory, and after two hours it would reach 90%, and eventually be killed if i don't restart it.

What i did was disabling logging to a file, and reduced the logs for console to error only, and it seems that solved the problem, since asterisk is consuming now 3%, compared to 50% yesterday in the same time.

I don't understund what is the relationship between logging and memory? Isn't there another way to reduce memory consumption while keeping logs in a file? The logs are rotated by the way.

Thank you

[issam]

The monologue continues

The memory consumption started to grow again, and at this rate, the memory will be saturated in a few hours. So that means that disabling the logs wasn't effective, but i have no idea why the memory usage didn't start growing until the aternoon.

I have 12G of ram, so it's not a question of not having enough memory

[gjoseph]

Yeah, I think this is related to @mkozusnik's issue. I'm investigating now.

[gjoseph]

Do you use realtime for voicemail or sip peers?
If so, do you have rtpcachefriends set to yes or no?

[issam]

No, i use sip.conf for sip peers, and don't use voicemail. I mainly do outbound calls, and just a few inbound calls that go to a queue. And i also use odbc extensively.
I just installed 13.11.2, i'll see tomorrow if there is any improvment.

[gjoseph]

OK, I think you have a different issue than @mkozusnik.
The output from 'core show taskprocessors' will at least tell us if the internal queues are backing up.

[issam]

Thank you for your time

Below is the output of core show taskprocessors. There is a line where "In queue" is high

subm:manager_topic-00000006 3942894 106211 106349 2700 3000

If this is related to AMI? because i do a lot of outbound calls using AMI Originate, sometimes 10-20 per second, and i also send some variables in the Originate action to use in the dialplan

---

Processor                                      Processed   In Queue  Max Depth  Low water High water
ast_msg_queue                                          0          0          0        450        500
CCSS_core                                              0          0          0        450        500
Sorcery                                                0          0          0        450        500
Sorcery-control                                        1          0          1        450        500
sorcery/bucket-00000000                                0          0          0        450        500
sorcery/file-00000001                                  0          0          0        450        500
stasis-core                                      6056113          0         11        450        500
stasis-core-control                             12113030          0         70        450        500
subm:ast_bridge_topic_all-cached-0000010a         142077          0         15        450        500
subm:ast_channel_topic_all-0000010e              2770482          0         39        450        500
subm:ast_channel_topic_all-cached-0000010b       3604842          0         46        450        500
subm:ast_device_state_topic-00000002              127209          0         46        450        500
subm:ast_device_state_topic-00000004              127208          0         25        450        500
subm:ast_device_state_topic-0000010d              126801          0         12        450        500
subm:ast_presence_state_topic_all-00000005             1          0          1        450        500
subm:ast_security-0000010c                         22085          0          1        450        500
subm:ast_system-00000008                              14          0         12        450        500
subm:ast_system-00000009                              13          0          3        450        500
subm:cdr_engine-00000003                         3746920          0        318       4500       5000
subm:endpoint_topic_all-cached-00000007          1824346          0         39        450        500
subm:manager_topic-00000006                      3942894     106211     106349       2700       3000
subp:SIP/1000-0000000a                               200          0          4        450        500
subp:SIP/1001-0000000b                                 2          0          2        450        500
subp:SIP/1002-0000000c                                 2          0          2        450        500
subp:SIP/1003-0000000d                                 6          0          2        450        500
subp:SIP/1004-0000000e                                26          0          2        450        500
subp:SIP/1005-0000000f                                 2          0          2        450        500
subp:SIP/1006-00000010                                 2          0          2        450        500
subp:SIP/1007-00000011                                 2          0          2        450        500
subp:SIP/1008-00000012                                 2          0          2        450        500
subp:SIP/1009-00000013                                 2          0          2        450        500
subp:SIP/1010-00000014                                 2          0          2        450        500
subp:SIP/5000-00000017                                 2          0          2        450        500
subp:SIP/5001-00000018                                 2          0          2        450        500
subp:SIP/5002-000000b2                                 2          0          2        450        500
subp:SIP/5003-00000039                                 2          0          2        450        500
subp:SIP/5004-00000019                                 2          0          2        450        500
subp:SIP/6001-0000001c                                 2          0          2        450        500
subp:SIP/6002-0000001a                                 2          0          2        450        500
subp:SIP/6003-0000001b                                 2          0          2        450        500
subp:SIP/6004-0000001d                                 2          0          2        450        500
subp:SIP/6005-00000020                                 2          0          2        450        500
subp:SIP/6006 -00000040                                2          0          2        450        500
subp:SIP/6008-00000038                                 2          0          2        450        500
subp:SIP/6009-0000001e                                 2          0          2        450        500
subp:SIP/6010-0000001f                                 2          0          2        450        500
subp:SIP/6011-00000021                                 2          0          2        450        500
subp:SIP/6012-00000022                                 2          0          2        450        500
subp:SIP/6013-00000032                                 2          0          2        450        500
subp:SIP/6014-00000033                                 2          0          2        450        500
subp:SIP/6015-00000034                                 2          0          2        450        500
subp:SIP/6016-0000003b                                 2          0          2        450        500
subp:SIP/6017-00000026                                 2          0          2        450        500
subp:SIP/6018-00000028                                 2          0          2        450        500
subp:SIP/6019-0000003a                                 2          0          2        450        500
subp:SIP/6020-00000027                                 2          0          2        450        500
subp:SIP/6021-00000029                                 2          0          2        450        500
subp:SIP/6022-0000002a                                 2          0          2        450        500
subp:SIP/6023-0000002b                                 2          0          2        450        500
subp:SIP/6024-0000002c                                 2          0          2        450        500
subp:SIP/6025-0000002d                                 2          0          2        450        500
subp:SIP/6026-00000031                                 2          0          2        450        500
subp:SIP/6027-00000035                                 2          0          2        450        500
subp:SIP/6028-00000030                                 2          0          2        450        500
subp:SIP/6029-00000036                                 2          0          2        450        500
subp:SIP/6030-00000037                                 2          0          2        450        500
subp:SIP/6031-0000003c                                 2          0          2        450        500
subp:SIP/6032-0000003d                                 2          0          2        450        500
subp:SIP/6033-0000003e                                 2          0          2        450        500
subp:SIP/6034-0000003f                                 2          0          2        450        500
subp:SIP/6035-00000041                                 2          0          2        450        500
subp:SIP/6036-00000042                                 2          0          2        450        500
subp:SIP/6037-00000043                                 2          0          2        450        500
subp:SIP/6038-00000044                                 2          0          2        450        500
subp:SIP/6039-00000045                                 2          0          2        450        500
subp:SIP/6040-00000046                                 2          0          2        450        500
subp:SIP/6041-00000047                                 2          0          2        450        500
subp:SIP/6042-00000048                                 2          0          2        450        500
subp:SIP/6043-00000049                                 2          0          2        450        500
subp:SIP/6044-0000004a                                 2          0          2        450        500
subp:SIP/6045-0000004b                                 2          0          2        450        500
subp:SIP/6046-00000059                                 2          0          2        450        500
subp:SIP/6047-0000005a                                 2          0          2        450        500
subp:SIP/6048-0000005b                                 2          0          2        450        500
subp:SIP/6049-0000005c                                 2          0          2        450        500
subp:SIP/6050-0000005d                                 2          0          2        450        500
subp:SIP/6051-0000005e                                 2          0          2        450        500
subp:SIP/6052-0000005f                                 2          0          2        450        500
subp:SIP/6053-00000060                                 2          0          2        450        500
subp:SIP/6054-00000061                                 2          0          2        450        500
subp:SIP/6055-00000062                                 2          0          2        450        500
subp:SIP/6056-00000063                                 2          0          2        450        500
subp:SIP/6057-00000064                                 2          0          2        450        500
subp:SIP/6058-00000065                                 2          0          2        450        500
subp:SIP/6059-00000066                                 2          0          2        450        500
subp:SIP/6060-00000067                                 2          0          2        450        500
subp:SIP/6061-00000068                                 2          0          2        450        500
subp:SIP/6062-00000069                                 2          0          2        450        500
subp:SIP/6063-0000006a                                 2          0          2        450        500
subp:SIP/6064-0000006b                                 2          0          2        450        500
subp:SIP/6065-0000006c                                 2          0          2        450        500
subp:SIP/6066-0000006d                                 2          0          2        450        500
subp:SIP/6067-0000006e                                 2          0          2        450        500
subp:SIP/6068-0000006f                                 2          0          2        450        500
subp:SIP/6069-00000070                                 2          0          2        450        500
subp:SIP/6070-00000071                                 2          0          2        450        500
subp:SIP/6071-00000072                                 2          0          2        450        500
subp:SIP/6072-00000073                                 2          0          2        450        500
subp:SIP/6073-00000074                                 2          0          2        450        500
subp:SIP/6074-00000075                                 2          0          2        450        500
subp:SIP/6075-00000076                                 2          0          2        450        500
subp:SIP/6076-00000077                                 2          0          2        450        500
subp:SIP/6077-00000078                                 2          0          2        450        500
subp:SIP/6078-00000079                                 2          0          2        450        500
subp:SIP/6079-0000007a                                 2          0          2        450        500
subp:SIP/6080-0000007b                                 2          0          2        450        500
subp:SIP/6081-0000007c                                 2          0          2        450        500
subp:SIP/6082-0000007d                                 2          0          2        450        500
subp:SIP/6083-0000007e                                 2          0          2        450        500
subp:SIP/6084-0000007f                                 2          0          2        450        500
subp:SIP/6085-00000080                                 2          0          2        450        500
subp:SIP/6086-00000081                                 2          0          2        450        500
subp:SIP/6087-00000082                                 2          0          2        450        500
subp:SIP/6088-00000083                                 2          0          2        450        500
subp:SIP/6089-00000084                                 2          0          2        450        500
subp:SIP/6090-00000085                                 2          0          2        450        500
subp:SIP/6091-00000086                                 2          0          2        450        500
subp:SIP/6092-00000087                                 2          0          2        450        500
subp:SIP/6093-00000088                                 2          0          2        450        500
subp:SIP/6094-00000089                                 2          0          2        450        500
subp:SIP/6095-0000008a                                 2          0          2        450        500
subp:SIP/6096-0000008b                                 2          0          2        450        500
subp:SIP/6097-0000008c                                 2          0          2        450        500
subp:SIP/6098-0000008d                                 2          0          2        450        500
subp:SIP/6099-0000008e                                 2          0          2        450        500
subp:SIP/6100-0000008f                                 2          0          2        450        500
subp:SIP/6101-00000090                                 2          0          2        450        500
subp:SIP/7000-00000058                                 2          0          2        450        500
subp:SIP/7005-0000004c                                 2          0          2        450        500
subp:SIP/7006-0000004d                                 2          0          2        450        500
subp:SIP/7007-0000004e                                 2          0          2        450        500
subp:SIP/7013-00000023                                 2          0          2        450        500
subp:SIP/7014-00000024                                 2          0          2        450        500
subp:SIP/7015-00000025                                 2          0          2        450        500
subp:SIP/7026-0000002e                                 2          0          2        450        500
subp:SIP/7027-0000002f                                 2          0          2        450        500
subp:SIP/7028-00000091                                 2          0          2        450        500
subp:SIP/7029-00000092                                 2          0          2        450        500
subp:SIP/7030-00000093                                 2          0          2        450        500
subp:SIP/7031-00000094                                 2          0          2        450        500
subp:SIP/7032-00000095                                 2          0          2        450        500
subp:SIP/7033-00000096                                 2          0          2        450        500
subp:SIP/7034-00000097                                 2          0          2        450        500
subp:SIP/7035-00000098                                 2          0          2        450        500
subp:SIP/7036-00000099                                 2          0          2        450        500
subp:SIP/7037-0000009a                                 2          0          2        450        500
subp:SIP/7038-0000009b                                 2          0          2        450        500
subp:SIP/7039-0000009c                                 2          0          2        450        500
subp:SIP/7040-0000009d                                 2          0          2        450        500
subp:SIP/7041-0000009e                                 2          0          2        450        500
subp:SIP/7042-0000009f                                 2          0          2        450        500
subp:SIP/7043-000000a0                                 2          0          2        450        500
subp:SIP/7044-000000a1                                 2          0          2        450        500
subp:SIP/7045-000000a2                                 2          0          2        450        500
subp:SIP/7046-000000a3                                 2          0          2        450        500
subp:SIP/7047-000000a4                                 2          0          2        450        500
subp:SIP/7048-000000a5                                 2          0          2        450        500
subp:SIP/7049-000000a6                                 2          0          2        450        500
subp:SIP/7050-000000a7                                 2          0          2        450        500
subp:SIP/7051-000000a8                                 2          0          2        450        500
subp:SIP/7052-000000a9                                 2          0          2        450        500
subp:SIP/7053-000000aa                                 2          0          2        450        500
subp:SIP/7054-000000ab                                 2          0          2        450        500
subp:SIP/7055-000000ac                                 2          0          2        450        500
subp:SIP/7056-000000ad                                 2          0          2        450        500
subp:SIP/7057-000000ae                                 2          0          2        450        500
subp:SIP/7058-000000af                                 2          0          2        450        500
subp:SIP/7059-000000b0                                 2          0          2        450        500
subp:SIP/7060-000000b1                                 2          0          2        450        500
subp:SIP/7090-000000b3                                 2          0          2        450        500
subp:SIP/9000-0000004f                                 2          0          1        450        500
subp:SIP/9001-00000050                                 2          0          2        450        500
subp:SIP/9002-00000051                                 2          0          2        450        500
subp:SIP/9003-00000052                                 2          0          2        450        500
subp:SIP/9004-00000053                                 2          0          2        450        500
subp:SIP/9005-00000054                                 2          0          2        450        500
subp:SIP/9006-00000055                                 2          0          2        450        500
subp:SIP/9007-00000056                                 2          0          2        450        500
subp:SIP/9008-00000057                                 2          0          2        450        500
subp:SIP/9900-000000b4                                 2          0          2        450        500
subp:SIP/9901-000000b5                               786          0         14        450        500
subp:SIP/9902-000000b6                              4134          0         12        450        500
subp:SIP/9903-000000b7                                 2          0          2        450        500
subp:SIP/9904-000000b8                              1025          0          9        450        500
subp:SIP/9905-000000b9                               161          0          3        450        500
subp:SIP/9906-000000ba                              4958          0         15        450        500
subp:SIP/9907-000000bb                              8802          0         15        450        500
subp:SIP/9908-000000bc                              1102          0          8        450        500
subp:SIP/9909-000000bd                              7871          0         12        450        500
subp:SIP/9910-000000be                                 2          0          2        450        500
subp:SIP/9911-000000bf                              5958          0         17        450        500
subp:SIP/9912-000000c0                              5189          0         17        450        500
subp:SIP/9913-000000c1                              7920          0         16        450        500
subp:SIP/9914-000000c2                              7617          0         19        450        500
subp:SIP/9915-000000c3                              2347          0         17        450        500
subp:SIP/9916-000000c4                              7303          0         20        450        500
subp:SIP/9917-000000c5                              6443          0         13        450        500
subp:SIP/9918-000000c6                              6928          0         15        450        500
subp:SIP/9919-000000c7                              5654          0         11        450        500
subp:SIP/9920-000000c8                              6440          0         17        450        500
subp:SIP/9921-000000c9                               155          0          5        450        500
subp:SIP/9922-000000ca                              5224          0         17        450        500
subp:SIP/9923-000000cb                              5916          0          9        450        500
subp:SIP/9924-000000cc                              9013          0         18        450        500
subp:SIP/9925-000000cd                             11702          0         17        450        500
subp:SIP/9926-000000ce                              9911          0         17        450        500
subp:SIP/9927-000000cf                             11530          0         20        450        500
subp:SIP/9928-000000d0                              3605          0         15        450        500
subp:SIP/9929-000000d1                              8587          0         11        450        500
subp:SIP/9930-000000d2                              7082          0         12        450        500
subp:SIP/9931-000000d3                              6875          0         18        450        500
subp:SIP/9932-000000d4                              4573          0          9        450        500
subp:SIP/9933-000000d5                                 2          0          2        450        500
subp:SIP/9934-000000d6                              4422          0         17        450        500
subp:SIP/9935-000000d7                              2890          0         12        450        500
subp:SIP/9936-000000d8                              7304          0         17        450        500
subp:SIP/9937-000000d9                              6983          0         17        450        500
subp:SIP/9938-000000da                              5843          0          9        450        500
subp:SIP/9939-000000db                                16          0          2        450        500
subp:SIP/9940-000000dc                                 2          0          2        450        500
subp:SIP/9941-000000dd                               722          0          4        450        500
subp:SIP/9942-000000de                                 2          0          2        450        500
subp:SIP/9943-000000df                                 2          0          2        450        500
subp:SIP/9944-000000e0                                 2          0          2        450        500
subp:SIP/9945-000000e1                                 2          0          2        450        500
subp:SIP/9946-000000e2                                 2          0          2        450        500
subp:SIP/9947-000000e3                                 2          0          2        450        500
subp:SIP/9948-000000e4                                 2          0          2        450        500
subp:SIP/9949-000000e5                                 2          0          1        450        500
subp:SIP/9950-000000e6                                 2          0          2        450        500
subp:SIP/9951-000000e7                                 2          0          2        450        500
subp:SIP/9952-000000e8                                 2          0          2        450        500
subp:SIP/9953-000000e9                                 2          0          2        450        500
subp:SIP/9954-000000ea                                 2          0          2        450        500
subp:SIP/9955-000000eb                                 2          0          2        450        500
subp:SIP/9956-000000ec                                 2          0          2        450        500
subp:SIP/9957-000000ed                                 2          0          2        450        500
subp:SIP/9958-000000ee                                 2          0          2        450        500
subp:SIP/9959-000000ef                                 2          0          2        450        500
subp:SIP/9960-000000f0                                 2          0          2        450        500
subp:SIP/9961-000000f1                                 2          0          2        450        500
subp:SIP/9962-000000f2                                 2          0          2        450        500
subp:SIP/9963-000000f3                                 2          0          2        450        500
subp:SIP/9964-000000f4                                 2          0          2        450        500
subp:SIP/9965-000000f5                                 2          0          2        450        500
subp:SIP/9966-000000f6                                 2          0          2        450        500
subp:SIP/9967-000000f7                                 2          0          2        450        500
subp:SIP/9968-000000f8                                 2          0          2        450        500
subp:SIP/9969-000000f9                                 2          0          2        450        500
subp:SIP/9970-000000fa                                 2          0          2        450        500
subp:SIP/9971-000000fb                                 2          0          2        450        500
subp:SIP/9972-000000fc                                 2          0          2        450        500
subp:SIP/9973-000000fd                                 2          0          2        450        500
subp:SIP/9974-000000fe                                 2          0          2        450        500
subp:SIP/9975-000000ff                                 2          0          2        450        500
subp:SIP/9976-00000100                                 2          0          2        450        500
subp:SIP/9977-00000101                                 2          0          2        450        500
subp:SIP/9978-00000102                                 2          0          2        450        500
subp:SIP/9979-00000103                                 2          0          2        450        500
subp:SIP/9980-00000104                                 2          0          2        450        500

277 taskprocessors

[gjoseph]

subm:manager_topic-00000006 3942894 106211 106349 2700 3000

Those are not good numbers.

If this is related to AMI? because i do a lot of outbound calls using AMI Originate, sometimes 10-20 per second, and i also send some variables in the Originate action to use in the dialplan

Yeah, something's not processing the internal messages related to manager and as the messages pile up, so does the memory.

Please open an issue at https://issues.asterisk.org. This is going to require more in-depth investigation.

[issam]

Ok, i found the problem.

It is indeed related to AMI. For each call connected, a web page is opened and there is a connection to manager via http. I log in and then execute an action. This is done for each call. The problem here is that i wasn't logging off. This didn't cause any problem in previous versions of Asterisk, may be the connection was timing out or something.

Anyway, now i just send a logoff action for each connection i open, and it seems that the problem is solved.

Thank you for your help

[gjoseph]

Anyway, now i just send a logoff action for each connection i open, and it seems that the problem is solved.

Good to hear you found a solution.

[fvizz]

Hi I’m new to the support forum and before to open a new thread, i try to continue this with a problem which seems related,I hope it is correct.
I use Asterisk 13.17.0 to manage calls and audio routing between 50-60 SIP users, the calls are placed and routed by
an application conneted wit AMI interface using Originate Message and variable setting; the application do only
one connection to asterisk and remains connected for all his life.
Each SIP user send register every 30 sec (set in sip.conf).
For 6 days everything is ok, then the CPU consumption pass from 20% to 45%, and the messages file show the follow message :

[Sep 12 00:35:41] WARNING[14126][C-00000015] taskprocessor.c: The ‘subm:manager_topic-00000006’ task processor queue reached 3000 scheduled tasks.
[Sep 12 00:35:50] WARNING[27816][C-00000024] taskprocessor.c: The ‘subm:manager_topic-00000006’ task processor queue reached 3000 scheduled tasks

again.

After 6 days asterisk becomes less responsive, i’m monitoring the beahviour.
Any suggestions would be strongly appreciated Thanks.

The show taskprocessor output after 7 day of activity is:

Connected to Asterisk 13.17.0 currently running on mercury (pid = 3859)
mercuryCLI>
mercuryCLI>
mercury*CLI> core show taskprocessors

Processor Processed In Queue Max Depth Low water High water
ast_msg_queue 0 0 0 450 500
CCSS_core 0 0 0 450 500
Sorcery 0 0 0 450 500
Sorcery-control 1 0 1 450 500
sorcery/bucket-00000000 0 0 0 450 500
sorcery/file-00000001 0 0 0 450 500
stasis-core 384557 0 2 450 500
stasis-core-control 769302 0 8 450 500
subm:ast_bridge_topic_all-cached-00000096 33 0 5 450 500
subm:ast_channel_topic_all-0000009a 388094 0 25 450 500
subm:ast_channel_topic_all-0000009d 388088 0 24 450 500
subm:ast_channel_topic_all-cached-00000097 389771 0 35 450 500
subm:ast_channel_topic_all-cached-00000098 389770 0 56 450 500
subm:ast_channel_topic_all-cached-00000099 389769 0 56 450 500
subm:ast_device_state_topic-00000002 365568 0 14 450 500
subm:ast_device_state_topic-00000004 365567 0 11 450 500
subm:ast_device_state_topic-0000009c 365548 0 4 450 500
subm:ast_parking-00000009 1 0 1 450 500
subm:ast_presence_state_topic_all-00000005 1 0 1 450 500
subm:ast_security-0000009b 728572 0 2 450 500
subm:ast_system-00000008 28119 0 31 450 500
subm:ast_system-0000000a 28118 0 6 450 500
subm:cdr_engine-00000003 389806 0 35 4500 5000
subm:endpoint_topic_all-cached-00000007 1096783 0 26 450 500
subm:manager_topic-00000006 1718271287 1 33986 2700 3000
subp:SIP/3101-0000000c 635 0 26 450 500
subp:SIP/3102-0000000d 2 0 1 450 500
subp:SIP/3103-0000000e 2 0 1 450 500
subp:SIP/3104-0000000f 43759 0 18 450 500
subp:SIP/3105-00000010 2 0 1 450 500
subp:SIP/3106-00000011 2 0 1 450 500
subp:SIP/3107-00000012 2 0 1 450 500
subp:SIP/3108-00000013 2 0 1 450 500
subp:SIP/3109-00000014 2 0 1 450 500
subp:SIP/3110-00000015 43148 0 8 450 500
subp:SIP/3111-00000016 43142 0 2 450 500
subp:SIP/3112-00000017 43136 0 3 450 500
subp:SIP/3113-00000018 43130 0 3 450 500
subp:SIP/3114-00000019 43126 0 6 450 500
subp:SIP/3115-0000001a 43120 0 3 450 500
subp:SIP/3116-0000001b 43116 0 2 450 500
subp:SIP/3117-0000001c 43110 0 2 450 500
subp:SIP/3118-0000001d 43106 0 3 450 500
subp:SIP/3119-0000001e 39288 0 2 450 500
subp:SIP/3120-0000001f 2 0 1 450 500
subp:SIP/3121-00000020 2 0 1 450 500
subp:SIP/3122-00000021 74 0 2 450 500
subp:SIP/3123-00000022 2 0 1 450 500
subp:SIP/3124-00000023 2 0 1 450 500
subp:SIP/3125-00000024 2 0 1 450 500
subp:SIP/3126-00000025 2 0 1 450 500
subp:SIP/3127-00000026 2 0 1 450 500
subp:SIP/3128-00000027 2 0 1 450 500
subp:SIP/3129-00000028 2 0 1 450 500
subp:SIP/3130-00000029 2 0 1 450 500
subp:SIP/3131-0000002a 2 0 1 450 500
subp:SIP/3132-0000002b 2 0 1 450 500
subp:SIP/3133-0000002c 2 0 1 450 500
subp:SIP/3134-0000002d 38908 0 30 450 500
subp:SIP/3135-0000002e 39266 0 23 450 500
subp:SIP/3136-0000002f 44267 0 33 450 500
subp:SIP/3137-00000030 39452 0 34 450 500
subp:SIP/3138-00000031 2 0 1 450 500
subp:SIP/3139-00000032 2 0 2 450 500
subp:SIP/3140-00000033 39218 0 3 450 500
subp:SIP/3141-00000034 2 0 2 450 500
subp:SIP/3142-00000035 2 0 2 450 500
subp:SIP/3143-00000036 2 0 1 450 500
subp:SIP/3144-00000037 2 0 2 450 500
subp:SIP/3145-00000038 2 0 2 450 500
subp:SIP/3146-00000039 2 0 2 450 500
subp:SIP/3147-0000003a 2 0 2 450 500
subp:SIP/3148-0000003b 2 0 2 450 500
subp:SIP/3149-0000003c 2 0 2 450 500
subp:SIP/3150-0000003d 2 0 2 450 500
subp:SIP/3151-0000003e 2 0 2 450 500
subp:SIP/3152-0000003f 2 0 1 450 500
subp:SIP/3153-00000040 2 0 2 450 500
subp:SIP/3154-00000041 2 0 2 450 500
subp:SIP/3155-00000042 2 0 2 450 500
subp:SIP/3156-00000043 2 0 2 450 500
subp:SIP/3157-00000044 2 0 2 450 500
subp:SIP/3158-00000045 2 0 2 450 500
subp:SIP/3159-00000046 2 0 2 450 500
subp:SIP/3160-00000047 2 0 2 450 500
subp:SIP/3161-00000048 2 0 2 450 500
subp:SIP/3162-00000049 2 0 2 450 500
subp:SIP/3163-0000004a 2 0 2 450 500
subp:SIP/3164-0000004b 2 0 2 450 500
subp:SIP/3165-0000004c 2 0 2 450 500
subp:SIP/3166-0000004d 2 0 2 450 500
subp:SIP/3167-0000004e 2 0 2 450 500
subp:SIP/3168-0000004f 2 0 2 450 500
subp:SIP/3169-00000050 2 0 2 450 500
subp:SIP/3170-00000051 2 0 2 450 500
subp:SIP/3171-00000052 2 0 2 450 500
subp:SIP/3172-00000053 2 0 2 450 500
subp:SIP/3173-00000054 2 0 2 450 500
subp:SIP/3174-00000055 2 0 2 450 500
subp:SIP/3175-00000056 2 0 2 450 500
subp:SIP/3176-00000057 2 0 2 450 500
subp:SIP/3177-00000058 2 0 2 450 500
subp:SIP/3178-00000059 2 0 2 450 500
subp:SIP/3179-0000005a 2 0 2 450 500
subp:SIP/3180-0000005b 2 0 2 450 500
subp:SIP/3181-0000005c 2 0 2 450 500
subp:SIP/3182-0000005d 2 0 2 450 500
subp:SIP/3183-0000005e 2 0 2 450 500
subp:SIP/3184-0000005f 2 0 2 450 500
subp:SIP/3185-00000060 2 0 2 450 500
subp:SIP/3186-00000061 2 0 2 450 500
subp:SIP/3187-00000062 2 0 2 450 500
subp:SIP/3188-00000063 2 0 2 450 500
subp:SIP/3189-00000064 2 0 2 450 500
subp:SIP/3190-00000065 2 0 2 450 500
subp:SIP/3191-00000066 2 0 2 450 500
subp:SIP/3192-00000067 2 0 2 450 500
subp:SIP/3193-00000068 2 0 2 450 500
subp:SIP/3194-00000069 2 0 2 450 500
subp:SIP/3195-0000006a 2 0 2 450 500
subp:SIP/3196-0000006b 2 0 2 450 500
subp:SIP/3197-0000006c 2 0 2 450 500
subp:SIP/3198-0000006d 2 0 2 450 500
subp:SIP/3199-0000006e 2 0 2 450 500
subp:SIP/3200-0000006f 2 0 2 450 500
subp:SIP/4100-00000094 2 0 2 450 500
subp:SIP/4200-00000095 370 0 2 450 500
subp:SIP/5000-00000070 2 0 2 450 500
subp:SIP/5001-00000071 58531 0 7 450 500
subp:SIP/5002-00000072 2 0 2 450 500
subp:SIP/5003-00000073 2 0 2 450 500
subp:SIP/5004-00000074 2 0 2 450 500
subp:SIP/5005-00000075 2 0 2 450 500
subp:SIP/5006-00000076 2 0 2 450 500
subp:SIP/5007-00000077 2 0 2 450 500
subp:SIP/5008-00000078 2 0 2 450 500
subp:SIP/5009-00000079 2 0 2 450 500
subp:SIP/5010-0000007a 2 0 2 450 500
subp:SIP/6000-0000007c 2 0 2 450 500
subp:SIP/6001-0000007d 2 0 2 450 500
subp:SIP/6002-0000007e 2 0 2 450 500
subp:SIP/6003-0000007f 2 0 2 450 500
subp:SIP/6004-00000080 2 0 2 450 500
subp:SIP/6005-00000081 2 0 2 450 500
subp:SIP/6006-00000082 2 0 2 450 500
subp:SIP/6007-00000083 2 0 2 450 500
subp:SIP/6008-00000084 2 0 2 450 500
subp:SIP/6009-00000085 2 0 2 450 500
subp:SIP/6010-00000086 2 0 2 450 500
subp:SIP/6011-00000087 2 0 2 450 500
subp:SIP/6012-00000088 2 0 2 450 500
subp:SIP/6013-00000089 2 0 2 450 500
subp:SIP/6014-0000008a 2 0 2 450 500
subp:SIP/6015-0000008b 2 0 2 450 500
subp:SIP/6016-0000008c 2 0 2 450 500
subp:SIP/6017-0000008d 2 0 2 450 500
subp:SIP/6018-0000008e 2 0 2 450 500
subp:SIP/6019-0000008f 2 0 2 450 500
subp:SIP/6020-00000090 2 0 2 450 500
subp:SIP/7000-0000007b 2 0 2 450 500
subp:SIP/7001-00000091 2 0 2 450 500
subp:SIP/7002-00000092 2 0 2 450 500
subp:SIP/7003-00000093 2 0 2 450 500
subp:SIP/svr-trunk0-0000000b 2 0 2 450 500

164 taskprocessors

[david551]

I would start a new thread, because:

1. you are adding to a rather old thread;

2. you haven’t provided evidence that they underlying cause is the same.

[fvizz]

Ok , I 'll do as you suggest.
Created a new Topic "help finding CPU consumption"
Thanks