Help finding a memory leak

New 13.5 installation on an i7 with 16 GB of RAM and 2x SSDs. Low usage, but memory grows about 8% per day under load. This site is closed after hours, so the usage stops growing at night. I can’t seem to track down where the memory is going. I read that at 13.8, the taskprocessor shows individual modules. If I can figure it out on 13.5, I’ll leave it there, otherwise, I’ll have to upgrade. Recompiled with malloc_debug. Never used Valgrind, so I’m trying to avoid it for now.

Any suggestions would be appreciated. Thanks.

> top - 16:12:42 up 1 day, 20:03,  1 user,  load average: 0.01, 0.04, 0.05
> Tasks: 128 total,   2 running, 126 sleeping,   0 stopped,   0 zombie
> %Cpu0  :  0.0 us,  0.0 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
> %Cpu1  :  0.7 us,  0.7 sy,  0.0 ni, 84.1 id, 14.6 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu2  :  0.0 us,  0.7 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> KiB Mem:  16316608 total,  4367240 used, 11949368 free,   188944 buffers
> KiB Swap: 19564540 total,        0 used, 19564540 free.  1150524 cached Mem

>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                         
> 13737 root      20   0 4880268 2.607g  18656 S   0.0 16.8   9:47.13 asterisk  

> asterisk-twc01*CLI> memory show summary
>     276861 bytes in        441 allocations in file /usr/src/asterisk-13.5.0/include/asterisk/strings.h
>      19147 bytes in        117 allocations in file /usr/src/asterisk-13.5.0/include/asterisk/threadstorage.h
>     211856 bytes in        372 allocations in file app.c
>        736 bytes in          1 allocations in file app_bridgewait.c
>        992 bytes in          1 allocations in file app_confbridge.c
>        320 bytes in          2 allocations in file app_meetme.c
>      61000 bytes in        177 allocations in file app_queue.c
>        352 bytes in          9 allocations in file app_stack.c
>      10336 bytes in          9 allocations in file app_voicemail.c
>      10090 bytes in        484 allocations in file asterisk.c
>    1300416 bytes in      16255 allocations in file astobj2.c
>       7296 bytes in         21 allocations in file bridge.c
>        248 bytes in          5 allocations in file bridge_basic.c
>        704 bytes in          2 allocations in file bridge_channel.c
>       1016 bytes in          1 allocations in file bucket.c
>        928 bytes in          8 allocations in file ccss.c
>      17825 bytes in         19 allocations in file cdr.c
>       1876 bytes in         23 allocations in file cdr_adaptive_odbc.c
>       6999 bytes in          7 allocations in file cel.c
>       5880 bytes in          3 allocations in file chan_dahdi.c
>    1070281 bytes in        787 allocations in file chan_sip.c
>      32280 bytes in         34 allocations in file channel.c
>       2780 bytes in          6 allocations in file channel_internal_api.c
>      10038 bytes in        507 allocations in file cli.c
>       5384 bytes in         40 allocations in file codec.c
>      25920 bytes in          1 allocations in file codec_resample.c
>      36824 bytes in         13 allocations in file confbridge/conf_config_parser.c
>       3509 bytes in         38 allocations in file config.c
>      45552 bytes in        214 allocations in file config_options.c
>        160 bytes in          1 allocations in file core_local.c
>      26605 bytes in         43 allocations in file data.c
>      12384 bytes in        164 allocations in file devicestate.c
>   12351752 bytes in      28402 allocations in file endpoints.c
>         96 bytes in          4 allocations in file features.c
>       6115 bytes in         37 allocations in file features_config.c
>       7920 bytes in         30 allocations in file file.c
>      20615 bytes in        309 allocations in file format.c
>       1016 bytes in          1 allocations in file format_cache.c
>    2639296 bytes in      18405 allocations in file format_cap.c
>      16012 bytes (     16012 cache) in         38 allocations in file frame.c
>        736 bytes in          1 allocations in file func_dialgroup.c
>       2148 bytes in         11 allocations in file func_odbc.c
>      28040 bytes in         27 allocations in file hashtab.c
>         15 bytes in          3 allocations in file http.c
>      67119 bytes in       1200 allocations in file indications.c
>       8228 bytes in          4 allocations in file io.c
>  106692879 bytes in     943204 allocations in file json.c
>       2296 bytes in          1 allocations in file libasteriskssl.c
>      27100 bytes in        188 allocations in file loader.c
>       8490 bytes in          9 allocations in file logger.c
>     182869 bytes in        292 allocations in file manager.c
>         56 bytes in          3 allocations in file max_forwards.c
>    2618200 bytes in      10403 allocations in file media_index.c
>         96 bytes in          2 allocations in file message.c
>        832 bytes in          2 allocations in file named_acl.c
>        144 bytes in          1 allocations in file parking.c
>         26 bytes in          2 allocations in file parking/parking_bridge_features.c
>    1766506 bytes in      12812 allocations in file pbx.c
>      27011 bytes in        715 allocations in file pbx_config.c
>       9312 bytes in          1 allocations in file pbx_realtime.c
>         56 bytes in          1 allocations in file presencestate.c
>     187000 bytes in       1450 allocations in file res_config_odbc.c
>        256 bytes in          1 allocations in file res_config_sqlite3.c
>        601 bytes in          6 allocations in file res_http_websocket.c
>       2406 bytes in          8 allocations in file res_musiconhold.c
>       5273 bytes in         90 allocations in file res_odbc.c
>       3156 bytes in         14 allocations in file res_parking.c
>      20000 bytes in          4 allocations in file res_rtp_asterisk.c
>        992 bytes in          1 allocations in file res_sorcery_memory_cache.c
>       8000 bytes in          5 allocations in file res_stasis.c
>        736 bytes in          1 allocations in file res_stasis_device_state.c
>       2176 bytes in          1 allocations in file res_stasis_recording.c
>       9152 bytes in          1 allocations in file res_timing_pthread.c
>        272 bytes in          2 allocations in file res_timing_timerfd.c
>       5840 bytes in         13 allocations in file rtp_engine.c
>      34096 bytes in        174 allocations in file sched.c
>        736 bytes in          1 allocations in file sig_pri.c
>       3166 bytes in         93 allocations in file sip/route.c
>      16264 bytes in          1 allocations in file smoother.c
>       9901 bytes in         44 allocations in file sorcery.c
>     424063 bytes in       2655 allocations in file stasis.c
>       2232 bytes in          2 allocations in file stasis/messaging.c
>    3086648 bytes in      11409 allocations in file stasis_bridges.c
>    3737492 bytes in      65556 allocations in file stasis_cache.c
>      12600 bytes in        105 allocations in file stasis_cache_pattern.c
>  314752865 bytes in     256168 allocations in file stasis_channels.c
>     642512 bytes in       6178 allocations in file stasis_endpoints.c
>   36156403 bytes in     265972 allocations in file stasis_message.c
>      68712 bytes in        496 allocations in file stasis_message_router.c
>      37104 bytes in          2 allocations in file stdtime/localtime.c
>    1370017 bytes in      11311 allocations in file strings.c
>  224364905 bytes in    7009335 allocations in file taskprocessor.c
>      37856 bytes in        368 allocations in file threadpool.c
>        240 bytes in          7 allocations in file timing.c
>      76096 bytes in         40 allocations in file translate.c
>        208 bytes in          2 allocations in file udptl.c
>       7190 bytes in         92 allocations in file utils.c
>     981762 bytes in       4746 allocations in file xmldoc.c
> 715760504 bytes allocated (16012 in caches) in 8672267 allocations
asterisk-twc01*CLI> core show taskprocessors

    +----- Processor -----+--- Processed ---+- In Queue -+- Max Depth -+
                 Sorcery                   0            0            0
f5b2497f-d761-45aa-bff7-b34ca0bb563c                   0            2            0
13a51170-a8ad-4782-95a5-04a868d475b4                   2            0            1
086dfd1a-5abd-49d9-a6ee-dd212c4ac42f                   0            2            0
627256dc-967e-46b2-831f-b83600894ebe                   0            2            0
5ded9ecf-1f24-43be-8619-35766cfe52ac                   0            2            0
8e343ebf-2f08-4baf-976c-b06a06eded88                   2            0            1
7b05e1d7-1577-4096-b550-491ea7a09c96                   2            0            1
6b44f7ed-1ee9-4312-b3df-2e9df39594b2                   0            2            0
0d66bcda-c027-4d92-ace0-0c79b62958cb                   0            2            0
cdf05104-1320-4d7f-b846-68d38614488e                   2            0            1
76da4a93-1012-44b9-9c9d-ea12eea6eef1                   2            0            1
fd9116c4-3783-41ec-822e-77d29fefb5ce                   2            0            1
25b5490e-d71e-49ae-890a-6808a524e48d                   2            0            1
d9d4641c-9752-4dd5-b51a-f4daffcef7ae                   0            2            0
06481733-264d-4d81-a9d6-01a23c7c4235                   2            0            1
9ab4bb9f-66ce-4bf8-bef5-5281aaf891db                   0            2            0
24df2279-8a6d-4180-afb1-c4170e7ead22                   0           13            0
43a8c6ee-c9bf-4f95-b724-37cb38cbce9b                   0            2            0
9a304ba5-f18c-4bb8-890e-22db74fc28ad                   0            5            0
4eb11227-1552-4ffd-8956-069d811b11b5                   0            5            0
cc970e8e-5bed-46f7-abf3-6844b698415b                   0         2997            0
3fa3d2f8-358d-4d9e-9753-c5fe152f5c1a                   0         6682            0
5c94c51c-e66d-452a-b9ca-57e16b380366                   0       157739            0
4b4973f2-c48e-424b-a1e0-73925362a293                   0         5968            0
98f78bdc-8df5-4349-a353-8722705e205e                   0         5676            0
ccfe24d1-365c-4253-9324-809b0bd8a73a                   0       143933            0
ce63dc3d-629e-47d3-bfcf-30988938ce51                   0       142772            0
2990d300-847d-4d11-99f0-6d7da3c086a0                   0         5422            0
17619c6f-ce2b-42c1-8d9b-f0ac72670580                   0            6            0
5ae9e9fb-52a6-41f8-bc27-e2e7f785c1fc                   0       128609            0
3c56d5af-d048-4e61-a818-3f99db953b6b                   0       123546            0
87f2d2ab-7eb6-4092-8983-0417a0c508cd                   0         4569            0
f313b467-2984-440c-94fb-63c7c5147634                   0         3945            0
4460040c-cfdd-4ad2-9570-1f6babf25d85                   0         3764            0
05c07b54-dd32-496f-b039-fed9e570f6f3                   0        79042            0
c38b00cf-034e-447e-9c74-ab9e2099990f                   0         2910            0
a8bc8d90-0eda-49d5-ac1c-cb0b5bc49fe8                   0        62959            0
e2a81a29-e4bd-4d01-9c64-570c7ecc9d45                   0         2293            0
36fa1df8-6204-4d95-97b2-75edb23478f9                   0        58160            0
475f4660-2b5f-4f1d-b269-f7a3aa6f01bc                   0         2173            0
cd6b2fc3-eb3b-4281-ad49-a12323a6524e                   0         2160            0
3ce23701-bddc-455c-ba64-6e32d2577d61                   0        34761            0
f7c4980f-6522-4910-ae39-25b22292440c                   0        23329            0
a8485122-c567-4528-a867-f3317013f34b                   0        17584            0
c4071c76-3a4d-4d3b-aceb-0c64f2faf39b                   0          621            0
5a55ffd8-c98a-4996-a1b2-0e16493da8ce                   0         9423            0
3cce938c-8518-4291-a264-1c2bd98869cb                   0         2422            0
774f91a4-8bc0-49b5-a6c9-aa064ae7b08d               20214            0            6
a38d8472-19fe-45f5-b2a8-b4fa905d1a26               33008            0            4
           app_voicemail                   0            0            0
9f21c0cc-5310-4016-b56e-e9224dc7b864                   2            0            1
5e0f09c9-bac3-47e2-93df-60c878a30660                   4            2            1
40c3706c-f252-477b-b93e-a656a3a858f1                   0            2            0
6d97d54b-de68-476c-a912-0ba1bc4d9a49                   4         2077            1
36c1ffd1-6e91-48e9-aa5f-a8c7c76620bb                   0            2            0
3993ab3c-2eb5-4023-9e91-a50eaec1bd7c                   0            2            0
acca420e-7738-41bd-b497-d45d1cbfa59d                   2            0            1
32d62fd1-9210-4f28-ba40-c79ec1428705                   4            2            1
faac1629-bfa5-439b-90f8-890fed194dee                   0            2            0
52245555-12d7-4c2e-8f3f-23cf9bf638b3                   0            2            0
0e9cd48b-9e88-489d-9cce-363e5b5239eb                   2            0            1
dd0d6ea6-a1e9-42c0-8d49-87824286c349                   0            2            0
c49c9d78-cb9c-45c8-af6f-94c7c44da05d                   0            2            0
ef8a3290-e4aa-43a3-b3b7-9da195694356                   0            2            0
041d8d2c-1de7-4b8e-a7e6-fe397e511063                   0            2            0
dfecc980-c92e-4ed2-86e4-631b8272f49f                   0            2            0
3a954fdc-cbe2-463c-8491-7ca89efadb5e                   2            0            1
c131d6b8-fc83-43bc-abb2-ec558b650994                   0            1            0
197883bd-206a-43d2-bf07-61dd0bad88b6                   0         5737            0
2e94b138-69de-4d17-8f76-2d7ced223f5d                   0         4466            0
600f323e-cb66-4f6f-9cf7-4a3141060cfb                   0            5            0
b0aef229-20d2-471f-8473-7c3a2ca6a30a                   0          676            0
db6589cf-5ba9-4471-957c-4212be43cf79                   0            1            0
023d2714-76a6-45f8-a367-a2c8bb9fc1f9                   0         1484            0
ddbb932e-bbd3-4913-9284-f9517b2d482d                   0          870            0
ca7f5169-36e5-48c5-9ed2-983e3adf2064                   0           20            0
32aa999d-4f91-49ce-9c59-30f6370e2fcf                   0            5            0
22082428-86ed-4e74-aead-d91b210a70bd                   0            5            0
0d3853fc-737a-4b91-84be-82224717f5f0                   0           42            0
e8ac5ddb-14e9-45d1-be4b-de61125d5fb6                   0       172448            0
d73dce9f-b0ed-4939-a952-472c9b0cf3dc                   0            2            0
25452705-074c-44d3-a8a2-5e70e0a632e9                   0          147            0
41d471d4-9d41-49b1-b57a-8abc978f2ca8                   0       158711            0
6d060883-d06f-4f89-9af0-fb22543e6b8a                   0       156242            0
fedea3c8-b9e7-47ee-a9fd-4c2aedd1a157                   0       150178            0
fa62f8c1-e945-4f35-8d62-3be852943760                   0       137874            0
c7b796a6-bec4-4333-82d4-2035f3eadee7                   0       129869            0
e5a76dc8-28da-4bfb-bc97-dcb6df0a90fb                   0         4851            0
e7714ae4-dea8-48d6-95f8-58eefa583ea4                   0         4362            0
fac0ab5c-a3ac-43c4-9081-41cd677939ce                   0       105878            0
6d783ca4-df2f-462c-a887-86fb0fd87ef3                   0        59066            0
f00b405b-5b6b-47fb-a7b8-4c8abfbcfb6a                   0        53720            0
141ece6c-2466-457a-94f6-18d9324e6248                   0        27487            0
b43adca3-97d8-4bd2-8613-45e0cd00ba8f                   0         1290            0
964b6662-c6bc-40cf-937e-ccbed38a8ef5                   0           77            0
     stasis-core-control                 695            0            5
c307d69e-3d14-4426-9191-968d1b8b0ea5              254140            0           63
e3265d5b-13ab-4736-95ee-fe2fdb2846fc                  32            0           29
f9bb967f-9b29-4404-9ab7-341254178ae7                  65            0            0
e9ca855a-3026-4fd0-9e41-055dd64fcdc9                   2            0            0
9d6c8df3-7bc5-41fa-9fd0-867dc09f73ae                   2            0            1
5be487df-3579-4676-9f79-4353c12c1a93                   0            2            0
e8aaed5b-e746-4034-8258-f885f8692708                   0            2            0
71451075-2416-4c32-9ddf-da1f52c6e1c6                   0            2            0
b67368ec-b92c-4aba-a504-e2086ede32c1                   2            0            1
803b397b-75a9-4f88-912c-5372c1b90d57                   0            2            0
f8da1de1-57e8-4203-b22b-bd13b5a45061                   4            2            1
551a633e-ea39-4f45-8835-df34f56efa0f                   0            2            0
1ef212de-e74d-41ab-a08d-c42e9c937f21                   4            2            1
b7b0c17e-36cb-408e-bcb2-6bc02e21c438                   2            0            1
9419956a-937d-4f6d-8937-6337f14ba1fa                   2            0            1
b894ca24-8d58-4890-adea-73a7148d5727                   0            2            0
e6dadd6c-9481-48b2-a86a-cee40d6fbb61                   0          183            0
6cb0e0a6-15f6-4113-94db-caecabc2e710                   0          660            0
90757e47-4848-4b0e-8f6c-7bf2cff5033d                   0           46            0
47616645-df07-4402-ad4b-f7486d3f3994                   0            5            0
2c6492d8-36e6-4c0f-a2b5-c85383cd59e6                   0         6209            0
7043d28c-2c37-4b6d-bec9-58f4f6ab7e1a                   0         6734            0
f7b68d8e-74c9-467f-8f70-ca9b1e77e65f                   0       171370            0
4c0eba40-9f6c-4e45-9a06-e7863860b65f                   0       145499            0
20892427-328f-4831-b531-fec90da66946                   0         5628            0
b5bb0620-307d-4cd0-8ced-e17c5868a016                   0       137388            0
71da14ea-2411-4b1f-a785-aa4d6a4a5d83                   0         5065            0
a928b817-8d90-4b19-94a6-91e0c0f418ba                   0       118887            0
2e2bf0e7-216d-45fb-926b-7b1c3b63db5e                   0       114669            0
4d1c7ac8-66f3-46e2-8232-bb8fd6e6346f                   0         2905            0
aab9056c-fc27-4fd6-97b8-e03b7c903f69                   0       102792            0
70d4b8b8-8c00-4151-b3aa-b335bc0ec89b                   0       100656            0
0cb8138d-c789-4700-a0f9-35905d954aa0                   0        96364            0
84cc90cd-01c9-462b-a949-59d725f51732                   0        72385            0
8f36102a-d03d-487c-bcc5-a8f4dfaa3dd8                   0        68841            0
f63d4ffa-1841-4d4a-9ce3-b078af9eda73                   0        54718            0
dfb6d079-6482-4805-bdc7-88ed529a367f                   0         1843            0
4ba3c20d-30f8-4f30-b8ec-e7df8ee7302d                   0         1595            0
aee03e74-7dc7-4de0-ade7-e6fec5e90457                   0         1128            0
           ast_msg_queue                   0            0            0
107ba5a2-2463-4feb-b709-75ea948a0ef5               20213            0            9
a1379e9a-1a0e-4385-9605-93f317e93c35              528538            0          105
               CCSS core                   0            0            0
189bbb15-0ed5-40f1-956f-2483c399c41f              166981            0            8
5db98c49-eefb-4d33-b948-6384e0b493fd              177308            0           35
9f9a063b-df09-40a7-be67-8510d2cafbc0                   0            2            0
03955e6a-2b2a-4de7-9740-48355aea22f1                   2            0            1
e714336b-0a94-43c5-849a-7e9c53acdba6                   4            2            1
6463f07d-50a6-4cc5-906f-5ac3e3e735be                   4            2            1
ba951100-51d7-46a0-9085-f2c558c88f2a                   0            2            0
79861a8a-8e75-4efe-8111-f5e1489a1c9b                   2            0            1
be5419fb-c69a-497a-bc49-f012d010162c                   2            0            1
d6283e81-cc05-4d30-a241-83e6d6d0b3dd                   2            0            1
c52e7564-4b74-49dc-b446-51541f99d43a                   2            0            1
e76a7352-6c66-416a-a619-ed813229afbd                   0            2            0
342e5286-c6da-405f-a9b4-9402fe664927                   2            0            1
a9046841-5fab-465b-a3d2-e34a71064087                   0            2            0
5d99d596-495f-42e7-84b6-295db3df355d                   2            0            1
cd8d6733-3c73-4bf5-9197-8e464dfc55f9                   0            2            0
0ced4ab1-077d-4785-a06a-50b629caa2b5                   0            2            0
011cc75b-50d0-4ea5-add2-2f2305c19648                   0            2            0
20c5890f-2f40-4467-ab80-68535f3b35ba                   0            5            0
28a331e8-44df-49b6-9fa8-0e894bd6896a                   0           12            0
b1272b5a-14d5-4bb9-9aba-1d9328765611                   0            5            0
b0e9a79e-5c0b-4966-8e33-69107bac8d1b                   0           24            0
60350c20-42a3-47b1-8620-78a96ac77a93                   0          737            0
1f44eb27-8bee-4644-9ab6-29b98691d0a2                   0           11            0
fd84ed68-91f4-40af-a33f-699397e5b376                   0            5            0
b172f10e-ae77-4bd6-a035-e560e17b254e                   0            5            0
f8de1d2a-73ba-4af9-809f-0c87986b96e1                   0            1            0
910d39fd-6d86-4f3c-a256-c66b0e846d97                   0       168294            0
8d4ffb0f-a14a-4761-8118-ce09bfa70bbc                   0         6123            0
d92d89dd-fed7-41a1-ad0e-26e0b21f7afb                   0         5995            0
f5d0551b-3c32-40cc-9b52-33e193bfa3ca                   0       153895            0
223c5f17-405a-474f-9241-c5f55ad1087b                   0       152100            0
44d50514-1ab8-405b-b1ed-9db81f6223f5                   0         5543            0
9d0b7dee-a1d3-49c0-8a0f-1d1b4091a734                   0         5519            0
ff24f254-f00d-414e-931d-98160f3b740b                   0       141988            0
35ed6c89-a2d1-4812-a503-1a448754dd14                   0         5407            0
60d9d5c4-8005-494f-9cd6-28a8560c7e2a                   0         5399            0
18e784ed-78d8-4cf1-af14-984c79fe6b75                   0       114061            0
6be25286-47c3-4823-955f-7a29a08a1291                   0         4291            0
214718e2-6785-460e-86ba-05073053178f                   0        99124            0
4a9de65a-0719-4640-bbc8-1fec77672031                   0        85779            0
58dcca78-e028-42d5-9d30-b667fa8cc63c                   0         3265            0
bc928e3d-726f-4858-a428-4733a193f095                   0         2902            0
5db215da-078f-47da-a180-f19d5d06e417                   0         2193            0
2add64b8-327d-422a-bb84-699e6b2ecf3d                   0        56509            0
5fc57367-1275-4b44-a0c3-9c5c71aed097                   0         1633            0
b2efe2b0-cad2-4f13-a6fd-be22a919e316                   0         1204            0
d23aa144-6c50-4b76-85db-cdfb7a2fe9cf                   0         1179            0
4c8299ff-3758-4a22-a1c6-d9144c390291                   0         1150            0
         Sorcery-control                   1            0            0
5032327b-fddc-4d91-8873-38f077a1a9ee                   1            0            0
b4e70b94-2597-42b1-9864-4de8a85b254f                   2            0            1
e07c1afd-5b72-4e30-b392-17607b5a4e03                   0            2            0
6b0f8e5f-73cc-4498-b94d-1e8442ef99ab                   2            0            1
5b3ffc72-43c9-4b40-bf30-13b5c54c8c8d                   2            0            1
14aabe25-8221-4b4e-b721-fdd3b9a8856f                   0            2            0
d027161b-8b23-496a-a8a6-bd945dfd7961                   0            2            0
40f6d04c-7add-423a-8926-d15267dc50d8                   2            0            1
6fde4854-6397-45c7-b8d2-07f596110697                   2            0            1
8db880ac-5b17-49d1-8da9-db05ca46a720                   2            0            1
e1c0908f-4b25-4eac-82bd-00c14390aa83                   2            0            1
8abe522d-ef81-48db-84b2-db0b397ebaa0                   0            2            0
2c526649-f981-4781-94df-a40c2e20fb5e                   2            0            1
c3318da8-8b40-4545-a4f5-ff278cc027e9                   0            2            0
1168708b-1248-47ac-9052-0162e77173be                   2            0            1
34f1060f-1ca8-4150-8ddc-51265dc65138                   0            2            0
10b62f7e-2661-4543-b06b-f9e2d6da234d                   0            2            0
e41f0876-7615-4475-a9fb-f67596a22e2b                   0            2            0
021fdae8-9c29-4ef1-8349-356100af60a6                   0            2            0
586433a4-3960-423d-acea-e65cd03bdf6a                 211       163307           15
80c3611a-2912-4982-95c5-fe4b8ddd8c86                   0            1            0
eb104cab-6cb4-4d87-81f2-4abeb3d269dd                   0            2            0
28716392-1048-4f35-b0ee-c8fc0fd6c234                   0         7248            0
b8b871a2-1da7-4803-bf24-f7969bdc8782                   0            5            0
8ff7d9ef-7c16-49bc-98e3-fb1b14915caa                   0         1638            0
ff1030a1-6678-4dbd-9136-b1ee2f4587c1                   0            5            0
b2825b8a-11fb-416a-887a-69dd6e236941                   0         6567            0
2ab42c08-3ade-4147-9272-6757439d2823                   0          997            0
be7bd7e5-3d3a-421f-b8cc-58c6449a0e4f                   0       155229            0
797ab4a5-3dd3-42e9-a9a8-d31fe087df2d                   0       154925            0
09961002-9668-48f1-9c4c-cfebcff53ae5                   0       146118            0
d7784ece-d017-44c9-8910-091071a512ae                   0         5588            0
11574e63-f28b-4627-9d9f-dcf07976192d                   0         5080            0
5a2f82f8-f9f4-4e52-857c-3e4407e744c6                   0       117884            0
017107e2-8939-4210-b610-cbd3549efab0                   0       112396            0
c5c42c64-dcf9-45aa-9a26-7812a90d7d1d                   0         3862            0
88a91d8a-46f0-4974-8c1f-998e679b11d0                   0         3571            0
4c5d51d2-ec47-4999-8f10-b913b4589b13                   0         3318            0
9d3f0353-75cf-4673-a270-b1f9852ee65a                   0        82668            0
d6d26ea3-696a-425c-80d1-eb1bfb06fc26                   0         3238            0
98cc8ead-743c-4df7-abe4-91650725748b                   0        69496            0
8360fb0a-e276-4281-af6a-b68dacf49c71                   0        44324            0
f3c537d9-95c4-44e1-b1e3-a303f716ec75                   0         1725            0
e1356521-beb9-498e-b839-2c44733ec118                   0        39799            0
74665e0a-1efb-498f-8915-cec6e91bb55a                   0        11588            0
7ff8f642-0858-4f54-905b-c3e9b9af0c18                   0          501            0
50a0f0ac-6cb7-4db3-83d3-589812c0ee1b                   0         4896            0
a8e5691d-cb9a-42ec-a7d9-29cdbb4be8b2                   0            0            0
c9c42a65-bfcb-4a26-b231-a9ac150ea058              487670            0           58
48df061c-22c6-4ec1-8730-4fb9e2aea62d              242523            0           37
ee1d6f16-4f3e-4ee4-9593-ae6b8c66c08b                   2            0            1
30c66e48-c004-40e3-948a-8055dbd7cd41                   0            2            0
305cc594-fe41-41f0-8596-e935cee6a20c                   2            0            1
49391552-fa46-44e0-8594-299aeec3d5eb                   2            0            1
059cd581-8184-473d-80da-026486d41d81                   0            2            0
209eb980-d049-43de-8c65-01a1f742c8f6                   4            2            1
612c9f17-0857-47eb-94bc-0e11b92f2897                   4            2            1
4d863044-5ee4-41f3-8d23-bbf00a4db251                   2            0            1
2ef7ed2f-a09e-440b-95ac-e074390aa9d0                   2            0            1
955adfd3-59e9-4493-9287-6465ac4ca1d2                   2            0            1
77987f75-6b1a-463f-92ae-666eca412c2c                   2            0            1
3101f8c0-270e-42fa-8e7e-7f82b6c6e859                   0          110            0
e42cdba3-8179-4d63-9209-a04dd0f27b88                   0        21604            0
cf7fb4a3-d1e0-4256-bb7b-d4f99362e40c                   0         8003            0
cebb438a-933a-4dba-9afe-fbee699a4ca3                   0            5            0
1e4586b3-b150-40cf-a64f-4a13dd92eb35                   0            2            0
b0dd0eac-5290-47bf-9a9d-cc9b9199129d                   0        16930            0
e744b707-05e4-4e2f-a0e3-c59ab5ce5f90                   0            5            0
185ce209-5b98-46b3-bf82-c0693dfa90d0                   0         6093            0
e344dfd7-5284-47b3-9432-0df50162ab4f                   0         5976            0
e7f47d3a-cae8-4022-8e16-77bc7d5be9b3                   0         5937            0
75b4b7a5-23ae-4baa-9211-d4d42667b469                   0       149652            0
605ab6f8-c673-457b-b8c4-2d32b52e738d                   0         5654            0
151ef842-cfe4-49bd-9ef0-2efd2bf859ba                   0       139571            0
3b5b3669-9c96-4499-b05e-dc662f93af86                   0         5148            0
6d437f6b-6565-4d16-b527-960ad0da76f3                   0       131614            0
11079c5b-0186-4989-bfc8-67b2f6a754fd                   0       130552            0
cdc56262-fa68-45eb-ab97-5688ce598ec0                   0         4486            0
42a8df35-a324-4a0e-9ece-774fb56ff434                   0       116914            0
c67ca120-1f05-472a-a044-d5cb8edc831a                   0         4375            0
9bd4662b-22be-45e3-9dc5-f536c7d9fb12                   0        82019            0
736f0fe1-9541-4960-80f3-95022f015fe2                   0         2863            0
ba6f9659-0953-448e-ab2e-51cbd0df7208                   0         2592            0
f07af5f7-d2fb-4b54-a5d5-e6099125e45b                   0         2334            0
62e6e118-5b71-4a05-ac94-7c45fcdd4edb                   0         1712            0
d5f0fe5e-891e-49a0-a8ea-e32145897029                   0        38244            0
04fa12c9-6a52-432b-a7d8-5121502a8ca4                   0        34206            0
0ce58b18-1ae3-4e4e-9e9e-4f2d26435934                   0        33626            0
b789c794-7545-4cab-a6d6-0628c0e2ce5e                   0         1414            0
9323e19c-5483-479f-851a-9a0e71381232                   0        29557            0
dda0e5b5-08cd-417f-a856-c08e54c5ee39                   0         1331            0
a4f70d91-9bbc-43e0-acda-9e0dab67f582                   0        23878            0
92e2ba58-9b1b-4428-bbd5-b3ada348df83                   0        22367            0
9390a33c-bb35-4a61-974c-f5ef9152164f                   0        21994            0
bb56ba02-3407-4733-92a8-a5be3bdc34eb                   0          853            0
22b5f1a2-f38f-48ac-91c2-b9b8fe1e606f                   0        13193            0
ab6ebe84-7354-4da4-a5a6-a22605a3d21a                   0            0            0
             stasis-core                 211          216            1
            hep_queue_tp                   0            0            0
1f29f548-cddd-4522-97bb-cce67d902087                  31            0            0
963794aa-b5ba-4da7-9c9d-fe2231cfdc92               11550            0            7
312c148a-e102-4e29-ac12-a1bc29f95dcc              242524            0           50
b3488649-0a1f-49ac-b86c-31a97a31656f               20168            0            7
507ae503-8459-497c-a9b6-4594478635a9                   0            2            0
db340da0-07c0-4d00-baf8-347e7fc75cf4                   0            2            0
b53ae1b0-aad3-49e4-b387-3ebb68ab70ee                   0            2            0
6376fe66-b484-42e0-9bc8-6e5b9726bdcc                   0            2            0
5b999b20-d2f6-47d6-ba0b-7e1a67f1f86c                   0            2            0
8afdc389-3308-432a-98e1-36f7f33fdcb6                   2          868            1
27fef2b0-7a90-45f9-a0ac-f265c17eaebe                   0            2            0
e8f1f716-479c-4f92-8edb-ab9b71b2b8dc                   0            2            0
d5669709-8926-468a-bc1d-147593c7a960                   2            0            1
a54a99b1-ba50-4b68-af1d-de9f75b5b0d1                   0            2            0
0e119a4b-81d2-47c6-9972-5b748a445052                   0            2            0
93c35950-7452-4796-afa8-0a01ff7ef734                   2            0            1
ee57d89a-3c53-40e0-b713-dc63e46a37cf                   0         1318            0
e53091e6-a1ff-47de-8306-858fdfe225dd                   0         2081            0
8da2658b-9cf8-4fcc-a5a5-3f3bf8115744                   0           80            0
48eb8c13-d836-4695-9471-27cf564a9cf7                   0            5            0
db110a29-5243-4347-a42d-27a0b98033bf                   0          692            0
55c4aacf-690d-4902-9427-03c8510a5b09                   0            5            0
58ad55a4-7ce8-4149-9a20-23af58a64d2c                   0            1            0
79055394-a088-481c-9403-1c9b27fdbe14                   0         1533            0
92f1b28b-6d8c-4343-afcd-b6dd615fd215                   0            8            0
31be366d-3643-4243-91b1-b34020003bc0                   0          146            0
3064fdf1-385b-4cd5-a9c4-ab8d9f09766c                   0            2            0
3ad9352f-51aa-48d6-af75-52235bc0d746                   0         5883            0
afaf4362-bd90-48b4-badc-6aeeb4193ef9                   0         5828            0
de3d5969-b3b4-4c32-823a-915f1f4fe021                   0         5815            0
3c4214ed-6a62-4b65-8ac7-e3abe6f02c22                   0       144827            0
25155935-c434-4c1d-a24e-9d150d595489                   0          233            0
8a57e627-3ab2-45a1-a5bc-fe9d397b30ed                   0         5107            0
2e5358d6-ab42-4758-bfcf-eb862c8a5913                   0         4528            0
b29de0ca-96f6-4a46-af60-838390702bb4                   0           12            0
f7103227-3108-4b93-83da-3d9d2724a7ef                   0         4013            0
37f28ce4-9d35-4386-841b-11a2586942e8                   0         3154            0
d0fa5b62-a784-492a-8325-67fd8d4502fb                   0        71932            0
9d63bb3b-9376-4b82-bf3d-c28a870e33a0                   0         2836            0
3706874c-577f-446c-b54c-9ce6c245578d                   0        54225            0
c9edb645-26d5-436c-a115-c1dd94e89e96                   0         2147            0
0710a0d4-9cb7-46ca-93aa-481f6f385e0f                   0         1615            0
357d97d9-173c-48c3-b2d3-347ef1ec16a5                   0        26022            0
0c745095-65d7-476e-82ac-8bee08cff544                   0          575            0
4276356a-47cc-4f8b-bd07-a8595ba2f92b                   0          252            0
    +---------------------+-----------------+------------+-------------+
    334 taskprocessors

There’s been a few memory leaks plugged since that version, and taskprocessor usage has been changed to include a descriptive identifier for each so you know what they are for. I can’t point to any fixes specifically for this, but trying to isolate between that and current would take awhile…

1 Like

I read up on the changes from 13.5 to 13.8 yesterday and noticed quite a few updates for memory leaks. It probably would be better to just bite the bullet and upgrade. I’ve got 3 Asterisk clusters set up, and only one is currently in use. I’ll upgrade one of the others and run through all the testing on Monday (and maybe Tuesday…the dialplan is rather large). Thanks for your response.

I upgraded a smaller box to 13.8. I’ll test it on Monday to see if there’s anything I need to change. I see some config options are deprecated. But, so far, so good.

asterisk-whconfig01*CLI> memory show summary 
    226108 bytes in        352 allocations in file /usr/src/asterisk-certified-13.8-cert2/include/asterisk/strings
      3429 bytes in         18 allocations in file /usr/src/asterisk-certified-13.8-cert2/include/asterisk/threads
       216 bytes in          2 allocations in file app.c
       736 bytes in          1 allocations in file app_bridgewait.c
       992 bytes in          1 allocations in file app_confbridge.c
       320 bytes in          2 allocations in file app_meetme.c
      6784 bytes in          2 allocations in file app_queue.c
      9312 bytes in          1 allocations in file app_voicemail.c
      8666 bytes in        446 allocations in file asterisk.c
    376160 bytes in       4702 allocations in file astobj2.c
       304 bytes in          2 allocations in file bridge.c
      1016 bytes in          1 allocations in file bucket.c
       896 bytes in          6 allocations in file ccss.c
     12904 bytes in          5 allocations in file cdr.c
      1876 bytes in         23 allocations in file cdr_adaptive_odbc.c
      6999 bytes in          7 allocations in file cel.c
      5880 bytes in          3 allocations in file chan_dahdi.c
     25360 bytes in         10 allocations in file channel.c
      9271 bytes in        463 allocations in file cli.c
      5384 bytes in         40 allocations in file codec.c
     25920 bytes in          1 allocations in file codec_resample.c
     36832 bytes in         13 allocations in file confbridge/conf_config_parser.c
      3210 bytes in         35 allocations in file config.c
     49496 bytes in        231 allocations in file config_options.c
       160 bytes in          1 allocations in file core_local.c
     24179 bytes in         39 allocations in file data.c
      1906 bytes in         26 allocations in file devicestate.c
      2496 bytes in          2 allocations in file endpoints.c
      2257 bytes in         13 allocations in file features_config.c
      7656 bytes in         29 allocations in file file.c
      8307 bytes in        117 allocations in file format.c
      1016 bytes in          1 allocations in file format_cache.c
   2297992 bytes in      15939 allocations in file format_cap.c
      2042 bytes (      2042 cache) in          4 allocations in file frame.c
       736 bytes in          1 allocations in file func_dialgroup.c
      1995 bytes in         10 allocations in file func_odbc.c
     27568 bytes in         26 allocations in file hashtab.c
       729 bytes in          3 allocations in file http.c
     67119 bytes in       1200 allocations in file indications.c
       600 bytes in          5 allocations in file json.c
      2296 bytes in          1 allocations in file libasteriskssl.c
     26245 bytes in        181 allocations in file loader.c
      8382 bytes in          6 allocations in file logger.c
    177319 bytes in        281 allocations in file manager.c
   2619440 bytes in      10408 allocations in file media_index.c
        96 bytes in          2 allocations in file message.c
       832 bytes in          2 allocations in file named_acl.c
      1760 bytes in          1 allocations in file named_locks.c
       144 bytes in          1 allocations in file parking.c
    906599 bytes in      10333 allocations in file pbx.c
    374594 bytes in        338 allocations in file pbx_app.c
     27011 bytes in        715 allocations in file pbx_config.c
    341664 bytes in        210 allocations in file pbx_functions.c
        36 bytes in          1 allocations in file pbx_variables.c
        56 bytes in          1 allocations in file presencestate.c
       224 bytes in          1 allocations in file res_ari.c
       256 bytes in          1 allocations in file res_config_sqlite3.c
       953 bytes in          8 allocations in file res_http_websocket.c
      2406 bytes in          8 allocations in file res_musiconhold.c
      3641 bytes in         59 allocations in file res_odbc.c
      3156 bytes in         14 allocations in file res_parking.c
       992 bytes in          1 allocations in file res_sorcery_memory_cache.c
      8000 bytes in          5 allocations in file res_stasis.c
       736 bytes in          1 allocations in file res_stasis_device_state.c
      2176 bytes in          1 allocations in file res_stasis_recording.c
      2128 bytes in         38 allocations in file rtp_engine.c
      9312 bytes in         14 allocations in file sched.c
       736 bytes in          1 allocations in file sig_pri.c
      9901 bytes in         44 allocations in file sorcery.c
     17134 bytes in        171 allocations in file stasis.c
      2232 bytes in          2 allocations in file stasis/messaging.c
     66224 bytes in         40 allocations in file stasis_cache.c
       360 bytes in          3 allocations in file stasis_cache_pattern.c
     16991 bytes in        241 allocations in file stasis_message.c
      3456 bytes in         18 allocations in file stasis_message_router.c
     18552 bytes in          1 allocations in file stdtime/localtime.c
      2584 bytes in          3 allocations in file strings.c
      8789 bytes in        112 allocations in file taskprocessor.c
     11192 bytes in         15 allocations in file threadpool.c
       184 bytes in          4 allocations in file timing.c
     16768 bytes in         34 allocations in file translate.c
       208 bytes in          2 allocations in file udptl.c
      3032 bytes in         37 allocations in file utils.c
   1005178 bytes in       4858 allocations in file xmldoc.c
   8968804 bytes allocated (2042 in caches) in 52001 selected allocations

   8968804 bytes in all allocations
    278286 bytes in deferred free large allocations
     55634 bytes in deferred free small allocations
    333920 bytes in deferred free allocations
   9302724 bytes in all allocations and deferred free allocations
asterisk-whconfig01*CLI> core show t
taskprocessors  threads         translation     
asterisk-whconfig01*CLI> core show taskprocessors 

Processor                                      Processed   In Queue  Max Depth  Low water High water
app_voicemail                                          0          0          0        450        500
ast_msg_queue                                          0          0          0        450        500
CCSS_core                                              0          0          0        450        500
Sorcery                                                0          0          0        450        500
Sorcery-control                                        1          0          1        450        500
sorcery/bucket-00000000                                0          0          0        450        500
sorcery/file-00000001                                  0          0          0        450        500
stasis-core                                            0          0          0        450        500
stasis-core-control                                   11          0          2        450        500
subm:ast_bridge_topic_all-cached-0000000a              1          0          1        450        500
subm:ast_channel_topic_all-0000000f                    1          0          1        450        500
subm:ast_channel_topic_all-cached-0000000b             4          0          1        450        500
subm:ast_channel_topic_all-cached-0000000c             3          0          1        450        500
subm:ast_device_state_topic-00000002                  23          0          1        450        500
subm:ast_device_state_topic-00000004                  22          0          1        450        500
subm:ast_device_state_topic-0000000e                   1          0          1        450        500
subm:ast_parking-00000009                              1          0          1        450        500
subm:ast_presence_state_topic_all-00000005             1          0          1        450        500
subm:ast_security-0000000d                            20          0          1        450        500
subm:ast_system-00000008                              30          0         29        450        500
subm:cdr_engine-00000003                               7          0          1       4500       5000
subm:endpoint_topic_all-cached-00000007                1          0          1        450        500
subm:manager_topic-00000006                           60          0          1       2700       3000

23 taskprocessors
1 Like

I upgraded one of the live servers to 13.8 with the same config I used for 13.5. I made some inbound calls. I received a warning when the first call came through stating that it exceeded the high water mark. It’s an inbound call from a PRI (using a Digium G200), to a context for from-pstn, it hands off to an auto-attendant, which I press an option to Dial to another auto-attendant, press a last option to Dial the queue. There are no agents logged in, so the call is sent to voicemail.

Here’s the warning:
[Aug 23 20:29:17] WARNING[32627][C-00000000]: taskprocessor.c:887 taskprocessor_push: The 'subp:SIP/gateway-twc01-000000ca' task processor queue reached 500 scheduled tasks.

Here's the taskprocessor list after the one call:
Processor                                      Processed   In Queue  Max Depth  Low water High water
app_voicemail                                          0          0          0        450        500
ast_msg_queue                                          0          0          0        450        500
CCSS_core                                              0          0          0        450        500
Sorcery                                                0          0          0        450        500
Sorcery-control                                        1          0          1        450        500
sorcery/bucket-00000000                                0          0          0        450        500
sorcery/file-00000001                                  0          0          0        450        500
stasis-core                                          165         52          3        450        500
stasis-core-control                                  437          0          3        450        500
subm:ast_bridge_topic_all-cached-0000000b              1          0          1        450        500
subm:ast_channel_topic_all-00000010                  394          0          8        450        500
subm:ast_channel_topic_all-cached-0000000c           534          0          3        450        500
subm:ast_channel_topic_all-cached-0000000d           533          0          4        450        500
subm:ast_device_state_topic-00000002                  94          0          5        450        500
subm:ast_device_state_topic-00000004                  93          0          5        450        500
subm:ast_device_state_topic-0000000f                  40          0          2        450        500
subm:ast_parking-00000009                              1          0          1        450        500
subm:ast_presence_state_topic_all-00000005             1          0          1        450        500
subm:ast_security-0000000e                           211          0          4        450        500
subm:ast_system-00000008                              31          0         29        450        500
subm:ast_system-0000000a                              30          0          1        450        500
subm:cdr_engine-00000003                             537          0          5       4500       5000
subm:endpoint_topic_all-cached-00000007             1067          0          3        450        500
subm:manager_topic-00000006                          965          0         14       2700       3000
subp:61008@from-internal-sip-00000042                  0          2          0        450        500
subp:61008@from-internal-sip-00000044                  0          2          0        450        500
subp:61008@from-internal-sip-00000073                  1          1          1        450        500
subp:61008@from-internal-sip-000000bd                  0          1          0        450        500
subp:61009@from-internal-sip-00000019                  0          2          0        450        500
subp:61009@from-internal-sip-0000001b                  0          2          0        450        500
subp:61009@from-internal-sip-00000086                  1          0          1        450        500
subp:61023@from-internal-sip-00000037                  0          2          0        450        500
subp:61023@from-internal-sip-00000039                  0          2          0        450        500
subp:61023@from-internal-sip-000000cc                  0          1          0        450        500
subp:61027@from-internal-sip-0000001d                  0          2          0        450        500
subp:61027@from-internal-sip-0000001f                  0          2          0        450        500
subp:61027@from-internal-sip-00000098                  1          1          1        450        500
subp:61027@from-internal-sip-000000bb                  0          1          0        450        500
subp:61028@from-internal-sip-0000007c                  0          2          0        450        500
subp:61028@from-internal-sip-0000007e                  0          2          0        450        500
subp:61028@from-internal-sip-000000ac                  0          1          0        450        500
subp:61029@from-internal-sip-0000005d                  0          2          0        450        500
subp:61029@from-internal-sip-0000005f                  0          2          0        450        500
subp:61029@from-internal-sip-00000068                  0          2          0        450        500
subp:61029@from-internal-sip-0000006a                  0          2          0        450        500
subp:61029@from-internal-sip-00000075                  0          2          0        450        500
subp:61029@from-internal-sip-00000077                  0          2          0        450        500
subp:61029@from-internal-sip-000000a8                  0          1          0        450        500
subp:61033@from-internal-sip-00000082                  0          2          0        450        500
subp:61033@from-internal-sip-00000084                  0          2          0        450        500
subp:61033@from-internal-sip-000000a6                  0          1          0        450        500
subp:61034@from-internal-sip-00000046                  0          2          0        450        500
subp:61034@from-internal-sip-00000048                  0          2          0        450        500
subp:61034@from-internal-sip-00000094                  0          2          0        450        500
subp:61034@from-internal-sip-00000096                  0          2          0        450        500
subp:61034@from-internal-sip-0000009e                  0          2          0        450        500
subp:61034@from-internal-sip-000000a0                  1          0          1        450        500
subp:61035@from-internal-sip-0000006f                  0          2          0        450        500
subp:61035@from-internal-sip-00000071                  0          2          0        450        500
subp:61035@from-internal-sip-00000088                  0          2          0        450        500
subp:61035@from-internal-sip-0000008a                  0          2          0        450        500
subp:61035@from-internal-sip-0000009a                  0          2          0        450        500
subp:61035@from-internal-sip-0000009c                  0          2          0        450        500
subp:61035@from-internal-sip-000000c9                  0          1          0        450        500
subp:61036@from-internal-sip-000000a2                  0          2          0        450        500
subp:61036@from-internal-sip-000000a4                  0          1          0        450        500
subp:61037@from-internal-sip-000000bf                  0          1          0        450        500
subp:61038@from-internal-sip-00000033                  0          2          0        450        500
subp:61038@from-internal-sip-00000035                  0          2          0        450        500
subp:61038@from-internal-sip-000000ae                  0          1          0        450        500
subp:61039@from-internal-sip-00000012                  0          2          0        450        500
subp:61039@from-internal-sip-00000014                  0          2          0        450        500
subp:61039@from-internal-sip-00000016                  0          2          0        450        500
subp:61039@from-internal-sip-00000025                  0          2          0        450        500
subp:61039@from-internal-sip-00000027                  0          2          0        450        500
subp:61039@from-internal-sip-00000029                  1          1          1        450        500
subp:61039@from-internal-sip-000000b0                  0          1          0        450        500
subp:71001@from-internal-sip-00000021                  0          2          0        450        500
subp:71001@from-internal-sip-00000023                  0          2          0        450        500
subp:71001@from-internal-sip-0000002b                  0          2          0        450        500
subp:71001@from-internal-sip-0000002d                  0          2          0        450        500
subp:71001@from-internal-sip-0000002f                  0          2          0        450        500
subp:71001@from-internal-sip-00000031                 18          2          1        450        500
subp:71001@from-internal-sip-0000003b                  0          2          0        450        500
subp:71001@from-internal-sip-0000003d                  0          2          0        450        500
subp:71001@from-internal-sip-0000004a                 13          2          1        450        500
subp:71001@from-internal-sip-00000056                  0          2          0        450        500
subp:71001@from-internal-sip-00000058                  0          2          0        450        500
subp:71001@from-internal-sip-00000064                  0          2          0        450        500
subp:71001@from-internal-sip-00000066                  0          2          0        450        500
subp:71001@from-internal-sip-0000008e                  0          2          0        450        500
subp:71001@from-internal-sip-00000090                  0          2          0        450        500
subp:71001@from-internal-sip-000000aa                  0          2          0        450        500
subp:71001@from-internal-sip-000000c1                  0          1          0        450        500
subp:SIP/61007-00000041                                2          0          2        450        500
subp:SIP/61007-00000043                                2          0          2        450        500
subp:SIP/61007-000000b3                                0          5          0        450        500
subp:SIP/61007-000000b4                                0          5          0        450        500
subp:SIP/61007-000000bc                                0          2          0        450        500
subp:SIP/61008-00000072                                2          2          2        450        500
subp:SIP/61009-00000018                                2          0          2        450        500
subp:SIP/61009-0000001a                                2          0          2        450        500
subp:SIP/61009-00000085                                2          2          2        450        500
subp:SIP/61023-00000036                                2          0          2        450        500
subp:SIP/61023-00000038                                2          0          2        450        500
subp:SIP/61023-000000c4                                0          5          0        450        500
subp:SIP/61023-000000c5                                0          5          0        450        500
subp:SIP/61023-000000cb                                0          4          0        450        500
subp:SIP/61024-0000002a                                2          0          2        450        500
subp:SIP/61024-0000002c                                2          0          2        450        500
subp:SIP/61024-00000049                                2          2          2        450        500
subp:SIP/61025-0000002e                                2          0          2        450        500
subp:SIP/61025-00000030                                2          2          2        450        500
subp:SIP/61026-0000003a                                2          0          2        450        500
subp:SIP/61026-0000003c                                2          0          2        450        500
subp:SIP/61026-00000055                                2          0          2        450        500
subp:SIP/61026-00000057                                2          0          2        450        500
subp:SIP/61026-0000008d                                2          0          2        450        500
subp:SIP/61026-0000008f                                2          0          2        450        500
subp:SIP/61026-000000a9                                0          4          0        450        500
subp:SIP/61027-0000001c                                2          0          2        450        500
subp:SIP/61027-0000001e                                2          0          2        450        500
subp:SIP/61027-000000b1                                0          5          0        450        500
subp:SIP/61027-000000b2                                0          5          0        450        500
subp:SIP/61027-000000b5                                0          5          0        450        500
subp:SIP/61027-000000b6                                0          5          0        450        500
subp:SIP/61027-000000ba                                0          2          0        450        500
subp:SIP/61028-0000007b                                2          0          2        450        500
subp:SIP/61028-0000007d                                2          0          2        450        500
subp:SIP/61028-000000ab                                0          4          0        450        500
subp:SIP/61029-0000005c                                2          0          2        450        500
subp:SIP/61029-0000005e                                2          0          2        450        500
subp:SIP/61029-00000067                                2          0          2        450        500
subp:SIP/61029-00000069                                2          0          2        450        500
subp:SIP/61029-00000074                                2          0          2        450        500
subp:SIP/61029-00000076                                2          0          2        450        500
subp:SIP/61029-000000a7                                0          4          0        450        500
subp:SIP/61032-00000097                                2          0          2        450        500
subp:SIP/61033-00000081                                2          0          2        450        500
subp:SIP/61033-00000083                                2          0          2        450        500
subp:SIP/61033-000000a5                                0          4          0        450        500
subp:SIP/61034-00000045                                2          0          2        450        500
subp:SIP/61034-00000047                                2          0          2        450        500
subp:SIP/61034-00000093                                2          0          2        450        500
subp:SIP/61034-00000095                                2          0          2        450        500
subp:SIP/61034-0000009d                                2          0          2        450        500
subp:SIP/61034-0000009f                                2          2          2        450        500
subp:SIP/61035-0000006e                                2          0          2        450        500
subp:SIP/61035-00000070                                2          0          2        450        500
subp:SIP/61035-00000087                                2          0          2        450        500
subp:SIP/61035-00000089                                2          0          2        450        500
subp:SIP/61035-00000099                                2          0          2        450        500
subp:SIP/61035-0000009b                                2          0          2        450        500
subp:SIP/61035-000000b7                                0          5          0        450        500
subp:SIP/61035-000000b8                                0          5          0        450        500
subp:SIP/61035-000000b9                                0          5          0        450        500
subp:SIP/61035-000000c2                                0          5          0        450        500
subp:SIP/61035-000000c3                                0          5          0        450        500
subp:SIP/61035-000000c6                                0          5          0        450        500
subp:SIP/61035-000000c7                                0          5          0        450        500
subp:SIP/61035-000000c8                                0          4          0        450        500
subp:SIP/61036-000000a1                                2          0          2        450        500
subp:SIP/61036-000000a3                                0          4          0        450        500
subp:SIP/61037-000000be                                0          4          0        450        500
subp:SIP/61038-00000032                                2          0          2        450        500
subp:SIP/61038-00000034                                2          0          2        450        500
subp:SIP/61038-000000ad                                0          2          0        450        500
subp:SIP/61039-00000011                                2          0          2        450        500
subp:SIP/61039-00000013                                4          0          2        450        500
subp:SIP/61039-00000015                                2          0          2        450        500
subp:SIP/61039-00000028                                2          0          2        450        500
subp:SIP/61041-00000024                                2          0          2        450        500
subp:SIP/61041-00000026                                2          0          2        450        500
subp:SIP/61041-000000af                                0          2          0        450        500
subp:SIP/61043-00000051                                2          0          2        450        500
subp:SIP/62966-00000020                                2          0          2        450        500
subp:SIP/62966-00000022                                2          0          2        450        500
subp:SIP/62966-00000063                                2          0          2        450        500
subp:SIP/62966-00000065                                2          0          2        450        500
subp:SIP/62966-000000c0                                0          2          0        450        500
subp:SIP/gateway-twc01-000000ca                        0        533          0        450        500

181 taskprocessors

After 6 calls, the SIP/gateway-twc01 In Queue value keeps going up.
subp:SIP/gateway-twc01-000000ca 0 **3244** 0 450 500

Is that normal behavior? I’d like to understand this better, but can’t seem to find any detailed doc on it.
I can’t reproduce this in my test servers.

I have no config file for the stasis or sorcery being used.

Seems like it’s one thing after another. Still getting warnings. Haven’t had time to research yet.

[Aug 24 08:51:49] WARNING[27905][C-00000038]: taskprocessor.c:887 taskprocessor_push: The ‘subp:SIP/61026-000000a9’ task processor queue reached 500 scheduled tasks.
[Aug 24 09:59:28] WARNING[21103][C-0000000d]: taskprocessor.c:887 taskprocessor_push: The ‘subp:ast_channel_topic_all-000000c3’ task processor queue reached 500 scheduled tasks.

Because the app keeps crashing.
Aug 24 08:52:14 asterisk-twc01 kernel: [564051.294597] asterisk[27984]: segfault at 16c ip 00007f5b311ba857 sp 00007f5b14604970 error 4 in app_queue.so[7f5b311a6000+34000]
Aug 24 09:44:37 asterisk-twc01 kernel: [567194.186199] asterisk[13634]: segfault at 16c ip 00007f54ad550857 sp 00007f544c8d8970 error 4 in app_queue.so[7f54ad53c000+34000]
Aug 24 09:53:46 asterisk-twc01 kernel: [567742.901995] asterisk[18352]: segfault at 16c ip 00007fdecfddd857 sp 00007fde8effe970 error 4 in app_queue.so[7fdecfdc9000+34000]

Found a patch and applied. Fingers crossed.
Asterisk 13.8-cert2

apps/app_queue.c line 4199

    if (call->member->in_call && call->lastqueue->wrapuptime) {

to:
if (call->member->in_call && call->lastqueue && call->lastqueue->wrapuptime) {

The symptoms of the issue appear to be this issue:
https://issues.asterisk.org/jira/browse/ASTERISK-26145

I’m basing this off of the fact that you have a lot of tasks queued in a taskprocesor but the max depth is less than the number of tasks currently queued. When this situation happens, the taskprocessor is blocked for some reason.

Processor                                      Processed   In Queue  Max Depth  Low water High water
subp:SIP/gateway-twc01-000000ca                        0        533          0        450        500

That patch appears to be for chan_pjsip. I’m still using chan_sip. Do you know if the patches for the taskprocessor will still apply? I suspect that they’ll just be unreferenced by chan_sip unless they’re explicitly called.

subp:SIP/gateway-twc01-00000017 236 40941 11 450 500

I use realtime sippeers, so my sip.conf is very slim. Providing in case I’m missing something that may help.

[general]
context=default

allowguest=no
srvlookup=no
udpbindaddr=0.0.0.0

disallow=all
allow=g722
allow=ulaw
allow=alaw
allow=gsm

rtcachefriends=yes

subscribecontext=hints
callcounter=yes
allowsubscribe=yes

match_auth_username=yes
trustrpid=yes

Ended up having to roll back to 13.5. Too many issues. It got to the point where the queue would no longer attempt to ring agents. Once I removed the agent, and added them back, they would take one call. The next call in the queue was just stuck.

I’m currently working an issue[1] for precisely that problem where agents sometimes stop receiving calls that I expect to be fixed in 13.11.0. As for your 13.8 troubles - are you using ODBC? If so that release contained changes which uncovered fundamental problems within UnixODBC and the UnixODBC connectors themselves[2]. Thanks to the community we were able to isolate what was going on and come up with a solution[3] which is in 13.10.0. If you are not using ODBC I would be very interested in seeing a backtrace[4] if at all possible when the system is in that state. That would allow us to see precisely what it is hanging on. I’ve also bookmarked this thread so when the fix for app_queue goes in I can post here.

[1] https://issues.asterisk.org/jira/browse/ASTERISK-26299
[2] http://lists.digium.com/pipermail/asterisk-dev/2016-June/075580.html
[3] http://blogs.asterisk.org/2016/06/15/asterisk-odbc-connections/
[4] https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace#GettingaBacktrace-GettingInformationForADeadlock

I am using ODBC Version: 2.2.14p2-5ubuntu5. The strategy for the queue in question is set to ringall (2-3 agents, typically). I have 6 asterisk servers set up in 3 clusters. Only one server is in “live” use, and I can’t reproduce the problems on the “test” server. It’s still at 13.8. I’ll try to see if I can reproduce it based on [1] tomorrow. Thanks for the update. I’m happy these problems are being worked.

I’ve been put in charge of replacing all of our various phone systems with a single application. I’ve already built the servers for two of our sites (only one live), I’ve built all of the databases, configuration tools, monitoring, and replication. I’d hate to move forward if it’s going to require a significant amount of maintenance. Once I get everything stable, I’d like to set it, and forget it. :slight_smile:

Sounds like 13.11.0 will be the right release for you then!

I’ll also add that the connector for ODBC plays a big part as well. If these differ between your test server and production it would definitely influence the behavior under 13.8.

It’s all the same installation. I created an internal repository for Ubuntu 14.04 that doesn’t pull updates since it seemed extremely stable during the testing. Same OS level, same packages, were the same asterisk version (from source)/config/database/etc.

Very interesting! Same amount of load and usage?

No. The live server is currently running a 5 physician office and a 3 phone lab; about 30 phones with 4 auto-attendants, 1 queue, a few ring groups, DID forwards, 4 parking zones (one zone, but I separate out the extensions), call pickup, etc. The calls enter the main auto-attendant via server-side forwards from 6 DIDs. There are 5 options; 1 goes directly to a phone, 3 go to secondary auto-attendants, and 1 is sent to a queue. The other AAs have a similar function. it handles between 600 and 800 calls per day M-F.

The test server only has about 4 phones set up for me to test with.

That’s likely why, without a similar load ODBC would not get exercised in the same way and could tolerate the usage.

Definitely plausible. Perhaps I’ll try to stress it. I had run sipp on a smaller system during testing, but the context didn’t have any odbc in it, just audio.

For your Queue usage are you using Local channels?