Possible memory leak?

Hi,

I’m currently running version 13.13.1 with Pjsip / sorcery caching and have (what appears to be) a memory leak. I’ve not had a core dump however after the asterisk process exhausts all memory it appears to stall / stop processing messages so haven’t been able to collect much debug. I’ve enabled the memory logging so collected a few samples from memory show summary, etc…

In the memory show allocations - I get a lot of these - not sure if it’s indicative of anything:

    32 bytes allocated by ast_sip_endpoint_alloc() line  2098 of res_pjsip/pjsip_configuration.c
    32 bytes allocated by init_media_configuration() line  2089 of res_pjsip/pjsip_configuration.c
    37 bytes allocated by ast_sip_auth_vector_init() line   521 of res_pjsip/pjsip_configuration.c
   464 bytes allocated by ast_sip_endpoint_alloc() line  2098 of res_pjsip/pjsip_configuration.c
   208 bytes allocated by init_info_configuration() line  2084 of res_pjsip/pjsip_configuration.c

The memory summary is:

pbx-02*CLI> memory show summary
   8905792 bytes in       2803 allocations in file ../src/pj/pool_policy_malloc.c
    263206 bytes in        398 allocations in file /opt/asterisk-13.13.1/include/asterisk/strings.h
    270906 bytes in        239 allocations in file /opt/asterisk-13.13.1/include/asterisk/threadstorage.h
      3920 bytes in         15 allocations in file acl.c
      3704 bytes in         11 allocations in file app.c
       232 bytes in          3 allocations in file app_agent_pool.c
       728 bytes in          1 allocations in file app_bridgewait.c
       984 bytes in          1 allocations in file app_confbridge.c
     25512 bytes in          2 allocations in file app_followme.c
     36011 bytes in         10 allocations in file app_minivm.c
      6768 bytes in          2 allocations in file app_queue.c
     13330 bytes in         11 allocations in file app_voicemail.c
       784 bytes in          5 allocations in file ari/config.c
     10802 bytes in        554 allocations in file asterisk.c
   6050808 bytes in      84039 allocations in file astobj2.c
       288 bytes in          2 allocations in file bridge.c
      1008 bytes in          1 allocations in file bucket.c
       848 bytes in          4 allocations in file ccss.c
     12880 bytes in          5 allocations in file cdr.c
      2289 bytes in         28 allocations in file cdr_adaptive_odbc.c
      7032 bytes in          9 allocations in file cel.c
      4080 bytes in          2 allocations in file cel_custom.c
      1828 bytes in          2 allocations in file chan_oss.c
       752 bytes in          1 allocations in file chan_pjsip.c
      4096 bytes in          1 allocations in file chan_unistim.c
  34263904 bytes in     160759 allocations in file channel.c
     12877 bytes in        644 allocations in file cli.c
      5824 bytes in         44 allocations in file codec.c
     25920 bytes in          1 allocations in file codec_resample.c
    150652 bytes in         72 allocations in file confbridge/conf_config_parser.c
     86283 bytes in        617 allocations in file config.c
    122384 bytes in        627 allocations in file config_options.c
       152 bytes in          1 allocations in file core_local.c
     15473 bytes in         25 allocations in file data.c
     20952 bytes in        229 allocations in file devicestate.c
    140824 bytes in        641 allocations in file endpoints.c
      2313 bytes in         17 allocations in file features_config.c
      8184 bytes in         31 allocations in file file.c
     10847 bytes in        180 allocations in file format.c
      1008 bytes in          1 allocations in file format_cache.c
  12116816 bytes in     163732 allocations in file format_cap.c
     15191 bytes (     15191 cache) in         45 allocations in file frame.c
       728 bytes in          1 allocations in file func_dialgroup.c
      6010 bytes in         21 allocations in file func_odbc.c
      6848 bytes in         16 allocations in file hashtab.c
       773 bytes in          4 allocations in file http.c
     63807 bytes in       1200 allocations in file indications.c
     24684 bytes in         12 allocations in file io.c
       656 bytes in          5 allocations in file json.c
      2296 bytes in          1 allocations in file libasteriskssl.c
     41770 bytes in        285 allocations in file loader.c
     20924 bytes in         10 allocations in file logger.c
    188553 bytes in        400 allocations in file manager.c
    262328 bytes in       1430 allocations in file media_index.c
        96 bytes in          2 allocations in file message.c
       816 bytes in          2 allocations in file named_acl.c
      1752 bytes in          1 allocations in file named_locks.c
    283233 bytes in       2648 allocations in file pbx.c
    479073 bytes in        583 allocations in file pbx_app.c
     10785 bytes in        471 allocations in file pbx_config.c
    400160 bytes in        415 allocations in file pbx_functions.c
      9304 bytes in          1 allocations in file pbx_realtime.c
       272 bytes in          2 allocations in file pjsip/cli_commands.c
        56 bytes in          1 allocations in file presencestate.c
       240 bytes in          1 allocations in file res_ari.c
       440 bytes in          1 allocations in file res_calendar.c
      3159 bytes in          7 allocations in file res_clialiases.c
       248 bytes in          1 allocations in file res_config_sqlite3.c
       712 bytes in          5 allocations in file res_hep.c
      1021 bytes in         10 allocations in file res_http_websocket.c
      2468 bytes in         10 allocations in file res_musiconhold.c
      3546 bytes in         55 allocations in file res_odbc.c
     10240 bytes in        302 allocations in file res_pjproject.c
       344 bytes in          7 allocations in file res_pjsip.c
     48744 bytes in        197 allocations in file res_pjsip/config_auth.c
       497 bytes in          3 allocations in file res_pjsip/config_global.c
      5048 bytes in         14 allocations in file res_pjsip/config_transport.c
   8363757 bytes in      33673 allocations in file res_pjsip/location.c
       352 bytes in          1 allocations in file res_pjsip/pjsip_cli.c
  82098230 bytes in     569275 allocations in file res_pjsip/pjsip_configuration.c
      1112 bytes in          4 allocations in file res_pjsip/pjsip_distributor.c
      1488 bytes in          6 allocations in file res_pjsip/pjsip_global_headers.c
     24033 bytes in        309 allocations in file res_pjsip/pjsip_options.c
      1008 bytes in          1 allocations in file res_pjsip/pjsip_scheduler.c
       117 bytes in          1 allocations in file res_pjsip_authenticator_digest.c
       296 bytes in          6 allocations in file res_pjsip_config_wizard.c
      3160 bytes in          7 allocations in file res_pjsip_endpoint_identifier_ip.c
       939 bytes in          6 allocations in file res_pjsip_exten_state.c
      2048 bytes in          1 allocations in file res_pjsip_history.c
     58744 bytes in        286 allocations in file res_pjsip_mwi.c
      4915 bytes in         43 allocations in file res_pjsip_notify.c
      3895 bytes in         16 allocations in file res_pjsip_outbound_registration.c
      3144 bytes in          3 allocations in file res_pjsip_pidf_body_generator.c
    162135 bytes in        783 allocations in file res_pjsip_pubsub.c
       639 bytes in          4 allocations in file res_pjsip_session.c
       984 bytes in          1 allocations in file res_pjsip_transport_management.c
       152 bytes in          1 allocations in file res_smdi.c
        25 bytes in          1 allocations in file res_sorcery_astdb.c
      5144 bytes in         38 allocations in file res_sorcery_config.c
      6888 bytes in          7 allocations in file res_sorcery_memory.c
     44790 bytes in        307 allocations in file res_sorcery_memory_cache.c
        80 bytes in          5 allocations in file res_sorcery_realtime.c
      7960 bytes in          5 allocations in file res_stasis.c
       728 bytes in          1 allocations in file res_stasis_device_state.c
      2168 bytes in          1 allocations in file res_stasis_playback.c
      2168 bytes in          1 allocations in file res_stasis_recording.c
       360 bytes in          2 allocations in file res_statsd.c
      9144 bytes in          1 allocations in file res_timing_pthread.c
      1824 bytes in         38 allocations in file rtp_engine.c
     52936 bytes in        334 allocations in file sched.c
  53206956 bytes in     115835 allocations in file sorcery.c
    324026 bytes in       3254 allocations in file stasis.c
      2224 bytes in          2 allocations in file stasis/messaging.c
    116661 bytes in        843 allocations in file stasis_cache.c
     12320 bytes in        110 allocations in file stasis_cache_pattern.c
     13200 bytes in          9 allocations in file stasis_channels.c
     73004 bytes in        681 allocations in file stasis_message.c
     25840 bytes in        232 allocations in file stasis_message_router.c
     37104 bytes in          2 allocations in file stdtime/localtime.c
   6492640 bytes in     221805 allocations in file stringfields.c
    231328 bytes in        148 allocations in file strings.c
     97509 bytes in       1211 allocations in file taskprocessor.c
     27768 bytes in        294 allocations in file threadpool.c
       184 bytes in          4 allocations in file timing.c
     16768 bytes in         34 allocations in file translate.c
       192 bytes in          2 allocations in file udptl.c
      9900 bytes in        124 allocations in file utils.c
   1079078 bytes in       5911 allocations in file xmldoc.c
 217141888 bytes allocated (15191 in caches) in 1379608 selected allocations

 217141888 bytes in all allocations
    921927 bytes in deferred free large allocations
     36375 bytes in deferred free small allocations
    958302 bytes in deferred free allocations
 218100190 bytes in all allocations and deferred free allocations

I’m happy to collect more debug however would need a bit more guidance on what’s needed.

What is an example of an endpoint and other configuration? I’d also suggest trying 13.14.0 as some memory leaks were fixed in it.

Hi,

I’ll see if I can get 13.14.0 on there soon - in terms of config, here’s the main bits - everything not mentioned is null, ie. default

Sorcery Config:

endpoint/cache=memory_cache,object_lifetime_stale=600,object_lifetime_maximum=1800,expire_on_reload=yes
endpoint=realtime,ps_endpoints
endpoint=config,pjsip.conf,criteria=type=endpoint
auth/cache=memory_cache,expire_on_reload=yes
auth=realtime,ps_auths
aor/cache=memory_cache,object_lifetime_stale=1500,object_lifetime_maximum=1800,expire_on_reload=yes
aor=realtime,ps_aors
aor=config,pjsip.conf,criteria=type=aor
domain_alias=realtime,ps_domain_aliases
contact=realtime,ps_contacts

Using ODBC with default options (pre-connect = yes).

Example endpoint:

mysql> select * from ps_endpoints where id = 'a211d0f5-94df-4e38-a804-6b0e24263fe7' \G
*************************** 1. row ***************************
                                id: a211d0f5-94df-4e38-a804-6b0e24263fe7
                         transport: transport-udp
                              aors: a211d0f5-94df-4e38-a804-6b0e24263fe7
                              auth: a211d0f5-94df-4e38-a804-6b0e24263fe7
                           context: company-xxx
                          disallow: all
                             allow: alaw
                      direct_media: no
                         dtmf_mode: info
                       force_rport: yes
                       ice_support: no
                   rewrite_contact: yes
                     rtp_symmetric: yes
                    send_diversion: yes
                          send_pai: yes
                         send_rpid: yes
                          callerid: 801
                  callerid_privacy: allowed
                            100rel: no
                  trust_id_inbound: no
                 trust_id_outbound: no
                  named_call_group: xxx-2
                named_pickup_group: xxx-2
              device_state_busy_at: 1
                    sub_min_expiry: 30
                       from_domain: pbx-02.xxx.com
                       accountcode: xxx

mysql> select * from ps_aors where id = 'a211d0f5-94df-4e38-a804-6b0e24263fe7' \G
*************************** 1. row ***************************
                  id: a211d0f5-94df-4e38-a804-6b0e24263fe7
             contact: NULL
  default_expiration: 720
           mailboxes: 102@company-xxx
        max_contacts: 2
  minimum_expiration: NULL
     remove_existing: no
   qualify_frequency: 30
authenticate_qualify: NULL
  maximum_expiration: NULL
      outbound_proxy: NULL
        support_path: NULL
     qualify_timeout: 10
 voicemail_extension: NULL

Other than that - the dialplan makes good use of Local channels though would hope there aren’t issues there!

If you’re able to suggest some ways to track down the source of the increased usage that’d be great.

The easiest way is to use the memory summary, do some controlled things, see what goes up and then identify each and try to explain why it went up. It requires digging into the code and understanding the scenario and what is going on.