Possible memory leak in Asterisk v16.29.0

Hi there!
I am struggling with Asterisk memory issues for quite long now, the problem has different nature: sometimes memory gets fully utilized in a matter of couple days, sometimes in weeks, one in common, it’s never freed on itself - the service should be restarted, or OOM killer will jump in and do its job.
At first, affected machines were at 4GB RAM, after adding +4, it just delayed the problems, but did not solve it, basically it just bought us a bit more time before we have to restart the service manually. Upgrade to the latest v16.29.0 did not help as well. It’s fairly hard to reproduce the same behaviour on a local instance, so this is a problem specific only to production.

I know that I can re-build Asterisk package with MALLOC_DEBUG flag to retrieve more specific info, but since this will concern production machines it might be problematic, of course, I can get more info/configs on demand, but if we could avoid re-building package - would be great.

Appreciate any help, thanks.

$ ps aux --sort -rss
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
asterisk 17914  4.9 83.5 9016412 6692700 ?     Ssl  лис18 166:39 /usr/sbin/asterisk -f -C /etc/asterisk/asterisk.conf
...

P.S. I don’t think that setting cron job for restarting Asterisk service or dropping cache is a good idea, rather an ugly hack, so I’m looking forward to any suggestions.

I think for anyone to progress this, you will need to reproduce it on a supported version of Asterisk, and provide details of what you are doing on your system that is unusual. Asterisk 16 has been security fixes only for over a month, now.

I don’t understand the reference to “dropping cache”. As far as I know, cache could only really apply to the the system cache, which deliberately grows to use most of the otherwise free memory, and does not represent a memory use problem. However, your ps does seem to show a very large usage for Asterisk, itself, although I have no idea of the provenance of the graphs.

I don’t think that I could emphasize anything unusual on Asterisk itself.
Production application is passing calls to it, using different modules, no Asterisk’s sorcery cache is configured.

Click here to see modules

I stripped modules description in order to comply with 32000 char limit.

<my_machine>*CLI> module show
Module                          Use Count  Status      Support Level
acl                             2          Running              core
app_adsiprog.so                 0          Running        deprecated
app_alarmreceiver.so            0          Not Running      extended
app_amd.so                      0          Not Running      extended
app_attended_transfer.so        0          Running          extended
app_authenticate.so             0          Running              core
app_blind_transfer.so           0          Running          extended
app_bridgeaddchan.so            0          Running              core
app_bridgewait.so               0          Running              core
app_cdr.so                      0          Running              core
app_chanisavail.so              0          Running          extended
app_channelredirect.so          0          Running              core
app_chanspy.so                  1          Running              core
app_controlplayback.so          0          Running              core
app_db.so                       0          Running              core
app_dial.so                     0          Running              core
app_dictate.so                  0          Running          extended
app_directed_pickup.so          0          Running              core
app_disa.so                     0          Running              core
app_dtmfstore.so                0          Running          extended
app_dumpchan.so                 0          Running              core
app_echo.so                     0          Running              core
app_exec.so                     0          Running              core
app_externalivr.so              0          Running          extended
app_forkcdr.so                  0          Running              core
app_getcpeid.so                 0          Running        deprecated
app_image.so                    0          Running        deprecated
app_macro.so                    0          Running        deprecated
app_mf.so                       0          Running          extended
app_milliwatt.so                0          Running              core
app_mixmonitor.so               0          Running              core
app_morsecode.so                0          Running          extended
app_nbscat.so                   0          Running        deprecated
app_originate.so                0          Running              core
app_playback.so                 0          Running              core
app_playtones.so                0          Running              core
app_privacy.so                  0          Running              core
app_read.so                     0          Running              core
app_readexten.so                0          Running              core
app_record.so                   0          Running              core
app_reload.so                   0          Running          extended
app_saycounted.so               0          Running          extended
app_sayunixtime.so              0          Running              core
app_senddtmf.so                 0          Running              core
app_sendtext.so                 0          Running              core
app_sf.so                       0          Running          extended
app_sms.so                      0          Running          extended
app_softhangup.so               0          Running              core
app_speech_utils.so             0          Running              core
app_stack.so                    0          Running              core
app_stasis.so                   12         Running              core
app_statsd.so                   0          Running          extended
app_stream_echo.so              0          Running              core
app_system.so                   0          Running              core
app_talkdetect.so               0          Running              core
app_test.so                     0          Running          extended
app_transfer.so                 0          Running              core
app_url.so                      0          Running        deprecated
app_userevent.so                0          Running              core
app_verbose.so                  0          Running              core
app_waitforcond.so              0          Running          extended
app_waitforring.so              0          Running          extended
app_waitforsilence.so           0          Running          extended
app_waituntil.so                0          Running              core
app_while.so                    0          Running              core
app_zapateller.so               0          Running          extended
bridge_builtin_features.so      1          Running              core
bridge_builtin_interval_feature 0          Running              core
bridge_holding.so               2          Running              core
bridge_native_rtp.so            0          Running              core
bridge_simple.so                11         Running              core
bridge_softmix.so               3          Running              core
ccss                            3          Running              core
cdr                             8          Running              core
cdr_adaptive_odbc.so            0          Running              core
cdr_csv.so                      0          Not Running      extended
cdr_custom.so                   0          Running              core
cdr_manager.so                  0          Not Running          core
cdr_odbc.so                     0          Not Running      extended
cel                             1          Running              core
chan_bridge_media.so            0          Running              core
chan_pjsip.so                   65         Running              core
chan_rtp.so                     0          Running              core
codec_a_mu.so                   0          Running              core
codec_adpcm.so                  0          Running              core
codec_alaw.so                   0          Running              core
codec_codec2.so                 0          Running              core
codec_g722.so                   0          Running              core
codec_g726.so                   0          Running              core
codec_gsm.so                    0          Running              core
codec_ilbc.so                   0          Running              core
codec_lpc10.so                  0          Running              core
codec_opus.so                   0          Running          extended
codec_resample.so               0          Running              core
codec_silk.so                   0          Running          extended
codec_siren14.so                0          Running          extended
codec_siren7.so                 0          Running          extended
codec_speex.so                  0          Running              core
codec_ulaw.so                   17         Running              core
dnsmgr                          2          Running              core
dsp                             1          Running              core
enum                            2          Running              core
extconfig                       14         Running              core
features                        1          Running              core
format_g719.so                  0          Running              core
format_g723.so                  0          Running              core
format_g726.so                  0          Running              core
format_g729.so                  0          Running              core
format_gsm.so                   0          Running              core
format_h263.so                  0          Running              core
format_h264.so                  0          Running              core
format_ilbc.so                  0          Running              core
format_ogg_opus.so              0          Running              core
format_ogg_speex.so             0          Running          extended
format_ogg_vorbis.so            0          Running              core
format_pcm.so                   3          Running              core
format_siren14.so               0          Running              core
format_siren7.so                0          Running              core
format_sln.so                   0          Running              core
format_vox.so                   0          Running          extended
format_wav.so                   0          Running              core
format_wav_gsm.so               0          Running              core
func_aes.so                     0          Running              core
func_base64.so                  0          Running              core
func_blacklist.so               0          Running              core
func_callcompletion.so          0          Running              core
func_callerid.so                0          Running              core
func_cdr.so                     1          Running              core
func_channel.so                 0          Running              core
func_config.so                  0          Running              core
func_cut.so                     1          Running              core
func_db.so                      0          Running              core
func_devstate.so                0          Running              core
func_dialgroup.so               0          Running              core
func_dialplan.so                0          Running              core
func_enum.so                    0          Running              core
func_env.so                     0          Running              core
func_extstate.so                0          Running              core
func_frame_drop.so              0          Running          extended
func_frame_trace.so             0          Running          extended
func_global.so                  0          Running              core
func_groupcount.so              1          Running              core
func_hangupcause.so             0          Running              core
func_holdintercept.so           0          Running              core
func_iconv.so                   0          Running              core
func_jitterbuffer.so            0          Running              core
func_json.so                    0          Running          extended
func_lock.so                    0          Running              core
func_logic.so                   0          Running              core
func_math.so                    0          Running              core
func_md5.so                     0          Running              core
func_module.so                  0          Running              core
func_odbc.so                    0          Not Running          core
func_periodic_hook.so           2          Running              core
func_pitchshift.so              0          Running          extended
func_pjsip_aor.so               0          Running              core
func_pjsip_contact.so           0          Running              core
func_pjsip_endpoint.so          0          Running              core
func_presencestate.so           0          Running              core
func_rand.so                    0          Running              core
func_realtime.so                0          Running              core
func_sayfiles.so                0          Running              core
func_scramble.so                0          Running              core
func_sha1.so                    0          Running              core
func_shell.so                   0          Running              core
func_sorcery.so                 0          Running              core
func_speex.so                   0          Running              core
func_sprintf.so                 0          Running              core
func_srv.so                     0          Running              core
func_strings.so                 0          Running              core
func_sysinfo.so                 0          Running              core
func_talkdetect.so              0          Running              core
func_timeout.so                 0          Running              core
func_uri.so                     1          Running              core
func_version.so                 0          Running              core
func_volume.so                  0          Running              core
http                            5          Running              core
indications                     1          Running              core
logger                          1          Running              core
manager                         1          Running              core
pbx_config.so                   0          Running              core
pbx_dundi.so                    0          Not Running      extended
pbx_loopback.so                 0          Running              core
pbx_realtime.so                 0          Running          extended
pbx_spool.so                    0          Running              core
plc                             1          Running              core
res_adsi.so                     2          Running        deprecated
res_agi.so                      1          Running              core
res_ari.so                      10         Running              core
res_ari_applications.so         0          Running              core
res_ari_asterisk.so             0          Running              core
res_ari_bridges.so              0          Running              core
res_ari_channels.so             9          Running              core
res_ari_device_states.so        0          Running              core
res_ari_endpoints.so            0          Running              core
res_ari_events.so               1          Running              core
res_ari_model.so                10         Running              core
res_ari_playbacks.so            0          Running              core
res_ari_recordings.so           0          Running              core
res_ari_sounds.so               0          Running              core
res_chan_stats.so               0          Running          extended
res_clialiases.so               0          Running              core
res_clioriginate.so             0          Running              core
res_config_odbc.so              0          Running              core
res_convert.so                  0          Running              core
res_crypto.so                   2          Running              core
res_format_attr_g729.so         1          Running              core
res_format_attr_h263.so         1          Running              core
res_format_attr_h264.so         1          Running              core
res_format_attr_ilbc.so         1          Running              core
res_format_attr_opus.so         1          Running              core
res_format_attr_silk.so         1          Running              core
res_format_attr_siren14.so      1          Running              core
res_format_attr_siren7.so       1          Running              core
res_format_attr_vp8.so          1          Running              core
res_hep.so                      2          Running          extended
res_hep_pjsip.so                0          Running          extended
res_hep_rtcp.so                 0          Running          extended
res_http_media_cache.so         0          Not Running          core
res_http_post.so                0          Running              core
res_http_websocket.so           4          Running              core
res_limit.so                    0          Running              core
res_manager_devicestate.so      0          Running              core
res_manager_presencestate.so    0          Running              core
res_monitor.so                  1          Running        deprecated
res_musiconhold.so              3          Running              core
res_mutestream.so               0          Running              core
res_mwi_devstate.so             0          Running              core
res_odbc.so                     5          Running              core
res_odbc_transaction.so         1          Running              core
res_pjproject.so                4          Running              core
res_pjsip.so                    58         Running              core
res_pjsip_acl.so                0          Running              core
res_pjsip_authenticator_digest. 0          Running              core
res_pjsip_caller_id.so          14         Running              core
res_pjsip_config_wizard.so      1          Running              core
res_pjsip_dialog_info_body_gene            Running              core
res_pjsip_diversion.so          14         Running              core
res_pjsip_dlg_options.so        13         Running              core
res_pjsip_dtmf_info.so          13         Running              core
res_pjsip_empty_info.so         13         Running              core
res_pjsip_endpoint_identifier_a 0          Running              core
res_pjsip_endpoint_identifier_i 0          Running              core
res_pjsip_endpoint_identifier_u 0          Running              core
res_pjsip_exten_state.so        0          Running              core
res_pjsip_header_funcs.so       13         Running              core
res_pjsip_history.so            0          Running          extended
res_pjsip_logger.so             0          Running              core
res_pjsip_messaging.so          13         Running              core
res_pjsip_mwi.so                0          Running              core
res_pjsip_mwi_body_generator.so 0          Running              core
res_pjsip_nat.so                13         Running              core
res_pjsip_one_touch_record_info 13         Running              core
res_pjsip_outbound_authenticato 0          Running              core
res_pjsip_outbound_publish.so   2          Running              core
res_pjsip_outbound_registration 0          Running              core
res_pjsip_path.so               13         Running              core
res_pjsip_pidf_body_generator.s 0          Running              core
res_pjsip_pidf_digium_body_supp 0          Running              core
res_pjsip_pidf_eyebeam_body_sup 0          Running              core
res_pjsip_publish_asterisk.so   0          Running              core
res_pjsip_pubsub.so             12         Running              core
res_pjsip_refer.so              14         Running              core
res_pjsip_registrar.so          0          Running              core
res_pjsip_rfc3326.so            13         Running              core
res_pjsip_sdp_rtp.so            13         Running              core
res_pjsip_send_to_voicemail.so  13         Running              core
res_pjsip_session.so            19         Running              core
res_pjsip_sips_contact.so       0          Running              core
res_pjsip_stir_shaken.so        0          Not Running          core
res_pjsip_t38.so                19         Running              core
res_pjsip_transport_websocket.s 13         Running              core
res_pjsip_xpidf_body_generator. 0          Running              core
res_realtime.so                 0          Running              core
res_remb_modifier.so            0          Running          extended
res_rtp_asterisk.so             7          Running              core
res_rtp_multicast.so            1          Running              core
res_security_log.so             0          Running              core
res_smdi.so                     0          Running          extended
res_sorcery_astdb.so            2          Running              core
res_sorcery_config.so           12         Running              core
res_sorcery_memory.so           7          Running              core
res_sorcery_memory_cache.so     0          Running              core
res_sorcery_realtime.so         7          Running              core
res_speech.so                   2          Running              core
res_srtp.so                     8          Running              core
res_stasis.so                   35         Running              core
res_stasis_answer.so            1          Running              core
res_stasis_device_state.so      1          Running              core
res_stasis_playback.so          3          Running              core
res_stasis_recording.so         4          Running              core
res_stasis_snoop.so             1          Running              core
res_statsd.so                   4          Running          extended
res_stir_shaken.so              0          Not Running          core
res_timing_pthread.so           0          Running          extended
res_timing_timerfd.so           33         Running              core
res_tonedetect.so               0          Running          extended
sounds                          1          Running              core
udptl                           2          Running              core
299 modules loaded

I don’t understand the reference to “dropping cache”.

I meant this sync; echo 3 > /proc/sys/vm/drop_caches.

although I have no idea of the provenance of the graphs.

These are from 2 production Asterisk machines that should help to see that memory spikes are not consistent over time, which, I think, might be tied to the number of proccessed calls, but something is not being cleaned up after call ends, and that’s why memory keeps growing, but barely being freed.

I thought 16.x is not that old after all, as you pointed out, it reached security maintenance only about month ago.

That’s accounted to the system, not Asterisk, and, by design, grows to fill nearly all available memory. It represents memory contents that is available for reloading from the disk, so can be thrown away, when an application needs more writeable space. Windows does the same thing.

Yeap, I mentioned it only to say that this is not an option of overcoming the problem, since I saw many people suggest it as one of the options.

It’s certainly better if you upgrade from 16, but in reality the differences between 16 and 18 are not as many as the numbers might imply, so some information from it is better than none if you can’t upgrade.

MALLOC_DEBUG is probably the way to go. Dump the output of “memory show summary” to a text file and then do so again periodically, or after you think you’ve encountered a leak. If you diff the files it should provide some context as to where it is, because one module may be increasing over time.

Once you’ve done that, you can narrow it down further using “memory show allocations” and grep the module name and then diff at different points in times to show you where the leaked memory was allocated.

At a minimum, that’s probably what’s needed to report an issue.

Note that you can’t report it against 16, though. It must be done against 18.