Possible memory leak in Asterisk v16.29.0

I don’t think that I could emphasize anything unusual on Asterisk itself.
Production application is passing calls to it, using different modules, no Asterisk’s sorcery cache is configured.

Click here to see modules

I stripped modules description in order to comply with 32000 char limit.

<my_machine>*CLI> module show
Module                          Use Count  Status      Support Level
acl                             2          Running              core
app_adsiprog.so                 0          Running        deprecated
app_alarmreceiver.so            0          Not Running      extended
app_amd.so                      0          Not Running      extended
app_attended_transfer.so        0          Running          extended
app_authenticate.so             0          Running              core
app_blind_transfer.so           0          Running          extended
app_bridgeaddchan.so            0          Running              core
app_bridgewait.so               0          Running              core
app_cdr.so                      0          Running              core
app_chanisavail.so              0          Running          extended
app_channelredirect.so          0          Running              core
app_chanspy.so                  1          Running              core
app_controlplayback.so          0          Running              core
app_db.so                       0          Running              core
app_dial.so                     0          Running              core
app_dictate.so                  0          Running          extended
app_directed_pickup.so          0          Running              core
app_disa.so                     0          Running              core
app_dtmfstore.so                0          Running          extended
app_dumpchan.so                 0          Running              core
app_echo.so                     0          Running              core
app_exec.so                     0          Running              core
app_externalivr.so              0          Running          extended
app_forkcdr.so                  0          Running              core
app_getcpeid.so                 0          Running        deprecated
app_image.so                    0          Running        deprecated
app_macro.so                    0          Running        deprecated
app_mf.so                       0          Running          extended
app_milliwatt.so                0          Running              core
app_mixmonitor.so               0          Running              core
app_morsecode.so                0          Running          extended
app_nbscat.so                   0          Running        deprecated
app_originate.so                0          Running              core
app_playback.so                 0          Running              core
app_playtones.so                0          Running              core
app_privacy.so                  0          Running              core
app_read.so                     0          Running              core
app_readexten.so                0          Running              core
app_record.so                   0          Running              core
app_reload.so                   0          Running          extended
app_saycounted.so               0          Running          extended
app_sayunixtime.so              0          Running              core
app_senddtmf.so                 0          Running              core
app_sendtext.so                 0          Running              core
app_sf.so                       0          Running          extended
app_sms.so                      0          Running          extended
app_softhangup.so               0          Running              core
app_speech_utils.so             0          Running              core
app_stack.so                    0          Running              core
app_stasis.so                   12         Running              core
app_statsd.so                   0          Running          extended
app_stream_echo.so              0          Running              core
app_system.so                   0          Running              core
app_talkdetect.so               0          Running              core
app_test.so                     0          Running          extended
app_transfer.so                 0          Running              core
app_url.so                      0          Running        deprecated
app_userevent.so                0          Running              core
app_verbose.so                  0          Running              core
app_waitforcond.so              0          Running          extended
app_waitforring.so              0          Running          extended
app_waitforsilence.so           0          Running          extended
app_waituntil.so                0          Running              core
app_while.so                    0          Running              core
app_zapateller.so               0          Running          extended
bridge_builtin_features.so      1          Running              core
bridge_builtin_interval_feature 0          Running              core
bridge_holding.so               2          Running              core
bridge_native_rtp.so            0          Running              core
bridge_simple.so                11         Running              core
bridge_softmix.so               3          Running              core
ccss                            3          Running              core
cdr                             8          Running              core
cdr_adaptive_odbc.so            0          Running              core
cdr_csv.so                      0          Not Running      extended
cdr_custom.so                   0          Running              core
cdr_manager.so                  0          Not Running          core
cdr_odbc.so                     0          Not Running      extended
cel                             1          Running              core
chan_bridge_media.so            0          Running              core
chan_pjsip.so                   65         Running              core
chan_rtp.so                     0          Running              core
codec_a_mu.so                   0          Running              core
codec_adpcm.so                  0          Running              core
codec_alaw.so                   0          Running              core
codec_codec2.so                 0          Running              core
codec_g722.so                   0          Running              core
codec_g726.so                   0          Running              core
codec_gsm.so                    0          Running              core
codec_ilbc.so                   0          Running              core
codec_lpc10.so                  0          Running              core
codec_opus.so                   0          Running          extended
codec_resample.so               0          Running              core
codec_silk.so                   0          Running          extended
codec_siren14.so                0          Running          extended
codec_siren7.so                 0          Running          extended
codec_speex.so                  0          Running              core
codec_ulaw.so                   17         Running              core
dnsmgr                          2          Running              core
dsp                             1          Running              core
enum                            2          Running              core
extconfig                       14         Running              core
features                        1          Running              core
format_g719.so                  0          Running              core
format_g723.so                  0          Running              core
format_g726.so                  0          Running              core
format_g729.so                  0          Running              core
format_gsm.so                   0          Running              core
format_h263.so                  0          Running              core
format_h264.so                  0          Running              core
format_ilbc.so                  0          Running              core
format_ogg_opus.so              0          Running              core
format_ogg_speex.so             0          Running          extended
format_ogg_vorbis.so            0          Running              core
format_pcm.so                   3          Running              core
format_siren14.so               0          Running              core
format_siren7.so                0          Running              core
format_sln.so                   0          Running              core
format_vox.so                   0          Running          extended
format_wav.so                   0          Running              core
format_wav_gsm.so               0          Running              core
func_aes.so                     0          Running              core
func_base64.so                  0          Running              core
func_blacklist.so               0          Running              core
func_callcompletion.so          0          Running              core
func_callerid.so                0          Running              core
func_cdr.so                     1          Running              core
func_channel.so                 0          Running              core
func_config.so                  0          Running              core
func_cut.so                     1          Running              core
func_db.so                      0          Running              core
func_devstate.so                0          Running              core
func_dialgroup.so               0          Running              core
func_dialplan.so                0          Running              core
func_enum.so                    0          Running              core
func_env.so                     0          Running              core
func_extstate.so                0          Running              core
func_frame_drop.so              0          Running          extended
func_frame_trace.so             0          Running          extended
func_global.so                  0          Running              core
func_groupcount.so              1          Running              core
func_hangupcause.so             0          Running              core
func_holdintercept.so           0          Running              core
func_iconv.so                   0          Running              core
func_jitterbuffer.so            0          Running              core
func_json.so                    0          Running          extended
func_lock.so                    0          Running              core
func_logic.so                   0          Running              core
func_math.so                    0          Running              core
func_md5.so                     0          Running              core
func_module.so                  0          Running              core
func_odbc.so                    0          Not Running          core
func_periodic_hook.so           2          Running              core
func_pitchshift.so              0          Running          extended
func_pjsip_aor.so               0          Running              core
func_pjsip_contact.so           0          Running              core
func_pjsip_endpoint.so          0          Running              core
func_presencestate.so           0          Running              core
func_rand.so                    0          Running              core
func_realtime.so                0          Running              core
func_sayfiles.so                0          Running              core
func_scramble.so                0          Running              core
func_sha1.so                    0          Running              core
func_shell.so                   0          Running              core
func_sorcery.so                 0          Running              core
func_speex.so                   0          Running              core
func_sprintf.so                 0          Running              core
func_srv.so                     0          Running              core
func_strings.so                 0          Running              core
func_sysinfo.so                 0          Running              core
func_talkdetect.so              0          Running              core
func_timeout.so                 0          Running              core
func_uri.so                     1          Running              core
func_version.so                 0          Running              core
func_volume.so                  0          Running              core
http                            5          Running              core
indications                     1          Running              core
logger                          1          Running              core
manager                         1          Running              core
pbx_config.so                   0          Running              core
pbx_dundi.so                    0          Not Running      extended
pbx_loopback.so                 0          Running              core
pbx_realtime.so                 0          Running          extended
pbx_spool.so                    0          Running              core
plc                             1          Running              core
res_adsi.so                     2          Running        deprecated
res_agi.so                      1          Running              core
res_ari.so                      10         Running              core
res_ari_applications.so         0          Running              core
res_ari_asterisk.so             0          Running              core
res_ari_bridges.so              0          Running              core
res_ari_channels.so             9          Running              core
res_ari_device_states.so        0          Running              core
res_ari_endpoints.so            0          Running              core
res_ari_events.so               1          Running              core
res_ari_model.so                10         Running              core
res_ari_playbacks.so            0          Running              core
res_ari_recordings.so           0          Running              core
res_ari_sounds.so               0          Running              core
res_chan_stats.so               0          Running          extended
res_clialiases.so               0          Running              core
res_clioriginate.so             0          Running              core
res_config_odbc.so              0          Running              core
res_convert.so                  0          Running              core
res_crypto.so                   2          Running              core
res_format_attr_g729.so         1          Running              core
res_format_attr_h263.so         1          Running              core
res_format_attr_h264.so         1          Running              core
res_format_attr_ilbc.so         1          Running              core
res_format_attr_opus.so         1          Running              core
res_format_attr_silk.so         1          Running              core
res_format_attr_siren14.so      1          Running              core
res_format_attr_siren7.so       1          Running              core
res_format_attr_vp8.so          1          Running              core
res_hep.so                      2          Running          extended
res_hep_pjsip.so                0          Running          extended
res_hep_rtcp.so                 0          Running          extended
res_http_media_cache.so         0          Not Running          core
res_http_post.so                0          Running              core
res_http_websocket.so           4          Running              core
res_limit.so                    0          Running              core
res_manager_devicestate.so      0          Running              core
res_manager_presencestate.so    0          Running              core
res_monitor.so                  1          Running        deprecated
res_musiconhold.so              3          Running              core
res_mutestream.so               0          Running              core
res_mwi_devstate.so             0          Running              core
res_odbc.so                     5          Running              core
res_odbc_transaction.so         1          Running              core
res_pjproject.so                4          Running              core
res_pjsip.so                    58         Running              core
res_pjsip_acl.so                0          Running              core
res_pjsip_authenticator_digest. 0          Running              core
res_pjsip_caller_id.so          14         Running              core
res_pjsip_config_wizard.so      1          Running              core
res_pjsip_dialog_info_body_gene            Running              core
res_pjsip_diversion.so          14         Running              core
res_pjsip_dlg_options.so        13         Running              core
res_pjsip_dtmf_info.so          13         Running              core
res_pjsip_empty_info.so         13         Running              core
res_pjsip_endpoint_identifier_a 0          Running              core
res_pjsip_endpoint_identifier_i 0          Running              core
res_pjsip_endpoint_identifier_u 0          Running              core
res_pjsip_exten_state.so        0          Running              core
res_pjsip_header_funcs.so       13         Running              core
res_pjsip_history.so            0          Running          extended
res_pjsip_logger.so             0          Running              core
res_pjsip_messaging.so          13         Running              core
res_pjsip_mwi.so                0          Running              core
res_pjsip_mwi_body_generator.so 0          Running              core
res_pjsip_nat.so                13         Running              core
res_pjsip_one_touch_record_info 13         Running              core
res_pjsip_outbound_authenticato 0          Running              core
res_pjsip_outbound_publish.so   2          Running              core
res_pjsip_outbound_registration 0          Running              core
res_pjsip_path.so               13         Running              core
res_pjsip_pidf_body_generator.s 0          Running              core
res_pjsip_pidf_digium_body_supp 0          Running              core
res_pjsip_pidf_eyebeam_body_sup 0          Running              core
res_pjsip_publish_asterisk.so   0          Running              core
res_pjsip_pubsub.so             12         Running              core
res_pjsip_refer.so              14         Running              core
res_pjsip_registrar.so          0          Running              core
res_pjsip_rfc3326.so            13         Running              core
res_pjsip_sdp_rtp.so            13         Running              core
res_pjsip_send_to_voicemail.so  13         Running              core
res_pjsip_session.so            19         Running              core
res_pjsip_sips_contact.so       0          Running              core
res_pjsip_stir_shaken.so        0          Not Running          core
res_pjsip_t38.so                19         Running              core
res_pjsip_transport_websocket.s 13         Running              core
res_pjsip_xpidf_body_generator. 0          Running              core
res_realtime.so                 0          Running              core
res_remb_modifier.so            0          Running          extended
res_rtp_asterisk.so             7          Running              core
res_rtp_multicast.so            1          Running              core
res_security_log.so             0          Running              core
res_smdi.so                     0          Running          extended
res_sorcery_astdb.so            2          Running              core
res_sorcery_config.so           12         Running              core
res_sorcery_memory.so           7          Running              core
res_sorcery_memory_cache.so     0          Running              core
res_sorcery_realtime.so         7          Running              core
res_speech.so                   2          Running              core
res_srtp.so                     8          Running              core
res_stasis.so                   35         Running              core
res_stasis_answer.so            1          Running              core
res_stasis_device_state.so      1          Running              core
res_stasis_playback.so          3          Running              core
res_stasis_recording.so         4          Running              core
res_stasis_snoop.so             1          Running              core
res_statsd.so                   4          Running          extended
res_stir_shaken.so              0          Not Running          core
res_timing_pthread.so           0          Running          extended
res_timing_timerfd.so           33         Running              core
res_tonedetect.so               0          Running          extended
sounds                          1          Running              core
udptl                           2          Running              core
299 modules loaded

I don’t understand the reference to “dropping cache”.

I meant this sync; echo 3 > /proc/sys/vm/drop_caches.

although I have no idea of the provenance of the graphs.

These are from 2 production Asterisk machines that should help to see that memory spikes are not consistent over time, which, I think, might be tied to the number of proccessed calls, but something is not being cleaned up after call ends, and that’s why memory keeps growing, but barely being freed.

I thought 16.x is not that old after all, as you pointed out, it reached security maintenance only about month ago.