Asterisk 16.5 Memory leak, cannot reproduce with debug

Hi there,

We discovered a memory leak with our Asterisk setup, every days when it processes calls & webrtc sessions, memory usage ramps up, it stays flat at out of office hours then continues to rises.

On my test setup I was able to get it to > 1.5Gb in less than 48 hours.

So, ready to report the issue, I recompiled asterisk with DONT_OPTIMIZE, COMPILE_DOUBLE, DEBUG_THREADS, DEBUG_FD_LEAKS, BETTER_BACKTRACES, MALLOC_DEBUG, BUILD_NATIVE flags.

The issue now is that I need to feed asterisk with test calls to let it eat some RAM before dumping the core, but I can’t!

Performance is really horrible, it runs barely at 20-30 simultaneous calls, and when I connect more than a couple of webrtc browsers, it starts to lag, then hangs, then crash.

I’m running on AWS with C5.2XLARGE instance (8CPU cores & 15Gb RAM), compiled without debug flags it can run more than 1000+ channels smoothly on the exact same instance size & AMI.

We are using mysql realtime (res_mysql) and everything is done with stasis (dialplan starts, then pass immediately to stasis).

Any clue on what is going on? I would love to reproduce the issue and post full dumps but at 15 simultaneous calls and 3-5 webrtc sessions it will take ages to fill the RAM.

Regards.

You don’t need to build with DEBUG_THREADS for this. You can try disabling that and see if it allows you to isolate the problem. You also haven’t stated which SIP channel driver you are using.

Hi,

I’m using PJSIP (bundled).

Without DEBUG_THREADS it can take 100-150 calls, it was unstable/crashing at 200 so i’m running with 20 webRTC sessions & 100 calls, RAM is slowing creeping up.

If I run ast_coredumper when RAM usage is about 500MB is it enough?

Getting MALLOC_DEBUG information would show where memory is allocated. A backtrace doesn’t provide that information directly.

I got this from memory_show_summary:

memory_show_symmary.txt (11.1 KB)

File for “memory show allocations” is too big (22M) so I zipped it:

memory_show_allocations.zip.txt (1.1 MB)

Is there something else that I can provide to help?

I’d suggest filing an issue[1] with complete details including description of what precisely you’re doing, configuration, configuration that reproduces it, everything.

[1] https://issues.asterisk.org/jira

Done, under reference ASTERISK-28523 :slight_smile: