We discovered a memory leak with our Asterisk setup, every days when it processes calls & webrtc sessions, memory usage ramps up, it stays flat at out of office hours then continues to rises.
On my test setup I was able to get it to > 1.5Gb in less than 48 hours.
So, ready to report the issue, I recompiled asterisk with DONT_OPTIMIZE, COMPILE_DOUBLE, DEBUG_THREADS, DEBUG_FD_LEAKS, BETTER_BACKTRACES, MALLOC_DEBUG, BUILD_NATIVE flags.
The issue now is that I need to feed asterisk with test calls to let it eat some RAM before dumping the core, but I can’t!
Performance is really horrible, it runs barely at 20-30 simultaneous calls, and when I connect more than a couple of webrtc browsers, it starts to lag, then hangs, then crash.
I’m running on AWS with C5.2XLARGE instance (8CPU cores & 15Gb RAM), compiled without debug flags it can run more than 1000+ channels smoothly on the exact same instance size & AMI.
We are using mysql realtime (res_mysql) and everything is done with stasis (dialplan starts, then pass immediately to stasis).
Any clue on what is going on? I would love to reproduce the issue and post full dumps but at 15 simultaneous calls and 3-5 webrtc sessions it will take ages to fill the RAM.
You don’t need to build with DEBUG_THREADS for this. You can try disabling that and see if it allows you to isolate the problem. You also haven’t stated which SIP channel driver you are using.
Without DEBUG_THREADS it can take 100-150 calls, it was unstable/crashing at 200 so i’m running with 20 webRTC sessions & 100 calls, RAM is slowing creeping up.
I’d suggest filing an issue[1] with complete details including description of what precisely you’re doing, configuration, configuration that reproduces it, everything.
