[HELP] Seeing Huge Memory Leak in Asterisk V1.2.6

#1

I recently installed the following Asterisk software from source on a Fedora Core 5 server:

asterisk-1.2.6
zaptel-1.2.5
libpri-1.2.2
asterisk-addons-1.2.2
asterisk-sounds-1.2.1
freepbx-2.0.1

However, I am seeing a huge memory leak when I leave the server running over night. On average I will see about 124 bytes loss every 5-7 minutes. At some point in the night, I will see a huge loss of free memory, of anywhere to 70-100MB. Then the system will go back to losing 124 bytes about every 5-7 minutes. The system will never actually run out of memory, but it will lose all but a few MB’s.

I am running this on a Dell Dimension 2350 with 1.8GHz CPU and 512MB RAM. The system is sitting idle during this time, so there are no calls taking place. I only have about 12 SIP extensions configured.

I am confident that it is not the OS leaking the memory, because I have other servers running Fedora Core 5, and I don’t see this memory leak.

Does anyone know what the problem might be? What logs would be needed to solve this problem?

I have also installed Asterisk@Home V2.6 on a different server and I see this same problem.

Any advice on figuring out this problem would be greatly appreciated.

Thanks,
Marc

#2

Welcome to the club !

This is MAJOR MAJOR bad huge issue.
Reported here, in other forums and on the bugtracker and digium.
Many have the same problem, including me.

I seems like it comes from the asterisk queue application, i loose 500MB RAM in 3 days… :imp:

There is something in asterisk not freeing reserved variable space, my guess.

Its reported and im PRAYING for a QUICK QUICK fix…

#3

Do you know what the Bugtracker/Issue Number is for this issue?

#4

6153, 6950 and some more i dont have the number but read earlier.

#5

Ok, i just figured something out:

Mnes, do you have MANY musicOnHold files which you rotate randomly ?

#6

Only what comes as default. So I think like 4 or 5.

#7

I figured tonite, that i lose much memory everytime asterisk’s MOH is caching/reading a musicfile and this memory isnt freed anymore !

I have 275 MB music in the folder…

#8

So, last night I setup the following crontab

0,5,10,15,20,25,30,35,40,45,50,55 * * * * date >> /root/memloss.log;free >> /root/memloss.log
0,5,10,15,20,25,30,35,40,45,50,55 * * * * date >> /root/output.txt;ps -eo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm,vsize,pcpu,pmem,rss >> /root/output.txt

So baiscally I get to see what the free memory is and what process are running every 5 minutes. I did this on two machines. One machine was running Asterisk v1.2.6 and the other v1.2.7.1. Both of them lost 75MB and 140MB respectively between the time of 4:00am and 4:05am. Before and after that time peroid the memory loss was still present, but I was only lossing a few bytes every 5 minutes.

So what happens during that time peroid? Cron jobs? The only cronjobs during that time are:

tmpwatch
slocate.cron
rpm
prelink
logrotate
backup
0anacron
00-makewhatis.cron
00-logwatch

The backup script is the only thing related to Asterisk but the script does nothing more that backup the mysql database and a few other things. Don’t think this is the source of the memory leak.

Also, I currently don’t see anything related to mpg123 running on either of my systems. So I don’t think it has to do with the music on hold feature. But I will disable that and see what happens.

#9

Good job buddy !

Can you change the cronjobs and split them to see per task, what is happening when ?

Or simply write down the mem, start a job manually (enter the cron command in a terminal) and take the mem again.

Should isolate the bad boy that way…

#10

RichardHH, I am not sure what you are exactly looking for here. Can you provide me with the commands you want me to run.

Also, over the weekend I confirmed when the leaks happen. I rebooted both of my servers on Friday afternoon. Both of them had memory leaks at the following times.

Sat: 4:00-4:05 am
Sun: 4:20-4:25 am

However didn’t see any leaks on Monday morning when I had stop the scripts by 7:45am.

There is still a smaller memory leak that occurs more often but the big one only seems to happen at these times.

#11

Hi, Has asterisk crashed on you (or the system as a whole)? And also, can you show a sample of a free command at what you feel is a low point?

#12

I have never seen the OS or asterisk crash yet because of this “problem”. Here is the output from “free” when the amount of free memory is low.

                  total       used       free     shared    buffers     cached

Mem: 254184 246708 7476 0 44120 143548
-/+ buffers/cache: 59040 195144
Swap: 779144 0 779144

I never see anything go to swap space. This leaves me to believe, that there truly is no memory leak. Which seems likely because no one else seems to be reporting this. Or the commands free and vmstat are reporting the incorrect values.

I am beging to think that this is not a problem anymore with asterisk. The leak only seems to occur during the times a cron job is running. I think the memory is being used by the cron job and free/vmstat are not reporting the actual available memory there after.

#13

yeah you are ok. most of your mem is cache/buffer. you are seeing a drop in free memory because the kernel is loading up your disk data into memory. It has no need to ‘free’ the memory since it has so much available already. Everything looks normal to me.

george