When doing performance testing memory is not freed

Hi,

I am using Asterisk 1.4.20.1 and sipp tool for performance testing. I have 1 GB RAM and swap 3 GB.

Here are the observations i made while testing.

calls 30 50 100 30 50
Memory(kb) 45448 50412 72308 74900 79436

This memory used by Asterisk server is calucated using pmap -x pid of process. Anybody pls give me some pointers on this. Used memory is not freed gettting increased continously.

I ran Asterisk valgrind found major leaks though.

Here is CPU info
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name : Intel® Pentium® D CPU 2.80GHz
stepping : 7
cpu MHz : 2793.156
cache size : 1024 KB
physical id : 0
siblings : 1
core id : 255
cpu cores : 1
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm pni monitor ds_cpl cid cx16 xtpr lahf_lm
bogomips : 5590.56

processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name : Intel® Pentium® D CPU 2.80GHz
stepping : 7
cpu MHz : 2793.156
cache size : 1024 KB
physical id : 1
siblings : 1
core id : 255
cpu cores : 1
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm pni monitor ds_cpl cid cx16 xtpr lahf_lm
bogomips : 5586.24

meminfo is

MemTotal: 1027544 kB
MemFree: 820884 kB
Buffers: 9412 kB
Cached: 146512 kB
SwapCached: 0 kB
Active: 94284 kB
Inactive: 88608 kB
HighTotal: 122816 kB
HighFree: 160 kB
LowTotal: 904728 kB
LowFree: 820724 kB
SwapTotal: 3004112 kB
SwapFree: 3004112 kB
Dirty: 200 kB
Writeback: 0 kB
Mapped: 41380 kB
Slab: 13900 kB
CommitLimit: 3517884 kB
Committed_AS: 142036 kB
PageTables: 688 kB
VmallocTotal: 118776 kB
VmallocUsed: 6064 kB
VmallocChunk: 112380 kB

Sorry i wrongly mentiond when i ran valgrind i did not find any memory leaks.

malloc doesn’t garbage collect, because there is no mechanism to find references to heap memory and adjust them. It may adjust the allocation when there are clear pages at the top of the heap, and some implementations may have means to return internal pages to demand zero status.

Hi David thanks for your reply. How i proceed debug this issue ?

I don’t think you have proved that there is an issue.

Pls tell me what kind of of info you need from me.

I am trying to show that there was continous allocation of memory by Asterisk, even when no calls are there memory usage is not coming down.

Your valgrind results tend to disprove this.

To demonstrate a leak, without using valgrind, you would need to run for a very long time with calls randomly coming and going. If the rate of increase tends to zero, all is well, if it tends to a positive, constant, non-zero value, then you can start worrying.

Initially memory will be relatively unfragmented. As calls turnover, the fragmentation will increase, but, eventually you should reach a point where there is always a large enough section of free memory to satisfy a request, or at least I think that is how malloc type allocators behave for random requests.

There are almost certainly academic papers on the subject.

I agree with your explanation that “As calls turnover, the fragmentation will increase, but, eventually you should reach a point where there is always a large enough section of free memory to satisfy a request, or at least I think that is how malloc type allocators behave for random requests.” .

In my case also it does the same when i made 100 calls it eats memory let us say 70Kbytes aftter that i hangup all calls , now hardly memory comes down to around 60Kb and now here if i make 50 calls again no increase or decrease in memory since it has already large enough memory section held by asterisk process. So it might be trying to use it.

But next time if i make 100 calls there is slight increase in memory usage again and after that slight decrease in memory, like this if i go on testing after few hours i may reach to max of RAM where system could malfunction and more over here is the point where asterisk process holding entire memory with it but this is not we intended.

If it is just fragmentation, the memory will never grow when its current size is greater than total allocated + (number of active allocations) * (largest allocation -1), where sizes include any memory manager overhead, and ignoring the roundup to an integral number of pages.

For some definition of lifetime worst case, I think it can’t exceed (2 * lifetime worst case total allocation) - corresponding number of active allocations, but that is getting too statistical for me to be completely sure.

Finally my problem seems to be solved !.

I made randomly SIP calls to Asterisk and observed several things here are the things …

When i made 100 calls top shows up total memory usage including cache/buffers/physical memory this is correct behaviour of TOP command.

After some time i hangup all calls Asterisk release memory to system but most of memory allocations are still held by cache/buffers which is also free memory. Here it behaves like this because it could expect next time more than 100 calls or near around that so it tried reserve some memory in cache/buffers for faster access which results for TOP to show more memory usage.

Next time if i make 50 calls no furthur decrease/increase in the memory as this is the expected behaviour of linux memory management technique.

The freely available RAM memory can be shown using free -m at bufffers/cache line. Any running process or application can use this memory.

I got this info from some forums and i concluded with my observations also. I feel nothing to worry about this behaviour.

Mr David please make some comments on this and give me your ideas on this.

total used free shared buffers cached
Mem: 1003 445 558 0 163 200
-/+ buffers/cache: 81 922
Swap: 2933 0 2933

buffers and cache section is acurate information you can reply on.

Cache and buffer space is really managed by Linux. Linux will aim for quite a low free memory, with the remainder used up by the cache (cached files will not be discarded until the memory is needed for something else), so a low free memory is not an indication of a problem; one actually has to look at the physical disk access rate, particularly reads, to get an idea as to whether you really have a memory problem at the OS level.

Anyway thanks for the pointers and explanation.

Bye
Ramaseshi
www.intellisofttech.com

A bug has just been fixed which resulted in a memory leak under some circumstances:

https://issues.asterisk.org/view.php?id=15973

The memory leak is in the Asterisk -1.6 version but i am using 1.4. I do not see any memory leaks with valgrind.