Call quality degrade after period of time - Asterisk 13.10.0

Asterisk
1

Hi

I have a PBX server:
Centos 5.11
Intel® Xeon® CPU E5-2630 v2 @ 2.60GHz x2
32Gig Memory
Software RAID
1000 SIP clients
50-80 Concurrent calls

We reboot the server on Sunday mornings and by Thursdays and Fridays the quality on the calls degrade so much that we are forced to restart asterisk. The quality on the calls are perfectly fine after the restart.

When looking at the memory usage on the server hardly 6Gig of the total memory is being used.
Could it be that Asterisk has a memory leak and not using any more memory?
Any one else experienced the same problem?
Could upgrading to Asterisk 13.11.2 resolve the problem?

Suggested stable Asterisk 13 version?

Regards

2

Memory leaks cause the faulty process to use indefinitely large amounts of memory, not to stop at a certain level of usage less than the total memory. Also how did you measure memory. Both Linux and Windows will try to make use of nearly all available memory for something. You need to measure the process used by a single process to establish a leak.

It would be helpful to understand the actual nature of the distortion to the audio (high jitter, lost packets, etc.).

3

Its extremely high jitter on the calls but only one way. The incoming callers voice has hight jitters while the users audio is perfectly fine.
We have excluded lost packets as the network.

That is exactly why the low memory is strange, why will Linux not use more of the available memory?

Another client with 300 sip extensions has 12Gig memory and Linux is using 98% of the memory. They are also running Asterisk 13.10.0 and they don’t seam to experience the same problem.

4

It could be a number of things. Bad ethernet cable, bad switch, too much I/O on the hard drives, high amount of transcoding, etc. It sounds like yours is internal to the server. How much memory is the asterisk process using (top)? Is there another process using a large amount of resources? I had a problem with dead locks using chan_sip and realtime voicemail. It continued to queue up events in the taskprocessor until it used all of the memory for the asterisk process. Digium released a patch last week that fixed my problem. I haven’t had any audio problems though.

5

Memory % are low:

Mem: 33236188k total, 5796420k used, 27439768k free, 413456k buffers
5455 root 15 0 611m 479m 9132 S 109.3 1.5 354:37.99 asterisk
17834 root 25 0 671m 68m 7544 S 0.0 0.2 0:04.06 java
6587 mysql 33 10 139m 27m 4312 S 0.0 0.1 6:55.48 mysqld
4708 root 12 -10 22520 21m 1732 S 0.0 0.1 0:00.01 iscsiuio
17700 root 27 10 21904 19m 1336 S 51.1 0.1 76:08.73 op_server.pl

I am think of recompiling asterisk but I also might want to upgrade to 13.11.2

Only FOP is using high amount of CPU, but no other program is using high amount of memory.

Regards

6

The cpu does seem high. Are you transcoding a lot?

7

SIP if forced to use G729 and IAX as well. I don’t expect to have high or even any transcoding.

8

Transcoding will happen if you generate in band tones, record the call, run conferences, or use whisper, even if both incoming and outgoing sides use the same codec.

Originate will specify SLIN, unless you tell it otherwise, so can result in transcodings.

You will get transcodings if you don’t have recordings in the codec being used.

9

Hi

I have narrowed down the problem to IAX.

SIP to SIP dialing is working perfectly fine.
SIP to IAX is causing the quality on the calls:
-------- LOCAL --------------------- -------- REMOTE --------------------
Channel RTT Jit Del Lost % Drop OOO Kpkts Jit Del Lost % Drop OOO Kpkts FirstMsg LastMsg
IAX2/bitco-503 3 77 123 1 0 1 49 0 0 40 0 0 0 0 0 Rx:NEW Rx:ACK
IAX2/bitco-3057 3 99 146 20 0 34 162 0 0 40 0 0 0 0 0 Rx:NEW Tx:ACK
IAX2/bitco-5712 3 91 142 0 0 3 21 0 0 40 0 0 0 0 0 Rx:NEW Rx:ACK
IAX2/bitco-6524 3 147 189 0 0 13 168 0 0 40 0 0 0 0 0 Rx:NEW Rx:ACK
IAX2/bitco-8255 3 107 161 829 1 108 340 0 0 40 0 0 0 0 0 Rx:NEW Tx:ACK
IAX2/bitco-14808 3 76 122 2 0 3 69 0 0 40 0 0 0 0 0 Rx:NEW Tx:ACK
IAX2/bitco-bloemfontein-1 1 91 147 23 0 40 156 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/bitco-kemptonpark-18 1 91 145 24 0 34 143 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/bitco-onlythebest-18 3 78 139 15 0 5 113 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/workforce-belville-1 3 88 134 194 0 41 229 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/workforce-fempower-1 1000 0 40 0 0 0 0 0 0 0 0 0 0 0 0 Tx:NEW Rx:ACK
IAX2/bitco-montaguegarden 3 93 150 14 0 14 104 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/workforce-fempower-2 3 83 137 11 0 6 211 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/bitco-qw-20470 3 80 137 2 0 9 103 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/workforce-fempower-2 3 98 154 3 0 4 38 0 0 40 0 0 0 0 0 Tx:NEW Rx:ACK
IAX2/bitco-wadeville-2096 1000 116 162 3 0 0 3 0 0 0 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/workforce-belville-2 1000 79 144 1 0 1 29 0 0 0 0 0 0 0 0 Tx:NEW Rx:ACK
IAX2/bitco-finance-3rdflo 3 78 133 43 0 59 1195 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/workforce-pinetown-2 3 79 136 52 0 64 1261 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/bitco-hr-22226 1 77 126 4 0 8 127 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/workforce-pinetown-2 1000 159 200 19 1 14 38 0 0 0 0 0 0 0 0 Tx:NEW Rx:ACK
IAX2/bitco-supersoftware- 1000 85 135 3 0 2 10 0 0 0 0 0 0 0 0 Tx:NEW Rx:ACK
IAX2/bitco-bolandstr-paar 1 99 153 3 0 9 72 0 0 40 0 0 0 0 0 Tx:NEW Rx:ACK
IAX2/bitco-accutech-23730 3 102 155 6 0 13 95 0 0 40 0 0 0 0 0 Tx:NEW Rx:ACK
IAX2/bitco-qw-24181 3 90 138 3 0 2 49 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/bitco-riebeek-24480 3 91 153 3 0 6 83 0 0 40 0 0 0 0 0 Tx:NEW Rx:ACK
IAX2/bitco-eppingroad-249 3 78 130 6 0 13 285 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/bitco-eeb-24974 3 74 120 1 0 10 108 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/bitco-upington-25408 3 80 120 5 0 4 56 0 0 40 0 0 0 0 0 Tx:NEW Rx:ACK
IAX2/bitco-onlythebest-27 80 90 140 29 0 71 493 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/bitco-riebeek-27977 1 80 122 39 0 59 991 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/bitco-richardsbay-29 3 102 156 63 0 63 449 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/workforce-pinetown-2 44 78 123 4 0 2 71 0 0 40 0 0 0 0 0 Tx:NEW Rx:ACK
IAX2/bitco-wadeville-3006 3 78 122 23 0 29 426 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/bitco-supersoftware- 3 100 148 179 4 72 522 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/bitco-houghton-30522 3 140 199 902 1 86 299 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/bitco-riebeek-30986 3 88 134 23 0 40 221 0 0 40 0 0 0 0 0 Tx:NEW Tx:ACK
IAX2/workforce-fempower-3 3 95 144 24 3 4 23 0 0 40 0 0 0 0 0 Tx:NEW Rx:ACK
IAX2/workforce-fempower-3 3 78 120 9 0 2 74 0 0 40 0 0 0 0 0 Tx:NEW Rx:ACK
IAX2/bitco-wadeville-3208 3 79 133 4 0 6 55 0 0 40 0 0 0 0 0 Tx:NEW Rx:ACK
IAX2/bitco-eastlondon-322 1000 0 40 0 0 0 0 0 0 0 0 0 0 0 0 Tx:NEW Tx:ACK

Can any one advise on what could cause IAX to experience this problem?

Thread 5: state=0, update=0, actions=115, func=’‘
Thread 15: state=0, update=0, actions=35, func=’‘
Thread 8: state=0, update=0, actions=111, func=’‘
Thread 0: state=0, update=0, actions=158, func=’‘
Thread 6: state=0, update=0, actions=113, func=’‘
Thread 10: state=0, update=0, actions=102, func=’‘
Thread 14: state=0, update=0, actions=35, func=’‘
Thread 17: state=0, update=0, actions=19, func=’‘
Thread 3: state=0, update=0, actions=149, func=’‘
Thread 12: state=0, update=0, actions=74, func=’‘
Thread 4: state=0, update=0, actions=149, func=’‘
Thread 7: state=0, update=0, actions=109, func=’‘
Thread 11: state=0, update=0, actions=90, func=’‘
Thread 2: state=0, update=0, actions=162, func=’‘
Thread 13: state=0, update=0, actions=34, func=’‘
Thread 9: state=0, update=0, actions=103, func=’‘
Thread 1: state=0, update=0, actions=148, func=’‘
Thread 16: state=0, update=0, actions=20, func=’'
10 of 10 threads accounted for with 18 dynamic threads

Asterisk just stopped process calls as well, I wasn’t able to make or receive any calls. I will still able to run command with in asterisk.

Regards

10

See https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace#GettingaBacktrace-GettingInformationForADeadlock for obtaining debugging information for a deadlock.

11

That’s the problem, asterisk never dead locks and then never provides a core dump.

12

You have to manually request a core dump on a deadlock.

13

Can you provide me with a command in asterisk that I can run to create a core dump?

14

Not in Asterisk. Use gcore, or attach gdb to the running program.

15

Just to add to the quality problem:

After restarting Asterisk the quality on the calls only appear after 2-3 days of constant calling(50-70 Concurrent calls). There aren’t any quality related problems before that, this must indicate to a timing issue over IAX.
Since we don’t restart the the fiscal PBX server must indicate to software problem and not hardware related.

Recap:

  1. SIP to SIP no quality problems are experienced
  2. SIP to IAX quality after 2-3 days of calls being processed
  3. IAX inbound audio quality problem, outbound audio is perfectly fine
  4. Hosted system connected directly to voip provider
  5. Updating to 13.11.2 hasn’t resolve the problem.

Suspected problems:

  1. Timing issue on IAX, high call volumes
  2. IAX miss match to voip provider (Asterisk version difference)

Low volume clients aren’t experiencing any symptoms, they can run for 7 days without quality related problems.

16

Thank you, I will try that the next time it happens.

17

Just had another confirmation, 2 days after Asterisk restart and call quality starts degrading again.

The following IAX settings have been tried without any improvement:
jitterbuffer=no
trunkmtu=1460
iaxthreadcount=20

Regards

18

Regards Mkozusnik, are you able to please send me the details of this patch I have 13.12.2 but I am experiencing the same problem, I have realtime odbc voicemail and using sip trunks
Thanks in advanced.

19

That patch was for 13.5. I believe it was included in 13.8 and newer. I’m using 13.12 also, but not having any of the same troubles.

1 Like
20

I wasn’t able to fix the problem, I could only get a work around for the problem. I am restarting asterisk once a day.