I’m running Asterisk 1.8.13 on Amazon EC2 Ubuntu 12 with sip trunking from Centurylink. I’ve got a simple ConfBridge setup which works fine, but over time develops HUGE latency. After about 5 minutes, there is a lag of about 3 seconds. After 10 minutes, the delay is closer to 6 seconds. New calls to the bridge do not experience this latency. If a participant drops off and dials back in, the latency disappears for that caller. The behavior is the same with or without jitterbuffering. Tests were done with 3 participants on the bridge. Calls in to the Echo app do not experience this. Calls bridged to outbound calls do no see this. Any suggestions on how I track down the source of this latency?
I am currently facing similar kind of problem of Latency growing over Time
I am using currently CentOs as opertating system and Asterisk® Open Source PBX asterisk 1.6.2.6 and for conferencing I am using app_konference 2.2 and pjsua1.8 as sip user agent or sip client on hard phones on local network (private network)
what happens is when any Konference() is created it works fine with latenct 250-300ms for some hour or hour and half after that there is 1-3 seconds of delay in audio
one more unusual thing I found during testing is asterisk is taking too much of memory which is continuously growing that is when I do top I get
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3161 root 15 0 293m 24m 13m S 0.7 0.6 152:40.32 Xorg
15977 root 15 0 174m 160m 4456 S 0.7 4.2 0:08.62 asterisk
3406 root 15 0 448m 402m 9.9m S 0.3 10.5 19:48.41 gnome-terminal and sip call which I was testing is continuous type of call so I kept it for 24 hrs next day I found that asterisk is taking 63% memory as follows
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15977 root 15 0 2449m 2.4g 4540 S 0.7 63.7 2:17.68 asterisk my asterisk settings are as follows
node2*CLI> core show settings
node2*CLI>
PBX Core settings
-----------------
Version: 1.6.2.6
Build Options: LOADABLE_MODULES, DEBUG_CHANNEL_LOCKS, BUSYDETECT_TONEONLY, BUSYDETECT_DEBUG
Maximum calls: Not set
Maximum open file handles: Not set
Verbosity: 3
Debug level: 0
Maximum load average: 0.000000
Minimum free memory: 0 MB
Startup time: 10:15:21
Last reload time: 10:15:21
System: Linux/2.6.18-274.7.1.el5PAE built by root on i686 2012-01-13 07:07:46 UTC
System name:
Entity ID: 84:2b:2b:a1:ff:1f
Default language: en
Language prefix: Enabled
User name and group: /
Executable includes: Disabled
Transcode via SLIN: Enabled
Internal timing: Disabled
Transmit silence during rec: Disabled
* Subsystems
-------------
Manager (AMI): Enabled
Web Manager (AMI/HTTP): Enabled
Call data records: Enabled
Realtime Architecture (ARA): Disabled
* Directories
-------------
Configuration file:
Configuration directory: /etc/asterisk
Module directory: /usr/lib/asterisk/modules
Spool directory: /var/spool/asterisk
Log directory: /var/log/asterisk
node2*CLI> actually I thought that this problem of audio latency is due to app_konference so I upgraded my app_Konference module version to 2.2 from 1.3 so what ever is causing this problem is somewhere related to asterisk than app_konference module thats what I think.
similar to what fizmod mentioned New calls(channels ) to join do not experience this latency. If a participant drops off and dials back in, the latency disappears for that caller.
as time grows delay in audio grows is there any thing I am doing wrong or configuring wrong please guide me to find out root cause of this problem.
any help is appreciated
Thanks & Regards
Pranav.
I am seeing a similar problem with asterisk 11 on Ubuntu 12.04 running on VMWare ESXi, although in my case it starts happening sooner. In my test setup with 3 phones connected over 4G cellular, I see the delay start increasing somewhere between 3-10 minutes in to the call.
As with fizmod and Pranav, if I disconnect and reconnect a phone from the conference, the delay is reset to normal for that phone. I have seen once or twice where the delay returns to normal amounts on its own.
I have seen reports of people having issues with running Asterisk in a VM, and that seems to be common with fizmod’s setup. However, I would have guessed that the VM led to momentary breaks in the audio rather than steadily increasing delay throughout the call.
I have searched the forums but didn’t see that anyone found a resolution to this. Fizmod, Pranav, were you ever able to resolve your issue?
Thanks,
Tim Langham
You probably need to take this up with Amazon. I don’t believe that Asterisk has any design concessions for virtual machines and virtual machines often distort time.
Hi
Not sure this is purely a Amazon issue, this was the same problem that existed with meetme if used without a HW timing source. only real option if used on production is to use app_Konference as this mixes the channels differently.
David, Ian,
Thank you for your replies. I’m actually running it in a VM that is on hardware I own, not in EC2, but maybe that doesn’t make much of a difference.
I have read in other places as well that there are issues with timing when running in a VM, but I would have expected that to be choppy audio rather than a large delay like this. Also, it sounded like ConfBridge was more dependent on wall time than a super-accurate timing source since it uses timerfd instead of dahdi, so I was thinking that VM timing issues wouldn’t have been such a big deal. It doesn’t appear that there are several seconds of clock drift in my VM over the time period of the call.
Are there any logs that I can collect that would help to indicate if it’s a timing problem that could be caused by running this in a VM?
Thanks,
Tim
I did some further testing and found that this is actually happening with the Echo application as well, and it’s even more pronounced there.
I left out that I am using Asterisk with softphones running on a cellular phone. I started to think that it could be a side effect of the jitter from the cellular network causing Asterisk to get behind in the audio and then never really being able to catch up. I enabled the jitter buffer for the user profile in confbridge.conf and now the performance is much better.
I still have periods of time where the delay will get up to about 3 seconds, but it recovers relatively quickly and will get back down to a few hundred ms eventually. This more or less matches up with my understanding of cellular data performance.