Long-delay echo present on calls to POTS, not in recordings

I’m running Asterisk 1.8.7.1 on CentOS 5.7 (installed with yum). We’re using Cisco SPA504g phones and a TDM410P with four FXO modules, two of them hooked up to lines provided by Comcast cable’s “digital voice” service (that is, the wires run from the RJ11’s on the card to the RJ11’s on the back of the cable modem).

Calls between SIP phones and softphones sound great, but calls involving the PTSN have horrible echo of an unusual sort: the person on the SIP phone hears their own voice repeated back, at full volume, about one second later. There is no echo heard on the POTS phone on the other end. If I record the call through Asterisk and FreePBX, both sides of the conversation are recorded but the echo is not present. I’ve run fxotune, made sure fxotune -s is in the startup script, and tried every available software echo canceller (including none) but nothing seems to make any difference.

The nature of the echo leads me to believe that at some point in the audio processing chain it is deliberately playing what I’ve just said back to me. What am I missing?

Is echo the wrong term? I’ve tried two things that seem (to me at least) to indicate that “echo” isn’t really what’s going on:

  1. I’ve dialed out on one FXO and called the number associated with the other FXO port and talked to the IVR. The strength and delay of the “echo” is the same as when I call a regular POTS phone. Wouldn’t they delay double, or at least be different, if it was being introduced by the TDM400P and was being routed through two of it’s channels?

  2. I’ve run dahdi_monitor -v on the dial-out channel, and the “echo” doesn’t register on the meters. Doesn’t this suggest it’s being introduced somewhere in the audio processing AFTER it comes in from the TDM card?

Also, latency is in the 1-30 ms range between the Asterisk server and the phones and CPU usage never goes above 4%, if that’s relevant.

While I give it a low probability of being related to what you’re seeing here based on your description (most notably, the 1 second delay doesn’t seem to fit) BUT…I have seen some cases where Automatic Gain Control on SIP phones will gain up echo that was suppressed (as opposed to eliminated) elsewhere in the path.

Might be worth checking if your SIP phones to have an AGC setting that you can play with. But again, based on all you’ve written…it’s a long shot.

Thanks, sruffell, what do you think of this in light of that possible explanation: The SIP phones do not experience the echo when on speakerphone, only on the handset. I assumed that was because the echo was within the range that the speakerphone’s echo cancellation could deal with, but could it be related to the gain issue you suggested?

I’ve just been made aware of another issue with calls to the PSTN: Calls to cell phones are find, but calls to landlines are very quiet. Could that be related?

If you only hear the echo when using the handset and not the speakerphone then it’s almost certain to be some setting on the phone itself. While I’ve not personally experienced what you’re describing, it would make sense that automatic gain control might be something a phone would do more aggressively when on a handset so as to not amplify any acoustic echo it was trying to suppress.

Regarding the differences between cell phones and landlines, who is reporting that landlines are quiet, those on the SIP phones or those out on the PSTN?

The echo is only ever heard from the SIP phones connected to out Asterisk server, and the level and delay are the same no matter if we’re calling or being called, and no matter whether the other end is local, long distance, or a cell phone. In addition to that, if the other end is a landline, then the volume level of the person on the landline, as heard on the SIP phone, is very low, but it’s normal if the person on the other end is on a cell phone. The person on the far end always hears us great, better then when we were using POTS phones, and never hears any echo.

To further complicate things: To test, I quickly threw together another server and rolled it with the latest FreePBX distro (with Asterisk 1.8 ). The only setting that changed on the phones is the ip address of the Asterisk server. Now the echo is the same volume, but happens only a few milliseconds after you speak. This is with the same FXO card.

Here’s my thought number 287 on the subject: The card is, apparently, a TDM400P, not a 410, as it doesn’t have the connectors for the hardware echo cancellation module. But dahdi_genconf and dahdi_hardware pick it up as a 410. I notice that they use different drivers, but /etc/dahdi/system.conf and /etc/asterisk/chan_dahdi.conf are both configured for the 410. Could that be the problem?

Since you do not have hardware echo cancellation, why not using software echo cancellation ? There are options for software echo cancellation which can handle long delay.

Well, I tried changing chan_dahdi.conf to reflect the module for the 400P, but it made no difference. I did notice, however, that the one-second delay is only occurring when I’m connecting to the Asterisk server from home. In the office the echo is the same volume, but very close after you speak, only a few milliseconds delayed. It doesn’t matter if I’m using the SPA504g’s or a softphone, so I don’t think it has to do with the phone itself. Is it some strange failure in the system’s attempt to provide sideband?

valer77 - I’ve tried no EC, MG2, and OSLEC (just compiled it), nothing seems to make any difference. Could it be a bad card? Or bad FXO modules? Our office is having fits over this. I already changed the interface to the PSTN once, from a Linksys SPA400 to the TDM400P. The SPA400 didn’t have echo that I recall, but there’s no way I found to map it’s FXO ports to a DID, which I need to do to route calls properly.

This is very strange since even if there is a hardware failure, I would expect the problem to be at least partially resolve using the built-in echo cancellation. May I suggest you try an external echo cancellation software and see if it solves your problem.

What do you mean that dahdi_genconf and dahdi_hardware pick it up as a 410? Although, I don’t think this is going to be related to your problem. If you don’t hear the echo when using the speakerphone, I would be very surprised if it has anything to do with the hardware / drivers.

sruffell:

[**********@*********** asterisk]$ dahdi_hardware pci:0000:03:00.0 wctdm24xxp- d161:8005 Wildcard TDM410P and the comments at the start of /etc/dahdi/system.conf (generated by dahdi_genconf) end in

# Span 1: WCTDM/0 "Wildcard TDM410P" (MASTER)
The 410 uses the wctdm24xxp module, the 400 uses wctdm. It doesn’t seem to matter which I select in FreePBX, although I don’t see where that actually goes into any of the config files that FreePBX includes in chan_dahdi.conf.

And, interestingly, while the speakerphone on the SPA504G’s will cancel the echo out, the speakerphone on my Galaxy Nexus (LTE) does not, even though it works fine for cellular calls.

I think you have a 410 and not a 400. What is the output of lcpci -d d161:*?

Could you also give me the output of the following commands:

dmesg -c > /dev/null
/etc/init.d/dahdi stop
modprobe wctdm24xxp
dahdi_genconf system
dahdi_cfg -vvf
cat /proc/dahdi/1
dmesg

[code][*****@****** dahdi]# lspci -d d161:
03:00.0 Ethernet controller: Digium, Inc. Wildcard TDM410 4-port analog card (rev 11)
[******@****** dahdi]# dmesg -c > /dev/null
[******@****** dahdi]# /etc/init.d/dahdi stop
Unloading DAHDI hardware modules: done
[******@****** dahdi]# modprobe wctdm24xxp
[******@****** dahdi]# dahdi_genconf system
[******@****** dahdi]# dahdi_cfg -vvf
DAHDI Tools Version - 2.5.0.1

DAHDI Version: 2.5.0.2
Echo Canceller(s): HWEC
Configuration

Channel map:

Channel 01: FXS Kewlstart (Default) (Echo Canceler: oslec) (Slaves: 01)
Channel 02: FXS Kewlstart (Default) (Echo Canceler: oslec) (Slaves: 02)
Channel 03: FXS Kewlstart (Default) (Echo Canceler: oslec) (Slaves: 03)
Channel 04: FXS Kewlstart (Default) (Echo Canceler: oslec) (Slaves: 04)

4 channels to configure.

Setting echocan for channel 1 to oslec
Setting echocan for channel 2 to oslec
Setting echocan for channel 3 to oslec
Setting echocan for channel 4 to oslec
[******@****** dahdi]# cat /proc/dahdi/1
Span 1: WCTDM/0 “Wildcard TDM410P” (MASTER)

       1 WCTDM/0/0 FXSKS (EC: OSLEC - INACTIVE)
       2 WCTDM/0/1 FXSKS (EC: OSLEC - INACTIVE)
       3 WCTDM/0/2 FXSKS RED (EC: OSLEC - INACTIVE)
       4 WCTDM/0/3 FXSKS RED (EC: OSLEC - INACTIVE)

[******@****** dahdi]# dmesg
dahdi: Telephony Interface Registered on major 196
dahdi: Version: 2.5.0.2
PCI: Enabling device 0000:03:00.0 (0000 -> 0003)
ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 20 (level, low) -> IRQ 58
wctdm24xxp 0000:03:00.0: Port 1: Installed – AUTO FXO (USA mode)
wctdm24xxp 0000:03:00.0: Port 2: Installed – AUTO FXO (USA mode)
wctdm24xxp 0000:03:00.0: Port 3: Installed – AUTO FXO (USA mode)
wctdm24xxp 0000:03:00.0: Port 4: Installed – AUTO FXO (USA mode)
wctdm24xxp 0000:03:00.0: Found a Wildcard TDM: Wildcard TDM410P (0 BRI spans, 4 analog channels)
dahdi_echocan_oslec: Registered echo canceler ‘OSLEC’
[/code]

There must be some variant of the 410 that doesn’t have the connectors for the hardware echo canceller. Channels 3 and 4 are RED because there is no line connected to them.

Just to make sure…your card doesn’t look like (but with all red modules and without the long purple module but with the headers for them):

If not, I would contact Digium Technical support. I’m not aware of any of these cards shipping without support for the VPM module.


Well, it's fixed, and here's the scoop:  After looking into it and fooling with it (and trying not to cuss it) I've determined that the card is a generic that's compatible with the module for the 410, but does not have the headers for the hardware echo module.  It's physically laid out like a 400P.

The echo turns out to be pretty standard hybrid imbalance with a twist - since we're using Comcast Digital Voice, the hybrid is only about 20' of cable away.  This gives the MG2 echo canceler fits.  Also, it turns out [b]fxotune[/b] wasn't really able to calibrate, because the delay between sending a digit and the fast busy signal, in which you have silence, isn't a nice 18 seconds like with the PSTN.  It's about 3.5 seconds.

I was able to find a local miliwatt test number that went to silent after 8 seconds, so I used that for [b]fxotune[/b].  That helped alot, probably cut the level of the echo in half.  But there was still some echo until I switched to HPEC.  Best $20 (for two channels) I've spent on this whole system.

The whole reason I'm writing this is this:  If anyone else is having problems with where to put the echo cancelation commands when you're using FreePBX (the lines you'd normally put in [b]chan_dahdi.conf[/b] if you weren't using FreePBX), they have to go in [b]chan_dahdi_channels_custom.conf[/b].  When I tried to put them in [b]chan_dahdi_general_custom.conf[/b] they had no effect.  I couldn't find that information anywhere.  Maybe I wasn't searching well enough, or maybe I should have known that from the layout of the default Asterisk [b]chan_dahdi.conf[/b] file, but if anyone else runs into this problem, may google send them here.

It's possible that I didn't need HPEC and OSLEC would have worked, if I'd known how to turn it on in Asterisk with FreePBX.  But what worked for me was to use these settings in [b]chan_dahdi_channels_custom.conf[/b] (after building, registering, and enabling HPEC, of course):
[code]echocancel=1024
echocancelwhenbridged=no
echotraining=800
rxgain=3.0
txgain=0.0[/code]

I also manually edited [b]/etc/dahdi/system.conf[/b] to reflect:
[code]# Span 1: WCTDM/0 "Wildcard TDM410P" (MASTER) 
fxsks=1
echocanceller=hpec,1
fxsks=2
echocanceller=hpec,2
fxsks=3
echocanceller=kb1,3
fxsks=4
echocanceller=kb1,4[/code]
rather than having all four channels on HPEC - as it comes out of [b]dahdi_genconf[/b] - since I only purchased two HPEC channel licenses and the other two channels aren't used anyway.

And, if anyone is interested, running HPEC at 1024 taps with one active channel barely nudges a 2.8GHz Pentium D to 2% CPU usage.

Well, it’s fixed, and here’s the scoop: After looking into it and fooling with it (and trying not to cuss it) I’ve determined that the card is a generic that’s compatible with the module for the 410, but does not have the headers for the hardware echo module. It’s physically laid out like a 400P.

The echo turns out to be pretty standard hybrid imbalance with a twist - since we’re using Comcast Digital Voice, the hybrid is only about 20’ of cable away. This gives the MG2 echo canceler fits. Also, it turns out fxotune wasn’t really able to calibrate, because the delay between sending a digit and the fast busy signal, in which you have silence, isn’t a nice 18 seconds like with the PSTN. It’s about 3.5 seconds.

I was able to find a local miliwatt test number that went to silent after 8 seconds, so I used that for fxotune. That helped alot, probably cut the level of the echo in half. But there was still some echo until I switched to HPEC. Best $20 (for two channels) I’ve spent on this whole system.

The whole reason I’m writing this is this: If anyone else is having problems with where to put the echo cancelation commands when you’re using FreePBX (the lines you’d normally put in chan_dahdi.conf if you weren’t using FreePBX), they have to go in chan_dahdi_channels_custom.conf. When I tried to put them in chan_dahdi_general_custom.conf they had no effect. I couldn’t find that information anywhere. Maybe I wasn’t searching well enough, or maybe I should have known that from the layout of the default Asterisk chan_dahdi.conf file, but if anyone else runs into this problem, may google send them here.

It’s possible that I didn’t need HPEC and OSLEC would have worked, if I’d known how to turn it on in Asterisk with FreePBX. But what worked for me was to use these settings in chan_dahdi_channels_custom.conf (after building, registering, and enabling HPEC, of course):

echocancel=1024 echocancelwhenbridged=no echotraining=800 rxgain=3.0 txgain=0.0

I also manually edited /etc/dahdi/system.conf to reflect:

# Span 1: WCTDM/0 "Wildcard TDM410P" (MASTER) fxsks=1 echocanceller=hpec,1 fxsks=2 echocanceller=hpec,2 fxsks=3 echocanceller=kb1,3 fxsks=4 echocanceller=kb1,4
rather than having all four channels on HPEC - as it comes out of dahdi_genconf - since I only purchased two HPEC channel licenses and the other two channels aren’t used anyway.

And, if anyone is interested, running HPEC at 1024 taps with one active channel barely nudges a 2.8GHz Pentium D to 2% CPU usage.