Corrupt .gsm files from MixMonitor?

Hi guys

I’m still using Asterisk 1.8.11.0 across 14 branches of our company. Old, and up to now extremely reliable - been running about three years with no problems. About 80 000 .gsm files per day are Mixmonitor recorded accross the company into .gsm format.

I recently started encountering a situation at one branch where the .gsm files produced by MixMonitor are corrupted and truncated. E. g. you can be talking to a caller, and the eventual duration of the call is five minutes - but the mixmonitor recording length is only 3 minutes 15 seconds long (for example).

The recording transmutes before this 3:15 cutoff point in a weird hiss of audio that sounds like background / white noise on an FM radio that is not properly tuned. The white noise ends a few seconds later, even though the conversation continues in reality. E. g. the audible conversation turns to white noise, then ends, long before the actual call ends.

Anybody encountered this before? I’ve got more or less identical hardware at all the sites, all using 1.8.11.0 on Centos 6.5 on the 2.6.32-431.23.3.el6.x86_64 kernel.

At the site with the problem, it does not happen to EVERY recording, but about 75% have the issue. My upstream VOIP trunk provider uses G729 with 20ms payload at all the sites.

None of the other sites exhibit the issue, except the problem site.

I have already replaced the particular server at the problem branch with a completely physically new unit, problem still persists. The previous server was running for seven months beforehand without exhibiting this problem. We have not changed the dialplan or any hardware on this machine in that time.

No relevant errors are apparent in the CLI or in the /var/log/asterisk/messages log file. The server’s /var/log/messages does not show any errors, no kernel panic logs, etc. Dmesg does not show anything strange either.

The system has a load average of 0.75 according to top, iotop gives a mean disk write utilisation of about 90 kb/s constantly. The system has 8GB of physical RAM which is fully used (7609 MB allocated out of 7748 total according to free), with nothing allocated on swap.

All my calls are made and recorded by passing a telephone number and channel name to

[macro-STDOUT]
;${ARG1} = channel
;${ARG2} = number
exten=>s,1,Macro(WAITCHANNEL)
exten=>s,n,Macro(WAITCDR)
exten=>s,n,Macro(VCRECORD,${MACRO_CONTEXT}X${CALLERID(num)}ACC${CDR(accountcode)},${ARG2})
exten=>s,n(dodial),Macro(VCCALLOUT,${ARG1},${ARG2},${ARG3},${ARG4})
exten=>s,n,NoOp(Setting Userfield after call completion)
exten=>s,n,Set(CDR(userfield)=${MIXMONITOR_FILENAME})
;exten=>s,1(dodial),Dial(${ARG1}/${ARG2},120,TL(300000:60000:30000))
exten=>s,dodial+101,Busy()
exten=>s,n,Hangup()
exten=>s,n,MacroExit

[macro-WAITCHANNEL]
exten=>s,1,Set(trycounter=0)
exten=>s,n,Set(trycounter=${MATH(${trycounter}+1,int)}) ; increment counter
exten=>s,n,GoToIf($[${trycounter} < 5]?testval:showval) ;only try 5 times
exten=>s,n(testval),GoToIf($["${CALLERID(num)}"=""]?2:showval)
exten=>s,n(showval),NoOp(CALLER ID ${CALLERID(num)})
exten=>s,n,MacroExit

[macro-WAITCDR]
exten=>s,1,Set(trycounter=0)
exten=>s,n,Set(trycounter=${MATH(${trycounter}+1,int)})
exten=>s,n,GoToIf($[${trycounter} < 5]?testacc:showacc) ;only try 5 times
exten=>s,n(testacc),GoToIf($["${CDR(accountcode)}"=""]?2:showacc)
exten=>s,n(showacc),NoOp(Account ${CDR(accountcode)})
exten=>s,n,MacroExit

[macro-VCRECORD] ; MACRO To setup Recording
;${ARG1} Description To Save
;${ARG2} Dialed Number
exten=>s,1,NoOp(Start of MixMonitor recording)
exten=>s,n,GotoIf($["${CDR(accountcode)}"=""]?setcdr:setchan)
exten=>s,n(setcdr),Set(${CDR(accountcode)}=${chanacc})
exten=>s,n(setchan),Set(${__chanacc}=${CDR(accountcode)})
exten=>s,n(startrec),Set(recDir=${STRFTIME(${EPOCH},,%y%m/%d)})
exten=>s,n,Set(recFile=${recDir}/${STRFTIME(${EPOCH},,%y%m%d%H%M%S)}D${ARG1}N${ARG2}ID${UNIQUEID}.gsm)
exten=>s,n,GoToIf($["${CDR(accountcode)}" = ""]?makedir:setacc)
exten=>s,n(setacc),set(recFile=${recDir}/${CDR(accountcode)}.gsm)
exten=>s,n(makedir),System(/bin/mkdir -p /var/spool/asterisk/monitor/${recDir})
exten=>s,n,MixMonitor(${recFile},a)
exten=>s,n,Set(AUDIOHOOK_INHERIT(MixMonitor)=yes)
exten=>s,n,Set(CDR(userfield)=${MIXMONITOR_FILENAME})
exten=>s,n,Set(__chanrecording=${recFile})
exten=>s,n(finrec),NoOp(Recording to ${MIXMONITOR_FILENAME})
exten=>s,n,UserEvent(RecordingToFile,Uniqueid: ${UNIQUEID},Channel: ${Channel},Filename: ${MIXMONITOR_FILENAME})
exten=>s,n,MacroExit

[macro-VCCALLOUT] ;macro to dial numbers
; ${ARG1} Channel To Use
; ${ARG2} Number To Dial
; ${ARG3} FailOver Channel
;${ARG4} FailOverNumber
exten=>s,1,set(accsplit=${CUT(CDR(accountcode),-,1)})
exten=>s,n,Set(timeLimit=${DB(AVIS/AGENT/LIMIT/${accsplit})})
exten=>s,n,NoOp(Passed var is ${CallLimit})
exten=>s,n,Set(timeLimit=${CallLimit})
exten=>s,n,NoOp(Passed ${ARG1} ${ARG2} ${ARG3} ${ARG4})
exten=>s,n,Set(callcycle=${DB(AVIS/CYCLE)})
exten=>s,n,NoOp(Changed passed to: ${ARG1} ${ARG2} ${ARG3} ${ARG4})
exten=>s,n,Set(DB(AVIS/CYCLE)=)
exten=>s,n,Goto(setchan)
exten=>s,n(noswap),Set(DB(AVIS/CYCLE)=1)
exten=>s,n(setchan),Set(chantouse=${ARG1})
exten=>s,n,Set(numtodial=${ARG2})
exten=>s,n,Goto(makecall)
exten=>s,n(failover),GoToIf($["${ARG3}" = ""]?endcall:dofailover)
exten=>s,n(dofailover),Set(chantouse=${ARG3})
exten=>s,n,Set(numtodial=${ARG4})
exten=>s,n,Set(ARG3="")
exten=>s,n,NoOp(Attempting Call On ${chantouse} ${numtodial} ARG3 is now ${ARG3})
exten=>s,n(makecall),GotoIf($["${timeLimit}" = ""]?dialNoLimit:dialLimit)
exten=>s,n(dialNoLimit),Dial(${chantouse}/${numtodial},120,jTL(300000:60000:30000))
exten=>s,n,NoOp(Dial Status: ${DIALSTATUS})
exten=>s,n,GoTo(s-${DIALSTATUS},1)
exten=>s,n(dialLimit),Dial(${chantouse}/${numtodial},120,jTL(${MATH(${timelimit}*60000,int)}:60000:30000))
exten=>s,n,NoOp(Dial Status: ${DIALSTATUS})
exten=>s,n,GoTo(s-${DIALSTATUS},1)
exten=>s,dialNoLimit+101,Goto(s-${DIALSTATUS},1)
exten=>s,dialLimit+101,Goto(s-${DIALSTATUS},1)
exten=>s,n(endcall),busy()
exten=>s,n,NoOp(Call Completed - setting userfield to recording)
exten=>s,n,Set(CDR(userfield)=${MIXMONITOR_FILENAME})
exten=>s,n,Hangup()
exten=>s,n,MacroExit

Any ideas or pointers as to why I’m getting corrupt .gsm Mixmonitor recordings?

Thanks

It could be that the SIP provider is sending audio to that location using a different media gateway than the rest of your boxes. Running briefly “rtp set debug on” – then very quickly “rtp set debug off” – would show that gateway’s IP address.

Have you tried another file format like wav instead of gsm?

Hi

Thanks for the reply!

Turns out my upstream provider had me connected to an experimental server on their side that they were using for random number generation testing.

It appears they increased the G729 payload to 70ms instead of the standard 20ms that they always use on their regular servers.

As far as I can determine this was what was causing the corrupt .gsm recordings. I got the IP of one of their regular production servers, edited sip.conf to register to that server instead of the experimental one, and the box is now running fine.

I did replace the entire server as well, but I think the problem was somehow caused by their testing server to which I was unwittingly SIP subscribed to…

I learned something again - it appears that if a trunk you’re sending and receiving G729 encoded voice / RTP on has a widely differing payload size from what Asterisk 1.8.11.0 is set to expect (I explicitly specify allow=G729:20 in my sip.conf for this connection) the audio itself may be fine bi-directionally but .gsm mixmonitored recordings of same audio will be corrupt…?

Anyway, thanks for taking the time to respond, much obliged!

Hi Guys

Erm… turns out this is NOT solved. Still getting lots of corrupt .gsm files from my 1.8.11.0 installation.

Relevant CLI error messages include

[Dec 9 17:19:08] WARNING[18366]: format_gsm.c:65 gsm_read: Short read (14) (Resource temporarily unavailable)!
[Dec 9 17:20:29] WARNING[10250]: format_gsm.c:65 gsm_read: Short read (15) (Resource temporarily unavailable)!

and

[Dec 8 11:44:07] WARNING[23787] file.c: Failed to write frame
[Dec 8 11:44:46] WARNING[26991] file.c: Failed to write frame

If i try to listen to a .gsm file I get these error messages and playback turns into white noise:

[Dec 9 17:45:28] WARNING[5901]: translate.c:181 framein: gsmtolin did not update samples 0
[Dec 9 17:45:28] WARNING[5901]: codec_gsm.c:104 gsmtolin_framein: Invalid GSM data (1)
[Dec 9 17:45:28] WARNING[5901]: translate.c:181 framein: gsmtolin did not update samples 0
[Dec 9 17:45:28] WARNING[5901]: codec_gsm.c:104 gsmtolin_framein: Invalid GSM data (1)
[Dec 9 17:45:28] WARNING[5901]: translate.c:181 framein: gsmtolin did not update samples 0

With this new / more information I’ll try another post on this topic and see if somebody can assist.