Hello there!
I am looking for some help with my issue related to Asterisk. I am using FreePBX distribution, but I think core of issue is related to pure Asterisk.
We have about 190 registered SIP phones and couple of trunks. FreePBX is storing quite lot of information in Asterisk’s database - about 22 per extension. We also added few features which are working with database also. Total number of lines it about 8000 and printing database writes a bit more than 600kB.
Now what happened to me - I wanted to check something in database, so I piped it into less and whole asterisk suddenly freezed. At first I didn’t mention this, but after I exited less command (after 6 min 30 sec), Asterisk wrote about 60k lines with Broken pipe (not just one process, 17 different pids):
[2014-10-14 12:21:10] ERROR[18784] utils.c: write() returned error: Broken pipe
[2014-10-14 12:21:10] ERROR[18784] utils.c: write() returned error: Broken pipe
[2014-10-14 12:21:10] ERROR[18784] utils.c: write() returned error: Broken pipe
Also many of channels has to be disconnected for lack of RTP (which means Asterisk didn’t accept any incoming packet during last 6 mins):
[2014-10-14 12:21:12] VERBOSE[19709] asterisk.c: -- Remote UNIX connection disconnected
[2014-10-14 12:21:12] WARNING[4155] chan_sip.c: Retransmission timeout reached on transmission 1997509719-5060-128@
BA.CEA.BAD.BAI for seqno 1271 (Critical Response) -- See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmiss
ions
Packet timed out after 171272ms with no response
[2014-10-14 12:21:12] NOTICE[4155] chan_sip.c: Disconnecting call 'SIP/1234-0005ab8c' for lack of RTP activity in 395 seconds
[2014-10-14 12:21:12] NOTICE[4155] chan_sip.c: Disconnecting call 'SIP/2345-0005ab82' for lack of RTP activity in 308 seconds
[2014-10-14 12:21:12] NOTICE[4155] chan_sip.c: Disconnecting call 'SIP/3456-0005aa45' for lack of RTP activity in 301 seconds
[2014-10-14 12:21:12] NOTICE[4155] chan_sip.c: Disconnecting call 'SIP/4657-0005ab65' for lack of RTP activity in 331 seconds
Also our monitoring system, which is executing command asterisk -rx "sip show peers" was timeouting. I also checked SIP dumps (tcpdump udp and port 5060) and found that asterisk didn’t send out any packet in specified time interval.
Core of this issue is due to large data Asterisk has to print from database (600kB) and I piped output (4kB kernel buffer) to less (64kB input buffer by default) and it looks like there are some global lock which freezes whole asterisk until “database show” command is completed.
I am using Asterisk 1.8.18.0 on CentOS 5.8.
Do you have any idea how to solve this problem? I didn’t searched changelogs if something like this is already fixed. Also I didn’t try to search source core, because I am not familiar with asterisk code architecture.
Thank you for comments and ideas