PRI got event: Alarm (4) on D-channel of span 5

Hi, I hope someone can assist with some more information on the log output below. We have been experiencing dropped incoming calls and after reviewing the logs I noticed that Red Alarm warnings and also peers becoming unreachable.

Would this indicate a possible local lan network issue ? I am not seeing that the network card on the asterisk server is flapping up and down but do see dropped packets on the network interface.

      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      RX packets:342956799 errors:0 dropped:960999 overruns:0 frame:0
      TX packets:295334936 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000 

The cards we currently have installed are:

ISDN controller: Digium, Inc. Wildcard B410 quad-BRI card (rev 01)
Communication controller: Digium, Inc. Wildcard TE220 dual-span T1/E1/J1 card 3.3V (PCI-Express) (5th gen) (rev 02)

[Nov 10 10:44:03] NOTICE[3040] chan_dahdi.c: PRI got event: Alarm (4) on D-channel of span 5
[Nov 10 10:44:03] WARNING[3040] sig_pri.c: Span 5: D-channel is down!
[Nov 10 10:44:03] WARNING[3041] chan_dahdi.c: Detected alarm on channel 30: Red Alarm
[Nov 10 10:44:03] WARNING[3041] chan_dahdi.c: Detected alarm on channel 31: Red Alarm
[Nov 10 10:44:03] WARNING[3041] chan_dahdi.c: Detected alarm on channel 32: Red Alarm
[Nov 10 10:44:03] WARNING[3041] chan_dahdi.c: Detected alarm on channel 33: Red Alarm
[Nov 10 10:44:03] WARNING[3041] chan_dahdi.c: Detected alarm on channel 34: Red Alarm
[Nov 10 10:44:03] WARNING[3041] chan_dahdi.c: Detected alarm on channel 35: Red Alarm
[Nov 10 10:44:03] WARNING[3041] chan_dahdi.c: Detected alarm on channel 36: Red Alarm
[Nov 10 10:44:03] WARNING[3041] chan_dahdi.c: Detected alarm on channel 37: Red Alarm
[Nov 10 10:44:03] WARNING[3041] chan_dahdi.c: Detected alarm on channel 38: Red Alarm
[Nov 10 10:44:03] WARNING[3041] chan_dahdi.c: Detected alarm on channel 39: Red Alarm
[Nov 10 10:44:03] WARNING[3041] chan_dahdi.c: Detected alarm on channel 40: Red Alarm
[Nov 10 10:44:03] WARNING[3041] chan_dahdi.c: Detected alarm on channel 41: Red Alarm
[Nov 10 10:44:03] WARNING[3041] chan_dahdi.c: Detected alarm on channel 42: Red Alarm
[Nov 10 10:44:03] WARNING[3041] chan_dahdi.c: Detected alarm on channel 43: Red Alarm
[Nov 10 10:44:03] VERBOSE[2877] asterisk.c: – Remote UNIX connection
[Nov 10 10:44:03] VERBOSE[10373] asterisk.c: – Remote UNIX connection disconnected
[Nov 10 10:44:09] NOTICE[3042] chan_sip.c: Peer ‘2107’ is now UNREACHABLE! Last qualify: 23
[Nov 10 10:44:10] NOTICE[3042] chan_sip.c: Peer ‘2125’ is now UNREACHABLE! Last qualify: 27
[Nov 10 10:44:10] NOTICE[3042] chan_sip.c: Peer ‘2132’ is now UNREACHABLE! Last qualify: 23
[Nov 10 10:44:11] NOTICE[3042] chan_sip.c: Peer ‘2157’ is now UNREACHABLE! Last qualify: 23
[Nov 10 10:44:14] NOTICE[3042] chan_sip.c: Peer ‘2179’ is now UNREACHABLE! Last qualify: 23
[Nov 10 10:44:15] NOTICE[3042] chan_sip.c: Peer ‘2166’ is now UNREACHABLE! Last qualify: 23
[Nov 10 10:44:15] NOTICE[3042] chan_sip.c: Peer ‘2167’ is now UNREACHABLE! Last qualify: 23
[Nov 10 10:44:15] NOTICE[3042] chan_sip.c: Peer ‘2120’ is now UNREACHABLE! Last qualify: 23
[Nov 10 10:44:16] NOTICE[3042] chan_sip.c: Peer ‘2198’ is now UNREACHABLE! Last qualify: 23

Then the following later in the logs

Nov 10 10:46:15] NOTICE[3042] chan_sip.c: Peer ‘2164’ is now Reachable. (8ms / 2000ms)
[Nov 10 10:46:16] NOTICE[3042] chan_sip.c: Peer ‘2137’ is now Reachable. (8ms / 2000ms)
[Nov 10 10:46:17] NOTICE[3042] chan_sip.c: Peer ‘2113’ is now Reachable. (307ms / 2000ms)
[Nov 10 10:46:17] NOTICE[3042] chan_sip.c: Peer ‘2214’ is now Reachable. (7ms / 2000ms)
[Nov 10 10:46:18] NOTICE[3042] chan_sip.c: Peer ‘2194’ is now Reachable. (13ms / 2000ms)
[Nov 10 10:46:19] NOTICE[3042] chan_sip.c: Peer ‘2138’ is now Reachable. (15ms / 2000ms)
[Nov 10 10:46:19] NOTICE[3042] chan_sip.c: Peer ‘2124’ is now Reachable. (15ms / 2000ms)
[Nov 10 10:46:40] NOTICE[3042] chan_sip.c: Peer ‘2112’ is now UNREACHABLE! Last qualify: 3021
[Nov 10 10:46:56] NOTICE[3042] chan_sip.c: Peer ‘2156’ is now Reachable. (22ms / 2000ms)
[Nov 10 10:46:57] NOTICE[3042] chan_sip.c: Peer ‘2189’ is now Reachable. (22ms / 2000ms)
[Nov 10 10:46:57] NOTICE[3042] chan_sip.c: Peer ‘2132’ is now UNREACHABLE! Last qualify: 3020
[Nov 10 10:46:58] NOTICE[3042] chan_sip.c: Peer ‘2179’ is now Reachable. (24ms / 2000ms)
[Nov 10 10:46:59] NOTICE[3042] chan_sip.c: Peer ‘2166’ is now Reachable. (20ms / 2000ms)
[Nov 10 10:46:59] NOTICE[3042] chan_sip.c: Peer ‘2167’ is now Reachable. (21ms / 2000ms)

Any assistance would be much appreciated.

You seem to have at least two different things going wrong here.

Red alarms generally indicate hardware problems on the T1/E1 connection.

I think dropped packets on the LAN interface indicates that the system is overloaded and has run out of buffers because it cannot process them fast enough.

Although 1,000,000 is a lot, it is only 0.3% of the traffic, which is probably acceptable if evenly distributed. If it it is concentrated at one time, it could explain the SIP unchreachables, otherwise you have a third problem, of a general LAN overload.

Whilst red alarms should be addressed early, I’m not sure that the D channel error is a secondary from those. It may represent yet another unrelated failure.

Hi

Are you running any batch jobs on the server ? or heavy mysql load. We have seen peers going unreachable on heavily loaded systems with high disk io.

atop my show something as to whats going on

Ian

Many thanks. I am not aware of any batch jobs but will add some additional monitoring to the server for disk IO usage.

Many thanks. With regards to possible hardware problems would this most likely be on the setup of the PRI card. Is there any additional logging that can be enabled in order to try and determine the possible causes ?