Working IAX2 tunnel just started showing UNREACHABLE

Hello All,

We have two anterisk servers connected with IAX2 tunnel that has been up for about 5 months
working fine. In the past week we have been seeing the local asterisk machine become
Status: UNREACHABLE. A restart fixed it but now everything we try to get it back fails.

So, went with this -
[root@newpbx ~]# asterisk -rx “module unload chan_iax2.so”
Unloaded chan_iax2.so

[root@newpbx ~]# asterisk -rx “module load chan_iax2.so”
Loaded chan_iax2.so

Still showing UNREACHABLE

[local]# asterisk -R -x ‘iax2 show peers’
Name/Username Host Mask Port Status Description
106-IAXpeer/101 10.153.0.3 (S) 255.255.255.255 4569 (T) UNKNOWN
1 iax2 peers [0 online, 1 offline, 0 unmonitored]

I can ping the machines and nmap is showing all necessary ports are open

[remote to local] linux-gnu 02:18:50 ~
$ ping 10.44.23.27
PING 10.44.23.27 (10.44.23.27) 56(84) bytes of data.
64 bytes from 10.44.23.27: icmp_seq=1 ttl=62 time=63.9 ms
64 bytes from 10.44.23.27: icmp_seq=2 ttl=62 time=62.4 ms
64 bytes from 10.44.23.27: icmp_seq=3 ttl=62 time=62.4 ms

and

[local to remote]# ping 10.153.0.3
PING 10.153.0.3 (10.153.0.3) 56(84) bytes of data.
From 10.44.23.1: icmp_seq=1 Redirect Host(New nexthop: 10.44.23.180)
64 bytes from 10.153.0.3: icmp_seq=1 ttl=62 time=64.3 ms
64 bytes from 10.153.0.3: icmp_seq=2 ttl=62 time=62.6 ms


from local to remote
nmap -sU -p4569 10.44.23.27
PORT STATE SERVICE
4569/udp open|filtered iax

from remote to local
nmap -sU -p4569 10.153.0.3
PORT STATE SERVICE
4569/udp open|filtered iax


[remote] linux-gnu 02:23:49 ~
$ asterisk -R -x ‘iax2 show peers’
Name/Username Host Mask Port Status Description
101-IAXpeer/106 10.44.23.27 (S) 255.255.255.255 4569 (T) OK (64 ms)

Now back to “unreachable” from our pbx, One more restart.

now

[local]# asterisk -R -x ‘iax2 show peers’
Name/Username Host Mask Port Status Description
106-IAXpeer/101 10.153.0.3 (S) 255.255.255.255 4569 (T) UNREACHABLE
1 iax2 peers [0 online, 1 offline, 0 unmonitored]

on the local machine – iax2 set debug on

Tx-Frame Retry[000] – OSeqno: 017 ISeqno: 146 Type: IAX Subclass: PING
Timestamp: 13042784ms SCall: 07737 DCall: 18822 [10.44.23.180:17827]
Rx-Frame Retry[ No] – OSeqno: 146 ISeqno: 017 Type: IAX Subclass: ACK
Timestamp: 13041132ms SCall: 18822 DCall: 07737 [10.44.23.180:17827]
Rx-Frame Retry[ No] – OSeqno: 146 ISeqno: 018 Type: IAX Subclass: PONG
Timestamp: 13042784ms SCall: 18822 DCall: 07737 [10.44.23.180:17827]
RR_JITTER : 0
RR_LOSS : 0
RR_PKTS : 1
RR_DELAY : 40
RR_DROPPED : 0
RR_OUTOFORDER : 0

Tx-Frame Retry[-01] – OSeqno: 018 ISeqno: 147 Type: IAX Subclass: ACK
Timestamp: 13042784ms SCall: 07737 DCall: 18822 [10.44.23.180:17827]
Rx-Frame Retry[ No] – OSeqno: 251 ISeqno: 250 Type: IAX Subclass: LAGRQ
Timestamp: 830015ms SCall: 26906 DCall: 15675 [10.44.23.180:17827]
Tx-Frame Retry[000] – OSeqno: 250 ISeqno: 252 Type: IAX Subclass: LAGRP
Timestamp: 830015ms SCall: 15675 DCall: 26906 [10.44.23.180:17827]
Rx-Frame Retry[ No] – OSeqno: 252 ISeqno: 251 Type: IAX Subclass: ACK
Timestamp: 830015ms SCall: 26906 DCall: 15675 [10.44.23.180:17827]
Tx-Frame Retry[000] – OSeqno: 251 ISeqno: 252 Type: IAX Subclass: LAGRQ
Timestamp: 830115ms SCall: 15675 DCall: 26906 [10.44.23.180:17827]
Rx-Frame Retry[ No] – OSeqno: 252 ISeqno: 252 Type: IAX Subclass: LAGRP
Timestamp: 830115ms SCall: 26906 DCall: 15675 [10.44.23.180:17827]
Tx-Frame Retry[-01] – OSeqno: 252 ISeqno: 253 Type: IAX Subclass: ACK
Timestamp: 830115ms SCall: 15675 DCall: 26906 [10.44.23.180:17827]
Tx-Frame Retry[000] – OSeqno: 120 ISeqno: 060 Type: IAX Subclass: PING
Timestamp: 25451726ms SCall: 06876 DCall: 30091 [10.44.23.180:17827]
Rx-Frame Retry[ No] – OSeqno: 060 ISeqno: 120 Type: IAX Subclass: PING
Timestamp: 25451893ms SCall: 30091 DCall: 06876 [10.44.23.180:17827]
Tx-Frame Retry[000] – OSeqno: 121 ISeqno: 061 Type: IAX Subclass: PONG
Timestamp: 25451893ms SCall: 06876 DCall: 30091 [10.44.23.180:17827]
RR_JITTER : 0
RR_LOSS : 0
RR_PKTS : 1
RR_DELAY : 40
RR_DROPPED : 0
RR_OUTOFORDER : 0

Rx-Frame Retry[ No] – OSeqno: 061 ISeqno: 121 Type: IAX Subclass: PONG
Timestamp: 25451726ms SCall: 30091 DCall: 06876 [10.44.23.180:17827]
RR_JITTER : 0
RR_LOSS : 0
RR_PKTS : 1
RR_DELAY : 40
RR_DROPPED : 0
RR_OUTOFORDER : 0

Tx-Frame Retry[-01] – OSeqno: 121 ISeqno: 062 Type: IAX Subclass: ACK
Timestamp: 25451726ms SCall: 06876 DCall: 30091 [10.44.23.180:17827]
Rx-Frame Retry[ No] – OSeqno: 061 ISeqno: 122 Type: IAX Subclass: ACK
Timestamp: 25451893ms SCall: 30091 DCall: 06876 [10.44.23.180:17827]
Rx-Frame Retry[ No] – OSeqno: 080 ISeqno: 024 Type: IAX Subclass: LAGRQ
Timestamp: 7859803ms SCall: 19424 DCall: 15084 [10.44.23.180:17827]
Tx-Frame Retry[000] – OSeqno: 024 ISeqno: 081 Type: IAX Subclass: LAGRP
Timestamp: 7859803ms SCall: 15084 DCall: 19424 [10.44.23.180:17827]
Tx-Frame Retry[000] – OSeqno: 025 ISeqno: 081 Type: IAX Subclass: LAGRQ
Timestamp: 7860996ms SCall: 15084 DCall: 19424 [10.44.23.180:17827]
Rx-Frame Retry[ No] – OSeqno: 081 ISeqno: 025 Type: IAX Subclass: ACK
Timestamp: 7859803ms SCall: 19424 DCall: 15084 [10.44.23.180:17827]
Rx-Frame Retry[ No] – OSeqno: 081 ISeqno: 026 Type: IAX Subclass: LAGRP
Timestamp: 7860996ms SCall: 19424 DCall: 15084 [10.44.23.180:17827]
Tx-Frame Retry[-01] – OSeqno: 026 ISeqno: 082 Type: IAX Subclass: ACK
Timestamp: 7860996ms SCall: 15084 DCall: 19424 [10.44.23.180:17827]
Rx-Frame Retry[ No] – OSeqno: 212 ISeqno: 174 Type: IAX Subclass: LAGRQ
Timestamp: 2309779ms SCall: 22385 DCall: 06390 [10.44.23.180:17827]
Tx-Frame Retry[000] – OSeqno: 174 ISeqno: 213 Type: IAX Subclass: LAGRP
Timestamp: 2309779ms SCall: 06390 DCall: 22385 [10.44.23.180:17827]
Rx-Frame Retry[ No] – OSeqno: 213 ISeqno: 174 Type: IAX Subclass: PING
Timestamp: 2309814ms SCall: 22385 DCall: 06390 [10.44.23.180:17827]
Tx-Frame Retry[000] – OSeqno: 175 ISeqno: 214 Type: IAX Subclass: PONG
Timestamp: 2309814ms SCall: 06390 DCall: 22385 [10.44.23.180:17827]
RR_JITTER : 0
RR_LOSS : 0
RR_PKTS : 1
RR_DELAY : 40
RR_DROPPED : 0
RR_OUTOFORDER : 0

Tx-Frame Retry[000] – OSeqno: 176 ISeqno: 214 Type: IAX Subclass: LAGRQ
Timestamp: 2310228ms SCall: 06390 DCall: 22385 [10.44.23.180:17827]
Rx-Frame Retry[ No] – OSeqno: 213 ISeqno: 175 Type: IAX Subclass: ACK
Timestamp: 2309779ms SCall: 22385 DCall: 06390 [10.44.23.180:17827]
Tx-Frame Retry[000] – OSeqno: 177 ISeqno: 214 Type: IAX Subclass: PING
Timestamp: 2310288ms SCall: 06390 DCall: 22385 [10.44.23.180:17827]
Rx-Frame Retry[ No] – OSeqno: 214 ISeqno: 176 Type: IAX Subclass: ACK
Timestamp: 2309814ms SCall: 22385 DCall: 06390 [10.44.23.180:17827]
Rx-Frame Retry[ No] – OSeqno: 214 ISeqno: 177 Type: IAX Subclass: LAGRP
Timestamp: 2310228ms SCall: 22385 DCall: 06390 [10.44.23.180:17827]
Tx-Frame Retry[-01] – OSeqno: 177 ISeqno: 215 Type: IAX Subclass: ACK
Timestamp: 2310228ms SCall: 06390 DCall: 22385 [10.44.23.180:17827]
Rx-Frame Retry[ No] – OSeqno: 215 ISeqno: 178 Type: IAX Subclass: PONG
Timestamp: 2310288ms SCall: 22385 DCall: 06390 [10.44.23.180:17827]
RR_JITTER : 0
RR_LOSS : 0
RR_PKTS : 1
RR_DELAY : 40
RR_DROPPED : 0
RR_OUTOFORDER : 0

Remote machine is working fine taking and receiving calls we just cannot route through the tunnel. Local machine is also taking and receiving calls.

This just started happening and there have been no changes to our network

Local asterisk is Asterisk 11.21.2,

Remote asterisk is Asterisk 13.20.0

Both machines are Debian Stretch

IAX2 tunnel config on the local machine

[106-IAXpeer]
host=10.153.0.3
username=101-IAXuser
secret=
type=friend
qualify=yes
trunk=yes
requirecalltoken=no
context=from-internal

[106-IAXpeer]
secret=
type=user
context=from-internal

IAX2 tunnel config on the remote machine

[101-IAXpeer]
host=10.44.23.27
username=106-IAXuser
secret=
type=friend
qualify=yes
trunk=yes
requirecalltoken=no
context=from-internal

[101-IAXuser]
secret =
type=user
context=from-internal


We had
qualifysmoothing=yes
qualifyfreqnotok=30000
qualifyfreqok=120000

on both machines but took it out while debugging.

Any help will be greatly appreciated, thanks for your time