Additional note on tcpdump log (a.k.a. watchtcpdump) - the last outbound message to flowroute appears at 15:16:23.659508. After that, the Asterisk PBX seems to just stop responding.
16:05:33.233233 - shows the invite for the test call I referenced (which returned “number not in service” from flowroute) without response.
Where are the packets being captured? Both sides are showing TTL=50, which either means that the TTL field is invalid, or the capture point is likely 14 hops from both side, which is quite a lot.
Asterisk isn’t seeing the OPTIONS requests that the system with which it is trying to register is using to test the connectivity. I would guess you are not port forwarding port 5060 on the router, and so inbound connectivity only exists for a short time after the router sees outbound activity. In that case, ideally, enable port forwarding on the router, for both 5060 and Asterisk RTP port range. You might be able keep the 5060 forwarding open by enabling qualify on Asterisk, so that it also sends OPTIONS at an interval less than that for the dynamic rule timeout on the firewall, but that might not get the media flowing quickly, for which you would need to configure port forwarding for the RTP range, and if you are doing that, you might as well do it for 5060.
Another possibility is that you are running something like fail2ban, which blocks senders who get repeated failures. That might explain why the OPTIONs don’t resume after re-registration, which I would expect if this were a dynamic rule timeout problem.
I suppose an Asterisk deadlock is possible, but it seems to be too selective to me.
I think “number not in service” is the result of flowroute not seeing responses to OPTIONS.
My connection detail:
Internet → router → Asterisk PBX
Router forwards 5060 tcp, udp; 10000-20000 udp to Asterisk PBX
Asterisk PBX firewall accepts 10000-20000 udp from any source (per flworoute instructions)
accepts 5060 only from flowroute IP addresses
tcpdump packet capture is taking place on the Asterisk PBX–5060 and RTP must be forwarded or the capture would be empty (of the specific messages).
I’m not running any 3rd-party apps (like fail2ban). I am using firewalld to manage the Asterisk PBX firewall.
So after reading your latest I’m looking at TTL and hops:
15:17:00.641426 IP (tos 0x0, ttl 50, id 30639, offset 0, flags [none], proto UDP (17), length 559)
ec2-34-226-36-33.compute-1.amazonaws.com.sip > myAsteriskPBX.sip: SIP, length: 531
OPTIONS sip:techprefix@my-ext-ip-addr:5060 SIP/2.0
Max-Forwards: 20
Record-Route: <sip:34.226.36.33;lr>
15:22:02.795914 IP (tos 0x0, ttl 50, id 13775, offset 0, flags [none], proto UDP (17), length 1225)
ec2-34-226-36-33.compute-1.amazonaws.com.sip > myAsteriskPBX.sip: SIP, length: 1197
INVITE sip:1myflowrouteDID@my-ext-ip-addr:5060 SIP/2.0
Record-Route: <sip:34.226.36.33;lr>
From: "PABST BRIAN" <sip:+17043455483@fl.gg>;tag=gK04116235
Max-Forwards: 66
From the above exerpt, TTL appears as 50. I notice in watchtcpip231222-13-29-08-cln.txt this is consistent for all messages to which Asterisk ignores (no response).
I picked the above to exerpt because you mentioned “more than 14 hops” and the capture shows 20 and 66 hops.
Clearly, messages are arriving as they are displayed. Perhaps Asterisk is ignoring them because they are passing a threshold? As you said, Asterisk isn’t seeing these messages–assuming I’m understanding.
Is there a way to tell Asterisk to respond to expired inbound messages (if that’s the issue)? Or, if this is the issue, what’s the normal process to resolution?
I know flowroute told me callers receive the “number not in service” recording because “your Asterisk server is not responding to us”.
I read your message a number of times–I hope I didn’t overlook antyhing in this response.
I really do appreciate you working through this with me. Thank you!
Messages show in “pjsip set logger on” as soon as they are received on the socket. If they don’t show up there then they aren’t coming off the socket, which would be outside of Asterisk. There’s nothing within Asterisk that controls that or would stop them from showing up in “pjsip set logger on”.
This has been a question I’ve had since I started packet capture with tcpdump:
If I see an invite in the tcpdump output but it doesn’t appear on the console (with pjsip set logger on) what is that? How do I troubleshoot it? That’s basically what I’m seeing–inbound messages displayed in tcpdump that Asterisk doesn’t “see”.
Here’s a comparison on a pass-fail from my logs:
Asterisk received and responded -
16:27:52.097336 IP (tos 0x0, ttl 52, id 8290, offset 0, flags [none], proto UDP (17), length 559)
ec2-34-226-36-34.compute-1.amazonaws.com.sip > p7-1456c.sip: SIP, length: 531
OPTIONS sip:techprefix@me-ext-ip-addr:5060 SIP/2.0
Max-Forwards: 20
Record-Route: <sip:34.226.36.34;lr>
Via: SIP/2.0/UDP 34.226.36.34:5060;branch=z9hG4bKd11f.7302746733cd5a6a5c326f3368f4f5f5.0
Via: SIP/2.0/UDP 52.40.141.109:5060;branch=z9hG4bK9612082
Route: <sip:34.226.36.34:5060;lr;received=sip:my-ext-ip-addr:5060>
From: sip:ping@invalid;tag=uloc-63881c7f-1f-6c061f56-0e1ec1f6-29be1638
To: sip:techprefix@my-ext-ip-addr:5060
Call-ID: 75694be1-8ebc17c9-5965ff8@0.0.0.0
CSeq: 1 OPTIONS
Content-Length: 0
Max-Forward: 10
tcpdump-only (didn't make it to the console) - no response -
15:50:11.107944 IP (tos 0x0, ttl 50, id 11642, offset 0, flags [none], proto UDP (17), length 559)
ec2-34-226-36-33.compute-1.amazonaws.com.sip > p7-1456c.sip: SIP, length: 531
OPTIONS sip:tecdhprefix@my-ext-ip-addr:5060 SIP/2.0
Max-Forwards: 20
Record-Route: <sip:34.226.36.33;lr>
Via: SIP/2.0/UDP 34.226.36.33:5060;branch=z9hG4bKd0c2.183dfe20ad4d06643cc2ac0139c48be5.0
Via: SIP/2.0/UDP 52.40.141.109:5060;branch=z9hG4bK7400876
Route: <sip:34.226.36.33:5060;lr;received=sip:my-ext-ip-addr:5060>
From: sip:ping@invalid;tag=uloc-63881c7f-1e-7eaf7a56-0e1ec1f6-7ea86628
To: sip:techprefix@my-ext-ip-addr:5060
Call-ID: 75694be1-d3b667b9-f3cfdf8@0.0.0.0
CSeq: 1 OPTIONS
Content-Length: 0
Max-Forward: 10
Line-by-line (a/b)
16:27:52.097336 IP (tos 0x0, ttl 52, id 8290, offset 0, flags [none], proto UDP (17), length 559)
15:50:11.107944 IP (tos 0x0, ttl 50, id 11642, offset 0, flags [none], proto UDP (17), length 559)
ec2-34-226-36-34.compute-1.amazonaws.com.sip > p7-1456c.sip: SIP, length: 531
ec2-34-226-36-33.compute-1.amazonaws.com.sip > p7-1456c.sip: SIP, length: 531
OPTIONS sip:techprefix@me-ext-ip-addr:5060 SIP/2.0
OPTIONS sip:techprefix@my-ext-ip-addr:5060 SIP/2.0
Max-Forwards: 20
Max-Forwards: 20
Record-Route: <sip:34.226.36.34;lr>
Record-Route: <sip:34.226.36.33;lr>
Via: SIP/2.0/UDP 34.226.36.34:5060;branch=z9hG4bKd11f.7302746733cd5a6a5c326f3368f4f5f5.0
Via: SIP/2.0/UDP 34.226.36.33:5060;branch=z9hG4bKd0c2.183dfe20ad4d06643cc2ac0139c48be5.0
Via: SIP/2.0/UDP 52.40.141.109:5060;branch=z9hG4bK9612082
Via: SIP/2.0/UDP 52.40.141.109:5060;branch=z9hG4bK7400876
Route: <sip:34.226.36.34:5060;lr;received=sip:my-ext-ip-addr:5060>
Route: <sip:34.226.36.33:5060;lr;received=sip:my-ext-ip-addr:5060>
From: sip:ping@invalid;tag=uloc-63881c7f-1f-6c061f56-0e1ec1f6-29be1638
From: sip:ping@invalid;tag=uloc-63881c7f-1e-7eaf7a56-0e1ec1f6-7ea86628
To: sip:techprefix@my-ext-ip-addr:5060
To: sip:techprefix@my-ext-ip-addr:5060
Call-ID: 75694be1-d3b667b9-f3cfdf8@0.0.0.0
Call-ID: 75694be1-d3b667b9-f3cfdf8@0.0.0.0
CSeq: 1 OPTIONS
CSeq: 1 OPTIONS
Content-Length: 0
Content-Length: 0
Max-Forward: 10
Max-Forward: 10
Either the INVITE was sent to an ip-address:port Asterisk is not listening to or maybe your firewall is dropping it. (tcpdump sees packets before the firewall.)
It typically means it is being blocked by the Linux firewall, although it could have the wrong port number. It could be a deadlock in Asterisk, but I think that would block everything on pjsip, including the REGISTER responses.
When the problem starts, Asterisk stops responding and the console shows nothing.
How do I detect and troubleshoot a deadlock? Yes, I’m searching now. 
A deadlock would mean there is a design error. That means the best you could do, unless you are prepared to debug and fix the source code, is avoid the situation that is triggering it.
Does it stop responding to ALL traffic? If the answer is no, then it’s not deadlocked. The output of “pjsip set logger on” occurs in the thread that receives all traffic.
As I noted, it is logging REGISTER responses, so the logging part of the thread is being reached after it stops seeing OPTIONS.
If it’s responding, it will have received the REGISTER requests so the thread that handles incoming traffic wouldn’t be deadlocked, which is where “pjsip set logger on” gets executed.
OK, during a fail I can see local network traffic. Based upon what I’ve learned here, that means it can’t be deadlock.
I )shudder( took down my firewall while observing the non-response issue and everything started working. Please note I have done this before and observed it faliing in the same way after time.
I hate leaving my firewall down but must eliminate the firewall as the issue.
I’ll post an update within the next 48-72 hours.
SIP messages are usually 5060 UDP (unless you’ve configured differently) so that’s the only ‘hole’ you need to poke in your firewall.
RTP (audio) is a separate UDP range, usually 10k → 20k. Opening that range is not much of a security risk, because nothing is listening to that range. Asterisk will only listen to the ports used in the active calls.
In general, ‘whitelisting’ known endpoints and blocking everything else is the best firewall policy.
Backing that up with fail2ban is also popular.
With tcpdump or the Asterisk console?
It sent the REGISTER request; it is receiving and logging the response. I think the implications are the same.
Sorry I wasn’t clear–yes, I see the local private network traffic on the console during the “fail.”
Here’s my firewall:
# firewall-cmd --zone=public --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: enp4s0
sources: 34.210.91.112/28 34.226.36.32/28 16.163.86.112/30 3.0.5.12/30 3.8.37.20/30 3.71.103.56/30 18.228.70.48/30 216.115.69.144/32
services: dhcpv6-client mdns ssh
ports: 5060/tcp 5060/udp 10000-20000/udp
protocols:
forward: yes
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
rule family="ipv4" port port="10000-20000" protocol="udp" accept
rule family="ipv4" source NOT address="192.168.1.0/24" drop
Flowroute details:
Identify: flowroute/flowroute
Match: 34.210.91.112/28 # US-West-OR(.sip.flowroute.com) 34.210.91.112-126
Match: 34.226.36.32/28 # US-East-VA 34.226.36.32-46
Match: 16.163.86.112/30 # AP-East-HK
Match: 3.0.5.12/30 # AP-SouthEast-SIN
Match: 3.8.37.20/30 # EU-West-LDN
Match: 3.71.103.56/30 # EU-Central-FRA
Match: 18.228.70.48/30 # SA-East-SP
Match: 216.115.69.144/32 # ????
as you can see, I have white-listed the flowroute IPs, opened 5060, and allowed any IP attempting 10000-20000/udp. I am planning on posting my flowroute white-list as general information but want to resolve this issue, 1st.
I am back from holidays and then after-holidays-recovery where I felt under the weather last week.
First, I’d like to thank sipvicious for the unrelenting assistance in testing my Asterisk PBX. I could sit on the edge of my seat for hours watching probe after probe attempt to identify a security issue.
Yes, I took down my firewall as previously noted.
I can confirm the firewall was part of the problem. For now, I will leave the firewall down until the primary issue is trapped and identified. The last time I saw the failure I just wasn’t feeling well enough to do anything. I will collect logs and post, here.
Thank you very much for your help in getting me to this point.
I have been unable to recreate this problem.
I’m going to attribute this to pebcak
Thank you very much for helping me work through this issue.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.