Asterisk - Source not in ICE active candidate list

Asterisk Asterisk WebRTC
1

Hi All,

I’ve recently setup a turn server and still carrying out some testing on it. My infrastructure is all of my webRTC browser phones are ideally going to be external to my network with Asterisk behind a strict NAT allowing no external connections into it.

I’ll then have my turn server sitting on my network and it will be locked down to only talk to my Asterisk server and my external phone browsers can hit my public facing turn server which should relay the traffic into Asterisk.

The issue I’ve got is that when my browser phones makes a call and it generates the ICE candidate list it has the public IP of my turn server but because the turn server and asterisk sit on the same network when the traffic is relayed into Asterisk it sees the source IP as the internal IP of the turn server which isn’t in the ICE candidate list and drops the connection with the above error.

I’m really looking for a way for me to be able to have this setup but Asterisk allow the connection from the internal IP address of turn server when only the external IP address is in the candidate list, I know I’m probably just missing a setting in the pjsip / rtp conf files or something easy like that.

I’m using Asterisk 20.9.2.

Any help on this is much appreciated.

Just to add a bit more information to this below are the errors that I can see in the Asterisk CLI showing the internal IP of my turn server but as stated the candidate list obviously only has the external IP as a candidate

[Jan 28 16:12:11] WARNING[1392331][C-0001b9ca] res_rtp_asterisk.c: 1769616723.498619: DTLS packet from 10.20.30.3:61930 dropped. Source not in ICE active candidate list.

2

Are you sure you need a TURN server at all? If you can control/access the router, simple destination nating would do the job, so long as you describe the NAT layout to Asterisk.

Running your Asterisk box “behind a strict NAT” and having the phones “external to my network”… is pretty much exactly the same as saying “You have a hosted PBX” - except you are the host. Asterisk has no problem in this configuration even with WSS for signaling and UDP for media.

The ice candidates are more about getting the correct live IP address to populate the SDP that the WebRTC Client sends to Asterisk. Asterisk should already know its own live IP address (provided to you by your ISP)

3

One of the apprehensions was to have Asterisk directly contactable by external phones, I believe this was attempted in the past and not implemented correctly which basically left the door open for hackers. The idea was to keep the PBX hidden in the network and only have the turn server exposed externally and use this has an extra layer of security so that, for RTP, they would have to negotiate turn and then get through to Asterisk but in hindsight this might have been a fool’s errand

4

Ok, I get it - you reduce your attack surface - if the TURN server is on a static IP address, then you can set to only allow inbound media traffic from that one IP, right? fair enough - however the effort may be in the wrong place.

Since you will need to have the SIP messages flow in from the outside you are probably opening the firewall to at least the HTTPS port - this is the more serious part. And worth more consideration than media.

Media is over UDP, and there is little to nothing that any hacker could do over UPD ports between 10,000 to 65,000. (Other than flood them). Also you don’t actually need to open the UDP ports for media either - Routers generally don’t allow inbound connections by default anyway, and if you use Asterisk right - you can get it to send and receive media on the same port… meaning the outflow of audio packets makes an inbound port for your webrtc media. (rtp_symmetric = yes). Although, there can be a standoff there, if two webrtc agents are talking to each other - in this case use (rtp_keepalive = 2).

Going back to signalling - The easiest is just to make that HTTPS port high, and arbitrary… (nothing under 1024). Generally you can configure the WebSocket int he WebRTC phone to anything you want.
The best, and probably the only way to fully protect yourself here is to issue your own CA Certificate, and install it onto your own WebRTC clients - then generate a server certificate, and host it on the HTTPS port. Make sure to validate the certificate and voilà! only your own WebRTC clients can connect to your Asterisk box using HTTPS (WSS://). If you have a public service and random WebRTC clients connecting… sorry you just have to deal with it.

Disclaimer: I’m not a security expert - seek advice.

Homework:

Also read this:

5

From a media injection perspective with WebRTC the media port is actually more restricted, so firewalling it is a bit of a shrug. The ICE negotiation requires a username/password exchanged in the signaling layer in SDP, packets won’t be accepted if they are not part of the ICE candidates, plus DTLS-SRTP is also negotiated which involves fingerprints also exchanged in the signaling layer in SDP. If none of this is correct the packets get dropped by Asterisk on ingress.

6

Yeah so reducing the attack surface was exactly where we were going. For the SIP messages these flow through our website into IIS which then reverse proxies the data into the asterisk box which means we don’t have any direct connection into Asterisk and gives us the added security of this only being accessible by users logged into our system with their own unique extension password etc all that normal stuff.

So it seems to make sense to keep the above reverse proxy we have but then open media ports as again there isn’t much anyone can do with these ports and have added security on these rather than faffing about with a turn server which seems quite overkill for what we’re trying to achieve?

7

Hi Jcolp,

I managed to get past this issue by setting an iptables rule on my turn server so that it overwrites the source of packets from the internal IP to the external IP which is in the ice candidate list and this stopped the error that I had initially but I now get the below:

[Feb 25 14:21:40] WARNING[1121245][C-00000001]: res_rtp_asterisk.c:3315 __rtp_recvfrom: 1772029295.1: DTLS packet from 127.0.0.1:13978 dropped. Source not in ICE active candidate list.

I can’t seem to figure out why Asterisk would be sending itself packets on the loopback IP address?

8

Asterisk won’t normally, unless you’ve done something to cause it or something else has happened. It’s most likely something specific to your setup, so you’d need to investigate further and look at how things are actually flowing.

9

Could this be an issue with the pjroject reinjection mechanism.

It’s my understanding this should overwriting the loopback IP address with the original source IP address for the DTLS candidate check but it seems like the DTLS candidate check fires before the loopback address gets substituted which is causing it to check the loopback IP address vs the candidate list and causing the mismatch

10

I have no idea what you mean by pjproject reinjection mechanism. Within Asterisk the logic works based on the source IP address + port for the DTLS packet as received when reading in the packet, against the ICE candidate list as communicated over SDP and any peer reflexive candidates learned.

11

When a call comes in via the TURN relay the packet arrives at Asterisk correctly from the external IP (confirmed via tcpdump on eth0) but I can then see on a tcpdump of the lo interface that the packet appears as 127.0.0.1:port > 127.0.0.1:port on the loopback. The socket is owned by the Asterisk process itself. This is what __rtp_recvfrom is reading with sa set to 127.0.0.1 which then fails the ICE candidate check. What is causing Asterisk to be sending this packet to itself on loopback and should the DTLS candidate check be accounting for this?

12

I don’t know, I’ve never seen it or heard anyone experience such issues. Are you configuring TURN within Asterisk itself? That could be why, and you’d be maybe the third person I’ve ever heard using it.

13

Within rtp.conf I have turnaddr, turnusername and turnpassword configured as Asterisk needs its own TURN relay candidate to advertise in the SDP. Without it Asterisk only offers a host candidate (Internal IP) and a srflx candidate (Public IP) which the browser can’t reach directly as there are no open ports on Asterisk from the outside. The browser on the other hand only has a working relay candidate via the TURN server so without Asterisk also having a relay candidate ICE can’t find a working path and the DTLS handshake times out.

14

That’s probably why then. You’re in seldom used uncharted for years territory. I’ve got nothing else to add there.

15

It looks to me as though your Open Media Ports vs Turn Server is very relevant to this, and the threads maybe ought to be combined, as it gives your reasoning for the unusual use of TURN.

16

Thanks David. I should’ve added some of this context as well so that it makes a bit more sense as to what I am trying to achieve.

I’m pretty stumped on how best to move forward now after these challenges, might just need to go down the route of scrapping the turn server and going forward with the open media ports

17

If you would like, Siperb is already a proxy, designed to work with Asterisk. If you opt for our transcoding service, we then send / receive media only over a single IP. It’s kind of doing exactly what you are setting up already.

18

After a bit of digging through the DTLS handshake check code I managed to get a solution for this. In res/res_rtp_asterisk.c within the __rtp_recvfrom() function, when TURN is configured in rtp.conf, pjproject processes incoming TURN relay packets and re-injects the decapsulated payload internally via the loopback interface (127.0.0.1). The problem is that the DTLS candidate source check fires before the loopback address substitution code runs, so by the time Asterisk checks whether the source address is in the ICE active candidate list it sees 127.0.0.1 rather than the real remote address, fails the check, and drops the packet.

The fix was to add a loopback substitution block immediately after the ast_debug_dtls line and before the #ifdef HAVE_PJPROJECT ICE candidate check. This checks whether the incoming source address matches the pjproject loopback re-injection address and if so replaces it with the real remote address before the candidate check runs. The candidate check then sees the actual TURN relay IP, finds it in the candidate list, and allows the packet through so the DTLS handshake can complete normally.

This is the original code:

ast_debug_dtls(3, "(%p) DTLS - __rtp_recvfrom rtp=%p - Got SSL packet '%d'\n", instance, rtp, *in);

/*
 * If ICE is in use, we can prevent a possible DOS attack
 * by allowing DTLS protocol messages (client hello, etc)
 * only from sources that are in the active remote
 * candidates list.
 */
#ifdef HAVE_PJPROJECT
        if (rtp->ice) {
                int pass_src_check = 0;
                int ix = 0;
                ...
                for (ix = 0; ix < rtp->ice->real_ice->rcand_cnt; ix++) {
                        pj_ice_sess_cand *rcand = &rtp->ice->real_ice->rcand[ix];
                        if (ast_sockaddr_pj_sockaddr_cmp(sa, &rcand->addr) == 0) {
                                pass_src_check = 1;
                                break;
                        }
                }
                if (!pass_src_check) {
                        ast_log(LOG_WARNING, "%s: DTLS packet from %s dropped. Source not in ICE active candidate list.\n",
                                ast_rtp_instance_get_channel_id(instance),
                                ast_sockaddr_stringify(sa));
                        return 0;  /* <-- PACKET DROPPED, 127.0.0.1 never in candidate list */
                }
        }
#endif

and this is the patch that I applied:

ast_debug_dtls(3, "(%p) DTLS - __rtp_recvfrom rtp=%p - Got SSL packet '%d'\n", instance, rtp, *in);

/* ===== START OF PATCH ===== */
#ifdef HAVE_PJPROJECT
        /* If this packet arrived via TURN/ICE loopback re-injection,
         * substitute the real remote address before the candidate check
         * otherwise the DTLS check will see 127.0.0.1 and drop the packet.
         */
        if (!ast_sockaddr_isnull(&rtp->rtp_loop) && !ast_sockaddr_cmp(&rtp->rtp_loop, sa)) {
                ast_rtp_instance_get_remote_address(instance, sa);
        } else if (rtcp && !ast_sockaddr_isnull(&rtp->rtcp_loop) && !ast_sockaddr_cmp(&rtp->rtcp_loop, sa)) {
                ast_sockaddr_copy(sa, &rtp->rtcp->them);
        }
#endif
/* ===== END OF PATCH ===== */

/*
 * If ICE is in use, we can prevent a possible DOS attack
 * by allowing DTLS protocol messages (client hello, etc)
 * only from sources that are in the active remote
 * candidates list.
 */
#ifdef HAVE_PJPROJECT
        if (rtp->ice) {
                ...
                if (!pass_src_check) {
                        ast_log(LOG_WARNING, "%s: DTLS packet from %s dropped. Source not in ICE active candidate list.\n",
                        return 0;  /* <-- Now sees real remote IP, not 127.0.0.1 */
                }
        }
#endif
19

You need to submit the corrections through the bug tracker, because you need to have “signed” a contributor licence agreement. Otherwise Joshua won’t be allowed to look at it, and it will never appear in the official version. The CLA gives Sangoma the right to exploit the update in their commercial products.

20

Thanks David, I’ve submitted this now through the bug tracker