Confbridge Video Mode: sfu

Asterisk Asterisk WebRTC
1

Hi All, im busy testing Asterisk 16 (Asterisk certified/16.8-cert3) with confbridge, and the SFU feature - to display all parties in a WebRTC session. I can create and use an audio conference, no problem, and can create a video conference with video_mode= follow_talker - these work fine. When on follow_talker it appears to change the source of the video to the speaker (most of the time). I have made sure to set the video codec to only vp8.

If I switch it to video_mode= sfu, I get only blank video stream.

Interestingly, if I enable the RTP debugger, I see the following:

Sent RTP packet to      XX.XX.17.199:64374 (via ICE) (type 100, seq 023090, ts 3613742623, len 000486)
Sent RTP packet to      XX.XX.17.199:52882 (via ICE) (type 111, seq 008988, ts 2480640, len 000138)
Sent RTP packet to      XX.XX.17.199:50946 (via ICE) (type 111, seq 025389, ts 2591136, len 000138)
Got  RTP packet from    XX.XX.17.199:52882 (type 111, seq 023180, ts 1606838904, len 000045)
Got  RTP packet from    XX.XX.17.199:50946 (type 111, seq 014780, ts 862568761, len 000013)
Got  RTP packet from   XX.XX.17.199:50946 (type 111, seq 014781, ts 862569721, len 000013)
Got  RTP packet from    XX.XX.17.199:64314 (type 96, seq 010597, ts 3613745683, len 000440)

I get the streams, in the re-invite, as a participant is added.

INVITE ...

v=0
o=- 1201072512 5 IN IP4 XX.XX.25.159
s=Asterisk
c=IN IP4 XX.XX.25.159
t=0 0
m=audio 18786 UDP/TLS/RTP/SAVPF 111 0 126
a=connection:existing
a=setup:actpass
a=fingerprint:SHA-256 42:14:5C:06:5A:01:13:F5:17:BD:10:FF:A3:3E:16
a=ice-ufrag:28b85c8663b7cb1018905d173dda2c76
a=ice-pwd:484708965a57207736a088d643ce8487
a=candidate:H3432a5cc 1 UDP 2130706431 XX.XX.165.204 18786 typ host
a=rtpmap:111 opus/48000/2
a=fmtp:111 useinbandfec=1
a=rtpmap:0 PCMU/8000
a=rtpmap:126 telephone-event/8000
a=fmtp:126 0-16
a=ptime:20
a=maxptime:20
a=sendrecv
a=rtcp-mux
m=video 11092 UDP/TLS/RTP/SAVPF 96
a=connection:existing
a=setup:actpass
a=fingerprint:SHA-256 42:14:5C:06:5A:01:13:F5:17:BD:10:FF:A3:3E:16:2C:36:7A
a=ice-ufrag:10284b6f3743ce2957f002d44f9459f3
a=ice-pwd:3972bc801443867a429c6f63626a115c
a=candidate:H3432a5cc 1 UDP 2130706431 XX.XX.165.204 11092 typ host
a=rtpmap:96 VP8/90000
a=sendrecv
a=rtcp-mux
m=video 12086 UDP/TLS/RTP/SAVPF 100
a=connection:new
a=setup:actpass
a=fingerprint:SHA-256 42:14:5C:06:5A:01:13:F5:17:BD:10:FF:A3:3E:16:2C:36:
a=ice-ufrag:388aedc44dfb40fa4551a96f140dd9ef
a=ice-pwd:284d61431df2506c3f1027f6718d0552
a=candidate:H3432a5cc 1 UDP 2130706431 XX.XX.165.204 12086 typ host
a=rtpmap:100 VP8/90000
a=sendonly
a=rtcp-mux

So it seems everything is doing what it needs to do, without any errors… it’s just blank video.

2

You don’t appear to be using bundle, which is enabled using the webrtc option in PJSIP. While the browsers may have code to support non-bundle for multiple video streams it is rarely used, so you would be in off-nominal territory.

3

Not sure what you mean by using bundle?

4

Bundle[1] is part of WebRTC, it bundles all media streams over a single UDP socket and distinguishes them using the SSRC which is part of SDP, instead of having separate for each stream. As I said while the browser may have rudimentary support for not doing it for multiple video - it is by no means widely tested or used.

[1] rfc8843

5

Ok, then how do it disable (or fix) this then?

6

The “webrtc” option in PJSIP automatically enables all WebRTC required functionality, including bundle. There is no separate bundle option because the bundle implementation was specifically written for WebRTC.

7

webrtc was not set, now it is set to “yes”, but it’s still doing the same thing… checking again.

8

If it’s still doing the same thing and the SDP shows “ssrc” attribute lines then you could upgrade to latest Asterisk 16 and see if it still happens, if it does then it’s likely something client/browser side and you’d have to investigate there.

9

Thanks, will take a look.

10

Now on Asterisk 16.21.1, and using: {bundlePolicy:"max-bundle"} as per RTCPeerConnection() - Web APIs | MDN with bundle=yes on the endpoint (also webrtc=yes).

Same behaviour (using Chrome and Edge).

Sent RTP packet to      XX.XX.17.199:50184 (via ICE) (type 96, seq 015120, ts 542681165, len 000538)
Sent RTP packet to      XX.XX.17.199:62056 (via ICE) (type 111, seq 009955, ts 1358400, len 000013)
Sent RTP packet to      XX.XX.17.199:50184 (via ICE) (type 111, seq 011534, ts 2775360, len 000138)
Sent RTP packet to      XX.XX.17.199:62056 (via ICE) (type 111, seq 009956, ts 1359360, len 000013)
Sent RTP packet to      XX.XX.17.199:50184 (via ICE) (type 111, seq 011535, ts 2776320, len 000138)

Got  RTP packet from    XX.XX.17.199:62056 (type 111, seq 002338, ts 1920834656, len 000045)
Got  RTP packet from    XX.XX.17.199:50184 (type 111, seq 034675, ts 2509750355, len 000013)

Sent RTP packet to      XX.XX.17.199:62056 (via ICE) (type 111, seq 009957, ts 1360320, len 000013)
Sent RTP packet to      XX.XX.17.199:50184 (via ICE) (type 111, seq 011536, ts 2777280, len 000138)

Got  RTP packet from    XX.XX.17.199:62056 (type 96, seq 010553, ts 542684135, len 000327)

Sent RTP packet to      XX.XX.17.199:50184 (via ICE) (type 96, seq 015121, ts 542684135, len 000315)

Got  RTP packet from    XX.XX.17.199:50184 (type 111, seq 034676, ts 2509751315, len 000013)
Got  RTP packet from    XX.XX.17.199:62056 (type 111, seq 002339, ts 1920835616, len 000045)

Sent RTP packet to      XX.XX.17.199:62056 (via ICE) (type 111, seq 009958, ts 1361280, len 000013)
Sent RTP packet to      XX.XX.17.199:50184 (via ICE) (type 111, seq 011537, ts 2778240, len 000138)

Got  RTP packet from    XX.XX.17.199:50184 (type 96, seq 011225, ts 2627547565, len 000655)

Sent RTP packet to      XX.XX.17.199:62056 (via ICE) (type 96, seq 031216, ts 2627547565, len 000643)

Got  RTP packet from    XX.XX.17.199:50184 (type 96, seq 011226, ts 2627547565, len 000655)

Sent RTP packet to      XX.XX.17.199:62056 (via ICE) (type 96, seq 031217, ts 2627547565, len 000643)
Sent RTP packet to      XX.XX.17.199:62056 (via ICE) (type 111, seq 009959, ts 1362240, len 000013)
Sent RTP packet to      XX.XX.17.199:50184 (via ICE) (type 111, seq 011538, ts 2779200, len 000138)

Got  RTP packet from    XX.XX.17.199:62056 (type 96, seq 010554, ts 542687105, len 000286)
Got  RTP packet from    XX.XX.17.199:62056 (type 111, seq 002340, ts 1920836576, len 000045)

Sent RTP packet to      XX.XX.17.199:50184 (via ICE) (type 96, seq 015122, ts 542687105, len 000274)

Got  RTP packet from    XX.XX.17.199:50184 (type 111, seq 034677, ts 2509752275, len 000013)

The RE_INVITE

v=0
o=- 2949188128 6 IN IP4 XX.XX.25.159
s=Asterisk
c=IN IP4 XX.XX.25.159
t=0 0
a=msid-semantic:WMS *
a=group:BUNDLE 0 1 video-2
m=audio 13278 UDP/TLS/RTP/SAVPF 111 0 126
a=connection:existing
a=setup:actpass
a=fingerprint:SHA-256 42:14:5C:06:5A:01:13:F5:17:BD:10:FF:A3:3E:16:2C:36:7A:A0:82:06:81:AA:FD:32:A9:F5:D8:54:44:0E:5A
a=ice-ufrag:029389872eb6ea0e59ae2d5915761888
a=ice-pwd:3b7b26a07ca20b5272501688183aab7b
a=candidate:H3432a5cc 1 UDP 2130706431 52.50.165.204 13278 typ host
a=candidate:Hb6de93cb 1 UDP 2130706431 fe80::4ff:a7ff:feb7:47e8 13278 typ host
a=rtpmap:111 opus/48000/2
a=fmtp:111 useinbandfec=1
a=rtpmap:0 PCMU/8000
a=rtpmap:126 telephone-event/8000
a=fmtp:126 0-16
a=ptime:20
a=maxptime:20
a=sendrecv
a=rtcp-mux
a=ssrc:1468228806 cname:209783db-3034-49b4-8d9d-8fffa9bfce77
a=msid:2aa50b8f-d0af-443d-b3ee-7dc07e287f5a ee80f03f-c277-4c81-94a0-bb583822aa3a
a=rtcp-fb:* transport-cc
a=mid:0
m=video 13278 UDP/TLS/RTP/SAVPF 96
a=connection:existing
a=setup:actpass
a=fingerprint:SHA-256 42:14:5C:06:5A:01:13:F5:17:BD:10:FF:A3:3E:16:2C:36:7A:A0:82:06:81:AA:FD:32:A9:F5:D8:54:44:0E:5A
a=ice-ufrag:029389872eb6ea0e59ae2d5915761888
a=ice-pwd:3b7b26a07ca20b5272501688183aab7b
a=rtpmap:96 VP8/90000
a=sendrecv
a=rtcp-mux
a=ssrc:1572160864 cname:67fdaf2a-9d30-4243-b382-54cd5cffdeef
a=msid:2aa50b8f-d0af-443d-b3ee-7dc07e287f5a 85188c69-ffda-49a3-828a-c253b36bbd70
a=rtcp-fb:* transport-cc
a=rtcp-fb:* ccm fir
a=rtcp-fb:* goog-remb
a=rtcp-fb:* nack
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=mid:1
m=video 0 UDP/TLS/RTP/SAVPF 96
a=mid:video-2

Endboint:

pjsip show endpoint TTbXG7XMOC 

 Endpoint:  <Endpoint/CID.....................................>  <State.....>  <Channels.>
    I/OAuth:  <AuthId/UserName...........................................................>
        Aor:  <Aor............................................>  <MaxContact>
      Contact:  <Aor/ContactUri..........................> <Hash....> <Status> <RTT(ms)..>
  Transport:  <TransportId........>  <Type>  <cos>  <tos>  <BindAddress..................>
   Identify:  <Identify/Endpoint.........................................................>
        Match:  <criteria.........................>
    Channel:  <ChannelId......................................>  <State.....>  <Time.....>
        Exten: <DialedExten...........>  CLCID: <ConnectedLineCID.......>
==========================================================================================

 Endpoint:  TTbXG7XMOC/800                                       Busy          1 of 1
     InAuth:  TTbXG7XMOC/TTbXG7XMOC
        Aor:  TTbXG7XMOC                                         1
      Contact:  TTbXG7XMOC/sip:0t0mhto4@172.31.18.8:36912; f1273b3123 Avail       256.993
    Channel: PJSIP/TTbXG7XMOC-00000003/ConfBridge                Up            00:07:06   
        Exten: s                         CLCID: "" <>


 ParameterName                      : ParameterValue
 ================================================================================
 100rel                             : yes
 accept_multiple_sdp_answers        : false
 accountcode                        : 
 acl                                : 
 aggregate_mwi                      : true
 allow                              : (opus|ulaw|vp8)
 allow_overlap                      : true
 allow_subscribe                    : true
 allow_transfer                     : true
 allow_unauthenticated_options      : false
 aors                               : TTbXG7XMOC
 asymmetric_rtp_codec               : false
 auth                               : TTbXG7XMOC
 bind_rtp_to_media_address          : false
 bundle                             : true
 call_group                         : 
 callerid                           : "Eight Hundred" <800>
 callerid_privacy                   : allowed_not_screened
 callerid_tag                       : 
 connected_line_method              : invite
 contact_acl                        : 
 context                            : from-extensions
 cos_audio                          : 0
 cos_video                          : 0
 device_state_busy_at               : 1
 direct_media                       : false
 direct_media_glare_mitigation      : none
 direct_media_method                : invite
 disable_direct_media_on_nat        : false
 dtls_auto_generate_cert            : No
 dtls_ca_file                       : 
 dtls_ca_path                       : 
 dtls_cert_file                     : /home/ubuntu/certs/innovateasterisk_com.crt
 dtls_cipher                        : 
 dtls_fingerprint                   : SHA-256
 dtls_private_key                   : /home/ubuntu/certs/innovateasterisk_com.key
 dtls_rekey                         : 0
 dtls_setup                         : actpass
 dtls_verify                        : Yes
 dtmf_mode                          : rfc4733
 fax_detect                         : false
 fax_detect_timeout                 : 0
 follow_early_media_fork            : true
 force_avp                          : false
 force_rport                        : true
 from_domain                        : 
 from_user                          : 
 g726_non_standard                  : false
 ice_support                        : true
 identify_by                        : username,ip
 ignore_183_without_sdp             : false
 inband_progress                    : false
 incoming_mwi_mailbox               : 
 language                           : 
 mailboxes                          : 
 max_audio_streams                  : 10
 max_video_streams                  : 10
 media_address                      : 
 media_encryption                   : dtls
 media_encryption_optimistic        : false
 media_use_received_transport       : true
 message_context                    : textmessages
 moh_passthrough                    : false
 moh_suggest                        : default
 mwi_from_user                      : 
 mwi_subscribe_replaces_unsolicited : no
 named_call_group                   : 
 named_pickup_group                 : 
 notify_early_inuse_ringing         : false
 one_touch_recording                : false
 outbound_auth                      : 
 outbound_proxy                     : 
 pickup_group                       : 
 preferred_codec_only               : false
 record_off_feature                 : automixmon
 record_on_feature                  : automixmon
 refer_blind_progress               : true
 rewrite_contact                    : true
 rpid_immediate                     : false
 rtcp_mux                           : true
 rtp_engine                         : asterisk
 rtp_ipv6                           : false
 rtp_keepalive                      : 0
 rtp_symmetric                      : true
 rtp_timeout                        : 60
 rtp_timeout_hold                   : 0
 sdp_owner                          : -
 sdp_session                        : Asterisk
 send_connected_line                : yes
 send_diversion                     : true
 send_history_info                  : false
 send_pai                           : false
 send_rpid                          : false
 set_var                            : 
 srtp_tag_32                        : false
 stir_shaken                        : false
 sub_min_expiry                     : 0
 subscribe_context                  : subscriptions
 suppress_q850_reason_headers       : false
 t38_udptl                          : false
 t38_udptl_ec                       : none
 t38_udptl_ipv6                     : false
 t38_udptl_maxdatagram              : 0
 t38_udptl_nat                      : false
 timers                             : yes
 timers_min_se                      : 90
 timers_sess_expires                : 1800
 tone_zone                          : 
 tos_audio                          : 0
 tos_video                          : 0
 transport                          : 
 trust_connected_line               : yes
 trust_id_inbound                   : false
 trust_id_outbound                  : false
 use_avpf                           : true
 use_ptime                          : false
 user_eq_phone                      : false
 voicemail_extension                : 
 webrtc                             : yes
11

The re-INVITE shows an audio stream, a video stream, and an unused video stream. You’d need to actually state the specific use case and step by step what you’re doing for me to understand whether that is expected or not.

I will say that we use this every day, so this more likely points to your specific use case or your WebRTC client doing something.

12

And turning on Asterisk core debug logging will tell you exactly what ConfBridge is doing when it comes to figuring out the video streams and re-INVITEs and that.

13

You could also test using Dana[1] and see if that works.

[1] GitHub - nimbleape/dana-the-stream-gatekeeper: React based front-end demo for Asterisk's SFU

14

I’m using SipJS - 0.11.6, and although it is quite an old version, I am able to configure the peer connection, and it’s generally not the job of the sipjs code to configure the SDP, it’s more Chrome’s job… so I was looking to see what exactly it was that was failing.

From reading up on bundle, i see that the default is balanced meaning: “instructs the browser to pick two tracks to send — one audio and one video” - as per your previous message, this is incompatible, and should be on max-bundle. I really thought this trick would do it.

I was also looking at the code for that cyber mega phone thing, and it doesn’t refer to this all, so it appears to be using browser defaults.

I will try use that code, as a test case, or even the Dana app you suggest, and see what that returns.

15

The WebRTC client still has to add streams, react to them appearing, that kind of stuff.

16

I’m sure, that’s why i’m quite surprised by all this.

Yes, this is the custom code that im busy with… I see the multiple streams and play them out with a video element, when there is one stream, it works fine, when there are multiple (using the same sort of code), it doesn’t ever play. It’s almost as if the video is somehow disabled (but it checked its says “live”).

17

I don’t know the browser side well, so I can’t really comment on that.

18

Without changing anything on the Asterisk side, just using the “Cyber Mega Phone 2K”, the conference SFU works fine.

This is the RE-INVITE when adding a 3rd participant

v=0
o=- 2604392200 8 IN IP4 XX.XX.25.159
s=Asterisk
c=IN IP4 XX.XX.25.159
t=0 0
a=msid-semantic:WMS *
a=group:BUNDLE 0 1 video-2 video-3
m=audio 12088 UDP/TLS/RTP/SAVPF 111 0 126
a=connection:existing
a=setup:actpass
a=fingerprint:SHA-256 42:14:5C:06:5A:01:13:F5:17:BD:10:FF:A3:3E:16:2C:36:7A:A0:82:06:81:AA:FD:32:A9:F5:D8:54:44:0E:5A
a=ice-ufrag:3695e2d071d40cd6139dadc100b5cf99
a=ice-pwd:4f851ff44524b253737f3c4b1098ff48
a=candidate:H3432a5cc 1 UDP 2130706431 XX.XX.165.204 12088 typ host
a=candidate:Hb6de93cb 1 UDP 2130706431 fe80::4ff:a7ff:feb7:47e8 12088 typ host
a=rtpmap:111 opus/48000/2
a=fmtp:111 useinbandfec=1
a=rtpmap:0 PCMU/8000
a=rtpmap:126 telephone-event/8000
a=fmtp:126 0-16
a=ptime:20
a=maxptime:20
a=sendrecv
a=rtcp-mux
a=ssrc:196047288 cname:ffd83dc1-010f-4de7-b44d-096b6394a45b
a=msid:12a4a5b7-aebb-4c31-9f86-8c6ce07ef122 245d0e91-305f-4d03-9375-9a6a4a5852d5
a=rtcp-fb:* transport-cc
a=mid:0
m=video 12088 UDP/TLS/RTP/SAVPF 96
a=connection:existing
a=setup:actpass
a=fingerprint:SHA-256 42:14:5C:06:5A:01:13:F5:17:BD:10:FF:A3:3E:16:2C:36:7A:A0:82:06:81:AA:FD:32:A9:F5:D8:54:44:0E:5A
a=ice-ufrag:3695e2d071d40cd6139dadc100b5cf99
a=ice-pwd:4f851ff44524b253737f3c4b1098ff48
a=rtpmap:96 VP8/90000
a=sendrecv
a=rtcp-mux
a=ssrc:1988854476 cname:a1c51493-b739-4e02-968b-b6a5f8870ca1
a=msid:12a4a5b7-aebb-4c31-9f86-8c6ce07ef122 7f1b8662-30ea-45e8-9773-fa7d2946c543
a=rtcp-fb:* transport-cc
a=rtcp-fb:* ccm fir
a=rtcp-fb:* goog-remb
a=rtcp-fb:* nack
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=mid:1
m=video 12088 UDP/TLS/RTP/SAVPF 96
a=connection:existing
a=setup:actpass
a=fingerprint:SHA-256 42:14:5C:06:5A:01:13:F5:17:BD:10:FF:A3:3E:16:2C:36:7A:A0:82:06:81:AA:FD:32:A9:F5:D8:54:44:0E:5A
a=ice-ufrag:3695e2d071d40cd6139dadc100b5cf99
a=ice-pwd:4f851ff44524b253737f3c4b1098ff48
a=rtpmap:96 VP8/90000
a=sendonly
a=rtcp-mux
a=ssrc:1898159152 cname:65264906-6479-4c8c-9a87-186a7b456643
a=msid:0159fe6b-6657-42d7-9f31-c4432143a2f9 8f3b077a-d82e-453e-a089-4418f47e60dc
a=rtcp-fb:* transport-cc
a=rtcp-fb:* ccm fir
a=rtcp-fb:* goog-remb
a=rtcp-fb:* nack
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=mid:video-2
m=video 12088 UDP/TLS/RTP/SAVPF 96
a=connection:existing
a=setup:actpass
a=fingerprint:SHA-256 42:14:5C:06:5A:01:13:F5:17:BD:10:FF:A3:3E:16:2C:36:7A:A0:82:06:81:AA:FD:32:A9:F5:D8:54:44:0E:5A
a=ice-ufrag:3695e2d071d40cd6139dadc100b5cf99
a=ice-pwd:4f851ff44524b253737f3c4b1098ff48
a=rtpmap:96 VP8/90000
a=fmtp:96 max-fr=60;max-fs=12288
a=sendonly
a=rtcp-mux
a=ssrc:585596634 cname:f5434ba1-abea-4577-a4da-aabbe23501b6
a=msid:2c6c2825-0c0b-4474-85e0-644f340d9f34 3e4a0b04-688b-41b8-b3ae-c6b6f9562bb5
a=rtcp-fb:* transport-cc
a=rtcp-fb:* ccm fir
a=rtcp-fb:* goog-remb
a=rtcp-fb:* nack
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=mid:video-3
19

I’m quite familiar with this side of things - i must be missing something.

20

I was… its working now (face palm emoji)

I it seems that an empty stream comes down as stream 0, its is probably something like the “self” stream, but its empty - i suppose its to be expected… since you dont need to see your self being streamed to your self.

Anyway, ignoring that stream, it working correctly now.

1 Like