Realtime Queue member endpoint becomes Invalid after receiving calls

Our setup:

  • WebRTC as the transport (http.conf with tlsenable, PJSIP endpoint with webrtc=true et al).
  • All PJSIP objects (endpoints, aors, etc) are loaded from sorcery/Realtime
  • Instead of a direct database connection, we interface with the Curl() function to do a remote read
  • Queues and queue members are also discovered this way

Before starting a call, the queue looks something like:

krossvm*CLI> queue show
7b9cc8df51c8e8ef486fa4db0380a1fe has 0 calls (max unlimited) in 'ringall' strategy (0s holdtime, 0s talktime), W:0, C:0, A:7, SL:0.0%, SL2:0.0% within 0s
   Members:
      95484335 (PJSIP/95484335) (ringinuse enabled) (realtime) (Not in use) has taken no calls yet (login was 1416 secs ago)
   No Callers

When we start a call we see something like the following logged for queues and members:

info:    Queues response body:
info:    name=7b9cc8df51c8e8ef486fa4db0380a1fe&autofill=yes&strategy=ringall&timeout=60&retry=10&wrapuptime=0&joinempty=inuse,ringing,invalid&leavewhenempty=&timeoutrestart=yes
info:    ::ffff:172.17.0.1 - POST /api/PreSharedKey/realtime/queues/single HTTP/1.1 200 168 - 11.101 ms
info:    queueMembers response body:
info:    queue_name=7b9cc8df51c8e8ef486fa4db0380a1fe&uniqueid=95484335&membername=95484335&penalty=0&paused=0&interface=PJSIP/95484335
info:    ::ffff:172.17.0.1 - POST /api/PreSharedKey/realtime/queue_members/multi HTTP/1.1 200 125 - 24.921 ms

According to the queue rules, if the member is not “inuse,ringing,invalid”, then they will receive the call.
Everything is working great so far.

The issue: Consistently after only a few calls are processed and hung up, the endpoint becomes
“Invalid”. Once in this state the endpoint is stuck. “core reload”, “pjsip reload” and “queue reload members” have no effect. The endpoint is still registered and keeping its registration active.

Example:

krossvm*CLI> queue show
7b9cc8df51c8e8ef486fa4db0380a1fe has 1 calls (max unlimited) in 'ringall' strategy (0s holdtime, 0s talktime), W:0, C:0, A:32, SL:0.0%, SL2:21.9% within 0s
   Members:
      95484335 (PJSIP/95484335) (ringinuse enabled) (realtime) (Invalid) has taken no calls yet (login was 1556 secs ago)
   Callers:
      1. PJSIP/7001-00000292 (wait: 0:02, prio: 0)

The endpoint is still registered and reachable though! If we originate to them, their line rings AND they become unstuck.

originate pjsip/95484335 application echo

After hanging up that call, the endpoint correctly returns to “Not in Use” and can receive queued calls.

Similar issues to this that didn’t help find a resolution:

Example endpoint:

Example Endpoint:
krossvm*CLI> pjsip show endpoint 95484335

 Endpoint:  <Endpoint/CID.....................................>  <State.....>  <Channels.>
    I/OAuth:  <AuthId/UserName...........................................................>
        Aor:  <Aor............................................>  <MaxContact>
      Contact:  <Aor/ContactUri..........................> <Hash....> <Status> <RTT(ms)..>
  Transport:  <TransportId........>  <Type>  <cos>  <tos>  <BindAddress..................>
   Identify:  <Identify/Endpoint.........................................................>
        Match:  <criteria.........................>
    Channel:  <ChannelId......................................>  <State.....>  <Time.....>
        Exten: <DialedExten...........>  CLCID: <ConnectedLineCID.......>
==========================================================================================

 Endpoint:  95484335                                             Not in use    0 of inf
     InAuth:  95484335/95484335
        Aor:  95484335                                           1
      Contact:  95484335/sip:s7ef1ind@192.168.1.227:61148; 87829a791a Avail        30.115
   Identify:  7b9cc8df51c8e8ef486fa4db0380a1fe/7b9cc8df51c8e8ef486fa4db0380a1fe
        Match: 192.168.1.96:5060/32


 ParameterName                      : ParameterValue
 ===================================================================================================
 100rel                             : yes
 accept_multiple_sdp_answers        : false
 accountcode                        :
 acl                                :
 aggregate_mwi                      : true
 allow                              : (opus|vp8|h264|vp9)
 allow_overlap                      : true
 allow_subscribe                    : true
 allow_transfer                     : true
 allow_unauthenticated_options      : false
 aors                               : 95484335
 asymmetric_rtp_codec               : false
 auth                               : 95484335
 bind_rtp_to_media_address          : false
 bundle                             : true
 call_group                         :
 callerid                           : <unknown>
 callerid_privacy                   : allowed_not_screened
 callerid_tag                       :
 codec_prefs_incoming_answer        : prefer:pending, operation:intersect, keep:all, transcode:allow
 codec_prefs_incoming_offer         : prefer:pending, operation:intersect, keep:all, transcode:allow
 codec_prefs_outgoing_answer        : prefer:pending, operation:intersect, keep:all, transcode:allow
 codec_prefs_outgoing_offer         : prefer:pending, operation:union, keep:all, transcode:allow
 connected_line_method              : invite
 contact_acl                        :
 context                            : from-internal
 cos_audio                          : 0
 cos_video                          : 0
 device_state_busy_at               : 0
 direct_media                       : false
 direct_media_glare_mitigation      : none
 direct_media_method                : invite
 disable_direct_media_on_nat        : false
 dtls_auto_generate_cert            : Yes
 dtls_ca_file                       :
 dtls_ca_path                       :
 dtls_cert_file                     :
 dtls_cipher                        :
 dtls_fingerprint                   : SHA-256
 dtls_private_key                   :
 dtls_rekey                         : 0
 dtls_setup                         : actpass
 dtls_verify                        : Yes
 dtmf_mode                          : rfc4733
 fax_detect                         : false
 fax_detect_timeout                 : 0
 follow_early_media_fork            : true
 force_avp                          : false
 force_rport                        : true
 from_domain                        :
 from_user                          :
 g726_non_standard                  : false
 ice_support                        : true
 identify_by                        : username,ip
 ignore_183_without_sdp             : false
 inband_progress                    : false
 incoming_call_offer_pref           : local
 incoming_mwi_mailbox               :
 language                           :
 mailboxes                          :
 max_audio_streams                  : 100
 max_video_streams                  : 100
 media_address                      :
 media_encryption                   : dtls
 media_encryption_optimistic        : false
 media_use_received_transport       : true
 message_context                    : message-from-internal
 moh_passthrough                    : false
 moh_suggest                        : default
 mwi_from_user                      :
 mwi_subscribe_replaces_unsolicited : no
 named_call_group                   :
 named_pickup_group                 :
 notify_early_inuse_ringing         : false
 one_touch_recording                : false
 outbound_auth                      :
 outbound_proxy                     :
 outgoing_call_offer_pref           : remote_merge
 pickup_group                       :
 preferred_codec_only               : false
 record_off_feature                 : automixmon
 record_on_feature                  : automixmon
 refer_blind_progress               : true
 rewrite_contact                    : false
 rpid_immediate                     : false
 rtcp_mux                           : true
 rtp_engine                         : asterisk
 rtp_ipv6                           : false
 rtp_keepalive                      : 60
 rtp_symmetric                      : false
 rtp_timeout                        : 300
 rtp_timeout_hold                   : 0
 sdp_owner                          : -
 sdp_session                        : Asterisk
 send_connected_line                : yes
 send_diversion                     : true
 send_history_info                  : false
 send_pai                           : false
 send_rpid                          : false
 set_var                            :
 srtp_tag_32                        : false
 stir_shaken                        : off
 stir_shaken_profile                :
 sub_min_expiry                     : 0
 subscribe_context                  :
 suppress_q850_reason_headers       : false
 t38_bind_udptl_to_media_address    : false
 t38_udptl                          : false
 t38_udptl_ec                       : none
 t38_udptl_ipv6                     : false
 t38_udptl_maxdatagram              : 0
 t38_udptl_nat                      : false
 timers                             : yes
 timers_min_se                      : 90
 timers_sess_expires                : 1800
 tone_zone                          :
 tos_audio                          : 0
 tos_video                          : 0
 transport                          :
 trust_connected_line               : yes
 trust_id_inbound                   : false
 trust_id_outbound                  : false
 use_avpf                           : true
 use_ptime                          : false
 user_eq_phone                      : false
 voicemail_extension                :
 webrtc                             : yes

Example extconfig.conf:

[settings]
ps_endpoints => curl,http://localhost:8080/api/realtime/endpoints
ps_auths => curl,http://localhost:8080/api/realtime/auths
ps_aors => curl,http://localhost:8080/api/realtime/aors
queues => curl,http://localhost:8080/api/realtime/queues
queue_members => curl,http://localhost:8080/api/realtime/queue_members

Example sorcery.conf:

[res_pjsip]
endpoint=realtime,ps_endpoints
endpoint=config,pjsip.conf,criteria=type=endpoint
auth=realtime,ps_auths
auth=config,pjsip.conf,criteria=type=auth
aor=realtime,ps_aors
aor=config,pjsip.conf,criteria=type=aor

full asterisk debug log as an endpoint goes invalid .txt (36.3 KB)

A video showing our steps to reproduce: realtime queue member endpoints become invalid.mkv - Google Drive

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.