Our setup:
- WebRTC as the transport (http.conf with tlsenable, PJSIP endpoint with webrtc=true et al).
- All PJSIP objects (endpoints, aors, etc) are loaded from sorcery/Realtime
- Instead of a direct database connection, we interface with the Curl() function to do a remote read
- Queues and queue members are also discovered this way
Before starting a call, the queue looks something like:
krossvm*CLI> queue show
7b9cc8df51c8e8ef486fa4db0380a1fe has 0 calls (max unlimited) in 'ringall' strategy (0s holdtime, 0s talktime), W:0, C:0, A:7, SL:0.0%, SL2:0.0% within 0s
Members:
95484335 (PJSIP/95484335) (ringinuse enabled) (realtime) (Not in use) has taken no calls yet (login was 1416 secs ago)
No Callers
When we start a call we see something like the following logged for queues and members:
info: Queues response body:
info: name=7b9cc8df51c8e8ef486fa4db0380a1fe&autofill=yes&strategy=ringall&timeout=60&retry=10&wrapuptime=0&joinempty=inuse,ringing,invalid&leavewhenempty=&timeoutrestart=yes
info: ::ffff:172.17.0.1 - POST /api/PreSharedKey/realtime/queues/single HTTP/1.1 200 168 - 11.101 ms
info: queueMembers response body:
info: queue_name=7b9cc8df51c8e8ef486fa4db0380a1fe&uniqueid=95484335&membername=95484335&penalty=0&paused=0&interface=PJSIP/95484335
info: ::ffff:172.17.0.1 - POST /api/PreSharedKey/realtime/queue_members/multi HTTP/1.1 200 125 - 24.921 ms
According to the queue rules, if the member is not “inuse,ringing,invalid”, then they will receive the call.
Everything is working great so far.
The issue: Consistently after only a few calls are processed and hung up, the endpoint becomes
“Invalid”. Once in this state the endpoint is stuck. “core reload”, “pjsip reload” and “queue reload members” have no effect. The endpoint is still registered and keeping its registration active.
Example:
krossvm*CLI> queue show
7b9cc8df51c8e8ef486fa4db0380a1fe has 1 calls (max unlimited) in 'ringall' strategy (0s holdtime, 0s talktime), W:0, C:0, A:32, SL:0.0%, SL2:21.9% within 0s
Members:
95484335 (PJSIP/95484335) (ringinuse enabled) (realtime) (Invalid) has taken no calls yet (login was 1556 secs ago)
Callers:
1. PJSIP/7001-00000292 (wait: 0:02, prio: 0)
The endpoint is still registered and reachable though! If we originate to them, their line rings AND they become unstuck.
originate pjsip/95484335 application echo
After hanging up that call, the endpoint correctly returns to “Not in Use” and can receive queued calls.
Similar issues to this that didn’t help find a resolution:
- Registered endpoint becomes unreachable
- Endpoints become unavaiable every few minutes - #27 by amilcarsilvestre
- PJSIP queue members showing as invalid - #13 by byoungdale
- Dynamic Queue Member(Agent)'s Status is Invalid - #5 by kharwell
Example endpoint:
Example Endpoint:
krossvm*CLI> pjsip show endpoint 95484335
Endpoint: <Endpoint/CID.....................................> <State.....> <Channels.>
I/OAuth: <AuthId/UserName...........................................................>
Aor: <Aor............................................> <MaxContact>
Contact: <Aor/ContactUri..........................> <Hash....> <Status> <RTT(ms)..>
Transport: <TransportId........> <Type> <cos> <tos> <BindAddress..................>
Identify: <Identify/Endpoint.........................................................>
Match: <criteria.........................>
Channel: <ChannelId......................................> <State.....> <Time.....>
Exten: <DialedExten...........> CLCID: <ConnectedLineCID.......>
==========================================================================================
Endpoint: 95484335 Not in use 0 of inf
InAuth: 95484335/95484335
Aor: 95484335 1
Contact: 95484335/sip:s7ef1ind@192.168.1.227:61148; 87829a791a Avail 30.115
Identify: 7b9cc8df51c8e8ef486fa4db0380a1fe/7b9cc8df51c8e8ef486fa4db0380a1fe
Match: 192.168.1.96:5060/32
ParameterName : ParameterValue
===================================================================================================
100rel : yes
accept_multiple_sdp_answers : false
accountcode :
acl :
aggregate_mwi : true
allow : (opus|vp8|h264|vp9)
allow_overlap : true
allow_subscribe : true
allow_transfer : true
allow_unauthenticated_options : false
aors : 95484335
asymmetric_rtp_codec : false
auth : 95484335
bind_rtp_to_media_address : false
bundle : true
call_group :
callerid : <unknown>
callerid_privacy : allowed_not_screened
callerid_tag :
codec_prefs_incoming_answer : prefer:pending, operation:intersect, keep:all, transcode:allow
codec_prefs_incoming_offer : prefer:pending, operation:intersect, keep:all, transcode:allow
codec_prefs_outgoing_answer : prefer:pending, operation:intersect, keep:all, transcode:allow
codec_prefs_outgoing_offer : prefer:pending, operation:union, keep:all, transcode:allow
connected_line_method : invite
contact_acl :
context : from-internal
cos_audio : 0
cos_video : 0
device_state_busy_at : 0
direct_media : false
direct_media_glare_mitigation : none
direct_media_method : invite
disable_direct_media_on_nat : false
dtls_auto_generate_cert : Yes
dtls_ca_file :
dtls_ca_path :
dtls_cert_file :
dtls_cipher :
dtls_fingerprint : SHA-256
dtls_private_key :
dtls_rekey : 0
dtls_setup : actpass
dtls_verify : Yes
dtmf_mode : rfc4733
fax_detect : false
fax_detect_timeout : 0
follow_early_media_fork : true
force_avp : false
force_rport : true
from_domain :
from_user :
g726_non_standard : false
ice_support : true
identify_by : username,ip
ignore_183_without_sdp : false
inband_progress : false
incoming_call_offer_pref : local
incoming_mwi_mailbox :
language :
mailboxes :
max_audio_streams : 100
max_video_streams : 100
media_address :
media_encryption : dtls
media_encryption_optimistic : false
media_use_received_transport : true
message_context : message-from-internal
moh_passthrough : false
moh_suggest : default
mwi_from_user :
mwi_subscribe_replaces_unsolicited : no
named_call_group :
named_pickup_group :
notify_early_inuse_ringing : false
one_touch_recording : false
outbound_auth :
outbound_proxy :
outgoing_call_offer_pref : remote_merge
pickup_group :
preferred_codec_only : false
record_off_feature : automixmon
record_on_feature : automixmon
refer_blind_progress : true
rewrite_contact : false
rpid_immediate : false
rtcp_mux : true
rtp_engine : asterisk
rtp_ipv6 : false
rtp_keepalive : 60
rtp_symmetric : false
rtp_timeout : 300
rtp_timeout_hold : 0
sdp_owner : -
sdp_session : Asterisk
send_connected_line : yes
send_diversion : true
send_history_info : false
send_pai : false
send_rpid : false
set_var :
srtp_tag_32 : false
stir_shaken : off
stir_shaken_profile :
sub_min_expiry : 0
subscribe_context :
suppress_q850_reason_headers : false
t38_bind_udptl_to_media_address : false
t38_udptl : false
t38_udptl_ec : none
t38_udptl_ipv6 : false
t38_udptl_maxdatagram : 0
t38_udptl_nat : false
timers : yes
timers_min_se : 90
timers_sess_expires : 1800
tone_zone :
tos_audio : 0
tos_video : 0
transport :
trust_connected_line : yes
trust_id_inbound : false
trust_id_outbound : false
use_avpf : true
use_ptime : false
user_eq_phone : false
voicemail_extension :
webrtc : yes
Example extconfig.conf:
[settings]
ps_endpoints => curl,http://localhost:8080/api/realtime/endpoints
ps_auths => curl,http://localhost:8080/api/realtime/auths
ps_aors => curl,http://localhost:8080/api/realtime/aors
queues => curl,http://localhost:8080/api/realtime/queues
queue_members => curl,http://localhost:8080/api/realtime/queue_members
Example sorcery.conf:
[res_pjsip]
endpoint=realtime,ps_endpoints
endpoint=config,pjsip.conf,criteria=type=endpoint
auth=realtime,ps_auths
auth=config,pjsip.conf,criteria=type=auth
aor=realtime,ps_aors
aor=config,pjsip.conf,criteria=type=aor
full asterisk debug log as an endpoint goes invalid .txt (36.3 KB)
A video showing our steps to reproduce: realtime queue member endpoints become invalid.mkv - Google Drive