So what am I missing? It seems like Asterisk still doesn’t accept new messages until the queue is cleared.
Your response was unclear. It wasn’t stated whether the change did or did not have an impact. I thought you were referring to before the change was made.
If you mean that even after the change has been made then PJSIP is still not accepting new INVITE requests, further investigation would likely need to be done for the specific scenario. A full debug log for example may provide more information.
Sorry about that. Yeah, the change has no affect.
what level of verbosity should the debug be set to?
And even with a full debug log it likely requires reproducing the specific scenario, stepping through the code, making a database behave the same way. You could file an issue for such but there is no time frame on when or if it would get looked at.
Understood.
So if I set PJSIP to only care about PJSIP taskprocessors, then does that mean that other taskprocessor overloads would not show up in the log? Or it will continue showing in the log, but in reality it won’t pause distributing messages.
They will continue showing up in the log. That is unrelated to the setting in PJSIP. The setting in PJSIP just tells PJSIP which ones to pay attention to when it happens. If it’s pjsip_only then it means only taskprocessors created in PJSIP world. As well the overload stuff only applies to new INVITEs and new stuff - SIP messages for existing dialogs and calls continue to be processed.
So let me make sure I correctly understand this.
If I set taskprocessor_overload_trigger=none then PJSIP will accept new INVITEs regardless of any overloads happening. Is that correct?
Thanks much
That is correct.
Got it. Well, I did that, but unfortunately PJSIP is not accepting new INVITEs when overloads happen.
I will try to set up a clean setup and create an issue in the tracker. (I am currently using v18.12.1)
That code hasn’t changed and has been confirmed to work, so it likely does work - but what you are experiencing is something after it and thus unrelated.
I didn’t get a chance to file an issue yet…
I worked on this a bit more, I still have taskprocessor_overload_trigger set to none. It seema to me like PJSIP is not ignoring the taskprocessor overloads.
For example, when creating a big amount of calls and the Confbridge queue gets overloaded, Asterisk (PJSIP) will not accept any new calls, until the alarm is cleared.
I watched it by running:
[root@yplab ~]# asterisk -x"core show taskprocessors" | grep 'Processor\|Confbridge'
Processor Processed In Queue Max Depth Low water High water
Confbridge/1000-00000421 366 533 695 450 500
Here’s when it started processing new INVITEs
[root@yplab ~]# asterisk -x"core show taskprocessors" | grep 'Processor\|Confbridge'
Processor Processed In Queue Max Depth Low water High water
Confbridge/1000-00000421 475 424 695 450 500
Further to this, I disabled CDR, I can destroy 2k channels at once and PJSIP is instantly processing the new INVITEs, meaning, I no longer get stasis/m:cdr overloads. I do see overloads during the process of the channels getting destroyed, such as stasis/m:devicestate and stasis/m:manager, but their queues clear so quick that I wasn’t able to capture the overloads and they had no impact on the calls whatsoever.
So I am afraid that PJSIP doesn’t actually ignore the overloads.
Did you enable debug level logging with at least a level of 3? The debug log will state that it is ignoring traffic due to the taskprocessor overload.
When you created the large number of channels, what was the state of the PJSIP traskprocessors as well?
I did not check. I’ll run this again tomorrow morning and check.
This produces a whole lot of messages, any particular strings to grep so I can locate the message a bit easier?
Thanks again for your time, Josh!
“Taskprocessor overload” will find it if present.
Indeed (pastebin):
[root@yplab ~]# tail -f /var/log/asterisk/full | grep -i processor
[2023-03-20 12:08:43] DEBUG[10291] res_pjsip/pjsip_distributor.c: Taskprocessor overload alert: Ignoring 'Request msg REGISTER/cseq=15 (rdata0x7f8e7c000948)'.
[2023-03-20 12:08:46] DEBUG[10291] res_pjsip/pjsip_distributor.c: Taskprocessor overload alert: Ignoring 'Request msg REGISTER/cseq=15 (rdata0x7f8e7c000948)'.
[2023-03-20 12:08:47] DEBUG[10291] res_pjsip/pjsip_distributor.c: Taskprocessor overload alert: Ignoring 'Request msg INVITE/cseq=1 (rdata0x7f8e7c000948)'.
[2023-03-20 12:08:50] DEBUG[10291] res_pjsip/pjsip_distributor.c: Taskprocessor overload alert: Ignoring 'Request msg REGISTER/cseq=15 (rdata0x7f8e7c000948)'.
[2023-03-20 12:08:54] DEBUG[10291] res_pjsip/pjsip_distributor.c: Taskprocessor overload alert: Ignoring 'Request msg REGISTER/cseq=15 (rdata0x7f8e7c000948)'.
[2023-03-20 12:08:58] DEBUG[10291] res_pjsip/pjsip_distributor.c: Taskprocessor overload alert: Ignoring 'Request msg REGISTER/cseq=15 (rdata0x7f8e7c000948)'.
[2023-03-20 12:09:02] DEBUG[10291] res_pjsip/pjsip_distributor.c: Taskprocessor overload alert: Ignoring 'Request msg REGISTER/cseq=15 (rdata0x7f8e7c000948)'.
[2023-03-20 12:09:03] DEBUG[10291] res_pjsip/pjsip_distributor.c: Taskprocessor overload alert: Ignoring 'Request msg INVITE/cseq=1 (rdata0x7f8e7c000948)'.
[2023-03-20 12:09:06] DEBUG[10291] res_pjsip/pjsip_distributor.c: Taskprocessor overload alert: Ignoring 'Request msg REGISTER/cseq=15 (rdata0x7f8e7c000948)'.
[2023-03-20 12:09:10] DEBUG[10291] res_pjsip/pjsip_distributor.c: Taskprocessor overload alert: Ignoring 'Request msg REGISTER/cseq=15 (rdata0x7f8e7c000948)'.
[2023-03-20 12:09:14] DEBUG[10291] res_pjsip/pjsip_distributor.c: Taskprocessor overload alert: Ignoring 'Request msg REGISTER/cseq=15 (rdata0x7f8e7c000948)'.
[2023-03-20 12:09:46] DEBUG[10291] res_pjsip/pjsip_distributor.c: Taskprocessor overload alert: Ignoring 'Request msg REGISTER/cseq=1 (rdata0x7f8e7c000948)'.
[2023-03-20 12:09:47] DEBUG[10291] res_pjsip/pjsip_distributor.c: Taskprocessor overload alert: Ignoring 'Request msg REGISTER/cseq=1 (rdata0x7f8e7c000948)'.
[2023-03-20 12:09:48] DEBUG[10291] res_pjsip/pjsip_distributor.c: Taskprocessor overload alert: Ignoring 'Request msg REGISTER/cseq=1 (rdata0x7f8e7c000948)'.
[2023-03-20 12:09:50] DEBUG[10291] res_pjsip/pjsip_distributor.c: Taskprocessor overload alert: Ignoring 'Request msg REGISTER/cseq=1 (rdata0x7f8e7c000948)'.
[2023-03-20 12:09:54] DEBUG[10291] res_pjsip/pjsip_distributor.c: Taskprocessor overload alert: Ignoring 'Request msg REGISTER/cseq=1 (rdata0x7f8e7c000948)'.
[2023-03-20 12:09:58] DEBUG[10291] res_pjsip/pjsip_distributor.c: Taskprocessor overload alert: Ignoring 'Request msg REGISTER/cseq=1 (rdata0x7f8e7c000948)'.
[2023-03-20 12:10:02] DEBUG[10291] res_pjsip/pjsip_distributor.c: Taskprocessor overload alert: Ignoring 'Request msg REGISTER/cseq=1 (rdata0x7f8e7c000948)'.
[2023-03-20 12:10:05] DEBUG[3175] taskprocessor.c: Taskprocessor 'Confbridge/1000-00000428' cleared the high water alert.
These were all local channels. Here’s the output of the the PJSIP task processors: PJSIP - core show taskprocessors - Pastebin.com
[root@yplab ~]# asterisk -x"core show taskprocessors" | grep 'Processor\|default\|distributor\|exten_state\|messaging\|pool'
Processor Processed In Queue Max Depth Low water High water
pjsip/default-0000000a 29 0 1 450 500
pjsip/default-0000000b 0 0 0 450 500
pjsip/default-0000000c 0 0 0 450 500
pjsip/default-0000000d 0 0 0 450 500
pjsip/default-0000000e 0 0 0 450 500
pjsip/default-0000000f 0 0 0 450 500
pjsip/default-00000010 0 0 0 450 500
pjsip/default-00000011 0 0 0 450 500
pjsip/distributor-000001fa 22 0 2 450 500
pjsip/distributor-000001fb 38 0 2 450 500
pjsip/distributor-000001fc 25 0 2 450 500
pjsip/distributor-000001fd 50 0 2 450 500
pjsip/distributor-000001fe 36 0 2 450 500
pjsip/distributor-000001ff 36 0 1 450 500
pjsip/distributor-00000200 38 0 2 450 500
pjsip/distributor-00000201 60 0 1 450 500
pjsip/distributor-00000202 28 0 1 450 500
pjsip/distributor-00000203 36 0 2 450 500
pjsip/distributor-00000204 28 0 1 450 500
pjsip/distributor-00000205 16 0 1 450 500
pjsip/distributor-00000206 36 0 2 450 500
pjsip/distributor-00000207 58 0 2 450 500
pjsip/distributor-00000208 36 0 2 450 500
pjsip/distributor-00000209 18 0 1 450 500
pjsip/distributor-0000020a 45 0 2 450 500
pjsip/distributor-0000020b 26 0 2 450 500
pjsip/distributor-0000020c 28 0 2 450 500
pjsip/distributor-0000020d 26 0 2 450 500
pjsip/distributor-0000020e 20264 0 2 450 500
pjsip/distributor-0000020f 52 0 2 450 500
pjsip/distributor-00000210 37 0 2 450 500
pjsip/distributor-00000211 42 0 2 450 500
pjsip/distributor-00000212 40 0 2 450 500
pjsip/distributor-00000213 28 0 1 450 500
pjsip/distributor-00000214 57 0 2 450 500
pjsip/distributor-00000215 22 0 1 450 500
pjsip/distributor-00000216 50 0 2 450 500
pjsip/distributor-00000217 26 0 1 450 500
pjsip/distributor-00000218 39 0 2 450 500
pjsip/exten_state 0 0 0 450 500
pjsip/messaging 0 0 0 450 500
pjsip/pool 122676 0 3 450 500
pjsip/pool-control 245740 0 5 450 500
sorcery/pool 10541 0 2 450 500
sorcery/pool-control 21443 0 2 450 500
stasis/pool 3008 0 2 450 500
stasis/pool-control 6068 0 3 450 500
It looks to me like all other PJSIP traskprocessors are endpoints and options.
What am I missing?
Then it would seem as though they were indeed ignored due to other taskprocessors, though the code is so simple I don’t see how[1]… unless something is fundamentally wrong or an assumption is incorrect. I’ve orchestrated the code to make sure that the configuration is read in correctly and applied, and it is. This isn’t possibly a case where FreePBX is changing it underneath you is it?
[1] asterisk/pjsip_distributor.c at 20 · asterisk/asterisk · GitHub
I’ve also put my instance into a permanent state of perceived overload, and with “none” or “pjsip_only” messages are not dropped. Setting it to “global” they do get dropped.
Tested on both 18 and 20.