Proper reload when pjsip and extensions are changed

vieridipaola · August 25, 2020, 11:31am

Hi,

I’m wondering what is the best way to reload SIP (pjsip) configuration when changing the file pjsip_wizard.conf, as well as extensions (dialplan).

I understand that “module reload” should be OK. Especially if:

module show like sip
Module                         Description                              Use Count  Status      Support Level
app_adsiprog.so                Asterisk ADSI Programming Application    0          Running        deprecated
chan_pjsip.so                  PJSIP Channel Driver                     0          Running              core
chan_sip.so                    Session Initiation Protocol (SIP)        0          Not Running      extended
func_pjsip_aor.so              Get information about a PJSIP AOR        0          Running              core
func_pjsip_contact.so          Get information about a PJSIP contact    0          Running              core
func_pjsip_endpoint.so         Get information about a PJSIP endpoint   0          Running              core
res_pjsip.so                   Basic SIP resource                       45         Running              core
res_pjsip_acl.so               PJSIP ACL Resource                       0          Running              core
res_pjsip_authenticator_digest.so PJSIP authentication resource            0          Running              core
res_pjsip_caller_id.so         PJSIP Caller ID Support                  1          Running              core
res_pjsip_config_wizard.so     PJSIP Config Wizard                      1          Running              core
res_pjsip_dialog_info_body_generator.so PJSIP Extension State Dialog Info+XML Pr 0          Running              core
res_pjsip_diversion.so         PJSIP Add Diversion Header Support       1          Running              core
res_pjsip_dlg_options.so       SIP OPTIONS in dialog handler            0          Running              core
res_pjsip_dtmf_info.so         PJSIP DTMF INFO Support                  0          Running              core
res_pjsip_empty_info.so        PJSIP Empty INFO Support                 0          Running              core
res_pjsip_endpoint_identifier_anonymous.so PJSIP Anonymous endpoint identifier      0          Running              core
res_pjsip_endpoint_identifier_ip.so PJSIP IP endpoint identifier             0          Running              core
res_pjsip_endpoint_identifier_user.so PJSIP username endpoint identifier       0          Running              core
res_pjsip_exten_state.so       PJSIP Extension State Notifications      0          Running              core
res_pjsip_header_funcs.so      PJSIP Header Functions                   0          Running              core
res_pjsip_history.so           PJSIP History                            0          Running          extended
res_pjsip_logger.so            PJSIP Packet Logger                      0          Running              core
res_pjsip_messaging.so         PJSIP Messaging Support                  0          Running              core
res_pjsip_mwi.so               PJSIP MWI resource                       0          Running              core
res_pjsip_mwi_body_generator.so PJSIP MWI resource                       0          Running              core
res_pjsip_nat.so               PJSIP NAT Support                        0          Running              core
res_pjsip_notify.so            CLI/AMI PJSIP NOTIFY Support             0          Not Running          core
res_pjsip_one_touch_record_info.so PJSIP INFO One Touch Recording Support   0          Running              core
res_pjsip_outbound_authenticator_digest.so PJSIP authentication resource            0          Running              core
res_pjsip_outbound_publish.so  PJSIP Outbound Publish Support           2          Running              core
res_pjsip_outbound_registration.so PJSIP Outbound Registration Support      0          Running              core
res_pjsip_path.so              PJSIP Path Header Support                0          Running              core
res_pjsip_phoneprov_provider.so PJSIP Phoneprov Provider                 0          Not Running      extended
res_pjsip_pidf_body_generator.so PJSIP Extension State PIDF Provider      0          Running              core
res_pjsip_pidf_digium_body_supplement.so PJSIP PIDF Digium presence supplement    0          Running              core
res_pjsip_pidf_eyebeam_body_supplement.so PJSIP PIDF Eyebeam supplement            0          Running              core
res_pjsip_publish_asterisk.so  PJSIP Asterisk Event PUBLISH Support     0          Running              core
res_pjsip_pubsub.so            PJSIP event resource                     12         Running              core
res_pjsip_refer.so             PJSIP Blind and Attended Transfer Suppor 1          Running              core
res_pjsip_registrar.so         PJSIP Registrar Support                  0          Running              core
res_pjsip_rfc3326.so           PJSIP RFC3326 Support                    0          Running              core
res_pjsip_sdp_rtp.so           PJSIP SDP RTP/AVP stream handler         0          Running              core
res_pjsip_send_to_voicemail.so PJSIP REFER Send to Voicemail Support    0          Running              core
res_pjsip_session.so           PJSIP Session resource                   17         Running              core
res_pjsip_sips_contact.so      UAC SIPS Contact support                 0          Running              core
res_pjsip_t38.so               PJSIP T.38 UDPTL Support                 0          Running              core
res_pjsip_transport_websocket.so PJSIP WebSocket Transport Support        0          Running              core
res_pjsip_xpidf_body_generator.so PJSIP Extension State PIDF Provider      0          Running              core
49 modules loaded

Well, maybe I still need to “dialplan reload” if I change something in extensions.conf, right?

However, after a “module reload” the new/updated pjsip account credentials are NOT loaded. SIP clients can only connect if I restart Asterisk completely. I suppose a pjsip reload might also work, but before doing so I would like to know why a “modules reload” isn’t enough.

Thanks

penguinpbx · September 4, 2020, 6:08pm

Yes.

Sadly, it is both not enough and too much at the same time!

I’ve found this to be a nice 1-2 punch to the PJSIP guts on some installs:

module reload res_pjsip.so
module reload res_pjsip_phoneprov_provider.so

vieridipaola · September 5, 2020, 1:21pm

Thanks, but I’m wondering how to solve the following problem.

I have a whole bunch of pjsip extensions configured withthe pjsip wizard. Now, I need to update (eg. change password) or add a new one. If I run the following command then it seems to work for the new/updated extension, but it also interrupts all the other extensions that may be online (eg. the webrtc clients get a websocket error and have to reconnect).

asterisk -rx "module reload res_pjsip.so"

How can I make effective an extension update or a new extension in pjsip WITHOUT disrupting the rest of the extensions?

I am using flat files, no realtime DB.

Thanks

penguinpbx · September 5, 2020, 9:15pm

Only WebRTC clients need reconnect ?

lordgurke · September 5, 2020, 10:21pm

I didn’t work that much with PJSIP — but is there no command like “pjsip reload”, like there is a “sip reload” with chan_sip?
That does not disrupt any calls.

vieridipaola · September 7, 2020, 7:44am

For now, they are moslty webrtc clients, yes. However, I might need to do the same for UDP SIP clients in the future.

There is no such command (“pjsip reload”). It must be something like this:
pjsip reload qualify {aor,endpoint} <id>

Please note that I’m using pjsip wizard, not pjsip – if that makes any difference, I don’t know.

vieridipaola · September 7, 2020, 8:00am

This is what happens when I try to reload the module:

*CLI> module reload res_pjsip.so
*CLI>
Disconnected from Asterisk server
Asterisk cleanly ending (0).
Executing last minute cleanups
Asterisk ending (0).

It doesn’t make any sense that the whole Asterisk PBX goes down and restarts just because I want to reload pjsip…

vieridipaola · September 12, 2020, 2:00pm

Whenever I run module reload res_pjsip.so or pjsip reload I get an Asterisk failure:

asterisk: Asterisk terminated with Signal: 6

It restarts automatically because I have a wrapper.
I guess it’s a bug.

david551 · September 12, 2020, 3:34pm

Signal 6 is Abort, which means the code has explicitly declared a crash. Normally this is the result of an assertion failure, i.e. the code has checked a condition which should be true, but wasn’t.

In that case, a message will have been written. Even if you aren’t capturing standard error, the message should be visible in the parameters in the backtrace from the bug.

Typical reasons for assertion failures are bugs relating to incorrect locking and more general memory corruption.

The normal guidance about retrying with the latest sub-version apply. NB many people misunderstand cert version and RC versions. The former are often not the most reliable versions and the latter should be assumed to be unsafe, unless you want to detect bugs early.

vieridipaola · September 14, 2020, 5:00am

I’m running Asterisk 16.13.0 which is the latest available in my distro. I’ve read it could be a segfault, as you say, which is bad… Before reporting this as a bug, I guess I’ll have to get a trace with gdb.
I don’t see any meaningful messages yet.
Thanks

jcolp · September 14, 2020, 9:53am

What distribution are you using? There’s been no reports of crashes from reloads in a long time. There is also a guide on the wiki[1] for collecting information.

[1] https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace

vieridipaola · September 23, 2020, 11:30pm

I’m using Gentoo Linux.

vieridipaola · September 24, 2020, 12:07am

I 'm running Asterisk with -g, but there are no core dump files in my system.

# ps -C asterisk u
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     26406  2.8  0.2 4305252 68220 pts/1   Sl   01:53   0:01 /usr/sbin/asterisk -C /etc/asterisk/asterisk.conf -f -g -U root -G asterisk

The process is either crashing or restarting because:

 # ps -C asterisk u ; asterisk -rx "module reload res_pjsip.so" ; sleep 10 ; ps -C asterisk u
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     26406  0.7  0.2 4305252 68224 pts/1   Sl   01:53   0:03 /usr/sbin/asterisk -C /etc/asterisk/asterisk.conf -f -g -U root -G asterisk
Asterisk ending (0).
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     27906 34.6  0.2 4304724 67604 pts/1   Sl   02:01   0:01 /usr/sbin/asterisk -C /etc/asterisk/asterisk.conf -f -g -U root -G asterisk

(the PID is different after reloading the module)

…and I’m supposing it’s crashing because ech time I reload the module as in the example above, I get this in /var/log/messages:

asterisk: Asterisk terminated with Signal: 6

So now I need to find out why a core dump is not generated here.

david551 · September 24, 2020, 9:38am

Signal 6 is “abort” which is where the the user space software explicitly declares that it has crashed. Usually that happens as the result of calling assert(), which is used to check the validity of some pre-condition. Assert failures in Asteirsk are normally due to memory corruption on incorrect locking.

You need to configure your system to create the core file, or to actually start the main Asterisk daemon under control of gdb, so you can see the message that assert is producing. It should also appear on standard error, in the context of the daemon.

vieridipaola · September 24, 2020, 1:02pm

This is what I see in gdb:

Thread 72 "asterisk" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffacc75700 (LWP 192586)]
0x00007ffff74b5881 in raise () from /lib64/libc.so.6

Then, I tried this:

(gdb) bt
#0  0x00007ffff74b5881 in raise () at /lib64/libc.so.6
#1  0x00007ffff749f55b in abort () at /lib64/libc.so.6
#2  0x00007ffff74f8118 in  () at /lib64/libc.so.6
#3  0x00007ffff74ff4ba in  () at /lib64/libc.so.6
#4  0x00007ffff7500ff5 in  () at /lib64/libc.so.6
#5  0x00007ffff4669a90 in add_extension
    (context=context@entry=0x5555570a4150, exten=exten@entry=0x7fff7c078565 "1001", priority=priority@entry=1, application=application@entry=0x7fff7c0774b1 "Gosub(default,${EXTEN},1(${HINT}))") at res_pjsip_config_wizard.c:478
#6  0x00007ffff4669c53 in add_hints
    (context=context@entry=0x7fff7c07741d "default", exten=exten@entry=0x7fff7c078565 "1001", application=application@entry=0x7fff7c0774b1 "Gosub(default,${EXTEN},1(${HINT}))", id=id@entry=0x7fff7c076ee0 "1001") at res_pjsip_config_wizard.c:521
#7  0x00007ffff466b89c in handle_endpoint (wiz=<optimized out>, otw=0x555556177640, sorcery=0x5555559fe310) at res_pjsip_config_wizard.c:723
#8  wizard_apply_handler (sorcery=sorcery@entry=0x5555559fe310, otw=otw@entry=0x555556177640, wiz=wiz@entry=0x7fff7c076ee0) at res_pjsip_config_wizard.c:1022
#9  0x00007ffff466c664 in object_type_loaded_observer (name=<optimized out>, sorcery=0x5555559fe310, object_type=0x555555c8a030 "endpoint", reloaded=<optimized out>)
    at res_pjsip_config_wizard.c:1116
#10 0x00005555556b4a80 in sorcery_object_load (obj=0x555555c8a030, arg=arg@entry=0x7fffacc74c30, flags=flags@entry=2) at sorcery.c:1357
#11 0x00005555555b9a7b in internal_ao2_traverse
    (self=0x5555559fe398, flags=flags@entry=OBJ_NODATA, cb_fn=cb_fn@entry=0x5555556b4660 <sorcery_object_load>, arg=arg@entry=0x7fffacc74c30, data=data@entry=0x0, type=type@entry=AO2_CALLBACK_DEFAULT, tag=0x555555754a20 "", file=0x5555557873d3 "sorcery.c", line=1415, func=0x555555787ea0 <__PRETTY_FUNCTION__.12112> "ast_sorcery_reload") at astobj2_container.c:328
#12 0x00005555555b9fec in __ao2_callback
    (c=<optimized out>, flags=flags@entry=OBJ_NODATA, cb_fn=cb_fn@entry=0x5555556b4660 <sorcery_object_load>, arg=arg@entry=0x7fffacc74c30, tag=tag@entry=0x555555754a20 "", file=file@entry=0x5555557873d3 "sorcery.c", line=1415, func=0x555555787ea0 <__PRETTY_FUNCTION__.12112> "ast_sorcery_reload") at astobj2_container.c:414
#13 0x00005555556b8334 in ast_sorcery_reload (sorcery=0x5555559fe310) at sorcery.c:1415
#14 0x00007ffff6753265 in ast_res_pjsip_reload_configuration () at res_pjsip/pjsip_configuration.c:2056
#15 0x00007ffff67308d9 in reload_configuration_task (obj=<optimized out>) at res_pjsip.c:5106
#16 0x00007ffff6730801 in sync_task (data=0x7fffacbf71f0) at res_pjsip.c:4623
#17 0x00005555556d87e2 in ast_taskprocessor_execute (tps=tps@entry=0x55555617eb50) at taskprocessor.c:1237
#18 0x00005555556e0400 in execute_tasks (data=0x55555617eb50) at threadpool.c:1356
#19 0x00005555556d87e2 in ast_taskprocessor_execute (tps=0x55555617b2c0) at taskprocessor.c:1237
#20 0x00005555556df7f4 in threadpool_execute (pool=0x55555617a880) at threadpool.c:367
#21 worker_active (worker=0x7fff84004b20) at threadpool.c:1137
#22 worker_start (arg=arg@entry=0x7fff84004b20) at threadpool.c:1056
#23 0x00005555556e6eeb in dummy_start (data=<optimized out>) at utils.c:1249
#24 0x00007ffff77c7f27 in start_thread () at /lib64/libpthread.so.0
#25 0x00007ffff7576bef in clone () at /lib64/libc.so.6

david551 · September 24, 2020, 1:38pm

You seem to be missing some of the debugging information for libc; look for a corresponding debug package.

Also, Asterisk is almost impossible to debug if built optimized, as is the case here. All the useful information ends up as “optimized out”.

vieridipaola · September 24, 2020, 4:22pm

Well, that’s really odd because according to my distro, Asterisk has been built with DONT_OPTIMIZE DEBUG_THREADS BETTER_BACKTRACES.
I’ll have to report it on Gentoo or take a better look at it.

vieridipaola · September 25, 2020, 10:24am

Until I get the debugging issues sorted out, the only thing I see on stderr is:

Reloading module 'res_pjsip.so' (Basic SIP resource)
double free or corruption (fasttop)
Aborted

vieridipaola · October 4, 2020, 5:46pm

In case someone else has the same problem this is actually a bug in Asterisk which is triggered only in certain conditions. It seems to have been fixed now according to JIRA.

system · November 3, 2020, 5:46pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.