Asterisk 1.6 hangs occasionally

Hi there,

my Asterisk is hanging up from time to time - sometimes it works for hours, sometimes ten minutes.
It simple stops reacting on incoming and outgoing calls, the CLI does not log anything, sip show peers works until some reload (dialplan reload for example) takes place, after that, the CLI does not execute any command.

Make sure that you are running the latest version.

Make sure that it is compiled with no optimisation and with lock debugging enabled.

The next time it stalls, atempt to start a new CLI process and run “core show locks”.

Then use gcore to get a core dump.

Create backtraces as described in doc/backtrace.txt.

Search issues.asterisk.org to see if there is a known bug that matches the details of yours.

Otherwise, create a new issue on issues.asterisk.org, attaching the information obtained above, and providing any clues as to what you were doing at the time, or what may be unusual about your configuration, trying to find a subject that better describes the circumstances, possibly using the core show locks output as a guide.

Hi there,

Originally I used an AsteriskNow installation, but when the need for core debugging came up, I decided to compile my own * (from the latest 1.6 branch) and install it last night.
Today the deadlock happened again when I was at the customers site.
They’d have killed me literally if I’d started to debug the * while they can’t use the phone, but at least I captured a core show locks.
I dare to post this longish dump here, maybe one of you with more experience than me have an idea where the problem might be. Actually, it happened twice, and both dumps looked the same (same locks, but in different order).

The locking cannot be reproduced by any means I know of, sometimes it happens twice within one hour, sometimes it does not occur within two days of heave usage.

=======================================================================
=== Currently Held Locks ==============================================

===
=== <lock#> (): (times locked)

=== Thread ID: 1081084224 (do_monitor started at [22650] chan_sip.c restart_monitor())
=== —> Lock #0 (chan_sip.c): MUTEX 22100 handle_request_do &netlock 0x2aaab9e51760 (1)
/usr/sbin/asterisk(ast_bt_get_addresses+0x1a) [0x4d7b8b]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9bbbc9c]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1dbc0]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1d981]
/usr/sbin/asterisk(ast_io_wait+0x1ba) [0x4cc6d0]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1f626]
/usr/sbin/asterisk [0x55bdc9]
/lib64/libpthread.so.0 [0x3cd440673d]
/lib64/libc.so.6(clone+0x6d) [0x3cd3cd3f6d]
=== —> Lock #1 (chan_sip.c): MUTEX 7499 find_call sip_pvt_ptr 0x16cf8a10 (1)
/usr/sbin/asterisk(ast_bt_get_addresses+0x1a) [0x4d7b8b]
/usr/sbin/asterisk [0x4407e9]
/usr/sbin/asterisk(_ao2_lock+0x53) [0x440b37]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9bd2923]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1dbd4]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1d981]
/usr/sbin/asterisk(ast_io_wait+0x1ba) [0x4cc6d0]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1f626]
/usr/sbin/asterisk [0x55bdc9]
/lib64/libpthread.so.0 [0x3cd440673d]
/lib64/libc.so.6(clone+0x6d) [0x3cd3cd3f6d]
=== —> Lock #2 (pbx.c): MUTEX 9352 ast_rdlock_contexts &conlock 0x822c80 (1)
/usr/sbin/asterisk(ast_bt_get_addresses+0x1a) [0x4d7b8b]
/usr/sbin/asterisk [0x4f12a2]
/usr/sbin/asterisk(ast_rdlock_contexts+0x23) [0x509072]
/usr/sbin/asterisk [0x4f8fac]
/usr/sbin/asterisk(ast_add_extension+0x2e) [0x50209f]
/usr/sbin/asterisk(ast_extension_state_add+0x291) [0x4f6347]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1b718]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1d07d]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1e0fb]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1d981]
/usr/sbin/asterisk(ast_io_wait+0x1ba) [0x4cc6d0]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1f626]
/usr/sbin/asterisk [0x55bdc9]
/lib64/libpthread.so.0 [0x3cd440673d]
/lib64/libc.so.6(clone+0x6d) [0x3cd3cd3f6d]
=== —> Lock #3 (pbx.c): WRLOCK 9365 ast_wrlock_context &con->lock 0x16d796a0 (1)
/usr/sbin/asterisk(ast_bt_get_addresses+0x1a) [0x4d7b8b]
/usr/sbin/asterisk [0x4f352f]
/usr/sbin/asterisk(ast_wrlock_context+0x2a) [0x5090c3]
/usr/sbin/asterisk [0x50323b]
/usr/sbin/asterisk(ast_add_extension2+0x7e) [0x502d5f]
/usr/sbin/asterisk(ast_add_extension+0x89) [0x5020fa]
/usr/sbin/asterisk(ast_extension_state_add+0x291) [0x4f6347]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1b718]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1d07d]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1e0fb]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1d981]
/usr/sbin/asterisk(ast_io_wait+0x1ba) [0x4cc6d0]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1f626]
/usr/sbin/asterisk [0x55bdc9]
/lib64/libpthread.so.0 [0x3cd440673d]
/lib64/libc.so.6(clone+0x6d) [0x3cd3cd3f6d]
=== —> Waiting for Lock #4 (pbx.c): WRLOCK 4080 ast_add_hint &(&hints)->lock 0x825070 (1)
/usr/sbin/asterisk(ast_bt_get_addresses+0x1a) [0x4d7b8b]
/usr/sbin/asterisk [0x4f352f]
/usr/sbin/asterisk [0x4f69ce]
/usr/sbin/asterisk [0x502cca]
/usr/sbin/asterisk [0x5026fc]
/usr/sbin/asterisk [0x5033e0]
/usr/sbin/asterisk(ast_add_extension2+0x7e) [0x502d5f]
/usr/sbin/asterisk(ast_add_extension+0x89) [0x5020fa]
/usr/sbin/asterisk(ast_extension_state_add+0x291) [0x4f6347]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1b718]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1d07d]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1e0fb]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1d981]
/usr/sbin/asterisk(ast_io_wait+0x1ba) [0x4cc6d0]
/usr/lib/asterisk/modules/chan_sip.so [0x2aaab9c1f626]
/usr/sbin/asterisk [0x55bdc9]
/lib64/libpthread.so.0 [0x3cd440673d]
/lib64/libc.so.6(clone+0x6d) [0x3cd3cd3f6d]
=== -------------------------------------------------------------------

=======================================================================