Asterisk crash more than 8x per day

Asterisk Asterisk Support
1

Hello,

Asterisk crash 8 to 10 times a day since approx 1 month. I follow the procedure to get a backtrace but I’m not very able to understand it.
I know that my version is very old (1.2.35) but used only for voicemail. Before this month, the problem occured only 1 time a month and was supportable.

Someone can have a look?:

code bt
#0 0x003e6300 in pthread_mutex_lock () from /lib/libpthread.so.0
#1 0x0060e1d7 in expire_register (data=0xb62b4dc0) at …/include/asterisk/lock.h:601
#2 0x080564e2 in ast_sched_runq (con=0x8203888) at sched.c:373
#3 0x0060e8f8 in do_monitor (data=0x0) at chan_sip.c:11867
#4 0x003e449b in start_thread () from /lib/libpthread.so.0
#5 0x0033b42e in clone () from /lib/libc.so.6
(gdb) bt full
#0 0x003e6300 in pthread_mutex_lock () from /lib/libpthread.so.0
No symbol table info available.
#1 0x0060e1d7 in expire_register (data=0xb62b4dc0) at …/include/asterisk/lock.h:601
next =
found = (struct sip_peer *) 0x0
prev =
peer =
PRETTY_FUNCTION = “expire_register”
#2 0x080564e2 in ast_sched_runq (con=0x8203888) at sched.c:373
x = 3
res = 1352808740
#3 0x0060e8f8 in do_monitor (data=0x0) at chan_sip.c:11867
res = 1
sip =
t = 1352808736
fastrestart = 1
lastpeernum = 4219
curpeernum = 4219
reloading =
PRETTY_FUNCTION = “do_monitor”
#4 0x003e449b in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#5 0x0033b42e in clone () from /lib/libc.so.6
No symbol table info available.
(gdb) thread apply all bt

Thread 1 (process 23935):
#0 0x003e6300 in pthread_mutex_lock () from /lib/libpthread.so.0
#1 0x0060e1d7 in expire_register (data=0xb62b4dc0) at …/include/asterisk/lock.h:601
#2 0x080564e2 in ast_sched_runq (con=0x8203888) at sched.c:373
#3 0x0060e8f8 in do_monitor (data=0x0) at chan_sip.c:11867
#4 0x003e449b in start_thread () from /lib/libpthread.so.0
#5 0x0033b42e in clone () from /lib/libc.so.6
(gdb) [/code]

2

That backtrace is only somewhat usuable. To have a usable backtrace, asterisk needs to be compiled with DONT_OPTIMIZE flag set in the compiler flags section when you do the make menuselect portion of the build.

From what the backtrace does show, it seems to be caused during an SIP device’s registration expiring. It appears to be peer 4219, in this particular crash. Do you have any other backtraces to post?

What has changed on your network in the last month that could cause these issues? Are your sip devices on a lan, over the internet, over a vpn?

Have you run memtest86 to see if your RAM is failing?

3

Hi,

Thanks for your reply. We don’t have done change within our network for the last few months. But we have many new users. So, one could cause these crashes.
I’m wondering if I can find somewhere the SIP device behind peer 4219? Does this information is in CDRs?

4

What info are you looking for from peer 4219?

sip show peer 4219 will tell you what the ip address of the device is and its useragent string that will tell you what kind of device / softphone version.

cat /var/log/asterisk/messages | grep "sip/4219" will show what interactions peer has had.

That particular device may not be the cause anyway. Have you reviewed any of the other backtraces to see if the are related to expire_register in lock.h?

5

I don’t find the peer in the logs and currently he is not connected or behind a new peer number.
I will backtrace the coredump this morning.

6

Here is the output of the backtrace done on the coredump today:

[code]Program terminated with signal 11, Segmentation fault.
#0 0x003e6300 in pthread_mutex_lock () from /lib/libpthread.so.0
(gdb) bt
#0 0x003e6300 in pthread_mutex_lock () from /lib/libpthread.so.0
#1 0x001ed1d7 in expire_register (data=0xaf032090) at …/include/asterisk/lock.h:601
#2 0x080564e2 in ast_sched_runq (con=0x95d1150) at sched.c:373
#3 0x001ed8f8 in do_monitor (data=0x0) at chan_sip.c:11867
#4 0x003e449b in start_thread () from /lib/libpthread.so.0
#5 0x0033b42e in clone () from /lib/libc.so.6
(gdb) bt full
#0 0x003e6300 in pthread_mutex_lock () from /lib/libpthread.so.0
No symbol table info available.
#1 0x001ed1d7 in expire_register (data=0xaf032090) at …/include/asterisk/lock.h:601
next =
found = (struct sip_peer *) 0x0
prev =
peer =
PRETTY_FUNCTION = “expire_register”
#2 0x080564e2 in ast_sched_runq (con=0x95d1150) at sched.c:373
x = 6
res = 0
#3 0x001ed8f8 in do_monitor (data=0x0) at chan_sip.c:11867
res = 1
sip =
t = 1353303426
fastrestart = 1
lastpeernum = 7044
curpeernum = 7044
reloading =
PRETTY_FUNCTION = “do_monitor”
#4 0x003e449b in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#5 0x0033b42e in clone () from /lib/libc.so.6
No symbol table info available.
(gdb) thread apply all bt

Thread 1 (process 747):
#0 0x003e6300 in pthread_mutex_lock () from /lib/libpthread.so.0
#1 0x001ed1d7 in expire_register (data=0xaf032090) at …/include/asterisk/lock.h:601
#2 0x080564e2 in ast_sched_runq (con=0x95d1150) at sched.c:373
#3 0x001ed8f8 in do_monitor (data=0x0) at chan_sip.c:11867
#4 0x003e449b in start_thread () from /lib/libpthread.so.0
#5 0x0033b42e in clone () from /lib/libc.so.6[/code]

It looks to be in expire too.

7

To complete, I see that just 3 minutes before crashing, queue_log has been written with the following:
1353303602|NONE|NONE|NONE|QUEUESTART|

I’m surprise because we don’t need queue and I don’t see in extensions.conf anything that ask a queue start.

8

Hi,

I back on that topic because were are still experiencing the problem. I recompiled with ‘dont-optimize’ flag and produced the following backtrace:

[code]GNU gdb Red Hat Linux (6.5-37.el5_2.2rh)
Copyright © 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type “show copying” to see the conditions.
There is absolutely no warranty for GDB. Type “show warranty” for details.
This GDB was configured as “i386-redhat-linux-gnu”…Using host libthread_db library “/lib/libthread_db.so.1”.

Reading symbols from /lib/libdl.so.2…done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libpthread.so.0…done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /usr/lib/libncurses.so.5…done.
Loaded symbols for /usr/lib/libncurses.so.5
Reading symbols from /lib/libm.so.6…done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libresolv.so.2…done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libssl.so.6…done.
Loaded symbols for /lib/libssl.so.6
Reading symbols from /lib/libc.so.6…done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2…done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/lib/libgssapi_krb5.so.2…done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/libkrb5.so.3…done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /lib/libcom_err.so.2…done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libk5crypto.so.3…done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /lib/libcrypto.so.6…done.
Loaded symbols for /lib/libcrypto.so.6
Reading symbols from /usr/lib/libz.so.1…done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /usr/lib/libkrb5support.so.0…done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1…done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /lib/libselinux.so.1…done.
Loaded symbols for /lib/libselinux.so.1
Reading symbols from /lib/libsepol.so.1…done.
Loaded symbols for /lib/libsepol.so.1
Reading symbols from /usr/lib/asterisk/modules/format_wav.so…done.
Loaded symbols for /usr/lib/asterisk/modules/format_wav.so
Reading symbols from /usr/lib/asterisk/modules/format_pcm.so…done.
Loaded symbols for /usr/lib/asterisk/modules/format_pcm.so
Reading symbols from /usr/lib/asterisk/modules/format_au.so…done.
Loaded symbols for /usr/lib/asterisk/modules/format_au.so
Reading symbols from /usr/lib/asterisk/modules/format_mp3.so…done.
Loaded symbols for /usr/lib/asterisk/modules/format_mp3.so
Reading symbols from /usr/lib/asterisk/modules/res_musiconhold.so…done.
Loaded symbols for /usr/lib/asterisk/modules/res_musiconhold.so
Reading symbols from /usr/lib/asterisk/modules/res_monitor.so…done.
Loaded symbols for /usr/lib/asterisk/modules/res_monitor.so
Reading symbols from /usr/lib/asterisk/modules/res_config_mysql.so…done.
Loaded symbols for /usr/lib/asterisk/modules/res_config_mysql.so
Reading symbols from /usr/lib/mysql/libmysqlclient.so.15…done.
Loaded symbols for /usr/lib/mysql/libmysqlclient.so.15
Reading symbols from /lib/libcrypt.so.1…done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libnsl.so.1…done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libnss_files.so.2…done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /usr/lib/asterisk/modules/res_agi.so…done.
Loaded symbols for /usr/lib/asterisk/modules/res_agi.so
Reading symbols from /usr/lib/asterisk/modules/res_crypto.so…done.
Loaded symbols for /usr/lib/asterisk/modules/res_crypto.so
Reading symbols from /usr/lib/asterisk/modules/res_features.so…done.
Loaded symbols for /usr/lib/asterisk/modules/res_features.so
Reading symbols from /usr/lib/asterisk/modules/res_indications.so…done.
Loaded symbols for /usr/lib/asterisk/modules/res_indications.so
Reading symbols from /usr/lib/asterisk/modules/res_adsi.so…done.
Loaded symbols for /usr/lib/asterisk/modules/res_adsi.so
Reading symbols from /usr/lib/asterisk/modules/pbx_functions.so…done.
Loaded symbols for /usr/lib/asterisk/modules/pbx_functions.so
Reading symbols from /usr/lib/asterisk/modules/pbx_loopback.so…done.
Loaded symbols for /usr/lib/asterisk/modules/pbx_loopback.so
Reading symbols from /usr/lib/asterisk/modules/pbx_config.so…done.
Loaded symbols for /usr/lib/asterisk/modules/pbx_config.so
Reading symbols from /usr/lib/asterisk/modules/pbx_spool.so…done.
Loaded symbols for /usr/lib/asterisk/modules/pbx_spool.so
Reading symbols from /usr/lib/asterisk/modules/pbx_dundi.so…done.
Loaded symbols for /usr/lib/asterisk/modules/pbx_dundi.so
Reading symbols from /usr/lib/asterisk/modules/pbx_ael.so…done.
Loaded symbols for /usr/lib/asterisk/modules/pbx_ael.so
Reading symbols from /usr/lib/asterisk/modules/pbx_realtime.so…done.
Loaded symbols for /usr/lib/asterisk/modules/pbx_realtime.so
Reading symbols from /usr/lib/asterisk/modules/chan_phone.so…done.
Loaded symbols for /usr/lib/asterisk/modules/chan_phone.so
Reading symbols from /usr/lib/asterisk/modules/chan_features.so…done.
Loaded symbols for /usr/lib/asterisk/modules/chan_features.so
Reading symbols from /usr/lib/asterisk/modules/chan_sip.so…done.
Loaded symbols for /usr/lib/asterisk/modules/chan_sip.so
Reading symbols from /usr/lib/asterisk/modules/chan_skinny.so…done.
Loaded symbols for /usr/lib/asterisk/modules/chan_skinny.so
Reading symbols from /usr/lib/asterisk/modules/chan_mgcp.so…done.
Loaded symbols for /usr/lib/asterisk/modules/chan_mgcp.so
Reading symbols from /usr/lib/asterisk/modules/chan_iax2.so…done.
Loaded symbols for /usr/lib/asterisk/modules/chan_iax2.so
Reading symbols from /usr/lib/asterisk/modules/chan_local.so…done.
Loaded symbols for /usr/lib/asterisk/modules/chan_local.so
Reading symbols from /usr/lib/asterisk/modules/chan_agent.so…done.
Loaded symbols for /usr/lib/asterisk/modules/chan_agent.so
Reading symbols from /usr/lib/asterisk/modules/format_sln.so…done.
Loaded symbols for /usr/lib/asterisk/modules/format_sln.so
Reading symbols from /usr/lib/asterisk/modules/app_directory.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_directory.so
Reading symbols from /usr/lib/asterisk/modules/app_nbscat.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_nbscat.so
Reading symbols from /usr/lib/asterisk/modules/app_mixmonitor.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_mixmonitor.so
Reading symbols from /usr/lib/asterisk/modules/app_adsiprog.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_adsiprog.so
Reading symbols from /usr/lib/asterisk/modules/app_lookupblacklist.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_lookupblacklist.so
Reading symbols from /usr/lib/asterisk/modules/app_cdr.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_cdr.so
Reading symbols from /usr/lib/asterisk/modules/app_setcidnum.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_setcidnum.so
Reading symbols from /usr/lib/asterisk/modules/format_g726.so…done.
Loaded symbols for /usr/lib/asterisk/modules/format_g726.so
Reading symbols from /usr/lib/asterisk/modules/app_festival.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_festival.so
Reading symbols from /usr/lib/asterisk/modules/app_while.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_while.so
Reading symbols from /usr/lib/asterisk/modules/func_enum.so…done.
Loaded symbols for /usr/lib/asterisk/modules/func_enum.so
Reading symbols from /usr/lib/asterisk/modules/format_wav_gsm.so…done.
Loaded symbols for /usr/lib/asterisk/modules/format_wav_gsm.so
Reading symbols from /usr/lib/asterisk/modules/app_voicemail.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_voicemail.so
Reading symbols from /usr/lib/asterisk/modules/app_txtcidname.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_txtcidname.so
Reading symbols from /usr/lib/asterisk/modules/codec_a_mu.so…done.
Loaded symbols for /usr/lib/asterisk/modules/codec_a_mu.so
Reading symbols from /usr/lib/asterisk/modules/app_image.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_image.so
Reading symbols from /usr/lib/asterisk/modules/format_g723.so…done.
Loaded symbols for /usr/lib/asterisk/modules/format_g723.so
Reading symbols from /usr/lib/asterisk/modules/app_dumpchan.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_dumpchan.so
Reading symbols from /usr/lib/asterisk/modules/app_zapateller.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_zapateller.so
Reading symbols from /usr/lib/asterisk/modules/app_macro.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_macro.so
Reading symbols from /usr/lib/asterisk/modules/cdr_csv.so…done.
Loaded symbols for /usr/lib/asterisk/modules/cdr_csv.so
Reading symbols from /usr/lib/asterisk/modules/app_setcallerid.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_setcallerid.so
Reading symbols from /usr/lib/asterisk/modules/app_saycountpl.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_saycountpl.so
Reading symbols from /usr/lib/asterisk/modules/app_lookupcidname.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_lookupcidname.so
Reading symbols from /usr/lib/asterisk/modules/app_stack.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_stack.so
Reading symbols from /usr/lib/asterisk/modules/app_url.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_url.so
Reading symbols from /usr/lib/asterisk/modules/app_sms.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_sms.so
Reading symbols from /usr/lib/asterisk/modules/app_waitforring.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_waitforring.so
Reading symbols from /usr/lib/asterisk/modules/app_chanisavail.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_chanisavail.so
Reading symbols from /usr/lib/asterisk/modules/cdr_custom.so…done.
Loaded symbols for /usr/lib/asterisk/modules/cdr_custom.so
Reading symbols from /usr/lib/asterisk/modules/app_db.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_db.so
Reading symbols from /usr/lib/asterisk/modules/app_hasnewvoicemail.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_hasnewvoicemail.so
Reading symbols from /usr/lib/asterisk/modules/app_md5.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_md5.so
Reading symbols from /usr/lib/asterisk/modules/app_waitforsilence.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_waitforsilence.so
Reading symbols from /usr/lib/asterisk/modules/app_system.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_system.so
Reading symbols from /usr/lib/asterisk/modules/codec_adpcm.so…done.
Loaded symbols for /usr/lib/asterisk/modules/codec_adpcm.so
Reading symbols from /usr/lib/asterisk/modules/app_dial.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_dial.so
Reading symbols from /usr/lib/asterisk/modules/app_privacy.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_privacy.so
Reading symbols from /usr/lib/asterisk/modules/app_eval.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_eval.so
Reading symbols from /usr/lib/asterisk/modules/app_chanspy.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_chanspy.so
Reading symbols from /usr/lib/asterisk/modules/format_gsm.so…done.
Loaded symbols for /usr/lib/asterisk/modules/format_gsm.so
Reading symbols from /usr/lib/asterisk/modules/app_alarmreceiver.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_alarmreceiver.so
Reading symbols from /usr/lib/asterisk/modules/app_record.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_record.so
Reading symbols from /usr/lib/asterisk/modules/app_controlplayback.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_controlplayback.so
Reading symbols from /usr/lib/asterisk/modules/app_random.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_random.so
Reading symbols from /usr/lib/asterisk/modules/app_groupcount.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_groupcount.so
Reading symbols from /usr/lib/asterisk/modules/app_sayunixtime.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_sayunixtime.so
Reading symbols from /usr/lib/asterisk/modules/func_callerid.so…done.
Loaded symbols for /usr/lib/asterisk/modules/func_callerid.so
Reading symbols from /usr/lib/asterisk/modules/format_g729.so…done.
Loaded symbols for /usr/lib/asterisk/modules/format_g729.so
Reading symbols from /usr/lib/asterisk/modules/format_vox.so…done.
Loaded symbols for /usr/lib/asterisk/modules/format_vox.so
Reading symbols from /usr/lib/asterisk/modules/app_setrdnis.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_setrdnis.so
Reading symbols from /usr/lib/asterisk/modules/app_parkandannounce.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_parkandannounce.so
Reading symbols from /usr/lib/asterisk/modules/app_disa.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_disa.so
Reading symbols from /usr/lib/asterisk/modules/format_h263.so…done.
Loaded symbols for /usr/lib/asterisk/modules/format_h263.so
Reading symbols from /usr/lib/asterisk/modules/app_verbose.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_verbose.so
Reading symbols from /usr/lib/asterisk/modules/cdr_addon_mysql.so…done.
Loaded symbols for /usr/lib/asterisk/modules/cdr_addon_mysql.so
Reading symbols from /usr/lib/asterisk/modules/app_forkcdr.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_forkcdr.so
Reading symbols from /usr/lib/asterisk/modules/app_settransfercapability.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_settransfercapability.so
Reading symbols from /usr/lib/asterisk/modules/app_ices.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_ices.so
Reading symbols from /usr/lib/asterisk/modules/app_mp3.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_mp3.so
Reading symbols from /usr/lib/asterisk/modules/func_uri.so…done.
Loaded symbols for /usr/lib/asterisk/modules/func_uri.so
Reading symbols from /usr/lib/asterisk/modules/cdr_manager.so…done.
Loaded symbols for /usr/lib/asterisk/modules/cdr_manager.so
Reading symbols from /usr/lib/asterisk/modules/app_transfer.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_transfer.so
Reading symbols from /usr/lib/asterisk/modules/app_setcdruserfield.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_setcdruserfield.so
Reading symbols from /usr/lib/asterisk/modules/app_sendtext.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_sendtext.so
Reading symbols from /usr/lib/asterisk/modules/app_authenticate.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_authenticate.so
Reading symbols from /usr/lib/asterisk/modules/codec_alaw.so…done.
Loaded symbols for /usr/lib/asterisk/modules/codec_alaw.so
Reading symbols from /usr/lib/asterisk/modules/app_getcpeid.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_getcpeid.so
Reading symbols from /usr/lib/asterisk/modules/app_exec.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_exec.so
Reading symbols from /usr/lib/asterisk/modules/format_jpeg.so…done.
Loaded symbols for /usr/lib/asterisk/modules/format_jpeg.so
Reading symbols from /usr/lib/asterisk/modules/format_pcm_alaw.so…done.
Loaded symbols for /usr/lib/asterisk/modules/format_pcm_alaw.so
Reading symbols from /usr/lib/asterisk/modules/app_directed_pickup.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_directed_pickup.so
Reading symbols from /usr/lib/asterisk/modules/app_echo.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_echo.so
Reading symbols from /usr/lib/asterisk/modules/app_dictate.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_dictate.so
Reading symbols from /usr/lib/asterisk/modules/app_readfile.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_readfile.so
Reading symbols from /usr/lib/asterisk/modules/app_softhangup.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_softhangup.so
Reading symbols from /usr/lib/asterisk/modules/app_milliwatt.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_milliwatt.so
Reading symbols from /usr/lib/asterisk/modules/app_externalivr.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_externalivr.so
Reading symbols from /usr/lib/asterisk/modules/codec_lpc10.so…done.
Loaded symbols for /usr/lib/asterisk/modules/codec_lpc10.so
Reading symbols from /usr/lib/asterisk/modules/app_userevent.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_userevent.so
Reading symbols from /usr/lib/asterisk/modules/app_test.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_test.so
Reading symbols from /usr/lib/asterisk/modules/format_ilbc.so…done.
Loaded symbols for /usr/lib/asterisk/modules/format_ilbc.so
Reading symbols from /usr/lib/asterisk/modules/app_playback.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_playback.so
Reading symbols from /usr/lib/asterisk/modules/app_cut.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_cut.so
Reading symbols from /usr/lib/asterisk/modules/app_addon_sql_mysql.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_addon_sql_mysql.so
Reading symbols from /usr/lib/asterisk/modules/codec_ulaw.so…done.
Loaded symbols for /usr/lib/asterisk/modules/codec_ulaw.so
Reading symbols from /usr/lib/asterisk/modules/app_enumlookup.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_enumlookup.so
Reading symbols from /usr/lib/asterisk/modules/app_math.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_math.so
Reading symbols from /usr/lib/asterisk/modules/app_setcidname.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_setcidname.so
Reading symbols from /usr/lib/asterisk/modules/codec_g726.so…done.
Loaded symbols for /usr/lib/asterisk/modules/codec_g726.so
Reading symbols from /usr/lib/asterisk/modules/app_realtime.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_realtime.so
Reading symbols from /usr/lib/asterisk/modules/app_talkdetect.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_talkdetect.so
Reading symbols from /usr/lib/asterisk/modules/app_queue.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_queue.so
Reading symbols from /usr/lib/asterisk/modules/app_read.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_read.so
Reading symbols from /usr/lib/asterisk/modules/codec_gsm.so…done.
Loaded symbols for /usr/lib/asterisk/modules/codec_gsm.so
Reading symbols from /usr/lib/asterisk/modules/app_senddtmf.so…done.
Loaded symbols for /usr/lib/asterisk/modules/app_senddtmf.so
Reading symbols from /lib/libgcc_s.so.1…done.
Loaded symbols for /lib/libgcc_s.so.1
Core was generated by `asterisk -g’.
Program terminated with signal 11, Segmentation fault.
#0 0x003e6300 in pthread_mutex_lock () from /lib/libpthread.so.0
(gdb) bt
#0 0x003e6300 in pthread_mutex_lock () from /lib/libpthread.so.0
#1 0x00f41495 in ast_mutex_lock (pmutex=0x5c) at …/include/asterisk/lock.h:601
#2 0x00f5373a in expire_register (data=0xb16e7260) at chan_sip.c:5907
#3 0x0805697e in ast_sched_runq (con=0x998b130) at sched.c:373
#4 0x00f6b94d in do_monitor (data=0x0) at chan_sip.c:11867
#5 0x003e449b in start_thread () from /lib/libpthread.so.0
#6 0x0033b42e in clone () from /lib/libc.so.6
(gdb) bt full
#0 0x003e6300 in pthread_mutex_lock () from /lib/libpthread.so.0
No symbol table info available.
#1 0x00f41495 in ast_mutex_lock (pmutex=0x5c) at …/include/asterisk/lock.h:601
No locals.
#2 0x00f5373a in expire_register (data=0xb16e7260) at chan_sip.c:5907
newcount = 0
peer = (struct sip_peer *) 0x0
PRETTY_FUNCTION = “expire_register”
#3 0x0805697e in ast_sched_runq (con=0x998b130) at sched.c:373
current = (struct sched *) 0xb6995378
tv = {tv_sec = 1360843585, tv_usec = 716996}
x = 3
res = 0
#4 0x00f6b94d in do_monitor (data=0x0) at chan_sip.c:11867
res = 1
sip = (struct sip_pvt *) 0xb6449640
peer = (struct sip_peer *) 0x0
t = 1360843585
fastrestart = 1
lastpeernum = 5030
curpeernum = 5031
reloading = 0
PRETTY_FUNCTION = “do_monitor”
#5 0x003e449b in start_thread () from /lib/libpthread.so.0
No symbol table info available.
#6 0x0033b42e in clone () from /lib/libc.so.6
No symbol table info available.
(gdb) thread apply all bt

Thread 1 (process 32340):
#0 0x003e6300 in pthread_mutex_lock () from /lib/libpthread.so.0
#1 0x00f41495 in ast_mutex_lock (pmutex=0x5c) at …/include/asterisk/lock.h:601
#2 0x00f5373a in expire_register (data=0xb16e7260) at chan_sip.c:5907
#3 0x0805697e in ast_sched_runq (con=0x998b130) at sched.c:373
#4 0x00f6b94d in do_monitor (data=0x0) at chan_sip.c:11867
#5 0x003e449b in start_thread () from /lib/libpthread.so.0
#6 0x0033b42e in clone () from /lib/libc.so.6
(gdb) quit[/code]

Is it more eloquent than the previous done before? Thanks in advance for any support/suggestions.

9

This looks like a problem with failing to lock correctly, leading to memory corruption. Such problems are difficult to debug and tend to crash some time after the actual fault.

Enabling thread debugging does have some performance impact, but may give you an error message closer to the primary errror.

10

Hello David,

Thank you for your feedback. I prefer to find the problem and have a couple of minutes with bad performance.
How can I enable thread debbuging?

11

It is another compiler option.

12

Ok. I done the previous compilation with “make dont-optimize”.
I can’t do “make menuselect”, there is no target for this.

How can I add this option?

13

I had missed the age of the version. You can’t add it to that version!

14

I already have these options in the Makefile:
DEBUG_THREADS = #-DDUMP_SCHEDULER #-DDEBUG_SCHEDULER #-DDEBUG_THREADS #-DDO_CRASH #-DDETECT_DEADLOCKS

So, I guess that Asterisk is already compile with the good debug options. Isn’t ?

15

David,

I’m afraid that these old debug on the Makefile could cause instability. I deactivate these (DEBUG_THREADS, DONT_OPTIMIZE and MALLOC_DEBUG).
I will see for a couple of hours if Asterisk goes more stable.

In addition of these crashes, I see that, sometimes, Asterisk doesn’t reply to an INVITE within the next 5 seconds, cause our proxy to CANCEL the call. Is there a limitation in “chan_sip” that could cause this behaviour? Should I authorize more thread or something like that ?

16

I don’t know enough about the limitations of that version.

I am not aware of any mechanism that can be safely used to limit the number of threads available to Asterisk. You will need one thread per call plus maybe half a dozen to a dozen more for various functions.

If you are forcing Asterisk into a situation where system calls are failing due to resource limits, it could well exerise buggy parts of the code.

17

Hello David,

I understand. Asterisk looks to be more stable after having commented the debugs options. Only problem which remains is, sometimes, non-answer on INVITE.
That mean that Asterisk take more than 5sec to reply with 1xx response.

Do you have any ideas on that?

18

Hello David,

Could it be related to the number of REGISTER that Asterisk can handle? There is no lot of concurrent call but hundred of REGISTER per minutes.
Is there any benchmark about this? I begin to look for this but can’t find for the moment.

19

Asterisk is known for not handling SIP Register packets too good when it comes to a lot of concurrent requests. In that case usually another box is used to take care of SIP Register requests (i.e. Kamailio).

How many clients are connecting to your server? The ammount of SIP Register packets that the Asterisk server is handling (according to your description) is quite a lot.

20

We use a SIP proxy to deal with the REGISTER, but these requests are also forwarded to Asterisk in order to deal with MWI. So, Asterisk know the contact of each customer to send us NOTIFY requests with MWI informations.
Asterisk treat arround ~200 REGISTER per seconds.