Astdb error on restart causes core dump, restart cycle

I am currently running 1.8.16 on Centos (2.6.18-194.el5PAE #1 SMP)

we have 6 of these servers running, and up to recently all was very good. Now one of them is having issues with a corrupted astdb

we run the command:

/usr/sbin/asterisk -rx “core restart gracefully”

and most of the time it is fine, but 2 times in the last 2 weeks, it does not restart. It ends up in a restart loop. we usually start the server via

/etc/init.d/asterisk start which also starts up the safe_asterisk, which is restarting it everytime it dies.

on further testing (via asterisk -vvvvvvvv), it drops a core. I check the core out and the line that caused the issue was:

aster_db_deltree() whilst it was parsing the iax.conf file.

so I deleted the file and restarted, and all was fine.

if I run, on a working server

asterisk -rx “database show”

it lists:

[color=#004040]
/IAX/Registry/55690233 : 83.244.251.250:1139:180
/IAX/Registry/55690237 : 83.244.251.250:1209:180
/IAX/Registry/55691013 : 83.244.251.250:1269:180
/IAX/Registry/55691017 : 83.244.251.250:4569:180
/IAX/Registry/55691025 : 83.244.251.250:1227:180
/IAX/Registry/55696401 : 82.43.57.188:4569:180

/IAX/Registry/55710445 : 81.106.221.248:41134:180
/IAX/Registry/55710446 : 81.106.221.248:15221:180
/IAX/Registry/55710456 : 83.244.251.250:1221:180
/IAX/Registry/55710457 : 83.244.251.250:1181:180
/IAX/Registry/55710499 : 81.106.221.248:41568:180
/IAX/Registry/55710514 : 213.177.248.10:49493:180
/SIP/Registry/1217379674 : 10.10.3.************************
/dundi/secret : **********************************
/dundi/secretexpiry : 1355327366
318 results found.[/color]

so I am not storing any data that was not included in my sip.conf or iax.conf

iax.conf:

[color=#004040]
;
; IAX Configuration for colossus
;

[general]
jitterbuffer=yes
forcejitterbuffer=yes
maxjitterbuffer=300
maxexcessbuffer=80
callerid=“Guest IAX User”
;allow=g729,alaw,ulaw,gsm
disallow=all
allow=alaw,speex,gsm
maxregexpire=3600
minregexpire=180

[hudzeniax]
type=friend
context=hudzeniax
;host=193.114.206.200
host=10.10.6.39
allow=speex,alaw
username=hudzeniax
secret=****************

[haliax]
type=friend
context=haliax
host=83.244.221.195
allow=speex,alaw
username=haliax
secret=****************

[daneeliax]
type=friend
context=daneeliax
host=217.35.17.217
username=daneeliax
secret=****************

[timiax]
username=timiax
type=friend
context=timiax
secret=****************
host=dynamic
callerid=“tim” <1217379201>

[geordi_iax]
username=proteus
type=friend
context=geordi
secret=****************
host=188.222.202.27
encryption=aes128

[proteus_iax]
username=proteus_iax
type=friend
context=proteus_iax
secret=****************
host=10.10.1.20

#include /home/voipoperator/iax_users.conf
[/color]

iax_users.conf: (this file has 315 users in it, and I have only pasted the top and the bottom part of the file)

[color=#004040]
[55660006]
type=friend
context=voipoperator
host=dynamic
username=55660006
secret=***************
requirecalltoken=no

[55662994]
type=friend
context=voipoperator
host=dynamic
username=55662994
secret=***************
requirecalltoken=no

[55663853]
type=friend
context=voipoperator
host=dynamic
username=55663853
secret=***************
requirecalltoken=no

[55673554]
type=friend
context=voipoperator
host=dynamic
username=55673554
secret=***************
requirecalltoken=no

[55710514]
type=friend
context=voipoperator
host=dynamic
username=55710514
secret=***************
requirecalltoken=no

[55710515]
type=friend
context=voipoperator
host=dynamic
username=55710515
secret=***************
requirecalltoken=no

[55710516]
type=friend
context=voipoperator
host=dynamic
username=55710516
secret=***************
requirecalltoken=no

[55710535]
type=friend
context=voipoperator
host=dynamic
username=55710535
secret=***************
requirecalltoken=no
[/color]

from the full log file:

[color=#004040]
[Dec 9 13:39:04] VERBOSE[31290] config.c: == Parsing ‘/etc/asterisk/iax.conf’: [Dec 9 13:39:04] VERBOSE[31290] config.c: == Found
[Dec 9 13:39:04] VERBOSE[31290] config.c: == Parsing ‘/home/voipoperator/iax_users.conf’: [Dec 9 13:39:04] VERBOSE[31290] config.c: == Found
[Dec 9 13:39:04] VERBOSE[31290] config.c: == Parsing ‘/etc/asterisk/users.conf’: [Dec 9 13:39:04] VERBOSE[31290] config.c: == Found
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: == Binding IAX2 to default address 0.0.0.0:4569
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: – Seeding ‘55701460’ at 84.45.119.6:60679 for 180
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: – Seeding ‘55701805’ at 81.106.221.223:4569 for 180
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: – Seeding ‘55702534’ at 84.45.119.6:39761 for 180
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: – Seeding ‘55702797’ at 84.45.119.6:53585 for 180
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: – Seeding ‘55703083’ at 195.171.174.82:40953 for 180
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: – Seeding ‘55703180’ at 195.171.174.82:36112 for 180
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: – Seeding ‘55703190’ at 195.171.174.82:37618 for 180
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: – Seeding ‘55704877’ at 94.101.113.101:16985 for 180
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: – Seeding ‘55706041’ at 84.45.119.6:58705 for 180
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: – Seeding ‘55706485’ at 84.45.119.6:44881 for 180
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: – Seeding ‘55707703’ at 84.45.119.6:30759 for 180
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: – Seeding ‘55708077’ at 213.123.52.98:9491 for 180
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: – Seeding ‘55708535’ at 84.45.119.6:50513 for 180
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: – Seeding ‘55709417’ at 213.177.248.10:20883 for 180
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: – Seeding ‘55709471’ at 41.134.234.178:16725 for 180
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: – Seeding ‘55709503’ at 213.177.248.10:46607 for 180
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: – Seeding ‘55709777’ at 94.101.113.101:17007 for 180
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: – Seeding ‘55710207’ at 84.45.119.6:28943 for 180
[Dec 9 13:39:04] VERBOSE[31290] pbx.c: == Registered application ‘IAX2Provision’
[Dec 9 13:39:04] VERBOSE[31290] pbx.c: == Registered custom function ‘IAXPEER’
[Dec 9 13:39:04] VERBOSE[31290] pbx.c: == Registered custom function ‘IAXVAR’
[Dec 9 13:39:04] VERBOSE[31290] manager.c: == Manager registered action IAXpeers
[Dec 9 13:39:04] VERBOSE[31290] manager.c: == Manager registered action IAXpeerlist
[Dec 9 13:39:04] VERBOSE[31290] manager.c: == Manager registered action IAXnetstats
[Dec 9 13:39:04] VERBOSE[31290] manager.c: == Manager registered action IAXregistry
[Dec 9 13:39:04] VERBOSE[31290] channel.c: == Registered channel type ‘IAX2’ (Inter Asterisk eXchange Driver (Ver 2))
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: == 10 helper threads started
[Dec 9 13:39:04] VERBOSE[31290] chan_iax2.c: == IAX Ready and Listening
[Dec 9 13:39:04] VERBOSE[31290] config.c: == Parsing ‘/etc/asterisk/iaxprov.conf’: [Dec 9 13:39:04] VERBOSE[31290] config.c: == Found
[Dec 9 13:39:04] VERBOSE[31290] iax2-provision.c: – Loaded provisioning template ‘default’
[Dec 9 13:39:04] NOTICE[31313] chan_sip.c: Peer ‘onesource-proartamis’ is now Reachable. (8ms / 2000ms)
[Dec 9 13:39:12] VERBOSE[31298] asterisk.c: – Remote UNIX connection
[Dec 9 13:39:23] VERBOSE[31330] asterisk.c: – Remote UNIX connection disconnected
[Dec 9 13:41:12] VERBOSE[31368] config.c: == Parsing ‘/etc/asterisk/logger.conf’: [Dec 9 13:41:12] VERBOSE[31368] config.c: == Found
[Dec 9 13:41:12] VERBOSE[31368] config.c: == Parsing ‘/etc/asterisk/asterisk.conf’: [Dec 9 13:41:12] VERBOSE[31368] config.c: == Found
[Dec 9 13:41:12] VERBOSE[31368] manager.c: == Manager registered action DataGet
[Dec 9 13:41:12] VERBOSE[31368] config.c: == Parsing ‘/etc/asterisk/codecs.conf’: [Dec 9 13:41:12] VERBOSE[31368] config.c: == Found
[Dec 9 13:41:12] VERBOSE[31368] loader.c: Asterisk Dynamic Loader Starting:
[Dec 9 13:41:12] VERBOSE[31368] config.c: == Parsing ‘/etc/asterisk/modules.conf’: [Dec 9 13:41:12] VERBOSE[31368] config.c: == Found
[Dec 9 13:41:12] VERBOSE[31368] config.c: == Parsing ‘/etc/asterisk/dnsmgr.conf’: [Dec 9 13:41:12] VERBOSE[31368] config.c: == Found
[Dec 9 13:41:12] VERBOSE[31368] config.c: == Parsing ‘/etc/asterisk/http.conf’: [Dec 9 13:41:12] VERBOSE[31368] config.c: == Found
[Dec 9 13:41:12] VERBOSE[31368] manager.c: == Manager registered action Ping
[/color]

sip.conf:

[color=#004040]
;
;sip config for Colossus
;
;

[general]
context=default ; Default context for incoming calls
allowguest=no ; Allow or reject guest calls (default is yes)

realm= office ; Realm for digest authentication
bindport=5060 ; UDP Port to bind to (SIP standard port is 5060)
bindaddr=0.0.0.0 ; IP address to bind to (0.0.0.0 binds to all)

disallow=all ; Disallow all codecs
allow=alaw ; We prefer alaw

;
;--------------------------- SIP timers ----------------------------------------------------
; These timers are used primarily in INVITE transactions.
; The default for Timer T1 is 500 ms or the measured run-trip time between
; Asterisk and the device if you have qualify=yes for the device.
;
t1min=100 ; Minimum roundtrip time for messages to monitored hosts
; Defaults to 100 ms
timert1=500 ; Default T1 timer
; Defaults to 500 ms or the measured round-trip
; time to a peer (qualify=yes).
timerb=10000 ; Call setup timer. If a provisional response is not received
; in this amount of time, the call will autocongest
; Defaults to 64*timert1

rtptimeout=60 ; Terminate call if 60 seconds of no RTP activity
useragent=Asterisk PBX
;dtmfmode = rfc2833
dtmmode = inband
;dtmfmmode = info
jbenable=yes
jbforce=yes
jbmaxsize=200
jblog=yes
jbimpl=adaptive
jbresyncthreshold=1000

checkmwi=10

register => 84419907:*************:84419907@sip.voiceflex.com

; allowexternaldomains=no

; For incoming calls only. We match on IP address of the
; proxy for incoming calls since we can not match on username
; (caller id)
[dumber]
type=peer
context=dumber; The context in extensions.conf we send calls to
host=###.###.###.###
canreinvite=no
qualify=no

[vega-london]
type=peer
context=vega-london
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[vega-london-abovenet]
type=peer
context=vega-london-abovenet
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[amiga-btethernet]
type=peer
context=st-internal
host=###.###.###.###
canreinvite=no
progressinband=yes
;qualify=yes

[amiga-inetc]
type=peer
context=st-internal
host=###.###.###.###
canreinvite=no
progressinband=yes

[st-btethernet]
type=peer
context=st-internal
host=###.###.###.###
canreinvite=no
progressinband=yes
;qualify=yes

[st-internal]
type=peer
context=st-internal
host=###.###.###.###
canreinvite=no
progressinband=yes
;qualify=yes

[st-inetc]
type=peer
context=st-internal
host=###.###.###.###
canreinvite=no
progressinband=yes
;qualify=yes

[st-fusion]
type=peer
context=st-internal
host=###.###.###.###
canreinvite=no
progressinband=yes

[st-exponential]
type=peer
context=st-internal
host=###.###.###.###
canreinvite=no
progressinband=yes
;qualify=yes

[electron-route1]
type=peer
context=ut-route1
host=###.###.###.###
canreinvite=no

[lisa-route1]
type=peer
context=ut-route1
host=###.###.###.###
canreinvite=no

[leo-route1]
type=peer
context=ut-route1
host=###.###.###.###
canreinvite=no

[bad-machine]
type=peer
context=st-internal
host=###.###.###.###
canreinvite=no

[abel]
type=peer
context=st-internal
host=###.###.###.###
canreinvite=no

[hudzen]
type=peer
context=st-internal
host=###.###.###.###
canreinvite=no

[holly]
type=peer
context=holly
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[in2tel-quintum]
type=peer
context=in2tel
host=###.###.###.###
canreinvite=no
progressinband=yes

[in2tel-caffreys]
type=peer
context=in2tel-caffreys
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[guinness]
type=peer
context=guinness
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[guinness-oxy]
type=peer
context=in2tel
host=###.###.###.###
canreinvite=no
dtmfmode=rfc2833
qualify=no
nat=yes

[irnbru]
type=peer
context=irnbru
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[vega-manchester]
type=peer
context=students-in
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[vega-reading]
type=peer
context=students-in
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[cgh-manchester]
type=peer
context=students-in
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[trinity-point]
type=peer
context=students-in
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[liberty-court]
type=peer
context=students-in
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[liberty-hall]
type=peer
context=students-in
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[queens-hospital]
type=peer
context=students-in
host=###.###.###.###

[hunter-court]
type=peer
context=students-in
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[the-beeches]
type=peer
context=students-in
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[atlantic-point]
type=peer
context=students-in
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[mill-point]
type=peer
context=students-in
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[prospect-point]
type=peer
context=students-in
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[quay-point]
type=peer
context=students-in
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[cambrian-point]
type=peer
context=students-in
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[severn-point]
type=peer
context=students-in
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

[2communications]
type=peer
context=2communications
host=###.###.###.###
canreinvite=no
dtmfmode=rfc2833
qualify=no

[voiceflex-infosys]
type=peer
context=voiceflex-infosys
host=###.###.###.###
canreinvite=no
insecure=yes
dtmfmode=rfc2833

[drltd-rcms]
type=peer
context=drltd-rcms
host=###.###.###.###
canreinvite=no
dtmfmode=rfc2833
qualify=no

[drltd-lync]
type=peer
context=drltd-lync
host=###.###.###.###
canreinvite=no
dtmfmode=rfc2833
qualify=no

;23702
[drltd-mobal]
type=peer
context=drltd-mobal
host=###.###.###.###
canreinvite=no
dtmfmode=rfc2833
nat=yes

[drltd-mobal-old]
type=peer
context=drltd-mobal
host=###.###.###.###
canreinvite=no
dtmfmode=rfc2833
nat=yes

[onesource-proartamis-test]
type=peer
context=onesource-proartamis
host=###.###.###.###
canreinvite=no
insecure=yes
dtmfmode=rfc2833
nat=yes

[onesource-proartamis]
type=peer
context=onesource-proartamis
host=###.###.###.###
canreinvite=no
insecure=yes
dtmfmode=rfc2833
nat=yes
qualify=yes

[markhennessyhome]
type=peer
context=in2tel
host=###.###.###.###
canreinvite=no
progressinband=yes
qualify=no

;tim.brookes
[1217379674]
type=peer
context=tim
host=###.###.###.###
insecure=port
canreinvite=no
secret=****************
nat=yes
progressinband=yes
;prematuremedia=yes
;mailbox=1217379670

;mark.hennessy
[1217379675]
type=peer
context=mark
host=###.###.###.###
insecure=port
canreinvite=no
secret=****************
nat=yes
;mailbox=1217379670

[55689637]
type=peer
context=voipoperator
host=###.###.###.###
insecure=port
canreinvite=no
secret=****************
nat=yes

[55689638]
type=peer
context=voipoperator
host=###.###.###.###
insecure=port
canreinvite=no
secret=****************

[telecomit1]
type=peer
context=telecomit
host=###.###.###.###
canreinvite=no
progressinband=yes

[telecomit2]
type=peer
context=telecomit
host=###.###.###.###
canreinvite=no
progressinband=yes

[telecomit3]
type=peer
context=telecomit
host=###.###.###.###
canreinvite=no
progressinband=yes

[telecomit4]
type=peer
context=telecomit
host=###.###.###.###
canreinvite=no
progressinband=yes

[telecomit5]
type=peer
context=telecomit
host=###.###.###.###
canreinvite=no
progressinband=yes

;account 38772
[ip3_dxi]
type=peer
context=ip3_dxi
host=###.###.###.###
canreinvite=no
progressinband=yes

[1223915969]
type=peer
context=geordi
host=###.###.###.###
insecure=port
canreinvite=no
secret=****************
nat=yes

[itlab]
type=peer
context=itlab
host=###.###.###.###
canreinvite=no
;insecure=yes
dtmfmode=rfc2833
nat=yes

[itlab2]
type=peer
context=itlab
host=###.###.###.###
canreinvite=no
;insecure=yes
dtmfmode=rfc2833
nat=yes

[itlab3]
type=peer
context=itlab
host=###.###.###.###
canreinvite=no
;insecure=yes
dtmfmode=rfc2833
nat=yes

[itlab4]
type=peer
context=itlab
host=###.###.###.###
canreinvite=no
;insecure=yes
dtmfmode=rfc2833
nat=yes

[dietchef]
type=peer
context=dietchef
host=###.###.###.###
canreinvite=no
dtmfmode=rfc2833
nat=yes

[mediaiq]
type=peer
context=mediaiq
host=94.143.253.110
canreinvite=no
dtmfmode=rfc2833
nat=yes

[mediaiq2]
type=peer
context=mediaiq
host=94.143.249.174
canreinvite=no
dtmfmode=rfc2833
nat=yes
[/color]

iaxprov.conf
[color=#004040]
[default]
codec=ulaw
flags=register,heartbeat
[/color]

Try "service asterisk stop “
then
"service asterisk start”

manually starting when using safe_asterisk can cause instability