Hi,
We are running Asterisk 13.4 PJSIP in a production environment and get random crashes. Everything seemed to be working fine for a couple of days and we now get crashes more and more.
I have also noticed that crashes happens even though no calls are made, so I don’t think it has anything to do with the dialplan. I’ve really tested everything in a test machine before moving my dialplan to PJSIP. Most of the phones use direct media, so the load on asterisk is low.
I get no errors when doing asterisk -cvvvvvvvvvvvvvv.
I have backtraces that I can provide if needed.
Any help will be greatly appreciated.
Our software:
Asterisk Realtime 13.4 PJSIP Driver
mysql Ver 14.14
pjproject 2.3
spandsp 0.0.6
jansson 2.7
CentOS 6.6 64 bits on Vmware
Number of endpoints : > 700
Numbers of calls : 3000/day
Our Hardware:
Phones : Cisco SPA514G FW: 7.5.7
ATA : Audiocodes MP124
T1 : Mediatrix 3532 ISDN to SIP gateway
CPU : Quadcore Intel® Xeon® CPU E5-2650
RAM : 3 GB
[root@dti-asterisk ~]# asterisk -cvvvvvvvvvvvvvv
[Jun 17 05:29:42] Asterisk 13.4.0, Copyright (C) 1999 - 2014, Digium, Inc. and others.
[Jun 17 05:29:42] Created by Mark Spencer <markster@digium.com>
[Jun 17 05:29:42] Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.
[Jun 17 05:29:42] This is free software, with components licensed under the GNU General Public
[Jun 17 05:29:42] License version 2 and other licenses; you are welcome to redistribute it under
[Jun 17 05:29:42] certain conditions. Type 'core show license' for details.
[Jun 17 05:29:42] =========================================================================
[Jun 17 05:29:42] [ Initializing Custom Configuration Options ]
[Jun 17 05:29:42] == Parsing '/etc/asterisk/extconfig.conf': Found
[Jun 17 05:29:42] == Binding voicemail to odbc/asterisk/voicemail_users
[Jun 17 05:29:42] == Binding ps_endpoints to odbc/asterisk/ps_endpoints
[Jun 17 05:29:42] == Binding ps_auths to odbc/asterisk/ps_auths
[Jun 17 05:29:42] == Binding ps_aors to odbc/asterisk/ps_aors
[Jun 17 05:29:42] == Binding ps_domain_aliases to odbc/asterisk/ps_domain_aliases
[Jun 17 05:29:42] == Binding ps_endpoint_id_ips to odbc/asterisk/ps_endpoint_id_ips
[Jun 17 05:29:42] == Binding ps_contacts to odbc/asterisk/ps_contacts
[Jun 17 05:29:42] == Registered 'audio' codec 'g723' at sample rate '8000' with id '1'
[Jun 17 05:29:42] == Created cached format with name 'g723'
[Jun 17 05:29:42] == Registered 'audio' codec 'ulaw' at sample rate '8000' with id '2'
[Jun 17 05:29:42] == Created cached format with name 'ulaw'
[Jun 17 05:29:42] == Registered 'audio' codec 'alaw' at sample rate '8000' with id '3'
[Jun 17 05:29:42] == Created cached format with name 'alaw'
[Jun 17 05:29:42] == Registered 'audio' codec 'gsm' at sample rate '8000' with id '4'
[Jun 17 05:29:42] == Created cached format with name 'gsm'
[Jun 17 05:29:42] == Registered 'audio' codec 'g726' at sample rate '8000' with id '5'
[Jun 17 05:29:42] == Created cached format with name 'g726'
[Jun 17 05:29:42] == Registered 'audio' codec 'g726aal2' at sample rate '8000' with id '6'
[Jun 17 05:29:42] == Created cached format with name 'g726aal2'
[Jun 17 05:29:42] == Registered 'audio' codec 'adpcm' at sample rate '8000' with id '7'
[Jun 17 05:29:42] == Created cached format with name 'adpcm'
[Jun 17 05:29:42] == Registered 'audio' codec 'slin' at sample rate '8000' with id '8'
[Jun 17 05:29:42] == Created cached format with name 'slin'
[Jun 17 05:29:42] == Registered 'audio' codec 'slin' at sample rate '12000' with id '9'
[Jun 17 05:29:42] == Created cached format with name 'slin12'
[Jun 17 05:29:42] == Registered 'audio' codec 'slin' at sample rate '16000' with id '10'
[Jun 17 05:29:42] == Created cached format with name 'slin16'
[Jun 17 05:29:42] == Registered 'audio' codec 'slin' at sample rate '24000' with id '11'
[Jun 17 05:29:42] == Created cached format with name 'slin24'
[Jun 17 05:29:42] == Registered 'audio' codec 'slin' at sample rate '32000' with id '12'
[Jun 17 05:29:42] == Created cached format with name 'slin32'
[Jun 17 05:29:42] == Registered 'audio' codec 'slin' at sample rate '44100' with id '13'
[Jun 17 05:29:42] == Created cached format with name 'slin44'
[Jun 17 05:29:42] == Registered 'audio' codec 'slin' at sample rate '48000' with id '14'
[Jun 17 05:29:42] == Created cached format with name 'slin48'
[Jun 17 05:29:42] == Registered 'audio' codec 'slin' at sample rate '96000' with id '15'
[Jun 17 05:29:42] == Created cached format with name 'slin96'
[Jun 17 05:29:42] == Registered 'audio' codec 'slin' at sample rate '192000' with id '16'
[Jun 17 05:29:42] == Created cached format with name 'slin192'
[Jun 17 05:29:42] == Registered 'audio' codec 'lpc10' at sample rate '8000' with id '17'
[Jun 17 05:29:42] == Created cached format with name 'lpc10'
[Jun 17 05:29:42] == Registered 'audio' codec 'g729' at sample rate '8000' with id '18'
[Jun 17 05:29:42] == Created cached format with name 'g729'
[Jun 17 05:29:42] == Registered 'audio' codec 'speex' at sample rate '8000' with id '19'
[Jun 17 05:29:42] == Created cached format with name 'speex'
[Jun 17 05:29:42] == Registered 'audio' codec 'speex' at sample rate '16000' with id '20'
[Jun 17 05:29:42] == Created cached format with name 'speex16'
[Jun 17 05:29:42] == Registered 'audio' codec 'speex' at sample rate '32000' with id '21'
[Jun 17 05:29:42] == Created cached format with name 'speex32'
[Jun 17 05:29:42] == Registered 'audio' codec 'ilbc' at sample rate '8000' with id '22'
[Jun 17 05:29:42] == Created cached format with name 'ilbc'
[Jun 17 05:29:42] == Registered 'audio' codec 'g722' at sample rate '16000' with id '23'
[Jun 17 05:29:42] == Created cached format with name 'g722'
[Jun 17 05:29:42] == Registered 'audio' codec 'siren7' at sample rate '16000' with id '24'
[Jun 17 05:29:42] == Created cached format with name 'siren7'
[Jun 17 05:29:42] == Registered 'audio' codec 'siren14' at sample rate '32000' with id '25'
[Jun 17 05:29:42] == Created cached format with name 'siren14'
[Jun 17 05:29:42] == Registered 'audio' codec 'testlaw' at sample rate '8000' with id '26'
[Jun 17 05:29:42] == Created cached format with name 'testlaw'
[Jun 17 05:29:42] == Registered 'audio' codec 'g719' at sample rate '48000' with id '27'
[Jun 17 05:29:42] == Created cached format with name 'g719'
[Jun 17 05:29:42] == Registered 'audio' codec 'opus' at sample rate '48000' with id '28'
[Jun 17 05:29:42] == Created cached format with name 'opus'
[Jun 17 05:29:42] == Registered 'image' codec 'jpeg' at sample rate '0' with id '29'
[Jun 17 05:29:42] == Created cached format with name 'jpeg'
[Jun 17 05:29:42] == Registered 'image' codec 'png' at sample rate '0' with id '30'
[Jun 17 05:29:42] == Created cached format with name 'png'
[Jun 17 05:29:42] == Registered 'video' codec 'h261' at sample rate '0' with id '31'
[Jun 17 05:29:42] == Created cached format with name 'h261'
[Jun 17 05:29:42] == Registered 'video' codec 'h263' at sample rate '0' with id '32'
[Jun 17 05:29:42] == Created cached format with name 'h263'
[Jun 17 05:29:42] == Registered 'video' codec 'h263p' at sample rate '0' with id '33'
[Jun 17 05:29:42] == Created cached format with name 'h263p'
[Jun 17 05:29:42] == Registered 'video' codec 'h264' at sample rate '0' with id '34'
[Jun 17 05:29:42] == Created cached format with name 'h264'
[Jun 17 05:29:42] == Registered 'video' codec 'mpeg4' at sample rate '0' with id '35'
[Jun 17 05:29:42] == Created cached format with name 'mpeg4'
[Jun 17 05:29:42] == Registered 'video' codec 'vp8' at sample rate '0' with id '36'
[Jun 17 05:29:42] == Created cached format with name 'vp8'
[Jun 17 05:29:42] == Registered 'text' codec 'red' at sample rate '0' with id '37'
[Jun 17 05:29:42] == Created cached format with name 'red'
[Jun 17 05:29:42] == Registered 'text' codec 't140' at sample rate '0' with id '38'
[Jun 17 05:29:42] == Created cached format with name 't140'
[Jun 17 05:29:42] == Registered 'audio' codec 'none' at sample rate '8000' with id '39'
[Jun 17 05:29:42] == Created cached format with name 'none'
[Jun 17 05:29:42] == Parsing '/etc/asterisk/asterisk.conf': Found
XSLT support not found. XML documentation may be incomplete.
[Jun 17 05:29:42] == Sorcery registered wizard 'bucket'
[Jun 17 05:29:42] == Sorcery registered wizard 'bucket_file'
[Jun 17 05:29:42] == Parsing '/etc/asterisk/sorcery.conf': Found
[Jun 17 05:29:42] == Parsing '/etc/asterisk/stasis.conf': Found
[Jun 17 05:29:42] == Parsing '/etc/asterisk/logger.conf': Found
[Jun 17 05:29:42] == Manager registered action DBGet
[Jun 17 05:29:42] == Manager registered action DBPut
[Jun 17 05:29:42] == Manager registered action DBDel
[Jun 17 05:29:42] == Manager registered action DBDelTree
[Jun 17 05:29:42] == Message handler 'dialplan' registered.
[Jun 17 05:29:42] == Registered custom function 'MESSAGE'
[Jun 17 05:29:42] == Registered custom function 'MESSAGE_DATA'
[Jun 17 05:29:42] == Registered application 'MessageSend'
[Jun 17 05:29:42] == Manager registered action MessageSend
[Jun 17 05:29:42] == Manager registered action DataGet
[Jun 17 05:29:42] == Registered channel type 'Surrogate' (Surrogate channel used to pull channel from an application)
[Jun 17 05:29:42] == Manager registered action BridgeTechnologyList
[Jun 17 05:29:42] == Manager registered action BridgeTechnologySuspend
[Jun 17 05:29:42] == Manager registered action BridgeTechnologyUnsuspend
[Jun 17 05:29:42] Asterisk Dynamic Loader Starting:
[Jun 17 05:29:42] == Parsing '/etc/asterisk/modules.conf': Found
[Jun 17 05:29:42] NOTICE[14501]: loader.c:1364 load_modules: 2 modules will be loaded.
[Jun 17 05:29:42] Loading res_odbc.so.
[Jun 17 05:29:42] == Parsing '/etc/asterisk/res_odbc.conf': Found
[Jun 17 05:29:42] NOTICE[14501]: res_odbc.c:1528 odbc_obj_connect: Connecting asterisk
[Jun 17 05:29:42] NOTICE[14501]: res_odbc.c:1567 odbc_obj_connect: res_odbc: Connected to asterisk [asterisk-connector]
[Jun 17 05:29:42] NOTICE[14501]: res_odbc.c:923 load_odbc_config: Registered ODBC class 'asterisk' dsn->[asterisk-connector]
[Jun 17 05:29:42] == Registered application 'ODBC_Commit'
[Jun 17 05:29:42] == Registered application 'ODBC_Rollback'
[Jun 17 05:29:42] == Registered custom function 'ODBC'
[Jun 17 05:29:42] NOTICE[14501]: res_odbc.c:1907 load_module: res_odbc loaded.
[Jun 17 05:29:42] == res_odbc.so => (ODBC resource)
[Jun 17 05:29:42] Loading res_config_odbc.so.
[Jun 17 05:29:42] == res_config_odbc.so => (Realtime ODBC configuration)
[Jun 17 05:29:42] == Parsing '/etc/asterisk/features.conf': Found
[Jun 17 05:29:42] == Registered custom function 'FEATURE'
[Jun 17 05:29:42] == Registered custom function 'FEATUREMAP'
[Jun 17 05:29:42] == Registered application 'Bridge'
[Jun 17 05:29:42] == Manager registered action Bridge
[Jun 17 05:29:42] == Parsing '/etc/asterisk/acl.conf': Found
[Jun 17 05:29:42] == Parsing '/etc/asterisk/indications.conf': Found
[Jun 17 05:29:42] -- Registered indication country 'at'
[Jun 17 05:29:42] -- Registered indication country 'au'
[Jun 17 05:29:42] -- Registered indication country 'bg'
[Jun 17 05:29:42] -- Registered indication country 'br'
[Jun 17 05:29:42] -- Registered indication country 'be'
[Jun 17 05:29:42] -- Registered indication country 'ch'
[Jun 17 05:29:42] -- Registered indication country 'cl'
[Jun 17 05:29:42] -- Registered indication country 'cn'
[Jun 17 05:29:42] -- Registered indication country 'cz'
[Jun 17 05:29:42] -- Registered indication country 'de'
[Jun 17 05:29:42] -- Registered indication country 'dk'
[Jun 17 05:29:42] -- Registered indication country 'ee'
[Jun 17 05:29:42] -- Registered indication country 'es'
[Jun 17 05:29:42] -- Registered indication country 'fi'
[Jun 17 05:29:42] -- Registered indication country 'fr'
[Jun 17 05:29:42] -- Registered indication country 'gr'
[Jun 17 05:29:42] -- Registered indication country 'hu'
[Jun 17 05:29:42] -- Registered indication country 'il'
[Jun 17 05:29:42] -- Registered indication country 'in'
[Jun 17 05:29:42] -- Registered indication country 'it'
[Jun 17 05:29:42] -- Registered indication country 'lt'
[Jun 17 05:29:42] -- Registered indication country 'jp'
[Jun 17 05:29:42] -- Registered indication country 'mx'
[Jun 17 05:29:42] -- Registered indication country 'my'
[Jun 17 05:29:42] -- Registered indication country 'nl'
[Jun 17 05:29:42] -- Registered indication country 'no'
[Jun 17 05:29:42] -- Registered indication country 'nz'
[Jun 17 05:29:42] -- Registered indication country 'ph'
[Jun 17 05:29:42] -- Registered indication country 'pl'
[Jun 17 05:29:42] -- Registered indication country 'pt'
[Jun 17 05:29:42] -- Registered indication country 'ru'
[Jun 17 05:29:42] -- Registered indication country 'se'
[Jun 17 05:29:42] -- Registered indication country 'sg'
[Jun 17 05:29:42] -- Registered indication country 'th'
[Jun 17 05:29:42] -- Registered indication country 'uk'
[Jun 17 05:29:42] -- Registered indication country 'us'
[Jun 17 05:29:42] -- Registered indication country 'us-old'
[Jun 17 05:29:42] -- Registered indication country 'tw'
[Jun 17 05:29:42] -- Registered indication country 've'
[Jun 17 05:29:42] -- Registered indication country 'za'
[Jun 17 05:29:42] -- Setting default indication country to 'us'
[Jun 17 05:29:42] == Parsing '/etc/asterisk/cdr.conf': Found
[Jun 17 05:29:42] NOTICE[14501]: cdr.c:4152 cdr_toggle_runtime_options: CDR simple logging enabled.
[Jun 17 05:29:42] == Parsing '/etc/asterisk/udptl.conf': Found
[Jun 17 05:29:42] Asterisk PBX Core Initializing
[Jun 17 05:29:42] == Registering builtin applications and functions:
[Jun 17 05:29:42] == Registered custom function 'EXCEPTION'
[Jun 17 05:29:42] == Registered custom function 'TESTTIME'
[Jun 17 05:29:42] == Registered application 'Answer'
[Jun 17 05:29:42] == Registered application 'BackGround'
[Jun 17 05:29:42] == Registered application 'Busy'
[Jun 17 05:29:42] == Registered application 'Congestion'
[Jun 17 05:29:42] == Registered application 'ExecIfTime'
[Jun 17 05:29:42] == Registered application 'Goto'
[Jun 17 05:29:42] == Registered application 'GotoIf'
[Jun 17 05:29:42] == Registered application 'GotoIfTime'
[Jun 17 05:29:42] == Registered application 'ImportVar'
[Jun 17 05:29:42] == Registered application 'Hangup'
[Jun 17 05:29:42] == Registered application 'Incomplete'
[Jun 17 05:29:42] == Registered application 'NoOp'
[Jun 17 05:29:42] == Registered application 'Proceeding'
[Jun 17 05:29:42] == Registered application 'Progress'
[Jun 17 05:29:42] == Registered application 'RaiseException'
[Jun 17 05:29:42] == Registered application 'Ringing'
[Jun 17 05:29:42] == Registered application 'SayAlpha'
[Jun 17 05:29:42] == Registered application 'SayAlphaCase'
[Jun 17 05:29:42] == Registered application 'SayDigits'
[Jun 17 05:29:42] == Registered application 'SayNumber'
[Jun 17 05:29:42] == Registered application 'SayPhonetic'
[Jun 17 05:29:42] == Registered application 'Set'
[Jun 17 05:29:42] == Registered application 'MSet'
[Jun 17 05:29:42] == Registered application 'SetAMAFlags'
[Jun 17 05:29:42] == Registered application 'Wait'
[Jun 17 05:29:42] == Registered application 'WaitExten'
[Jun 17 05:29:42] == Manager registered action ShowDialPlan
[Jun 17 05:29:42] == Manager registered action ExtensionStateList
[Jun 17 05:29:42] == Registered channel type 'Local' (Local Proxy Channel Driver)
[Jun 17 05:29:42] == Manager registered action LocalOptimizeAway
[Jun 17 05:29:42] == Parsing '/etc/asterisk/cel.conf': Found
[Jun 17 05:29:42] == Manager registered action Ping
[Jun 17 05:29:42] == Manager registered action Events
[Jun 17 05:29:42] == Manager registered action Logoff
[Jun 17 05:29:42] == Manager registered action Login
[Jun 17 05:29:42] == Manager registered action Challenge
[Jun 17 05:29:42] == Manager registered action Hangup
[Jun 17 05:29:42] == Manager registered action Status
[Jun 17 05:29:42] == Manager registered action Setvar
[Jun 17 05:29:42] == Manager registered action Getvar
[Jun 17 05:29:42] == Manager registered action GetConfig
[Jun 17 05:29:42] == Manager registered action GetConfigJSON
[Jun 17 05:29:42] == Manager registered action UpdateConfig
[Jun 17 05:29:42] == Manager registered action CreateConfig
[Jun 17 05:29:42] == Manager registered action ListCategories
[Jun 17 05:29:42] == Manager registered action Redirect
[Jun 17 05:29:42] == Manager registered action Atxfer
[Jun 17 05:29:42] == Manager registered action Originate
[Jun 17 05:29:42] == Manager registered action Command
[Jun 17 05:29:42] == Manager registered action ExtensionState
[Jun 17 05:29:42] == Manager registered action PresenceState
[Jun 17 05:29:42] == Manager registered action AbsoluteTimeout
[Jun 17 05:29:42] == Manager registered action MailboxStatus
[Jun 17 05:29:42] == Manager registered action MailboxCount
[Jun 17 05:29:42] == Manager registered action ListCommands
[Jun 17 05:29:42] == Manager registered action SendText
[Jun 17 05:29:42] == Manager registered action UserEvent
[Jun 17 05:29:42] == Manager registered action WaitEvent
[Jun 17 05:29:42] == Manager registered action CoreSettings
[Jun 17 05:29:42] == Manager registered action CoreStatus
[Jun 17 05:29:42] == Manager registered action Reload
[Jun 17 05:29:42] == Manager registered action LoggerRotate
[Jun 17 05:29:42] == Manager registered action CoreShowChannels
[Jun 17 05:29:42] == Manager registered action ModuleLoad
[Jun 17 05:29:42] == Manager registered action ModuleCheck
[Jun 17 05:29:42] == Manager registered action AOCMessage
[Jun 17 05:29:42] == Manager registered action Filter
[Jun 17 05:29:42] == Manager registered action BlindTransfer
[Jun 17 05:29:42] == Registered custom function 'AMI_CLIENT'
[Jun 17 05:29:42] == Parsing '/etc/asterisk/manager.conf': Found
[Jun 17 05:29:42] == Registered application 'CallCompletionRequest'
[Jun 17 05:29:42] == Registered application 'CallCompletionCancel'
[Jun 17 05:29:42] == Parsing '/etc/asterisk/ccss.conf': Found
[Jun 17 05:29:42] == Parsing '/etc/asterisk/ccss.conf': Found
[Jun 17 05:29:42] Asterisk Dynamic Loader Starting:
[Jun 17 05:29:42] == Parsing '/etc/asterisk/modules.conf': Found
[Jun 17 05:29:42] NOTICE[14501]: loader.c:1364 load_modules: 207 modules will be loaded.
[Jun 17 05:29:42] Loading res_statsd.so.
[Jun 17 05:29:42] == Parsing '/etc/asterisk/statsd.conf': Found
[Jun 17 05:29:42] == res_statsd.so => (Statsd client support)
[Jun 17 05:29:42] Loading res_sorcery_realtime.so.
[Jun 17 05:29:42] == Sorcery registered wizard 'realtime'
[Jun 17 05:29:42] == res_sorcery_realtime.so => (Sorcery Realtime Object Wizard)
[Jun 17 05:29:42] Loading res_sorcery_astdb.so.
[Jun 17 05:29:42] == Sorcery registered wizard 'astdb'
[Jun 17 05:29:42] == res_sorcery_astdb.so => (Sorcery Astdb Object Wizard)
[Jun 17 05:29:42] Loading res_pjsip_config_wizard.so.
[Jun 17 05:29:42] == res_pjsip_config_wizard.so => (PJSIP Config Wizard)
[Jun 17 05:29:42] Loading res_sorcery_memory.so.
[Jun 17 05:29:42] == Sorcery registered wizard 'memory'
[Jun 17 05:29:42] == res_sorcery_memory.so => (Sorcery In-Memory Object Wizard)
[Jun 17 05:29:42] Loading res_sorcery_config.so.
[Jun 17 05:29:42] == Sorcery registered wizard 'config'
[Jun 17 05:29:42] == res_sorcery_config.so => (Sorcery Configuration File Object Wizard)
[Jun 17 05:29:42] Loading res_pjsip_log_forwarder.so.
05:29:42.670 os_core_unix.c !pjlib 2.3 for POSIX initialized
[Jun 17 05:29:42] == res_pjsip_log_forwarder.so => (PJSIP Log Forwarder)
[Jun 17 05:29:42] Loading res_pjsip.so.
[Jun 17 05:29:42] == Parsing '/etc/asterisk/sorcery.conf': Found
[Jun 17 05:29:42] == Parsing '/etc/asterisk/pjsip.conf': Found
[Jun 17 05:29:42] == Manager registered action PJSIPShowEndpoints
[Jun 17 05:29:42] == Manager registered action PJSIPShowEndpoint
[Jun 17 05:29:42] == Parsing '/etc/asterisk/pjsip.conf': Found
[Jun 17 05:29:42] == Parsing '/etc/asterisk/pjsip_wizard.conf': Found
[Jun 17 05:29:42] == Parsing '/etc/asterisk/pjsip.conf': Found
[Jun 17 05:29:42] == Parsing '/etc/asterisk/pjsip_wizard.conf': Found
[Jun 17 05:29:42] == Parsing '/etc/asterisk/pjsip.conf': Found
[Jun 17 05:29:42] == Parsing '/etc/asterisk/pjsip.conf': Found