Segmentation Fault - Asterisk will not start

After updating from asterisk 15 this lovely system decided it could no longer run. I am hoping the community has a bit of advice on getting this back online again.

This was a dist upgrade from Stretch to Buster and an Asterisk version 15 to 16.2.1

root@spk-asterisk:/etc/asterisk# asterisk -vvvvvvvvvvc
Asterisk 16.2.1~dfsg-1+deb10u2, Copyright (C) 1999 - 2018, Digium, Inc. and others.
Created by Mark Spencer <markster@digium.com>
Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.
This is free software, with components licensed under the GNU General Public
License version 2 and other licenses; you are welcome to redistribute it under
certain conditions. Type 'core show license' for details.
=========================================================================
Manager registered action DBGet
Manager registered action DBPut
Manager registered action DBDel
Manager registered action DBDelTree
PBX UUID: 5bfe946b-460c-4760-bc87-f914dfaf54a4
Registered 'audio' codec 'amr' at sample rate '8000' with id '1'
Created cached format with name 'amr'
Registered 'audio' codec 'amrwb' at sample rate '16000' with id '2'
Created cached format with name 'amrwb'
Registered 'audio' codec 'codec2' at sample rate '8000' with id '3'
Created cached format with name 'codec2'
Registered 'audio' codec 'g723' at sample rate '8000' with id '4'
Created cached format with name 'g723'
Registered 'audio' codec 'ulaw' at sample rate '8000' with id '5'
Created cached format with name 'ulaw'
Registered 'audio' codec 'alaw' at sample rate '8000' with id '6'
Created cached format with name 'alaw'
Registered 'audio' codec 'gsm' at sample rate '8000' with id '7'
Created cached format with name 'gsm'
Registered 'audio' codec 'g726' at sample rate '8000' with id '8'
Created cached format with name 'g726'
Registered 'audio' codec 'g726aal2' at sample rate '8000' with id '9'
Created cached format with name 'g726aal2'
Registered 'audio' codec 'adpcm' at sample rate '8000' with id '10'
Created cached format with name 'adpcm'
Registered 'audio' codec 'slin' at sample rate '8000' with id '11'
Created cached format with name 'slin'
Registered 'audio' codec 'slin' at sample rate '12000' with id '12'
Created cached format with name 'slin12'
Registered 'audio' codec 'slin' at sample rate '16000' with id '13'
Created cached format with name 'slin16'
Registered 'audio' codec 'slin' at sample rate '24000' with id '14'
Created cached format with name 'slin24'
Registered 'audio' codec 'slin' at sample rate '32000' with id '15'
Created cached format with name 'slin32'
Registered 'audio' codec 'slin' at sample rate '44100' with id '16'
Created cached format with name 'slin44'
Registered 'audio' codec 'slin' at sample rate '48000' with id '17'
Created cached format with name 'slin48'
Registered 'audio' codec 'slin' at sample rate '96000' with id '18'
Created cached format with name 'slin96'
Registered 'audio' codec 'slin' at sample rate '192000' with id '19'
Created cached format with name 'slin192'
Registered 'audio' codec 'lpc10' at sample rate '8000' with id '20'
Created cached format with name 'lpc10'
Registered 'audio' codec 'g729' at sample rate '8000' with id '21'
Created cached format with name 'g729'
Registered 'audio' codec 'speex' at sample rate '8000' with id '22'
Created cached format with name 'speex'
Registered 'audio' codec 'speex' at sample rate '16000' with id '23'
Created cached format with name 'speex16'
Registered 'audio' codec 'speex' at sample rate '32000' with id '24'
Created cached format with name 'speex32'
Registered 'audio' codec 'ilbc' at sample rate '8000' with id '25'
Created cached format with name 'ilbc'
Registered 'audio' codec 'g722' at sample rate '16000' with id '26'
Created cached format with name 'g722'
Registered 'audio' codec 'siren7' at sample rate '16000' with id '27'
Created cached format with name 'siren7'
Registered 'audio' codec 'siren14' at sample rate '32000' with id '28'
Created cached format with name 'siren14'
Registered 'audio' codec 'testlaw' at sample rate '8000' with id '29'
Created cached format with name 'testlaw'
Registered 'audio' codec 'g719' at sample rate '48000' with id '30'
Created cached format with name 'g719'
Registered 'audio' codec 'opus' at sample rate '48000' with id '31'
Created cached format with name 'opus'
Registered 'image' codec 'jpeg' at sample rate '0' with id '32'
Created cached format with name 'jpeg'
Registered 'image' codec 'png' at sample rate '0' with id '33'
Created cached format with name 'png'
Registered 'video' codec 'h261' at sample rate '1000' with id '34'
Created cached format with name 'h261'
Registered 'video' codec 'h263' at sample rate '1000' with id '35'
Created cached format with name 'h263'
Registered 'video' codec 'h263p' at sample rate '1000' with id '36'
Created cached format with name 'h263p'
Registered 'video' codec 'h264' at sample rate '1000' with id '37'
Created cached format with name 'h264'
Registered 'video' codec 'mpeg4' at sample rate '1000' with id '38'
Created cached format with name 'mpeg4'
Registered 'video' codec 'vp8' at sample rate '1000' with id '39'
Created cached format with name 'vp8'
Registered 'video' codec 'vp9' at sample rate '1000' with id '40'
Created cached format with name 'vp9'
Registered 'text' codec 'red' at sample rate '0' with id '41'
Created cached format with name 'red'
Registered 'text' codec 't140' at sample rate '0' with id '42'
Created cached format with name 't140'
Registered 'image' codec 't38' at sample rate '0' with id '43'
Created cached format with name 't38'
Registered 'audio' codec 'none' at sample rate '8000' with id '44'
Created cached format with name 'none'
Registered 'audio' codec 'silk' at sample rate '8000' with id '45'
Created cached format with name 'silk8'
Registered 'audio' codec 'silk' at sample rate '12000' with id '46'
Created cached format with name 'silk12'
Registered 'audio' codec 'silk' at sample rate '16000' with id '47'
Created cached format with name 'silk16'
Registered 'audio' codec 'silk' at sample rate '24000' with id '48'
Created cached format with name 'silk24'
Sorcery registered wizard 'bucket'
Sorcery registered wizard 'bucket_file'
  == Message handler 'dialplan' registered.
  == Registered custom function 'MESSAGE'
  == Registered custom function 'MESSAGE_DATA'
  == Registered application 'MessageSend'
  == Manager registered action MessageSend
  == Registered channel type 'Surrogate' (Surrogate channel used to pull channel from an application)
  == Manager registered action BridgeTechnologyList
  == Manager registered action BridgeTechnologySuspend
  == Manager registered action BridgeTechnologyUnsuspend
  == Registered DNS resolver 'system' with priority '2147483647'
 Asterisk PBX Core Initializing
  == Registering builtin functions:
  == Registered custom function 'EXCEPTION'
  == Registered custom function 'TESTTIME'
  == Manager registered action ShowDialPlan
  == Manager registered action ExtensionStateList
  == Registered application 'Answer'
  == Registered application 'BackGround'
  == Registered application 'Busy'
  == Registered application 'Congestion'
  == Registered application 'ExecIfTime'
  == Registered application 'Goto'
  == Registered application 'GotoIf'
  == Registered application 'GotoIfTime'
  == Registered application 'ImportVar'
  == Registered application 'Hangup'
  == Registered application 'Incomplete'
  == Registered application 'NoOp'
  == Registered application 'Proceeding'
  == Registered application 'Progress'
  == Registered application 'RaiseException'
  == Registered application 'Ringing'
  == Registered application 'SayAlpha'
  == Registered application 'SayAlphaCase'
  == Registered application 'SayDigits'
  == Registered application 'SayNumber'
  == Registered application 'SayPhonetic'
  == Registered application 'SetAMAFlags'
  == Registered application 'Wait'
  == Registered application 'WaitDigit'
  == Registered application 'WaitExten'
  == Registered application 'Set'
  == Registered application 'MSet'
  == Registered channel type 'Local' (Local Proxy Channel Driver)
  == Manager registered action LocalOptimizeAway
 Asterisk Dynamic Loader Starting:
[Nov 16 14:38:47] NOTICE[1266]: loader.c:2230 load_modules: 338 modules will be loaded.
Segmentation fault
root@spk-asterisk:/etc/asterisk# asterisk -V
Asterisk 16.2.1~dfsg-1+deb10u2
root@spk-asterisk:/etc/asterisk#

As a work around I attempted to reinstall asterisk and it yielded this…

root@spk-asterisk:~# apt reinstall asterisk
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 2,288 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://ftp.us.debian.org/debian buster/main amd64 asterisk amd64 1:16.2.1~dfsg-1+deb10u2 [2,288 kB]
Fetched 2,288 kB in 2s (1,057 kB/s)
(Reading database ... 70507 files and directories currently installed.)
Preparing to unpack .../asterisk_1%3a16.2.1~dfsg-1+deb10u2_amd64.deb ...
Unpacking asterisk (1:16.2.1~dfsg-1+deb10u2) over (1:16.2.1~dfsg-1+deb10u2) ...
Setting up asterisk (1:16.2.1~dfsg-1+deb10u2) ...
Job for asterisk.service failed because a fatal signal was delivered causing the control process to dump core.
See "systemctl status asterisk.service" and "journalctl -xe" for details.
invoke-rc.d: initscript asterisk, action "restart" failed.
● asterisk.service - Asterisk PBX
   Loaded: loaded (/lib/systemd/system/asterisk.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: core-dump) since Tue 2021-11-16 15:06:22 PST; 226ms ago
     Docs: man:asterisk(8)
  Process: 1367 ExecStart=/usr/sbin/asterisk -g -f -p -U asterisk (code=dumped, signal=SEGV)
 Main PID: 1367 (code=dumped, signal=SEGV)

Nov 16 15:06:22 spk-asterisk systemd[1]: asterisk.service: Failed with result 'core-dump'.
Nov 16 15:06:22 spk-asterisk systemd[1]: Failed to start Asterisk PBX.
dpkg: error processing package asterisk (--configure):
 installed asterisk package post-installation script subprocess returned error exit status 1
Processing triggers for systemd (241-7~deb10u8) ...
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for libc-bin (2.28-10) ...
Errors were encountered while processing:
 asterisk
E: Sub-process /usr/bin/dpkg returned an error code (1)

And this…

root@spk-asterisk:~# systemctl status asterisk.service
● asterisk.service - Asterisk PBX
   Loaded: loaded (/lib/systemd/system/asterisk.service; enabled; vendor preset: enabled)
   Active: failed (Result: core-dump) since Tue 2021-11-16 15:06:31 PST; 4min 32s ago
     Docs: man:asterisk(8)
  Process: 3068 ExecStart=/usr/sbin/asterisk -g -f -p -U asterisk (code=dumped, signal=SEGV)
 Main PID: 3068 (code=dumped, signal=SEGV)

Nov 16 15:06:30 spk-asterisk systemd[1]: asterisk.service: Failed with result 'core-dump'.
Nov 16 15:06:30 spk-asterisk systemd[1]: Failed to start Asterisk PBX.
Nov 16 15:06:31 spk-asterisk systemd[1]: asterisk.service: Service RestartSec=1s expired, scheduling restart.
Nov 16 15:06:31 spk-asterisk systemd[1]: asterisk.service: Scheduled restart job, restart counter is at 6.
Nov 16 15:06:31 spk-asterisk systemd[1]: Stopped Asterisk PBX.
Nov 16 15:06:31 spk-asterisk systemd[1]: asterisk.service: Start request repeated too quickly.
Nov 16 15:06:31 spk-asterisk systemd[1]: asterisk.service: Failed with result 'core-dump'.
Nov 16 15:06:31 spk-asterisk systemd[1]: Failed to start Asterisk PBX.

I would clear out ‘astmoddir’ and recompile.

If that still fails, I would delete anything that ‘find / -xdev | grep --ignore-case asterisk’ shows and re-install from scratch.

using “apt remove asterisk” I removed it from the system then used the original 15.6.2 source dir to recompile what was there. That worked and its back online again. I am still wanting to get it updated only this time I will not be doing it in the middle of a work day when people rely on the server. I take full responsibility for that one. Lesson learned!!

Telephony is one of those things that isn’t perceived as all that important until it isn’t there.

A ‘dev’ environment where you can load your production dialplan and make a test call, even on a VM, can save a lot of heartache.