Unable to start safe_asterisk (Asterisk exited on signal 4)

Asterisk Asterisk Support
1

Hi there,

I hope someone can help me out here cause I’m at the point of pulling my hair out. I’ve tried installing Asterisk with my own documentation which works fine on all servers I tried it on so far and I’ve used the documentation from http://www.powerpbx.org/content/rhel-asterisk-freepbx-install-guide-rhel-v6-asterisk-v18-freepbx-v29 which also hasn’t given me any problems until now.
Because of this I don’t think this is a Asterisk problem nor is it a CentOS problem, but maybe something with the hardware provided by my VPS host http://www.cloudvps.com. The VPS is XEN based.

uname -r 2.6.32-220.4.1.el6.x86_64
However later, or xen, versions of the kernel gave me the same problem.

cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 45 model name : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz stepping : 7 cpu MHz : 2300.092 cache size : 15360 KB fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm rep_good aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes avx hypervisor lahf_lm ida arat xsaveopt pln pts dts bogomips : 4600.18 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management:

The problem:
Once I get to the point in the documentation where I have to give the command:

I get the following output:

/usr/sbin/safe_asterisk: line 147: 32428 Illegal instruction (core dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY} Asterisk ended with exit status 132 Asterisk exited on signal 4. cat: /var/run/asterisk/asterisk.pid: No such file or directory Automatically restarting Asterisk. /usr/sbin/safe_asterisk: line 147: 32435 Illegal instruction (core dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY} Asterisk ended with exit status 132 Asterisk exited on signal 4. cat: /var/run/asterisk/asterisk.pid: No such file or directory Automatically restarting Asterisk. /usr/sbin/safe_asterisk: line 147: 32442 Illegal instruction (core dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY} Asterisk ended with exit status 132 Asterisk exited on signal 4. cat: /var/run/asterisk/asterisk.pid: No such file or directory Automatically restarting Asterisk. ^C [root@sip freepbx-2.10.0]# /usr/sbin/safe_asterisk: line 147: 32449 Illegal instruction (core dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY} Asterisk ended with exit status 132 Asterisk exited on signal 4. cat: /var/run/asterisk/asterisk.pid: No such file or directory Automatically restarting Asterisk.

This goes on and on in a loop untill I press enter and give the command:

Asterisk does not produce any logfiles /var/log/asterisk from I can debug this. I’m really lost here, I googled my @ss of today, but I could find similar situations but no solutions. I have 2 weeks to solve to or cancel my subscribtion with this VPS service provider and refund my money.

I really hope someone is able to help me out of the woods here, and if I need to produce more information just let me know.

Thanks in advance.

Cheers,

R. Roeleveld.

2

Hi

what user and group is safe_asterisk running as and what is asterisk trying to start as.

are the varibles set to start it correctly ?

can you “su asterisk” also is selinux diabled ?

Ian

3

Signal 4, on LInux, is normally illegal instruction. That is typically the result of running a binary compiled for a different processor variant than is actually being used.

4

[quote=“ianplain”]Hi

what user and group is safe_asterisk running as and what is asterisk trying to start as.

are the varibles set to start it correctly ?

can you “su asterisk” also is selinux diabled ?

Ian[/quote]

Hi Ian,

Thanks a lot for responding this fast. I’ve checked the /usr/sbin/safe_asterisk file permissions and these are:

-rwxr-xr-x 1 root root 5527 Aug  6 22:02 /usr/sbin/safe_asterisk

I have changed the user permissions for safe_asterisk to asterisk and left group permissions the same:

-rwxr-xr-x 1 asterisk root 5527 Aug  6 22:02 /usr/sbin/safe_asterisk

This gave me the exact same result. I can do “su asterisk”. When I try to start asterisk safe from within “su asterisk” the error remains as well.
I have now changed back the permissions to what they where.

-rwxr-xr-x 1 root root 5527 Aug  6 22:02 /usr/sbin/safe_asterisk

SELinux is indeed disabled.
What exactly do you mean with:

Thanks in advance,

Cheers,

R. Roeleveld.

5

Hi David,

Thank you also for responding this fast. You are indeed correct, Signal 4 indicates an illegal instruction. You suggestion seems the most viable as I never had any problems running the previous mentioned manual with asterisk 10.4 to the latest 10.7. I have also read similar posts on forums regarding the same issue which had to do with the person trying to run asterisk on a non supported processor, I think it was a pentium I.
However my server uses a:

Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz processor

which (according to Intel) was released Q1 of 2012. So it’s one of the latest processors available.
Is there anyway I can check if asterisk compiled for the correct processor, or at least force asterisk to compile using this processor?

Thanks in advance, cheers,

R. Roeleveld.

6

These options on configure should force the processor assumptions:

System types:
–build=BUILD configure for building on BUILD [guessed]
–host=HOST cross-compile to build programs to run on HOST [BUILD]

Getting a backtrace, from the core dump, should indicate which instruction is causing the problem and which module it is in.

7

Sorry in advance if what I’m going to ask now is really stupid :stuck_out_tongue:

I’ve did a make clean all in the asterisk directory and then did ./configure --build=BUILD --host=HOST which gave the following output:

checking build system type... Invalid configuration `BUILD': machine `BUILD' not recognized configure: error: /bin/sh ./config.sub BUILD failed

I assume instead of BUILD and HOST I have to fill in something matching my system. Could you explain to me what they mean? And how I could figure out what to type there.

I know where the core dumps are being stored, but how would I make those readable?

Cheers,

R. Roeleveld.

8

Ok, I found somewhere on a webpage regarding asterisk:

Do you think this is suitable for my processor?

Cheers,

R. Roeleveld.

9

Those examples are for 64 bit processors. I thought the Xeon’s were 32 bit. If 32 bit, i386-linux-gnu is likely to generate the most conservative code.

Google “asterisk wiki backtrace” for information on getting backtraces. You may need to use additional gdb commands to find the actual machine instruction.

10

gcc.gnu.org/onlinedocs/gcc/i386- … tions.html

11

[quote=“ictinc”]Ok, I found somewhere on a webpage regarding asterisk:

[/quote]

I’ve removed the asterisk files and recompiled with the above flags, however the problem remains.

On http://ark.intel.com/products/64593/Intel-Xeon-Processor-E5-2630-(15M-Cache-2_30-GHz-7_20-GTs-Intel-QPI) it states that the processor has a 64 bits instruction set, so I assume the flags I set where correct.

I could retry with the flags you mentioned, see if that will make any difference…

Cheers

12

Is there any possibility that you have some old modules in the Asterisk modules directory?

Otherwise, this could be a compiler bug. I assume that running a 32 bit application should be OK, so using a conservative architecture specification ought to help, but won’t give the best performance.

13

Hi there,

Just to be sure I’ve tried:

however with no luck. Still the same issue arises.

I have to go into a meeting right now, when I get back I’ll have a look into the asterisk trace.

thanks for bearing with me so far.

Cheers…

14

[quote=“david55”]Is there any possibility that you have some old modules in the Asterisk modules directory?

Otherwise, this could be a compiler bug. I assume that running a 32 bit application should be OK, so using a conservative architecture specification ought to help, but won’t give the best performance.[/quote]

I suspected it to be a gcc bug, gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC)

However I tried installing a higher and lower version, but without luck…

I’m now thinking if it would be wise to switch from CentOS 6.3 to Debian 6 or Ubuntu 12.04 LTS.

Cheers,

R. Roeleveld.

15

Hi there,

Well I’m really confused right now. I’ve installed Ubuntu 12.04 LTS and installed Asterisk 10.7 and tried 1.8.15.

Both are doing showing the same problems as with CentOS 6.3 although now I’m using gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5).

Could it be that there’s just something weird with the hardware of my provider?

Any ideas where I could find out where it’s going wrong?

Cheers,

R. Roeleveld.

16

You need to get a backtrace and identify the machine instruction that is being faulted. You may only get the source line, by default, so you may have to explicitly disassemble it. I’m sorry, but I can’t remember the gdb commands to disassemble it, off the top of my head.

17

Ok, thanks… I’ll try doing that in the morning. For now I’ve installed CentOS 5.7 which works like a charm…
I’ll watch it for tonight to see if it remains stable, and return to CentOS 6.3 tomorrow.

I’ll post the output here…

Thanks you so much for your help so far…!!

18

Hi David,

Sorry it took me a bit longer than I expected, however I do have the output of the backtrace for you. I hope you can make sense of it, cause I can’t. However I’m not a developer though… :stuck_out_tongue:

Thanks in advance for having a look at this for me…

# gdb -se "asterisk" -ex "bt full" -ex "thread apply all bt" --batch -c core.7059 > /tmp/backtrace.txt

[New Thread 7059]
[New Thread 7060]
[Thread debugging using libthread_db enabled]
Core was generated by `/usr/sbin/asterisk -f -g -vvvg -c'.
Program terminated with signal 4, Illegal instruction.
#0  0x0000000000561c73 in timesub (timep=0x7fff00276430, offset=7200, sp=0x1fa3170, tmp=0x7fff00276690) at stdtime/localtime.c:1784
1784		seconds = tdays * SECSPERDAY + 0.5;
#0  0x0000000000561c73 in timesub (timep=0x7fff00276430, offset=7200, sp=0x1fa3170, tmp=0x7fff00276690) at stdtime/localtime.c:1784
        lp = 0x7c0000007f
        tdays = 220
        idays = 32695
        rem = 82692
        y = 2012
        ip = 0x19ea0
        corr = 0
        hit = 0
        i = -1
        seconds = 140426549161600
#1  0x00000000005610c8 in localsub (timep=0x7fff00276680, offset=0, tmp=0x7fff00276690, sp=0x1fa3170) at stdtime/localtime.c:1539
        ttisp = 0x1fa5dc0
        i = 11
        result = 0x6552d4
        t = {tv_sec = 1344466692, tv_usec = 357126}
#2  0x000000000056116e in ast_localtime (timep=0x7fff00276680, tmp=0x7fff00276690, zone=0x0) at stdtime/localtime.c:1555
        sp = 0x1fa3170
#3  0x00000000004f1f41 in ast_log (level=5, file=0x61a874 "asterisk.c", line=3495, function=0x61edc0 "main", fmt=0x6404ef "%s") at logger.c:1249
        logmsg = 0x1fa2990
        buf = 0x1fa2780
        tm = {tm_sec = 0, tm_min = 0, tm_hour = 0, tm_mday = 0, tm_mon = 0, tm_year = 0, tm_wday = 0, tm_yday = 0, tm_isdst = 0, tm_gmtoff = 0, tm_zone = 0x0, tm_usec = 0}
        now = {tv_sec = 1344466692, tv_usec = 357126}
        res = 486
        ap = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = 0x7fff00276830, reg_save_area = 0x7fff00276760}}
        datestring = "\210\021\026\233\267\177\000\000\246\203A\000\000\000\000\000\017\204\035\346\000\000\000\000:\236\364\232\267\177\000\000\000\000\000\000\000\000\000\000\220\375\024\233\267\177\000\000\001", '\000' <repeats 15 times>, "\001\000\000\000\000\000\000\000\210\021\026\233\267\177\000\000\210\021\026\233\267\177\000\000\000\000\000\000\000\000\000\000\220\375\024\233\267\177\000\000\020f'\000\377\177\000\000\b%\372\001\000\000\000\000\000g'\000\377\177\000\000@h'\000\377\177\000\000\240g'\000\377\177\000\000o\002", '\000' <repeats 14 times>, "\b%\372\001\000\000\000\000\262|b\232\267\177\000\000\001\200\255\373\000\000\000\000\b%\372\001\000\000\000\000\b%\372\001\000\000\000\000\b%\372\001\000\000\000\000\b%\372\001\000\000\000\000\356&\372\001\000\000\000\000w'\372\001\000\000\000\000\b%\372\001\000\000\000\000w'\372\001", '\000' <repeats 11 times>
        __PRETTY_FUNCTION__ = "ast_log"
#4  0x00000000004f350c in __ast_verbose_ap (file=0x61a874 "asterisk.c", line=3495, func=0x61edc0 "main", fmt=0x7fff00276840 "\177Asterisk %s, Copyright (C) 1999 - 2012 Digium, Inc. and others.\nCreated by Mark Spencer <markster@digium.com>\nAsterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.\nThis i"..., ap=0x7fff00276b90) at logger.c:1523
        buf = 0x1fa24f0
        res = 486
#5  0x00000000004f3608 in __ast_verbose (file=0x61a874 "asterisk.c", line=3495, func=0x61edc0 "main", fmt=0x61cc70 "Asterisk %s, Copyright (C) 1999 - 2012 Digium, Inc. and others.\nCreated by Mark Spencer <markster@digium.com>\nAsterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.\nThis is"...) at logger.c:1531
        ap = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x7fff00276c80, reg_save_area = 0x7fff00276bb0}}
#6  0x00000000004431cc in main (argc=5, argv=0x7fff00278278) at asterisk.c:3495
        c = -1
        filename = "/root/.asterisk_history", '\000' <repeats 56 times>
        hostname = "sip.ictinc.nl", '\000' <repeats 50 times>
        tmp = '\000' <repeats 24 times>, "\004\000\000\000\061\000\000\000[\000\000\000|\000\000\000w\000\000\000n\000\000\000\346\004\000\000\001\000\000\000 \223\212\000\000\000\000\000@\373\001\000\000\000\000\000\300\324\371\001\000\000\000"
        xarg = 0x0
        x = 5
        f = 0x1f9d4d0
        sigs = {__val = {140733195976592, 140426557657480, 140733195976616, 6884670984, 0, 4263512, 140426545467768, 140426557603840, 140426530126832, 140426557595648, 140733195976680, 4294967295, 4, 1, 0, 96}}
        num = 32695
        isroot = 1
        rundir_exists = 0
        buf = 0x439960 "\353\001\220\311\303UH\211\345H\203\354@H\211}\310A\270\201\250a"
        runuser = 0x0
        rungroup = 0x0
        remotesock = 0x0
        moduleresult = 808466742
        l = {rlim_cur = 0, rlim_max = 140426557586832}
        __PRETTY_FUNCTION__ = "main"
        __func__ = "main"

Thread 2 (Thread 0x7fb79b14a700 (LWP 7060)):
#0  0x00007fb79a6933dd in read () from /lib64/libc.so.6
#1  0x000000000055e2de in inotify_daemon (data=0x0) at stdtime/localtime.c:294
#2  0x0000000000576ee0 in dummy_start (data=0x1f9c270) at utils.c:1014
#3  0x00007fb799725851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fb79a6a06dd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fb79b14b7c0 (LWP 7059)):
#0  0x0000000000561c73 in timesub (timep=0x7fff00276430, offset=7200, sp=0x1fa3170, tmp=0x7fff00276690) at stdtime/localtime.c:1784
#1  0x00000000005610c8 in localsub (timep=0x7fff00276680, offset=0, tmp=0x7fff00276690, sp=0x1fa3170) at stdtime/localtime.c:1539
#2  0x000000000056116e in ast_localtime (timep=0x7fff00276680, tmp=0x7fff00276690, zone=0x0) at stdtime/localtime.c:1555
#3  0x00000000004f1f41 in ast_log (level=5, file=0x61a874 "asterisk.c", line=3495, function=0x61edc0 "main", fmt=0x6404ef "%s") at logger.c:1249
#4  0x00000000004f350c in __ast_verbose_ap (file=0x61a874 "asterisk.c", line=3495, func=0x61edc0 "main", fmt=0x7fff00276840 "\177Asterisk %s, Copyright (C) 1999 - 2012 Digium, Inc. and others.\nCreated by Mark Spencer <markster@digium.com>\nAsterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.\nThis i"..., ap=0x7fff00276b90) at logger.c:1523
#5  0x00000000004f3608 in __ast_verbose (file=0x61a874 "asterisk.c", line=3495, func=0x61edc0 "main", fmt=0x61cc70 "Asterisk %s, Copyright (C) 1999 - 2012 Digium, Inc. and others.\nCreated by Mark Spencer <markster@digium.com>\nAsterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details.\nThis is"...) at logger.c:1531
#6  0x00000000004431cc in main (argc=5, argv=0x7fff00278278) at asterisk.c:3495
# gdb set logging on -se "asterisk" -c core.7059 | tee /tmp/backtrace01.txt

GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
[New Thread 7059]
[New Thread 7060]
Core was generated by `/usr/sbin/asterisk -f -g -vvvg -c'.
Program terminated with signal 4, Illegal instruction.
#0  0x0000000000561c73 in ?? ()
e[?1034h(gdb) bt
#0  0x0000000000561c73 in ?? ()
#1  0x00007fff00276690 in ?? ()
#2  0x0000000001fa3170 in ?? ()
#3  0x0000000000001c20 in ?? ()
#4  0x00007fff00276430 in ?? ()
#5  0x0000000000000004 in ?? ()
#6  0x000007dc000007dc in ?? ()
#7  0x0000007c0000007f in ?? ()
#8  0x00000000000000dc in ?? ()
#9  0x00007fb79a946e80 in ?? ()
#10 0x0000000000014304 in ?? ()
#11 0x0000000000019ea0 in ?? ()
#12 0x0000000000000000 in ?? ()
(gdb) bt full
#0  0x0000000000561c73 in ?? ()
No symbol table info available.
#1  0x00007fff00276690 in ?? ()
No symbol table info available.
#2  0x0000000001fa3170 in ?? ()
No symbol table info available.
#3  0x0000000000001c20 in ?? ()
No symbol table info available.
#4  0x00007fff00276430 in ?? ()
No symbol table info available.
#5  0x0000000000000004 in ?? ()
No symbol table info available.
#6  0x000007dc000007dc in ?? ()
No symbol table info available.
#7  0x0000007c0000007f in ?? ()
No symbol table info available.
#8  0x00000000000000dc in ?? ()
No symbol table info available.
#9  0x00007fb79a946e80 in ?? ()
No symbol table info available.
#10 0x0000000000014304 in ?? ()
No symbol table info available.
#11 0x0000000000019ea0 in ?? ()
No symbol table info available.
#12 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) thread apply all bt

Thread 2 (Thread 7060):
#0  0x00007fb79a6933dd in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 1 (Thread 7059):
#0  0x0000000000561c73 in ?? ()
#1  0x00007fff00276690 in ?? ()
#2  0x0000000001fa3170 in ?? ()
#3  0x0000000000001c20 in ?? ()
#4  0x00007fff00276430 in ?? ()
#5  0x0000000000000004 in ?? ()
#6  0x000007dc000007dc in ?? ()
#7  0x0000007c0000007f in ?? ()
#8  0x00000000000000dc in ?? ()
#9  0x00007fb79a946e80 in ?? ()
#10 0x0000000000014304 in ?? ()
#11 0x0000000000019ea0 in ?? ()
#12 0x0000000000000000 in ?? ()
(gdb) quit

I really hope this has produced some useful information. I have never had to do this before as Asterisk has never let me down. :stuck_out_tongue: I got these commands from:
https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace So I’ll assume it’s what you needed from me. If you need me to produce more information just let me know.

Best regards,

R. Roeleveld.

19

The second trace has no debug information, so is of no use. I’m surprised that the first one generates an illegal instruction, but you need to issue the command:

disassemble

and page through until you reach the line corresponding to 0x0561c73 in the backtrace you have. It will disassemble the whole function.

By the way, does CentOS 5.7 support 64 bit; maybe that works because you get a 32 bit build.

20

Hi there,

Thanks for your response… After searching the internet and trying out several commands the only command and output I can come up with is the following:

[root@sip backtrace]# gdb disas -se "asterisk" -c core.7059 | tee backtrace_new.txt GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... disas: No such file or directory. Missing separate debuginfo for the main executable file Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/04/70f567637573e5b25c054280a4f9b7f70613c1 [New Thread 7059] [New Thread 7060] Core was generated by `/usr/sbin/asterisk -f -g -vvvg -c'. Program terminated with signal 4, Illegal instruction. #0 0x0000000000561c73 in ?? ()

I’m pretty sure this is not what you where hoping to receive from me. I know I’m asking a lot now, please bear with me as I’ve never done this before and would like to learn from this, however would you happen to have the exact command for me to get the right information. I’ve searched for asterisk disassembly, gdb disassembly but came up almost empty handed until now. I will keep searching in the mean time tough.

Cheers, and thanks.

R. Roeleveld.