One of the asterisk boxes suddenly “crashed” now twice in two days. The others are fine at more then 8 million calls processed in 6 months.
The problem is that it didnt actual crash - it just kinda stopped. I could issue some commands on the CLI, but couldnt issue a “sip reload”, or “core restart now”. These commands just did nothing. so i exited the cli, and here you can see what i did:
[code]# service asterisk restart
Stopping safe_asterisk: [ OK ]
Shutting down asterisk: *** glibc detected *** /usr/sbin/asterisk: corrupted double-linked list: 0x00007fa42c53c470 ***
======= Backtrace: =========
/lib64/libc.so.6[0x36cd6760e6]
/lib64/libc.so.6[0x36cd678e74]
/usr/sbin/asterisk[0x43e010]
/usr/sbin/asterisk[0x4f547f]
/usr/sbin/asterisk[0x43dfdc]
/usr/sbin/asterisk[0x43e12e]
/usr/sbin/asterisk[0x43f092]
/usr/sbin/asterisk[0x43faca]
/usr/sbin/asterisk[0x43dfdc]
/usr/sbin/asterisk[0x4ef34a]
/usr/sbin/asterisk[0x43789b]
/usr/sbin/asterisk[0x43bbce]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x36cd61ecdd]
/usr/sbin/asterisk[0x41ea49]
======= Memory map: ========
00400000-00680000 r-xp 00000000 fd:00 32246723 /usr/sbin/asterisk
00880000-0089f000 rw-p 00280000 fd:00 32246723 /usr/sbin/asterisk
0089f000-008da000 rw-p 00000000 00:00 0
00a9e000-00aa9000 rw-p 0029e000 fd:00 32246723 /usr/sbin/asterisk
012be000-031ce000 rw-p 00000000 00:00 0 [heap]
36cce00000-36cce20000 r-xp 00000000 fd:00 42205598 /lib64/ld-2.12.so
36cd01f000-36cd020000 r–p 0001f000 fd:00 42205598 /lib64/ld-2.12.so
36cd020000-36cd021000 rw-p 00020000 fd:00 42205598 /lib64/ld-2.12.so
36cd021000-36cd022000 rw-p 00000000 00:00 0
36cd200000-36cd202000 r-xp 00000000 fd:00 42205606 /lib64/libdl-2.12.so
36cd202000-36cd402000 —p 00002000 fd:00 42205606 /lib64/libdl-2.12.so
36cd402000-36cd403000 r–p 00002000 fd:00 42205606 /lib64/libdl-2.12.so
36cd403000-36cd404000 rw-p 00003000 fd:00 42205606 /lib64/libdl-2.12.so
36cd600000-36cd78a000 r-xp 00000000 fd:00 42205599 /lib64/libc-2.12.so
36cd78a000-36cd989000 —p 0018a000 fd:00 42205599 /lib64/libc-2.12.so
36cd989000-36cd98d000 r–p 00189000 fd:00 42205599 /lib64/libc-2.12.so
36cd98d000-36cd98e000 rw-p 0018d000 fd:00 42205599 /lib64/libc-2.12.so
36cd98e000-36cd993000 rw-p 00000000 00:00 0
36cda00000-36cda17000 r-xp 00000000 fd:00 42205604 /lib64/libpthread-2.12.so
36cda17000-36cdc17000 —p 00017000 fd:00 42205604 /lib64/libpthread-2.12.so
36cdc17000-36cdc18000 r–p 00017000 fd:00 42205604 /lib64/libpthread-2.12.so
36cdc18000-36cdc19000 rw-p 00018000 fd:00 42205604 /lib64/libpthread-2.12.so
36cdc19000-36cdc1d000 rw-p 00000000 00:00 0
36cde00000-36cde83000 r-xp 00000000 fd:00 42205600 /lib64/libm-2.12.so
36cde83000-36ce082000 —p 00083000 fd:00 42205600 /lib64/libm-2.12.so
36ce082000-36ce083000 r–p 00082000 fd:00 42205600 /lib64/libm-2.12.so
36ce083000-36ce084000 rw-p 00083000 fd:00 42205600 /lib64/libm-2.12.so
36ce200000-36ce215000 r-xp 00000000 fd:00 42205601 /lib64/libz.so.1.2.3
36ce215000-36ce414000 —p 00015000 fd:00 42205601 /lib64/libz.so.1.2.3
36ce414000-36ce415000 r–p 00014000 fd:00 42205601 /lib64/libz.so.1.2.3
36ce415000-36ce416000 rw-p 00015000 fd:00 42205601 /lib64/libz.so.1.2.3
36ce600000-36ce607000 r-xp 00000000 fd:00 42205613 /lib64/librt-2.12.so
36ce607000-36ce806000 —p 00007000 fd:00 42205613 /lib64/librt-2.12.so
36ce806000-36ce807000 r–p 00006000 fd:00 42205613 /lib64/librt-2.12.so
36ce807000-36ce808000 rw-p 00007000 fd:00 42205613 /lib64/librt-2.12.so
36cee00000-36cee3f000 r-xp 00000000 fd:00 32246175 /usr/lib64/libjpeg.so.62.0.0
36cee3f000-36cf03f000 —p 0003f000 fd:00 32246175 /usr/lib64/libjpeg.so.62.0.0
36cf03f000-36cf040000 rw-p 0003f000 fd:00 32246175 /usr/lib64/libjpeg.so.62.0.0
36cf040000-36cf050000 rw-p 00000000 00:00 0
36cf200000-36cf216000 r-xp 00000000 fd:00 42205620 /lib64/libresolv-2.12.so
36cf216000-36cf416000 —p 00016000 fd:00 42205620 /lib64/libresolv-2.12.so
36cf416000-36cf417000 r–p 00016000 fd:00 42205620 /lib64/libresolv-2.12.so
36cf417000-36cf418000 rw-p 00017000 fd:00 42205620 /lib64/libresolv-2.12.so
36cf418000-36cf41a000 rw-p 00000000 00:00 0
36cf600000-36cf61d000 r-xp 00000000 fd:00 42205635 /lib64/libtinfo.so.5.7
36cf61d000-36cf81d000 —p 0001d000 fd:00 42205635 /lib64/libtinfo.so.5.7
36cf81d000-36cf821000 rw-p 0001d000 fd:00 42205635 /lib64/libtinfo.so.5.7
36cfa00000-36cfa5d000 r-xp 00000000 fd:00 42205607 /lib64/libfreebl3.so
36cfa5d000-36cfc5c000 —p 0005d000 fd:00 42205607 /lib64/libfreebl3.so
36cfc5c000-36cfc5d000 r–p 0005c000 fd:00 42205607 /lib64/libfreebl3.so
36cfc5d000-36cfc5e000 rw-p 0005d000 fd:00 42205607 /lib64/libfreebl3.so
36cfc5e000-36cfc62000 rw-p 00000000 00:00 0
36cfe00000-36cfe07000 r-xp 00000000 fd:00 42205608 /lib64/libcrypt-2.12.so
36cfe07000-36d0007000 —p 00007000 fd:00 42205608 /lib64/libcrypt-2.12.so
36d0007000-36d0008000 r–p 00007000 fd:00 42205608 /lib64/libcrypt-2.12.so
36d0008000-36d0009000 rw-p 00008000 fd:00 42205608 /lib64/libcrypt-2.12.so
36d0009000-36d0037000 rw-p 00000000 00:00 0
36d0200000-36d0219000 r-xp 00000000 fd:00 32247869 /usr/lib64/libsasl2.so.2.0.23
36d0219000-36d0418000 —p 00019000 fd:00 32247869 /usr/lib64/libsasl2.so.2.0.23
36d0418000-36d0419000 r–p 00018000 fd:00 32247869 /usr/lib64/libsasl2.so.2.0.23
36d0419000-36d041a000 rw-p 00019000 fd:00 32247869 /usr/lib64/libsasl2.so.2.0.23
36d0600000-36d0616000 r-xp 00000000 fd:00 42205327 /lib64/libnsl-2.12.so
36d0616000-36d0815000 —p 00016000 fd:00 42205327 /lib64/libnsl-2.12.so
36d0815000-36d0816000 r–p 00015000 fd:00 42205327 /lib64/libnsl-2.12.so
36d0816000-36d0817000 rw-p 00016000 fd:00 42205327 /lib64/libnsl-2.12.so
36d0817000-36d0819000 rw-p 00000000 00:00 0
36d0a00000-36d0a04000 r-xp 00000000 fd:00 42205627 /lib64/libplc4.so
[FAILED]
Starting asterisk:
service asterisk restart
Stopping safe_asterisk: [ OK ]
Shutting down asterisk: [ OK ]
Starting asterisk:
#[/code]
Because it just “froze up” and didn’t crash - i don’t have the crash dump or anything - the only information is whats on the screen.
I Googled a bit and found similar reports - mostly leading to memory corruption, with some saying things like “show channels” was causing the issue.
The actual “corrupted double-linked list”, seems to come from:
malloc/malloc.c inside the glibc’s code
1543 /* Take a chunk off a bin list */
1544 #define unlink(P, BK, FD) { \
1545 FD = P->fd; \
1546 BK = P->bk; \
1547 if (__builtin_expect (FD->bk != P || BK->fd != P, 0)) \
1548 malloc_printerr (check_action, "corrupted double-linked list", P); \
1549 else { \
1550 FD->bk = BK; \
1551 BK->fd = FD; \
Does anyone have any idea what could be causing the glibc to throw this error - I’m assuming asterisk is asking malloc to perform some memory operation like a delete etc and its either deleted already or its already in use. Also this could just have been a side effect of the shutdown call while Asterisk was “hung”… not sure.
Any ideas??