Getting weird Segfaults

system · July 17, 2012, 3:47am

Hello All,

We have been getting weird segfaults from one of our asterisk servers:

asterisk[29214]: segfault at 00002aaabbd3d5a0 rip 000000000045163c rsp 0000000040b4c400 error 4
asterisk[29438]: segfault at 00002aaabbd3d5a0 rip 000000000045163c rsp 0000000042628400 error 4
asterisk[30506]: segfault at 00002aaabbd3d5e8 rip 00000000004499f3 rsp 00000000427ab448 error 4
asterisk[30602]: segfault at 00002aaabbd3d5e8 rip 00000000004499f3 rsp 0000000040108448 error 4
asterisk[18706]: segfault at 00005555582b6f98 rip 0000003cc3873ba6 rsp 000000004085ae70 error 6
asterisk[29359]: segfault at 00002aaabffaa5a8 rip 0000003cc3872cc8 rsp 0000000040d97e70 error 6
asterisk[17860]: segfault at 00002aaabbd3d5a0 rip 000000000045163c rsp 0000000040a48400 error 4
asterisk[18049]: segfault at 00002aaabbd3d560 rip 00002aaaab529846 rsp 00000000426673e0 error 4
asterisk[2440]: segfault at 00002aaabbd3d5a0 rip 000000000045163c rsp 0000000042e03400 error 4
asterisk[9794]: segfault at 00005555745d1178 rip 0000003cc3872cc8 rsp 00000000403d0870 error 6
asterisk[31205]: segfault at 00002aaabbd3d5a0 rip 000000000045163c rsp 0000000042a16400 error 4
asterisk[16637]: segfault at 00002aaadaaa4d08 rip 0000003cc3873ba6 rsp 0000000042e0fbc0 error 6

Any help or suggestions would be appreciated. The issue tracker closed for the 1.6.x Branch so I can’t ping anyone there.

Wondering if anyone has any ideas? My 1st thought was a wonky h323 module, but we have that running elsewhere without issue.

We are running asterisk 1.6.2, compiled locally on the server. We have 1 x Sangoma A104, 4 port T1 card. Dell’s Hardware viewer and dmesg shows no T1 drops or any other hardware issues.

system · July 17, 2012, 3:54am

Here’s the backtrace:

[code][New Thread 16637]
[New Thread 16635]
[New Thread 16634]
[New Thread 16631]
[New Thread 16628]
[New Thread 16627]
[New Thread 16593]
[New Thread 16568]
[New Thread 16564]
[New Thread 16530]
[New Thread 16529]
[New Thread 16522]
[New Thread 16507]
[New Thread 16502]
[New Thread 16500]
[New Thread 16477]
[New Thread 16459]
[New Thread 16423]
[New Thread 16417]
[New Thread 16372]
[New Thread 16369]
[New Thread 15450]
[New Thread 15413]
[New Thread 15022]
[New Thread 13966]
[New Thread 13787]
[New Thread 11031]
[New Thread 31415]
[New Thread 24468]
[New Thread 24466]
[New Thread 24464]
[New Thread 24461]
[New Thread 24264]
[New Thread 24263]
[New Thread 24262]
[New Thread 24261]
[New Thread 24260]
[New Thread 24259]
[New Thread 24258]
[New Thread 24257]
[New Thread 24256]
[New Thread 24255]
[New Thread 24254]
[New Thread 24253]
[New Thread 24252]
[New Thread 24251]
[New Thread 24250]
[New Thread 24249]
[New Thread 24248]
[New Thread 24246]
[New Thread 24245]
[New Thread 24244]
[New Thread 24243]
[New Thread 24242]
[New Thread 24239]
[New Thread 24238]
[New Thread 24237]
[New Thread 24236]
[New Thread 24235]
[New Thread 24234]
[New Thread 24233]
[New Thread 24232]
[New Thread 24231]
[New Thread 24230]
[New Thread 24229]
[New Thread 24228]
[New Thread 24225]
[New Thread 24224]
[New Thread 24223]
[New Thread 24222]
[New Thread 24221]
[New Thread 24220]
[New Thread 24219]
[New Thread 24218]
[New Thread 24213]
[Thread debugging using libthread_db enabled]
Core was generated by /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c'. Program terminated with signal 11, Segmentation fault. #0 0x0000003cc3873ba6 in _int_malloc () from /lib64/libc.so.6 #0 0x0000003cc3873ba6 in _int_malloc () from /lib64/libc.so.6 No symbol table info available. #1 0x0000003cc3874cde in malloc () from /lib64/libc.so.6 No symbol table info available. #2 0x000000000048b0c0 in _ast_malloc (size=160) at /usr/src/asterisk-1.6.2.14/include/asterisk/utils.h:439 __PRETTY_FUNCTION__ = "_ast_malloc" #3 ast_smoother_new (size=160) at frame.c:181 s = 0x0 __PRETTY_FUNCTION__ = "ast_smoother_new" #4 0x00000000004cf04f in ast_rtp_write (rtp=0x2aaac819d9d0, _f=0x2aaac80151b8) at rtp.c:3898 fmt = {bits = 4, name = 0x53f668 "ulaw", samplespersecond = 8000, desc = 0x53f66d "G.711 u-law", fr_len = 80, min_ms = 10, max_ms = 150, inc_ms = 10, def_ms = 20, flags = 0, cur_ms = 20} f = <value optimized out> codec = 0 subclass = 4 __PRETTY_FUNCTION__ = "ast_rtp_write" #5 0x00002aaabba23653 in ?? () from /usr/lib/asterisk/modules/chan_ooh323.so No symbol table info available. #6 0x000000000045273b in ast_write (chan=0x2aaaac14f9a0, fr=0x2aaac80ca840) at channel.c:3835 res = -939024912 f = 0x1f4 count = <value optimized out> __PRETTY_FUNCTION__ = "ast_write" #7 0x0000000000487b86 in ast_readaudio_callback (s=0x2aaac8069de8) at file.c:751 fr = 0x2aaac80ca840 whennext = 160 __PRETTY_FUNCTION__ = "ast_readaudio_callback" #8 0x0000000000487d46 in ast_playstream (s=0x2aaac8000020) at file.c:858 res = <value optimized out> #9 0x00002aaaab529319 in handle_streamfile (chan=0x2aaaac14f9a0, agi=0x42e11870, argc=<value optimized out>, argv=<value optimized out>) at res_agi.c:1222 res = <value optimized out> fs = 0x2aaac8069de8 vfs = 0x0 sample_offset = 0 max_length = 65449 edigits = <value optimized out> __PRETTY_FUNCTION__ = "handle_streamfile" #10 0x00002aaaab527ab7 in agi_handle_command (chan=0x2aaaac14f9a0, agi=0x42e11870, buf=<value optimized out>, dead=0) at res_agi.c:2803 argv = {0x42e10460 "STREAM", 0x42e10467 "FILE", 0x42e1046c "custom/spops/spOpsWelcomePrompt", 0x42e1048c "", 0x0, 0x42e10410 "\234\030\341B", 0x42e10330 "D\357R\253\252*", 0x12978a70 "vent: Start\r\nChannel: OOH323/myfriend1-c805dff0\r\nCommandId: 569237870\r\nCommand: STREAM FILE \"custom/spops/spOpsWelcomePrompt\" \"\"\r\n\r\n", 0x12978ba6 "ECORD_ID", 0x0, 0x12978a30 "_\001", 0x42e10380 "a\004\341B", 0x54d930 "Channel: %s\r\nVariable: %s\r\nValue: %s\r\nUniqueid: %s\r\n", 0x42e10290 "\263d", 0x0, 0x4f29f5 "\211ƋD$\030I\213]", 0x0, 0x0, 0x546451 "\r\n", 0xd0000008c <Address 0xd0000008c out of bounds>, 0x8c <Address 0x8c out of bounds>, 0x8c <Address 0x8c out of bounds>, 0x3000000018 <Address 0x3000000018 out of bounds>, 0x42e100e0 "\240\002\341B", 0x0, 0x2aaacc089a07 "", 0x2aaacc089908 "200 result=0\n", 0x2aaacc089908 "200 result=0\n", 0x42e10270 "", 0x42e10110 "\001\200\255\373\252*", 0x0, 0x2aaacc089a07 "", 0x2aaacc089908 "200 result=0\n", 0x2aaacc089908 "200 result=0\n", 0x42e102a0 "\030", 0x2aaaab52edb5 "\n", 0xff <Address 0xff out of bounds>, 0x42e10310 "\030", 0x42e10110 "\001\200\255\373\252*", 0x3cc386988a "\211\302H\215\205", <incomplete sequence \360>, 0x2aaafbad8001 <Address 0x2aaafbad8001 out of bounds>, 0x2aaacc089908 "200 result=0\n", 0x2aaacc089908 "200 result=0\n", 0x2aaacc089908 "200 result=0\n", 0x2aaacc089908 "200 result=0\n", 0x2aaacc089909 "00 result=0\n", 0x2aaacc089a07 "", 0x2aaacc089908 "200 result=0\n", 0x2aaacc089a07 "", 0x0, 0x0, 0x0, 0x0, 0x0, 0x51580 <Address 0x51580 out of bounds>, 0x0, 0x50000000 <Address 0x50000000 out of bounds>, 0x0, 0xffffffff <Address 0xffffffff out of bounds>, 0x42e10380 "a\004\341B", 0x54d930 "Channel: %s\r\nVariable: %s\r\nValue: %s\r\nUniqueid: %s\r\n", 0x0, 0x0, 0x42e10330 "D\357R\253\252*", 0xffffffff <Address 0xffffffff out of bounds>, 0xc <Address 0xc out of bounds>, 0x2 <Address 0x2 out of bounds>, 0x3cc3b50460 "", 0x0, 0x64 <Address 0x64 out of bounds>, 0x500064b3 <Address 0x500064b3 out of bounds>, 0x3cc38cb198 "뿉\302H\213\005\005^(", 0xffffffff <Address 0xffffffff out of bounds>, 0x42e10400 "", 0x4 <Address 0x4 out of bounds>, 0x450db4 "\205\300\211\306\017\205\235\003", 0x500064b3 <Address 0x500064b3 out of bounds>, 0x4fef6c "\203", <incomplete sequence \370>, 0x500064b3 <Address 0x500064b3 out of bounds>, 0x4b1a <Address 0x4b1a out of bounds>, 0x0, 0x100000000 <Address 0x100000000 out of bounds>, 0x800000000 <Address 0x800000000 out of bounds>, 0x9ffffffff <Address 0x9ffffffff out of bounds>, 0x0, 0x1 <Address 0x1 out of bounds>, 0x0, 0x3cc388c2dd "=\001\360\377\377s\005H\203\304\b\303H\213\r\270L,", 0x500064b3 <Address 0x500064b3 out of bounds>, 0x4ff29e "H\213t$\030H\213L$\020H\272\317\367S㥛\304 H\201\306@B\017", 0x3000000018 <Address 0x3000000018 out of bounds>, 0x4b1a <Address 0x4b1a out of bounds>, 0x500064b3 <Address 0x500064b3 out of bounds>, 0x4b1c <Address 0x4b1c out of bounds>, 0x300000100 <Address 0x300000100 out of bounds>, 0x30000010f <Address 0x30000010f out of bounds>, 0x300000113 <Address 0x300000113 out of bounds>, 0x10003000000fb <Address 0x10003000000fb out of bounds>, 0xfb <Address 0xfb out of bounds>, 0x42e11460 "\001", 0x2aaac81acde0 "\210&\255\373\252*", 0x2aaaab5275e7 "H\213\234$", <incomplete sequence \350>, 0x2aaac81acde0 "\210&\255\373\252*", 0x3cc386bc6c "H\203\370\377H\211\302tN\203#\357H\211\203\220", 0x3000000018 <Address 0x3000000018 out of bounds>, 0x42e10410 "\234\030\341B", 0x42e10330 "D\357R\253\252*", 0x2aaacc0898f0 "", 0x2aaaab52ef44 "SUCCESS", 0x3cc38c6289 "H\213D$\bH\203\304(H=\001\360\377\377s\001\303H\213\r\006\255(", 0x0, 0x31 <Address 0x31 out of bounds>, 0x2aaaabfef000 "STREAM FILE \"custom/spops/spOpsWelcomePrompt\" \"\"\n", 0x1000 <Address 0x1000 out of bounds>, 0x2aaac81acde0 "\210&\255\373\252*", 0x30 <Address 0x30 out of bounds>, 0x2aaaabfef001 "TREAM FILE \"custom/spops/spOpsWelcomePrompt\" \"\"\n", 0x7fe <Address 0x7fe out of bounds>, 0x42e10461 "TREAM", 0x2aaac81acde0 "\210&\255\373\252*", 0x42e10461 "TREAM", 0x3cc3862716 "L+l$\020I\211^\bH\203\304\030[]K\215\004,A\\A]A^A_\303H\203<$", 0x0, 0x1c81acde0 <Address 0x1c81acde0 out of bounds>, 0x42e10460 "STREAM", 0x2aaac81acde0 "\210&\255\373\252*", 0x42e10460 "STREAM", 0x0} argc = 4 res = <value optimized out> c = 0x2aaaab7367a8 command_id = 569237870 __PRETTY_FUNCTION__ = "agi_handle_command" #11 0x00002aaaab52e200 in run_agi (chan=0x2aaaac14f9a0, data=<value optimized out>, enhanced=<value optimized out>, dead=0) at res_agi.c:2963 __PRETTY_FUNCTION__ = "run_agi" #12 agi_exec_full (chan=0x2aaaac14f9a0, data=<value optimized out>, enhanced=<value optimized out>, dead=0) at res_agi.c:3249 status = 0 res = AGI_RESULT_SUCCESS buf = "agi://localhost/REDACTED", '\000' <repeats 2013 times> fds = {251, 251} efd = -1 pid = -1 args = {argc = 1, argv = 0x42e11468, arg = {0x42e10c60 "agi://localhost/REDACTED", 0x0 <repeats 127 times>}} agi = {fd = 251, audio = -1, ctrl = 251, fast = 1, speech = 0x0} __PRETTY_FUNCTION__ = "agi_exec_full" #13 0x00000000004b308d in pbx_exec (c=0x2aaaac14f9a0, app=0x12343370, data=0x42e139c0) at pbx.c:1352 res = <value optimized out> u = 0x2aaac8095520 saved_c_appl = 0x0 saved_c_data = 0x0 __PRETTY_FUNCTION__ = "pbx_exec" #14 0x00000000004bf39c in pbx_extension_helper (c=0x2aaaac14f9a0, con=<value optimized out>, context=0x2aaaac14fd60 "REDACTED", exten=0x2aaaac14fdb0 "s", priority=2, label=0x0, callerid=0x2aaaac1d8690 "REDACTED", action=E_SPAWN, found=0x42e1603c, combined_find_spawn=1) at pbx.c:3715 e = <value optimized out> app = 0x12343370 res = <value optimized out> q = {incstack = {0x0 <repeats 128 times>}, stacklen = 0, status = 5, swo = 0x0, data = 0x0, foundcontext = 0x2aaaac14fd60 "sp-ops-welcome"} passdata = "agi://localhost/REDACTED\000 EXTEN is REDACTED, Caller: REDACTED\"\000\064\065\060\071\070\065\063\064\"\000csv/ast_h323.csv", '\000' <repeats 33 times>"\240, :\341B", '\000' <repeats 12 times>"\307, \327\002̪*\000\000)\311\002̪*\000\000)\311\002̪*\000\000\060<\341B\000\000\000\000WJ\274\253\252*\000\000\236\016\000\000\000\000\000\000\220<\341B\000\000\000\000\240:\341B\000\000\000\000\212\230\206\303<\000\000\000\001\200\255\373\000\000\000\000)\311\002̪*\000\000)\311\002̪*\000\000)\311\002̪*\000\000"... matching_action = 0 __PRETTY_FUNCTION__ = "pbx_extension_helper" #15 0x00000000004c2122 in ast_spawn_extension (c=0x2aaaac14f9a0, args=0x0) at pbx.c:4193 No locals. #16 __ast_pbx_run (c=0x2aaaac14f9a0, args=0x0) at pbx.c:4287 digit = 0 invalid = <value optimized out> dst_exten = "8\341B\000\000\000\000@\341B", '\000' <repeats 12 times>, "H\341B", ‘\000’ <repeats 20 times>, “\030”, ‘\000’ <repeats 47 times>, “P\341B\000\000\000\000X\341B\000\000\000\000``\341B\000\000\000\000h\341B\000\000\000\000x\341B\000\000\000\000\000\000\000\000\004\000\000\000\200`\341B\000\000\000\000\024\323\000\304<”, ‘\000’ <repeats 27 times>"\340, )\265\303<", ‘\000’ <repeats 11 times>, “p\030\254\022\000\000\000\000\220\a\002\000\000\000\000\000\340)\265\303<\000\000\000\030\000\000\000\000\000\000\000]I\207\303<”, ‘\000’ <repeats 11 times>"\360, \201\036\254\252*\000"
pos = 0
found = 1
res = 0
error = 0
PRETTY_FUNCTION = “__ast_pbx_run”
#17 0x00000000004c36fb in pbx_thread (data=0x2aaac8000020) at pbx.c:4574
No locals.
#18 0x00000000004fddbc in dummy_start (data=) at utils.c:968
__cancel_buf = {__cancel_jmp_buf = {{__cancel_jmp_buf = {46912520487408, -7514000970290468248, 0, 3, 5234016, 4096,
-7514000969265050888, -7514000970294315062}, __mask_was_saved = 0}}, __pad = {0x42e161d0, 0x0, 0x3cc3b4d880, 0x3cc3b4d888}}
__cancel_arg = 0x42e16940
not_first_call =
ret =
#19 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#20 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 75 (Thread 0x2ba8e3bb7b80 (LWP 24213)):
#0 0x0000003cc38c627b in read () from /lib64/libc.so.6
#1 0x000000000050cba0 in read_char (el=0x120585a0, cp=0x7fff4cb5683f “”) at read.c:298
#2 0x0000000000507b60 in el_getc (el=0x120585a0, cp=0x7fff4cb5683f “”) at read.c:350
#3 0x0000000000507fc0 in read_getcmd (el=0x120585a0, nread=0x7fff4cb58d2c) at read.c:243
#4 el_gets (el=0x120585a0, nread=0x7fff4cb58d2c) at read.c:446
#5 0x0000000000434c1d in main (argc=, argv=0x7fff4cb58e48) at asterisk.c:3798
#6 0x0000003cc381d994 in __libc_start_main () from /lib64/libc.so.6
#7 0x000000000041a419 in pthread_attr_setdetachstate ()
#8 0x00007fff4cb58e38 in ?? ()
#9 0x0000000000000000 in ?? ()

Thread 74 (Thread 24218):
#0 0x0000003cc400aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000004f3921 in ast_cond_wait (data=) at /usr/src/asterisk-1.6.2.14/include/asterisk/lock.h:1778
#2 tps_processing_function (data=) at taskprocessor.c:293
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 73 (Thread 24219):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x000000000042ffb7 in listener (unused=) at asterisk.c:1287
#2 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#3 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 72 (Thread 24220):
#0 0x0000003cc38c62fb in write () from /lib64/libc.so.6
#1 0x0000003cc386bab3 in _IO_new_file_write () from /lib64/libc.so.6
#2 0x0000003cc386b9c6 in _IO_new_do_write () from /lib64/libc.so.6
#3 0x0000003cc386ce22 in _IO_new_file_xsputn () from /lib64/libc.so.6
#4 0x0000003cc3861af8 in fputs () from /lib64/libc.so.6
#5 0x000000000043117b in console_verboser (s=) at asterisk.c:1741
#6 0x000000000049cfd5 in logger_print_verbose (data=) at logger.c:963
#7 logger_thread (data=) at logger.c:997
#8 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#9 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#10 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 71 (Thread 24221):
#0 0x0000003cc38c627b in read () from /lib64/libc.so.6
#1 0x00000000004eefa3 in inotify_daemon (data=) at stdtime/localtime.c:259
#2 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#3 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 70 (Thread 24222):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x00000000004fdd55 in ast_wait_for_input (fd=, ms=5000) at utils.c:1057
#2 0x00000000004f4101 in ast_tcptls_server_root (data=) at tcptls.c:248
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 69 (Thread 24223):
#0 0x0000003cc400aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000468ef9 in ast_cond_wait (data=) at /usr/src/asterisk-1.6.2.14/include/asterisk/lock.h:1778
#2 do_devstate_changes (data=) at devicestate.c:543
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 68 (Thread 24224):
#0 0x0000003cc400aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000004f3921 in ast_cond_wait (data=) at /usr/src/asterisk-1.6.2.14/include/asterisk/lock.h:1778
#2 tps_processing_function (data=) at taskprocessor.c:293
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 67 (Thread 24225):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x00000000004842ee in do_parking_thread (ignore=) at features.c:3301
#2 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#3 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 66 (Thread 24228):
#0 0x0000003cc400aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaab2674983 in do_timing (arg=) at /usr/src/asterisk-1.6.2.14/include/asterisk/lock.h:1778
#2 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#3 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 65 (Thread 24229):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x00002aaabf7ff0ed in pri_dchannel (vpri=) at chan_dahdi.c:12683
#2 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#3 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 64 (Thread 24230):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x00002aaabf7ff0ed in pri_dchannel (vpri=) at chan_dahdi.c:12683
#2 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#3 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 63 (Thread 24231):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x00002aaabf7ff0ed in pri_dchannel (vpri=) at chan_dahdi.c:12683
#2 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#3 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 62 (Thread 24232):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x00002aaabf7ff0ed in pri_dchannel (vpri=) at chan_dahdi.c:12683
#2 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#3 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 61 (Thread 24233):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x00002aaabf80eddb in do_monitor (data=) at chan_dahdi.c:9674
#2 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#3 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 60 (Thread 24234):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000496400 in ast_io_wait (ioc=0x2aaaac0c6670, howlong=1000) at io.c:275
#2 0x00002aaabe7cebf8 in do_monitor (data=) at chan_mgcp.c:3462
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 59 (Thread 24235):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x00002aaabcf9b44e in do_monitor (data=) at chan_phone.c:1081
#2 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#3 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 58 (Thread 24236):
#0 0x0000003cc38d47bb in accept () from /lib64/libc.so.6
#1 0x00002aaabc5572a9 in accept_thread (ignore=) at chan_skinny.c:6398
#2 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#3 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 57 (Thread 24237):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000496400 in ast_io_wait (ioc=0x2aaaac0d49d0, howlong=1000) at io.c:275
#2 0x00002aaabc55722d in do_monitor (data=) at chan_skinny.c:6443
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 56 (Thread 24238):
#0 0x0000003cc38cd212 in select () from /lib64/libc.so.6
#1 0x00002aaabba36e98 in ooMonitorChannels () from /usr/lib/asterisk/modules/chan_ooh323.so
#2 0x00002aaabbb0b499 in ooh323c_stack_thread () from /usr/lib/asterisk/modules/chan_ooh323.so
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 55 (Thread 24239):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000496400 in ast_io_wait (ioc=0x12357000, howlong=95) at io.c:275
#2 0x00002aaabba2823c in ?? () from /usr/lib/asterisk/modules/chan_ooh323.so
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 54 (Thread 24242):
#0 0x0000003cc400aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000004f3921 in ast_cond_wait (data=) at /usr/src/asterisk-1.6.2.14/include/asterisk/lock.h:1778
#2 tps_processing_function (data=) at taskprocessor.c:293
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 53 (Thread 24243):
#0 0x0000003cc389a3f1 in nanosleep () from /lib64/libc.so.6
#1 0x00002aaab6f13aa4 in lock_broker (unused=) at func_lock.c:192
#2 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#3 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 52 (Thread 24244):
#0 0x0000003cc400aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000004f3921 in ast_cond_wait (data=) at /usr/src/asterisk-1.6.2.14/include/asterisk/lock.h:1778
#2 tps_processing_function (data=) at taskprocessor.c:293
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 51 (Thread 24245):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000496400 in ast_io_wait (ioc=0x123927f0, howlong=1000) at io.c:275
#2 0x00002aaaabbb11a0 in do_monitor (data=) at chan_sip.c:22683
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 50 (Thread 24246):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000496400 in ast_io_wait (ioc=0x123c3c50, howlong=1000) at io.c:275
#2 0x00002aaab000cc52 in do_monitor (data=) at chan_unistim.c:4568
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 49 (Thread 24248):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000496400 in ast_io_wait (ioc=0x1241e300, howlong=1000) at io.c:275
#2 0x00002aaab103337d in network_thread (ignore=) at pbx_dundi.c:2179
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 48 (Thread 24249):
#0 0x0000003cc389a3f1 in nanosleep () from /lib64/libc.so.6
#1 0x0000003cc389a214 in sleep () from /lib64/libc.so.6
#2 0x00002aaab103f317 in process_precache (ign=) at pbx_dundi.c:2256
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 47 (Thread 24250):
#0 0x0000003cc389a3f1 in nanosleep () from /lib64/libc.so.6
#1 0x0000003cc389a214 in sleep () from /lib64/libc.so.6
#2 0x00002aaab1030ca2 in process_clearcache (ignore=) at pbx_dundi.c:2219
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 46 (Thread 24251):
#0 0x0000003cc389a3f1 in nanosleep () from /lib64/libc.so.6
#1 0x00002aaab021e1ec in scan_thread (unused=) at pbx_spool.c:452
#2 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#3 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 45 (Thread 24252):
#0 0x0000003cc400b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000004ed448 in ast_cond_timedwait (data=) at /usr/src/asterisk-1.6.2.14/include/asterisk/lock.h:1783
#2 sched_run (data=) at sched.c:109
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 44 (Thread 24253):
#0 0x0000003cc400aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaac45c80e3 in ast_cond_wait (data=0x12867e20) at /usr/src/asterisk-1.6.2.14/include/asterisk/lock.h:1778
#2 iax2_process_thread (data=0x12867e20) at chan_iax2.c:11219
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 43 (Thread 24254):
#0 0x0000003cc400aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaac45c80e3 in ast_cond_wait (data=0x12868fd0) at /usr/src/asterisk-1.6.2.14/include/asterisk/lock.h:1778
#2 iax2_process_thread (data=0x12868fd0) at chan_iax2.c:11219
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 42 (Thread 24255):
#0 0x0000003cc400aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaac45c80e3 in ast_cond_wait (data=0x1286a180) at /usr/src/asterisk-1.6.2.14/include/asterisk/lock.h:1778
#2 iax2_process_thread (data=0x1286a180) at chan_iax2.c:11219
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 41 (Thread 24256):
#0 0x0000003cc400aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaac45c80e3 in ast_cond_wait (data=0x1286b330) at /usr/src/asterisk-1.6.2.14/include/asterisk/lock.h:1778
#2 iax2_process_thread (data=0x1286b330) at chan_iax2.c:11219
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 40 (Thread 24257):
#0 0x0000003cc400aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaac45c80e3 in ast_cond_wait (data=0x1286c4e0) at /usr/src/asterisk-1.6.2.14/include/asterisk/lock.h:1778
#2 iax2_process_thread (data=0x1286c4e0) at chan_iax2.c:11219
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 39 (Thread 24258):
#0 0x0000003cc400aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaac45c80e3 in ast_cond_wait (data=0x1286d690) at /usr/src/asterisk-1.6.2.14/include/asterisk/lock.h:1778
#2 iax2_process_thread (data=0x1286d690) at chan_iax2.c:11219
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 38 (Thread 24259):
#0 0x0000003cc400aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaac45c80e3 in ast_cond_wait (data=0x1286e840) at /usr/src/asterisk-1.6.2.14/include/asterisk/lock.h:1778
#2 iax2_process_thread (data=0x1286e840) at chan_iax2.c:11219
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 37 (Thread 24260):
#0 0x0000003cc400aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaac45c80e3 in ast_cond_wait (data=0x1286f9f0) at /usr/src/asterisk-1.6.2.14/include/asterisk/lock.h:1778
#2 iax2_process_thread (data=0x1286f9f0) at chan_iax2.c:11219
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 36 (Thread 24261):
#0 0x0000003cc400aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaac45c80e3 in ast_cond_wait (data=0x12870ba0) at /usr/src/asterisk-1.6.2.14/include/asterisk/lock.h:1778
#2 iax2_process_thread (data=0x12870ba0) at chan_iax2.c:11219
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 35 (Thread 24262):
#0 0x0000003cc400aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaac45c80e3 in ast_cond_wait (data=0x12871d50) at /usr/src/asterisk-1.6.2.14/include/asterisk/lock.h:1778
#2 iax2_process_thread (data=0x12871d50) at chan_iax2.c:11219
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 34 (Thread 24263):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000496400 in ast_io_wait (ioc=0x12862ca0, howlong=-1) at io.c:275
#2 0x00002aaac45a222d in network_thread (ignore=) at chan_iax2.c:11718
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

[/code]

system · July 17, 2012, 3:55am

And the rest of the backtrace:

[code]Thread 33 (Thread 24264):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x000000000042dd85 in monitor_sig_flags (unused=) at asterisk.c:3062
#2 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#3 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 32 (Thread 24461):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x00000000004fdd55 in ast_wait_for_input (fd=, ms=-1) at utils.c:1057
#2 0x00000000004a45e7 in get_input (s=0x40e03030) at manager.c:3160
#3 do_message (s=0x40e03030) at manager.c:3198
#4 0x00000000004a4a30 in session_do (data=) at manager.c:3275
#5 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#6 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#7 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 31 (Thread 24464):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x00000000004fdd55 in ast_wait_for_input (fd=, ms=-1) at utils.c:1057
#2 0x00000000004a45e7 in get_input (s=0x40e7f030) at manager.c:3160
#3 do_message (s=0x40e7f030) at manager.c:3198
#4 0x00000000004a4a30 in session_do (data=) at manager.c:3275
#5 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#6 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#7 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 30 (Thread 24466):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x00000000004fdd55 in ast_wait_for_input (fd=, ms=-1) at utils.c:1057
#2 0x00000000004a45e7 in get_input (s=0x41d86030) at manager.c:3160
#3 do_message (s=0x41d86030) at manager.c:3198
#4 0x00000000004a4a30 in session_do (data=) at manager.c:3275
#5 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#6 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#7 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 29 (Thread 24468):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x00000000004fdd55 in ast_wait_for_input (fd=, ms=-1) at utils.c:1057
#2 0x00000000004a45e7 in get_input (s=0x4131e030) at manager.c:3160
#3 do_message (s=0x4131e030) at manager.c:3198
#4 0x00000000004a4a30 in session_do (data=) at manager.c:3275
#5 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#6 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#7 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 28 (Thread 31415):
#0 0x0000003cc400aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000439e19 in ast_cond_wait (ign=) at /usr/src/asterisk-1.6.2.14/include/asterisk/lock.h:1778
#2 autoservice_run (ign=) at autoservice.c:97
#3 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#4 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 27 (Thread 11031):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x000000000042fc44 in netconsole (vconsole=) at asterisk.c:1234
#2 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#3 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 26 (Thread 13787):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x404eee40, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x404eef4c) at channel.c:2190
#2 0x0000000000451377 in ast_waitfor_n (c=0x404eeb70, n=6, ms=) at channel.c:2448
#3 0x00000000004d2fae in bridge_native_loop (c0=0x2aaac80dcc40, c1=0x2aaacc05dd00, flags=0, fo=0x404ef400, rc=0x404ef3f8, timeoutms=-1)
at rtp.c:4073
#4 ast_rtp_bridge (c0=0x2aaac80dcc40, c1=0x2aaacc05dd00, flags=0, fo=0x404ef400, rc=0x404ef3f8, timeoutms=-1) at rtp.c:4591
#5 0x0000000000456cc2 in ast_channel_bridge (c0=0x2aaac80dcc40, c1=0x0, config=0x404f0620, fo=0x404ef400, rc=0x404ef3f8) at channel.c:5457
#6 0x000000000047a945 in ast_bridge_call (chan=0x2aaac80dcc40, peer=0x2aaacc05dd00, config=0x404f0620) at features.c:2655
#7 0x00002aaab8f75bbd in dial_exec_full (chan=0x2aaac80dcc40, data=, peerflags=0x404f08d0, continue_exec=0x0)
at app_dial.c:2269
#8 0x00002aaab8f78996 in dial_exec (chan=0x404eeb70, data=0x6) at app_dial.c:2353
#9 0x00000000004b308d in pbx_exec (c=0x2aaac80dcc40, app=0x123aae70, data=0x404f29c0) at pbx.c:1352
#10 0x00000000004bf39c in pbx_extension_helper (c=0x2aaac80dcc40, con=, context=0x2aaac80dd000 “sp-routing-transfer”,
exten=0x2aaac80dd050 “s”, priority=2, label=0x0, callerid=0x2aaac804d7e0 “7022185548”, action=E_SPAWN, found=0x404f503c,
combined_find_spawn=1) at pbx.c:3715
#11 0x00000000004c2122 in ast_spawn_extension (c=0x2aaac80dcc40, args=0x0) at pbx.c:4193
#12 __ast_pbx_run (c=0x2aaac80dcc40, args=0x0) at pbx.c:4287
#13 0x00000000004c36fb in pbx_thread (data=0x404eeb70) at pbx.c:4574
#14 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#15 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 25 (Thread 13966):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x42c1fe40, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x42c1ff4c) at channel.c:2190
#2 0x0000000000451377 in ast_waitfor_n (c=0x42c1fb70, n=6, ms=) at channel.c:2448
#3 0x00000000004d2fae in bridge_native_loop (c0=0x2aaaac122430, c1=0x2aaacc066a40, flags=0, fo=0x42c20400, rc=0x42c203f8, timeoutms=-1)
at rtp.c:4073
#4 ast_rtp_bridge (c0=0x2aaaac122430, c1=0x2aaacc066a40, flags=0, fo=0x42c20400, rc=0x42c203f8, timeoutms=-1) at rtp.c:4591
#5 0x0000000000456cc2 in ast_channel_bridge (c0=0x2aaacc066a40, c1=0x0, config=0x42c21620, fo=0x42c20400, rc=0x42c203f8) at channel.c:5457
#6 0x000000000047a945 in ast_bridge_call (chan=0x2aaacc066a40, peer=0x2aaaac122430, config=0x42c21620) at features.c:2655
#7 0x00002aaab8f75bbd in dial_exec_full (chan=0x2aaacc066a40, data=, peerflags=0x42c218d0, continue_exec=0x0)
at app_dial.c:2269
#8 0x00002aaab8f78996 in dial_exec (chan=0x42c1fb70, data=0x6) at app_dial.c:2353
#9 0x00000000004b308d in pbx_exec (c=0x2aaacc066a40, app=0x123aae70, data=0x42c239c0) at pbx.c:1352
#10 0x00000000004bf39c in pbx_extension_helper (c=0x2aaacc066a40, con=, context=0x2aaacc066e00 “sp-routing-transfer”,
exten=0x2aaacc066e50 “s”, priority=2, label=0x0, callerid=0x2aaacc0164c0 “3606784448”, action=E_SPAWN, found=0x42c2603c,
combined_find_spawn=1) at pbx.c:3715
#11 0x00000000004c2122 in ast_spawn_extension (c=0x2aaacc066a40, args=0x0) at pbx.c:4193
#12 __ast_pbx_run (c=0x2aaacc066a40, args=0x0) at pbx.c:4287
#13 0x00000000004c36fb in pbx_thread (data=0x42c1fb70) at pbx.c:4574
#14 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#15 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 24 (Thread 15022):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x000000000042fc44 in netconsole (vconsole=) at asterisk.c:1234
#2 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#3 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 23 (Thread 15413):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x429b3e40, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x429b3f4c) at channel.c:2190
#2 0x0000000000451377 in ast_waitfor_n (c=0x429b3b70, n=6, ms=) at channel.c:2448
#3 0x00000000004d2fae in bridge_native_loop (c0=0x2aaac8053650, c1=0x2aaac81b78d0, flags=0, fo=0x429b4400, rc=0x429b43f8, timeoutms=-1)
at rtp.c:4073
#4 ast_rtp_bridge (c0=0x2aaac8053650, c1=0x2aaac81b78d0, flags=0, fo=0x429b4400, rc=0x429b43f8, timeoutms=-1) at rtp.c:4591
#5 0x0000000000456cc2 in ast_channel_bridge (c0=0x2aaac8053650, c1=0x0, config=0x429b5620, fo=0x429b4400, rc=0x429b43f8) at channel.c:5457
#6 0x000000000047a945 in ast_bridge_call (chan=0x2aaac8053650, peer=0x2aaac81b78d0, config=0x429b5620) at features.c:2655
#7 0x00002aaab8f75bbd in dial_exec_full (chan=0x2aaac8053650, data=, peerflags=0x429b58d0, continue_exec=0x0)
at app_dial.c:2269
#8 0x00002aaab8f78996 in dial_exec (chan=0x429b3b70, data=0x6) at app_dial.c:2353
#9 0x00000000004b308d in pbx_exec (c=0x2aaac8053650, app=0x123aae70, data=0x429b79c0) at pbx.c:1352
#10 0x00000000004bf39c in pbx_extension_helper (c=0x2aaac8053650, con=, context=0x2aaac8053a10 “sp-routing-transfer”,
exten=0x2aaac8053a60 “s”, priority=2, label=0x0, callerid=0x2aaac80e6af0 “8183488663”, action=E_SPAWN, found=0x429ba03c,
combined_find_spawn=1) at pbx.c:3715
#11 0x00000000004c2122 in ast_spawn_extension (c=0x2aaac8053650, args=0x0) at pbx.c:4193
#12 __ast_pbx_run (c=0x2aaac8053650, args=0x0) at pbx.c:4287
#13 0x00000000004c36fb in pbx_thread (data=0x429b3b70) at pbx.c:4574
#14 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#15 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 22 (Thread 15450):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x40932e40, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x40932f4c) at channel.c:2190
#2 0x0000000000451377 in ast_waitfor_n (c=0x40932b70, n=6, ms=) at channel.c:2448
#3 0x00000000004d2fae in bridge_native_loop (c0=0x123e5fa0, c1=0x2aaac8193aa0, flags=0, fo=0x40933400, rc=0x409333f8, timeoutms=-1)
at rtp.c:4073
#4 ast_rtp_bridge (c0=0x123e5fa0, c1=0x2aaac8193aa0, flags=0, fo=0x40933400, rc=0x409333f8, timeoutms=-1) at rtp.c:4591
#5 0x0000000000456cc2 in ast_channel_bridge (c0=0x123e5fa0, c1=0x0, config=0x40934620, fo=0x40933400, rc=0x409333f8) at channel.c:5457
#6 0x000000000047a945 in ast_bridge_call (chan=0x123e5fa0, peer=0x2aaac8193aa0, config=0x40934620) at features.c:2655
#7 0x00002aaab8f75bbd in dial_exec_full (chan=0x123e5fa0, data=, peerflags=0x409348d0, continue_exec=0x0)
at app_dial.c:2269
#8 0x00002aaab8f78996 in dial_exec (chan=0x40932b70, data=0x6) at app_dial.c:2353
#9 0x00000000004b308d in pbx_exec (c=0x123e5fa0, app=0x123aae70, data=0x409369c0) at pbx.c:1352
#10 0x00000000004bf39c in pbx_extension_helper (c=0x123e5fa0, con=, context=0x123e6360 “sp-routing-transfer”,
exten=0x123e63b0 “s”, priority=2, label=0x0, callerid=0x12530170 “8182269291”, action=E_SPAWN, found=0x4093903c, combined_find_spawn=1)
at pbx.c:3715
#11 0x00000000004c2122 in ast_spawn_extension (c=0x123e5fa0, args=0x0) at pbx.c:4193
#12 __ast_pbx_run (c=0x123e5fa0, args=0x0) at pbx.c:4287
#13 0x00000000004c36fb in pbx_thread (data=0x40932b70) at pbx.c:4574
#14 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#15 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 21 (Thread 16369):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x433dfe40, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x433dff4c) at channel.c:2190
#2 0x0000000000451377 in ast_waitfor_n (c=0x433dfb70, n=6, ms=) at channel.c:2448
#3 0x00000000004d2fae in bridge_native_loop (c0=0x2aaacc060d20, c1=0x2aaaac2046e0, flags=0, fo=0x433e0400, rc=0x433e03f8, timeoutms=-1)
at rtp.c:4073
#4 ast_rtp_bridge (c0=0x2aaacc060d20, c1=0x2aaaac2046e0, flags=0, fo=0x433e0400, rc=0x433e03f8, timeoutms=-1) at rtp.c:4591
#5 0x0000000000456cc2 in ast_channel_bridge (c0=0x2aaaac2046e0, c1=0x0, config=0x433e1620, fo=0x433e0400, rc=0x433e03f8) at channel.c:5457
#6 0x000000000047a945 in ast_bridge_call (chan=0x2aaaac2046e0, peer=0x2aaacc060d20, config=0x433e1620) at features.c:2655
#7 0x00002aaab8f75bbd in dial_exec_full (chan=0x2aaaac2046e0, data=, peerflags=0x433e18d0, continue_exec=0x0)
at app_dial.c:2269
#8 0x00002aaab8f78996 in dial_exec (chan=0x433dfb70, data=0x6) at app_dial.c:2353
#9 0x00000000004b308d in pbx_exec (c=0x2aaaac2046e0, app=0x123aae70, data=0x433e39c0) at pbx.c:1352
#10 0x00000000004bf39c in pbx_extension_helper (c=0x2aaaac2046e0, con=, context=0x2aaaac204aa0 “sp-routing-transfer”,
exten=0x2aaaac204af0 “s”, priority=2, label=0x0, callerid=0x2aaaac0fc0e0 “5032565000”, action=E_SPAWN, found=0x433e603c,
combined_find_spawn=1) at pbx.c:3715
#11 0x00000000004c2122 in ast_spawn_extension (c=0x2aaaac2046e0, args=0x0) at pbx.c:4193
#12 __ast_pbx_run (c=0x2aaaac2046e0, args=0x0) at pbx.c:4287
#13 0x00000000004c36fb in pbx_thread (data=0x433dfb70) at pbx.c:4574
#14 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#15 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 20 (Thread 16372):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x42ba3e40, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x42ba3f4c) at channel.c:2190
#2 0x0000000000451377 in ast_waitfor_n (c=0x42ba3b70, n=6, ms=) at channel.c:2448
#3 0x00000000004d2fae in bridge_native_loop (c0=0x2aaacc132970, c1=0x2aaacc09ba50, flags=0, fo=0x42ba4400, rc=0x42ba43f8, timeoutms=-1)
at rtp.c:4073
#4 ast_rtp_bridge (c0=0x2aaacc132970, c1=0x2aaacc09ba50, flags=0, fo=0x42ba4400, rc=0x42ba43f8, timeoutms=-1) at rtp.c:4591
#5 0x0000000000456cc2 in ast_channel_bridge (c0=0x2aaacc132970, c1=0x0, config=0x42ba5620, fo=0x42ba4400, rc=0x42ba43f8) at channel.c:5457
#6 0x000000000047a945 in ast_bridge_call (chan=0x2aaacc132970, peer=0x2aaacc09ba50, config=0x42ba5620) at features.c:2655
#7 0x00002aaab8f75bbd in dial_exec_full (chan=0x2aaacc132970, data=, peerflags=0x42ba58d0, continue_exec=0x0)
at app_dial.c:2269
#8 0x00002aaab8f78996 in dial_exec (chan=0x42ba3b70, data=0x6) at app_dial.c:2353
#9 0x00000000004b308d in pbx_exec (c=0x2aaacc132970, app=0x123aae70, data=0x42ba79c0) at pbx.c:1352
#10 0x00000000004bf39c in pbx_extension_helper (c=0x2aaacc132970, con=, context=0x2aaacc132d30 “sp-routing-transfer”,
exten=0x2aaacc132d80 “s”, priority=2, label=0x0, callerid=0x2aaacc132e80 “4045523365”, action=E_SPAWN, found=0x42baa03c,
combined_find_spawn=1) at pbx.c:3715
#11 0x00000000004c2122 in ast_spawn_extension (c=0x2aaacc132970, args=0x0) at pbx.c:4193
#12 __ast_pbx_run (c=0x2aaacc132970, args=0x0) at pbx.c:4287
#13 0x00000000004c36fb in pbx_thread (data=0x42ba3b70) at pbx.c:4574
#14 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#15 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 19 (Thread 16417):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x430f7e40, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x430f7f4c) at channel.c:2190
#2 0x0000000000451377 in ast_waitfor_n (c=0x430f7b70, n=6, ms=) at channel.c:2448
#3 0x00000000004d2fae in bridge_native_loop (c0=0x2aaaac15f3c0, c1=0x2aaac81a1280, flags=0, fo=0x430f8400, rc=0x430f83f8, timeoutms=-1)
at rtp.c:4073
#4 ast_rtp_bridge (c0=0x2aaaac15f3c0, c1=0x2aaac81a1280, flags=0, fo=0x430f8400, rc=0x430f83f8, timeoutms=-1) at rtp.c:4591
#5 0x0000000000456cc2 in ast_channel_bridge (c0=0x2aaaac15f3c0, c1=0x0, config=0x430f9620, fo=0x430f8400, rc=0x430f83f8) at channel.c:5457
#6 0x000000000047a945 in ast_bridge_call (chan=0x2aaaac15f3c0, peer=0x2aaac81a1280, config=0x430f9620) at features.c:2655
#7 0x00002aaab8f75bbd in dial_exec_full (chan=0x2aaaac15f3c0, data=, peerflags=0x430f98d0, continue_exec=0x0)
at app_dial.c:2269
#8 0x00002aaab8f78996 in dial_exec (chan=0x430f7b70, data=0x6) at app_dial.c:2353
#9 0x00000000004b308d in pbx_exec (c=0x2aaaac15f3c0, app=0x123aae70, data=0x430fb9c0) at pbx.c:1352
#10 0x00000000004bf39c in pbx_extension_helper (c=0x2aaaac15f3c0, con=, context=0x2aaaac15f780 “sp-routing-transfer”,
exten=0x2aaaac15f7d0 “s”, priority=2, label=0x0, callerid=0x2aaaac1531f0 “9144851363”, action=E_SPAWN, found=0x430fe03c,
combined_find_spawn=1) at pbx.c:3715
#11 0x00000000004c2122 in ast_spawn_extension (c=0x2aaaac15f3c0, args=0x0) at pbx.c:4193
#12 __ast_pbx_run (c=0x2aaaac15f3c0, args=0x0) at pbx.c:4287
#13 0x00000000004c36fb in pbx_thread (data=0x430f7b70) at pbx.c:4574
#14 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#15 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 18 (Thread 16423):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x417d0e40, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x417d0f4c) at channel.c:2190
#2 0x0000000000451377 in ast_waitfor_n (c=0x417d0b70, n=6, ms=) at channel.c:2448
#3 0x00000000004d2fae in bridge_native_loop (c0=0x2aaacc023b60, c1=0x2aaaac10d360, flags=0, fo=0x417d1400, rc=0x417d13f8, timeoutms=-1)
at rtp.c:4073
#4 ast_rtp_bridge (c0=0x2aaacc023b60, c1=0x2aaaac10d360, flags=0, fo=0x417d1400, rc=0x417d13f8, timeoutms=-1) at rtp.c:4591
#5 0x0000000000456cc2 in ast_channel_bridge (c0=0x2aaaac10d360, c1=0x0, config=0x417d2620, fo=0x417d1400, rc=0x417d13f8) at channel.c:5457
#6 0x000000000047a945 in ast_bridge_call (chan=0x2aaaac10d360, peer=0x2aaacc023b60, config=0x417d2620) at features.c:2655
#7 0x00002aaab8f75bbd in dial_exec_full (chan=0x2aaaac10d360, data=, peerflags=0x417d28d0, continue_exec=0x0)
at app_dial.c:2269
#8 0x00002aaab8f78996 in dial_exec (chan=0x417d0b70, data=0x6) at app_dial.c:2353
#9 0x00000000004b308d in pbx_exec (c=0x2aaaac10d360, app=0x123aae70, data=0x417d49c0) at pbx.c:1352
#10 0x00000000004bf39c in pbx_extension_helper (c=0x2aaaac10d360, con=, context=0x2aaaac10d720 “sp-routing-transfer”,
exten=0x2aaaac10d770 “s”, priority=2, label=0x0, callerid=0x2aaaac1de920 “8162207483”, action=E_SPAWN, found=0x417d703c,
combined_find_spawn=1) at pbx.c:3715
#11 0x00000000004c2122 in ast_spawn_extension (c=0x2aaaac10d360, args=0x0) at pbx.c:4193
#12 __ast_pbx_run (c=0x2aaaac10d360, args=0x0) at pbx.c:4287
#13 0x00000000004c36fb in pbx_thread (data=0x417d0b70) at pbx.c:4574
#14 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#15 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 17 (Thread 16459):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x4116ee40, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x4116ef4c) at channel.c:2190
#2 0x0000000000451377 in ast_waitfor_n (c=0x4116eb70, n=6, ms=) at channel.c:2448
#3 0x00000000004d2fae in bridge_native_loop (c0=0x2aaaac2b3580, c1=0x2aaac80d8a60, flags=0, fo=0x4116f400, rc=0x4116f3f8, timeoutms=-1)
at rtp.c:4073
#4 ast_rtp_bridge (c0=0x2aaaac2b3580, c1=0x2aaac80d8a60, flags=0, fo=0x4116f400, rc=0x4116f3f8, timeoutms=-1) at rtp.c:4591
#5 0x0000000000456cc2 in ast_channel_bridge (c0=0x2aaaac2b3580, c1=0x0, config=0x41170620, fo=0x4116f400, rc=0x4116f3f8) at channel.c:5457
#6 0x000000000047a945 in ast_bridge_call (chan=0x2aaaac2b3580, peer=0x2aaac80d8a60, config=0x41170620) at features.c:2655
#7 0x00002aaab8f75bbd in dial_exec_full (chan=0x2aaaac2b3580, data=, peerflags=0x411708d0, continue_exec=0x0)
at app_dial.c:2269
#8 0x00002aaab8f78996 in dial_exec (chan=0x4116eb70, data=0x6) at app_dial.c:2353
#9 0x00000000004b308d in pbx_exec (c=0x2aaaac2b3580, app=0x123aae70, data=0x411729c0) at pbx.c:1352
#10 0x00000000004bf39c in pbx_extension_helper (c=0x2aaaac2b3580, con=, context=0x2aaaac2b3940 “sp-routing-transfer”,
exten=0x2aaaac2b3990 “s”, priority=2, label=0x0, callerid=0x2aaaac16d680 “7035955995”, action=E_SPAWN, found=0x4117503c,
combined_find_spawn=1) at pbx.c:3715
#11 0x00000000004c2122 in ast_spawn_extension (c=0x2aaaac2b3580, args=0x0) at pbx.c:4193
#12 __ast_pbx_run (c=0x2aaaac2b3580, args=0x0) at pbx.c:4287
#13 0x00000000004c36fb in pbx_thread (data=0x4116eb70) at pbx.c:4574
#14 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#15 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 16 (Thread 16477):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x42aabe40, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x42aabf4c) at channel.c:2190
#2 0x0000000000451377 in ast_waitfor_n (c=0x42aabb70, n=6, ms=) at channel.c:2448
#3 0x00000000004d2fae in bridge_native_loop (c0=0x2aaaac1f0ac0, c1=0x2aaacc00f820, flags=0, fo=0x42aac400, rc=0x42aac3f8, timeoutms=-1)
at rtp.c:4073
#4 ast_rtp_bridge (c0=0x2aaaac1f0ac0, c1=0x2aaacc00f820, flags=0, fo=0x42aac400, rc=0x42aac3f8, timeoutms=-1) at rtp.c:4591
#5 0x0000000000456cc2 in ast_channel_bridge (c0=0x2aaaac1f0ac0, c1=0x0, config=0x42aad620, fo=0x42aac400, rc=0x42aac3f8) at channel.c:5457
#6 0x000000000047a945 in ast_bridge_call (chan=0x2aaaac1f0ac0, peer=0x2aaacc00f820, config=0x42aad620) at features.c:2655
#7 0x00002aaab8f75bbd in dial_exec_full (chan=0x2aaaac1f0ac0, data=, peerflags=0x42aad8d0, continue_exec=0x0)
at app_dial.c:2269
#8 0x00002aaab8f78996 in dial_exec (chan=0x42aabb70, data=0x6) at app_dial.c:2353
#9 0x00000000004b308d in pbx_exec (c=0x2aaaac1f0ac0, app=0x123aae70, data=0x42aaf9c0) at pbx.c:1352
#10 0x00000000004bf39c in pbx_extension_helper (c=0x2aaaac1f0ac0, con=, context=0x2aaaac1f0e80 “sp-routing-transfer”,
exten=0x2aaaac1f0ed0 “s”, priority=2, label=0x0, callerid=0x2aaaac109e80 “9548642774”, action=E_SPAWN, found=0x42ab203c,
combined_find_spawn=1) at pbx.c:3715
#11 0x00000000004c2122 in ast_spawn_extension (c=0x2aaaac1f0ac0, args=0x0) at pbx.c:4193
#12 __ast_pbx_run (c=0x2aaaac1f0ac0, args=0x0) at pbx.c:4287
#13 0x00000000004c36fb in pbx_thread (data=0x42aabb70) at pbx.c:4574
#14 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#15 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 15 (Thread 16500):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x43000040, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x430000ec) at channel.c:2190
#2 0x000000000045638e in ast_channel_bridge (c0=0x2aaacc061340, c1=0x2aaacc0392f0, config=0x43001620, fo=0x43000400, rc=0x430003f8)
at channel.c:2448
#3 0x000000000047a945 in ast_bridge_call (chan=0x2aaacc061340, peer=0x2aaacc0392f0, config=0x43001620) at features.c:2655
#4 0x00002aaab8f75bbd in dial_exec_full (chan=0x2aaacc061340, data=, peerflags=0x430018d0, continue_exec=0x0)
at app_dial.c:2269
#5 0x00002aaab8f78996 in dial_exec (chan=0x42fffe00, data=0x4) at app_dial.c:2353
#6 0x00000000004b308d in pbx_exec (c=0x2aaacc061340, app=0x123aae70, data=0x430039c0) at pbx.c:1352
#7 0x00000000004bf39c in pbx_extension_helper (c=0x2aaacc061340, con=, context=0x2aaacc061700 “csr-transfer”,
exten=0x2aaacc061750 “s”, priority=8, label=0x0, callerid=0x2aaacc046950 “9736526353”, action=E_SPAWN, found=0x4300603c,
combined_find_spawn=1) at pbx.c:3715
#8 0x00000000004c2122 in ast_spawn_extension (c=0x2aaacc061340, args=0x0) at pbx.c:4193
#9 __ast_pbx_run (c=0x2aaacc061340, args=0x0) at pbx.c:4287
#10 0x00000000004c36fb in pbx_thread (data=0x42fffe00) at pbx.c:4574
#11 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#12 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#13 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 14 (Thread 16502):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x42e8be40, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x42e8bf4c) at channel.c:2190
#2 0x0000000000451377 in ast_waitfor_n (c=0x42e8bb70, n=6, ms=) at channel.c:2448
#3 0x00000000004d2fae in bridge_native_loop (c0=0x2aaaac1137b0, c1=0x2aaacc023630, flags=0, fo=0x42e8c400, rc=0x42e8c3f8, timeoutms=-1)
at rtp.c:4073
#4 ast_rtp_bridge (c0=0x2aaaac1137b0, c1=0x2aaacc023630, flags=0, fo=0x42e8c400, rc=0x42e8c3f8, timeoutms=-1) at rtp.c:4591
#5 0x0000000000456cc2 in ast_channel_bridge (c0=0x2aaaac1137b0, c1=0x0, config=0x42e8d620, fo=0x42e8c400, rc=0x42e8c3f8) at channel.c:5457
#6 0x000000000047a945 in ast_bridge_call (chan=0x2aaaac1137b0, peer=0x2aaacc023630, config=0x42e8d620) at features.c:2655
#7 0x00002aaab8f75bbd in dial_exec_full (chan=0x2aaaac1137b0, data=, peerflags=0x42e8d8d0, continue_exec=0x0)
at app_dial.c:2269
#8 0x00002aaab8f78996 in dial_exec (chan=0x42e8bb70, data=0x6) at app_dial.c:2353
#9 0x00000000004b308d in pbx_exec (c=0x2aaaac1137b0, app=0x123aae70, data=0x42e8f9c0) at pbx.c:1352
#10 0x00000000004bf39c in pbx_extension_helper (c=0x2aaaac1137b0, con=, context=0x2aaaac113b70 “sp-routing-transfer”,
exten=0x2aaaac113bc0 “s”, priority=2, label=0x0, callerid=0x2aaaac276300 “16022888842”, action=E_SPAWN, found=0x42e9203c,
combined_find_spawn=1) at pbx.c:3715
#11 0x00000000004c2122 in ast_spawn_extension (c=0x2aaaac1137b0, args=0x0) at pbx.c:4193
#12 __ast_pbx_run (c=0x2aaaac1137b0, args=0x0) at pbx.c:4287
#13 0x00000000004c36fb in pbx_thread (data=0x42e8bb70) at pbx.c:4574
#14 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#15 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 13 (Thread 16507):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x40387e40, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x40387f4c) at channel.c:2190
#2 0x0000000000451377 in ast_waitfor_n (c=0x40387b70, n=6, ms=) at channel.c:2448
#3 0x00000000004d2fae in bridge_native_loop (c0=0x2aaacc109690, c1=0x2aaaac2a0cb0, flags=0, fo=0x40388400, rc=0x403883f8, timeoutms=-1)
at rtp.c:4073
#4 ast_rtp_bridge (c0=0x2aaacc109690, c1=0x2aaaac2a0cb0, flags=0, fo=0x40388400, rc=0x403883f8, timeoutms=-1) at rtp.c:4591
#5 0x0000000000456cc2 in ast_channel_bridge (c0=0x2aaaac2a0cb0, c1=0x0, config=0x40389620, fo=0x40388400, rc=0x403883f8) at channel.c:5457
#6 0x000000000047a945 in ast_bridge_call (chan=0x2aaaac2a0cb0, peer=0x2aaacc109690, config=0x40389620) at features.c:2655
#7 0x00002aaab8f75bbd in dial_exec_full (chan=0x2aaaac2a0cb0, data=, peerflags=0x403898d0, continue_exec=0x0)
at app_dial.c:2269
#8 0x00002aaab8f78996 in dial_exec (chan=0x40387b70, data=0x6) at app_dial.c:2353
#9 0x00000000004b308d in pbx_exec (c=0x2aaaac2a0cb0, app=0x123aae70, data=0x4038b9c0) at pbx.c:1352
#10 0x00000000004bf39c in pbx_extension_helper (c=0x2aaaac2a0cb0, con=, context=0x2aaaac2a1070 “sp-routing-transfer”,
exten=0x2aaaac2a10c0 “s”, priority=2, label=0x0, callerid=0x2aaaac0ffd60 “6783537688”, action=E_SPAWN, found=0x4038e03c,
combined_find_spawn=1) at pbx.c:3715
#11 0x00000000004c2122 in ast_spawn_extension (c=0x2aaaac2a0cb0, args=0x0) at pbx.c:4193
#12 __ast_pbx_run (c=0x2aaaac2a0cb0, args=0x0) at pbx.c:4287
#13 0x00000000004c36fb in pbx_thread (data=0x40387b70) at pbx.c:4574
#14 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#15 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 12 (Thread 16522):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x4345be40, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x4345bf4c) at channel.c:2190
#2 0x0000000000451377 in ast_waitfor_n (c=0x4345bb70, n=6, ms=) at channel.c:2448
#3 0x00000000004d2fae in bridge_native_loop (c0=0x2aaaac22f8e0, c1=0x2aaacc091120, flags=0, fo=0x4345c400, rc=0x4345c3f8, timeoutms=-1)
at rtp.c:4073
#4 ast_rtp_bridge (c0=0x2aaaac22f8e0, c1=0x2aaacc091120, flags=0, fo=0x4345c400, rc=0x4345c3f8, timeoutms=-1) at rtp.c:4591
#5 0x0000000000456cc2 in ast_channel_bridge (c0=0x2aaaac22f8e0, c1=0x0, config=0x4345d620, fo=0x4345c400, rc=0x4345c3f8) at channel.c:5457
#6 0x000000000047a945 in ast_bridge_call (chan=0x2aaaac22f8e0, peer=0x2aaacc091120, config=0x4345d620) at features.c:2655
#7 0x00002aaab8f75bbd in dial_exec_full (chan=0x2aaaac22f8e0, data=, peerflags=0x4345d8d0, continue_exec=0x0)
at app_dial.c:2269
#8 0x00002aaab8f78996 in dial_exec (chan=0x4345bb70, data=0x6) at app_dial.c:2353
#9 0x00000000004b308d in pbx_exec (c=0x2aaaac22f8e0, app=0x123aae70, data=0x4345f9c0) at pbx.c:1352
#10 0x00000000004bf39c in pbx_extension_helper (c=0x2aaaac22f8e0, con=, context=0x2aaaac22fca0 “sp-routing-transfer”,
exten=0x2aaaac22fcf0 “s”, priority=2, label=0x0, callerid=0x2aaaac275eb0 “7192908459”, action=E_SPAWN, found=0x4346203c,
combined_find_spawn=1) at pbx.c:3715
#11 0x00000000004c2122 in ast_spawn_extension (c=0x2aaaac22f8e0, args=0x0) at pbx.c:4193
#12 __ast_pbx_run (c=0x2aaaac22f8e0, args=0x0) at pbx.c:4287
#13 0x00000000004c36fb in pbx_thread (data=0x4345bb70) at pbx.c:4574
#14 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#15 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 11 (Thread 16529):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x4028fe40, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x4028ff4c) at channel.c:2190
#2 0x0000000000451377 in ast_waitfor_n (c=0x4028fb70, n=6, ms=) at channel.c:2448
#3 0x00000000004d2fae in bridge_native_loop (c0=0x2aaaac175620, c1=0x2aaacc08c650, flags=0, fo=0x40290400, rc=0x402903f8, timeoutms=-1)
at rtp.c:4073
#4 ast_rtp_bridge (c0=0x2aaaac175620, c1=0x2aaacc08c650, flags=0, fo=0x40290400, rc=0x402903f8, timeoutms=-1) at rtp.c:4591
#5 0x0000000000456cc2 in ast_channel_bridge (c0=0x2aaaac175620, c1=0x0, config=0x40291620, fo=0x40290400, rc=0x402903f8) at channel.c:5457
#6 0x000000000047a945 in ast_bridge_call (chan=0x2aaaac175620, peer=0x2aaacc08c650, config=0x40291620) at features.c:2655
#7 0x00002aaab8f75bbd in dial_exec_full (chan=0x2aaaac175620, data=, peerflags=0x402918d0, continue_exec=0x0)
at app_dial.c:2269
#8 0x00002aaab8f78996 in dial_exec (chan=0x4028fb70, data=0x6) at app_dial.c:2353
#9 0x00000000004b308d in pbx_exec (c=0x2aaaac175620, app=0x123aae70, data=0x402939c0) at pbx.c:1352
#10 0x00000000004bf39c in pbx_extension_helper (c=0x2aaaac175620, con=, context=0x2aaaac1759e0 “sp-routing-transfer”,
exten=0x2aaaac175a30 “s”, priority=2, label=0x0, callerid=0x2aaaac1d69a0 “8314843892”, action=E_SPAWN, found=0x4029603c,
combined_find_spawn=1) at pbx.c:3715
#11 0x00000000004c2122 in ast_spawn_extension (c=0x2aaaac175620, args=0x0) at pbx.c:4193
#12 __ast_pbx_run (c=0x2aaaac175620, args=0x0) at pbx.c:4287
#13 0x00000000004c36fb in pbx_thread (data=0x4028fb70) at pbx.c:4574
#14 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#15 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 10 (Thread 16530):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x4030be40, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x4030bf4c) at channel.c:2190
#2 0x0000000000451377 in ast_waitfor_n (c=0x4030bb70, n=6, ms=) at channel.c:2448
#3 0x00000000004d2fae in bridge_native_loop (c0=0x2aaacc06bd60, c1=0x2aaaac29cf30, flags=0, fo=0x4030c400, rc=0x4030c3f8, timeoutms=-1)
at rtp.c:4073
#4 ast_rtp_bridge (c0=0x2aaacc06bd60, c1=0x2aaaac29cf30, flags=0, fo=0x4030c400, rc=0x4030c3f8, timeoutms=-1) at rtp.c:4591
#5 0x0000000000456cc2 in ast_channel_bridge (c0=0x2aaaac29cf30, c1=0x0, config=0x4030d620, fo=0x4030c400, rc=0x4030c3f8) at channel.c:5457
#6 0x000000000047a945 in ast_bridge_call (chan=0x2aaaac29cf30, peer=0x2aaacc06bd60, config=0x4030d620) at features.c:2655
#7 0x00002aaab8f75bbd in dial_exec_full (chan=0x2aaaac29cf30, data=, peerflags=0x4030d8d0, continue_exec=0x0)
at app_dial.c:2269
#8 0x00002aaab8f78996 in dial_exec (chan=0x4030bb70, data=0x6) at app_dial.c:2353
#9 0x00000000004b308d in pbx_exec (c=0x2aaaac29cf30, app=0x123aae70, data=0x4030f9c0) at pbx.c:1352
#10 0x00000000004bf39c in pbx_extension_helper (c=0x2aaaac29cf30, con=, context=0x2aaaac29d2f0 “sp-routing-transfer”,
exten=0x2aaaac29d340 “s”, priority=2, label=0x0, callerid=0x2aaaac2a1660 “3609301176”, action=E_SPAWN, found=0x4031203c,
combined_find_spawn=1) at pbx.c:3715
#11 0x00000000004c2122 in ast_spawn_extension (c=0x2aaaac29cf30, args=0x0) at pbx.c:4193
#12 __ast_pbx_run (c=0x2aaaac29cf30, args=0x0) at pbx.c:4287
#13 0x00000000004c36fb in pbx_thread (data=0x4030bb70) at pbx.c:4574
#14 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#15 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 9 (Thread 16564):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x4073be40, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x4073bf4c) at channel.c:2190
#2 0x0000000000451377 in ast_waitfor_n (c=0x4073bb70, n=6, ms=) at channel.c:2448
#3 0x00000000004d2fae in bridge_native_loop (c0=0x1290e9d0, c1=0x129980d0, flags=0, fo=0x4073c400, rc=0x4073c3f8, timeoutms=-1)
at rtp.c:4073
#4 ast_rtp_bridge (c0=0x1290e9d0, c1=0x129980d0, flags=0, fo=0x4073c400, rc=0x4073c3f8, timeoutms=-1) at rtp.c:4591
#5 0x0000000000456cc2 in ast_channel_bridge (c0=0x129980d0, c1=0x0, config=0x4073d620, fo=0x4073c400, rc=0x4073c3f8) at channel.c:5457
#6 0x000000000047a945 in ast_bridge_call (chan=0x129980d0, peer=0x1290e9d0, config=0x4073d620) at features.c:2655
#7 0x00002aaab8f75bbd in dial_exec_full (chan=0x129980d0, data=, peerflags=0x4073d8d0, continue_exec=0x0)
at app_dial.c:2269
#8 0x00002aaab8f78996 in dial_exec (chan=0x4073bb70, data=0x6) at app_dial.c:2353
#9 0x00000000004b308d in pbx_exec (c=0x129980d0, app=0x123aae70, data=0x4073f9c0) at pbx.c:1352
#10 0x00000000004bf39c in pbx_extension_helper (c=0x129980d0, con=, context=0x12998490 “sp-routing-transfer”,
exten=0x129984e0 “s”, priority=2, label=0x0, callerid=0x1252cef0 “3145372862”, action=E_SPAWN, found=0x4074203c, combined_find_spawn=1)
at pbx.c:3715
#11 0x00000000004c2122 in ast_spawn_extension (c=0x129980d0, args=0x0) at pbx.c:4193
#12 __ast_pbx_run (c=0x129980d0, args=0x0) at pbx.c:4287
#13 0x00000000004c36fb in pbx_thread (data=0x4073bb70) at pbx.c:4574
#14 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#15 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 8 (Thread 16568):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x4140fe40, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x4140ff4c) at channel.c:2190
#2 0x0000000000451377 in ast_waitfor_n (c=0x4140fb70, n=6, ms=) at channel.c:2448
#3 0x00000000004d2fae in bridge_native_loop (c0=0x2aaacc0aecc0, c1=0x2aaacc0e7e20, flags=0, fo=0x41410400, rc=0x414103f8, timeoutms=-1)
at rtp.c:4073
#4 ast_rtp_bridge (c0=0x2aaacc0aecc0, c1=0x2aaacc0e7e20, flags=0, fo=0x41410400, rc=0x414103f8, timeoutms=-1) at rtp.c:4591
#5 0x0000000000456cc2 in ast_channel_bridge (c0=0x2aaacc0aecc0, c1=0x0, config=0x41411620, fo=0x41410400, rc=0x414103f8) at channel.c:5457
#6 0x000000000047a945 in ast_bridge_call (chan=0x2aaacc0aecc0, peer=0x2aaacc0e7e20, config=0x41411620) at features.c:2655
#7 0x00002aaab8f75bbd in dial_exec_full (chan=0x2aaacc0aecc0, data=, peerflags=0x414118d0, continue_exec=0x0)
at app_dial.c:2269
#8 0x00002aaab8f78996 in dial_exec (chan=0x4140fb70, data=0x6) at app_dial.c:2353
#9 0x00000000004b308d in pbx_exec (c=0x2aaacc0aecc0, app=0x123aae70, data=0x414139c0) at pbx.c:1352
#10 0x00000000004bf39c in pbx_extension_helper (c=0x2aaacc0aecc0, con=, context=0x2aaacc0af080 “sp-routing-transfer”,
exten=0x2aaacc0af0d0 “s”, priority=2, label=0x0, callerid=0x2aaacc0fbb60 “4157316408”, action=E_SPAWN, found=0x4141603c,
combined_find_spawn=1) at pbx.c:3715
#11 0x00000000004c2122 in ast_spawn_extension (c=0x2aaacc0aecc0, args=0x0) at pbx.c:4193
#12 __ast_pbx_run (c=0x2aaacc0aecc0, args=0x0) at pbx.c:4287
#13 0x00000000004c36fb in pbx_thread (data=0x4140fb70) at pbx.c:4574
#14 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#15 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 7 (Thread 16593):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x415e0e40, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x415e0f4c) at channel.c:2190
#2 0x0000000000451377 in ast_waitfor_n (c=0x415e0b70, n=6, ms=) at channel.c:2448
#3 0x00000000004d2fae in bridge_native_loop (c0=0x2aaac8020400, c1=0x2aaac8078900, flags=0, fo=0x415e1400, rc=0x415e13f8, timeoutms=-1)
at rtp.c:4073
#4 ast_rtp_bridge (c0=0x2aaac8020400, c1=0x2aaac8078900, flags=0, fo=0x415e1400, rc=0x415e13f8, timeoutms=-1) at rtp.c:4591
#5 0x0000000000456cc2 in ast_channel_bridge (c0=0x2aaac8078900, c1=0x0, config=0x415e2620, fo=0x415e1400, rc=0x415e13f8) at channel.c:5457
#6 0x000000000047a945 in ast_bridge_call (chan=0x2aaac8078900, peer=0x2aaac8020400, config=0x415e2620) at features.c:2655
#7 0x00002aaab8f75bbd in dial_exec_full (chan=0x2aaac8078900, data=, peerflags=0x415e28d0, continue_exec=0x0)
at app_dial.c:2269
#8 0x00002aaab8f78996 in dial_exec (chan=0x415e0b70, data=0x6) at app_dial.c:2353
#9 0x00000000004b308d in pbx_exec (c=0x2aaac8078900, app=0x123aae70, data=0x415e49c0) at pbx.c:1352
#10 0x00000000004bf39c in pbx_extension_helper (c=0x2aaac8078900, con=, context=0x2aaac8078cc0 “sp-routing-transfer”,
exten=0x2aaac8078d10 “s”, priority=2, label=0x0, callerid=0x2aaac805b4e0 “3057257991”, action=E_SPAWN, found=0x415e703c,
combined_find_spawn=1) at pbx.c:3715
#11 0x00000000004c2122 in ast_spawn_extension (c=0x2aaac8078900, args=0x0) at pbx.c:4193
#12 __ast_pbx_run (c=0x2aaac8078900, args=0x0) at pbx.c:4287
#13 0x00000000004c36fb in pbx_thread (data=0x415e0b70) at pbx.c:4574
#14 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#15 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 6 (Thread 16627):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x414e8e38, n=1, fds=0x414e8ea0, nfds=1, exception=0x0, outfd=0x414e8e48, ms=0x414e8e4c)
at channel.c:2190
#2 0x0000000000486928 in waitstream_core (c=0x1290c9a0, breakon=0x414e9482 “0123456789”, forward=0x529587 “”, reverse=0x529587 “”,
skip_ms=, audiofd=-1, cmdfd=141, context=0x0) at file.c:1218
#3 0x0000000000486d6f in ast_waitstream_full (c=0x414e8cb0, breakon=0x4 <Address 0x4 out of bounds>, audiofd=1342203059,
cmdfd=) at file.c:1326
#4 0x00002aaaab52a453 in handle_getoption (chan=0x1290c9a0, agi=0x414ea870, argc=, argv=)
at res_agi.c:1282
#5 0x00002aaaab527ab7 in agi_handle_command (chan=0x1290c9a0, agi=0x414ea870, buf=, dead=0) at res_agi.c:2803
#6 0x00002aaaab52e200 in run_agi (chan=0x1290c9a0, data=, enhanced=, dead=0) at res_agi.c:2963
#7 agi_exec_full (chan=0x1290c9a0, data=, enhanced=, dead=0) at res_agi.c:3249
#8 0x00000000004b308d in pbx_exec (c=0x1290c9a0, app=0x12343370, data=0x414ec9c0) at pbx.c:1352
#9 0x00000000004bf39c in pbx_extension_helper (c=0x1290c9a0, con=, context=0x1290cd60 “sp-ops-leads-menu”,
exten=0x1290cdb0 “s”, priority=3, label=0x0, callerid=0x1291eec0 “5162337868”, action=E_SPAWN, found=0x414ef03c, combined_find_spawn=1)
at pbx.c:3715
#10 0x00000000004c2122 in ast_spawn_extension (c=0x1290c9a0, args=0x0) at pbx.c:4193
#11 __ast_pbx_run (c=0x1290c9a0, args=0x0) at pbx.c:4287
#12 0x00000000004c36fb in pbx_thread (data=0x414e8cb0) at pbx.c:4574
#13 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#14 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#15 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 5 (Thread 16628):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x41bdee40, n=2, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x41bdef4c) at channel.c:2190
#2 0x0000000000451377 in ast_waitfor_n (c=0x41bdeb70, n=6, ms=) at channel.c:2448
#3 0x00000000004d2fae in bridge_native_loop (c0=0x1287e480, c1=0x2aaacc0e14c0, flags=0, fo=0x41bdf400, rc=0x41bdf3f8, timeoutms=-1)
at rtp.c:4073
#4 ast_rtp_bridge (c0=0x1287e480, c1=0x2aaacc0e14c0, flags=0, fo=0x41bdf400, rc=0x41bdf3f8, timeoutms=-1) at rtp.c:4591
#5 0x0000000000456cc2 in ast_channel_bridge (c0=0x2aaacc0e14c0, c1=0x0, config=0x41be0620, fo=0x41bdf400, rc=0x41bdf3f8) at channel.c:5457
#6 0x000000000047a945 in ast_bridge_call (chan=0x2aaacc0e14c0, peer=0x1287e480, config=0x41be0620) at features.c:2655
#7 0x00002aaab8f75bbd in dial_exec_full (chan=0x2aaacc0e14c0, data=, peerflags=0x41be08d0, continue_exec=0x0)
at app_dial.c:2269
#8 0x00002aaab8f78996 in dial_exec (chan=0x41bdeb70, data=0x6) at app_dial.c:2353
#9 0x00000000004b308d in pbx_exec (c=0x2aaacc0e14c0, app=0x123aae70, data=0x41be29c0) at pbx.c:1352
#10 0x00000000004bf39c in pbx_extension_helper (c=0x2aaacc0e14c0, con=, context=0x2aaacc0e1880 “sp-routing-transfer”,
exten=0x2aaacc0e18d0 “s”, priority=2, label=0x0, callerid=0x2aaacc069aa0 “7202359831”, action=E_SPAWN, found=0x41be503c,
combined_find_spawn=1) at pbx.c:3715
#11 0x00000000004c2122 in ast_spawn_extension (c=0x2aaacc0e14c0, args=0x0) at pbx.c:4193
#12 __ast_pbx_run (c=0x2aaacc0e14c0, args=0x0) at pbx.c:4287
#13 0x00000000004c36fb in pbx_thread (data=0x41bdeb70) at pbx.c:4574
#14 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#15 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#16 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 4 (Thread 16631):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x42d93e38, n=1, fds=0x42d93ea0, nfds=1, exception=0x0, outfd=0x42d93e48, ms=0x42d93e4c)
at channel.c:2190
#2 0x0000000000486928 in waitstream_core (c=0x2aaaac1f6c10, breakon=0x42d94482 “0123456789”, forward=0x529587 “”, reverse=0x529587 “”,
skip_ms=, audiofd=-1, cmdfd=249, context=0x0) at file.c:1218
#3 0x0000000000486d6f in ast_waitstream_full (c=0x42d93cb0, breakon=0x4 <Address 0x4 out of bounds>, audiofd=1342203059,
cmdfd=) at file.c:1326
#4 0x00002aaaab52a453 in handle_getoption (chan=0x2aaaac1f6c10, agi=0x42d95870, argc=, argv=)
at res_agi.c:1282
#5 0x00002aaaab527ab7 in agi_handle_command (chan=0x2aaaac1f6c10, agi=0x42d95870, buf=, dead=0) at res_agi.c:2803
#6 0x00002aaaab52e200 in run_agi (chan=0x2aaaac1f6c10, data=, enhanced=, dead=0) at res_agi.c:2963
#7 agi_exec_full (chan=0x2aaaac1f6c10, data=, enhanced=, dead=0) at res_agi.c:3249
#8 0x00000000004b308d in pbx_exec (c=0x2aaaac1f6c10, app=0x12343370, data=0x42d979c0) at pbx.c:1352
#9 0x00000000004bf39c in pbx_extension_helper (c=0x2aaaac1f6c10, con=, context=0x2aaaac1f6fd0 “sp-ops-leads-menu”,
exten=0x2aaaac1f7020 “s”, priority=3, label=0x0, callerid=0x2aaaac23a5d0 “6106566867”, action=E_SPAWN, found=0x42d9a03c,
combined_find_spawn=1) at pbx.c:3715
#10 0x00000000004c2122 in ast_spawn_extension (c=0x2aaaac1f6c10, args=0x0) at pbx.c:4193
#11 __ast_pbx_run (c=0x2aaaac1f6c10, args=0x0) at pbx.c:4287
#12 0x00000000004c36fb in pbx_thread (data=0x42d93cb0) at pbx.c:4574
#13 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#14 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#15 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 3 (Thread 16634):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x427c58a8, n=1, fds=0x0, nfds=0, exception=0x0, outfd=0x0, ms=0x427c58b4) at channel.c:2190
#2 0x000000000045548f in ast_waitfor (chan=0x2aaaac1ecfe0, ms=657, cond=0, data=0x0) at channel.c:2455
#3 ast_safe_sleep_conditional (chan=0x2aaaac1ecfe0, ms=657, cond=0, data=0x0) at channel.c:1379
#4 0x00000000004b308d in pbx_exec (c=0x2aaaac1ecfe0, app=0x120704b0, data=0x427c79c0) at pbx.c:1352
#5 0x00000000004bf39c in pbx_extension_helper (c=0x2aaaac1ecfe0, con=,
context=0x2aaaac1ed3a0 “senior-care-speed-bump”, exten=0x2aaaac1ed3f0 “s”, priority=2, label=0x0, callerid=0x2aaaac1d1370 “4802757932”,
action=E_SPAWN, found=0x427ca03c, combined_find_spawn=1) at pbx.c:3715
#6 0x00000000004c2122 in ast_spawn_extension (c=0x2aaaac1ecfe0, args=0x0) at pbx.c:4193
#7 __ast_pbx_run (c=0x2aaaac1ecfe0, args=0x0) at pbx.c:4287
#8 0x00000000004c36fb in pbx_thread (data=0x427c5740) at pbx.c:4574
#9 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#10 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#11 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 2 (Thread 16635):
#0 0x0000003cc38cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000450db4 in ast_waitfor_nandfds (c=0x4165ce38, n=1, fds=0x4165cea0, nfds=1, exception=0x0, outfd=0x4165ce48, ms=0x4165ce4c)
at channel.c:2190
#2 0x0000000000486928 in waitstream_core (c=0x2aaacc00ebc0, breakon=0x4165d486 “0123456789”, forward=0x529587 “”, reverse=0x529587 “”,
skip_ms=, audiofd=-1, cmdfd=147, context=0x0) at file.c:1218
#3 0x0000000000486d6f in ast_waitstream_full (c=0x4165ccb0, breakon=0x4 <Address 0x4 out of bounds>, audiofd=1342203059,
cmdfd=) at file.c:1326
#4 0x00002aaaab52a453 in handle_getoption (chan=0x2aaacc00ebc0, agi=0x4165e870, argc=, argv=)
at res_agi.c:1282
#5 0x00002aaaab527ab7 in agi_handle_command (chan=0x2aaacc00ebc0, agi=0x4165e870, buf=, dead=0) at res_agi.c:2803
#6 0x00002aaaab52e200 in run_agi (chan=0x2aaacc00ebc0, data=, enhanced=, dead=0) at res_agi.c:2963
#7 agi_exec_full (chan=0x2aaacc00ebc0, data=, enhanced=, dead=0) at res_agi.c:3249
#8 0x00000000004b308d in pbx_exec (c=0x2aaacc00ebc0, app=0x12343370, data=0x416609c0) at pbx.c:1352
#9 0x00000000004bf39c in pbx_extension_helper (c=0x2aaacc00ebc0, con=, context=0x2aaacc00ef80 “sp-ops-main-menu”,
exten=0x2aaacc00efd0 “s”, priority=2, label=0x0, callerid=0x2aaacc02d950 “4108418288”, action=E_SPAWN, found=0x4166303c,
combined_find_spawn=1) at pbx.c:3715
#10 0x00000000004c2122 in ast_spawn_extension (c=0x2aaacc00ebc0, args=0x0) at pbx.c:4193
#11 __ast_pbx_run (c=0x2aaacc00ebc0, args=0x0) at pbx.c:4287
#12 0x00000000004c36fb in pbx_thread (data=0x4165ccb0) at pbx.c:4574
#13 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#14 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#15 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x42e16940 (LWP 16637)):
#0 0x0000003cc3873ba6 in _int_malloc () from /lib64/libc.so.6
#1 0x0000003cc3874cde in malloc () from /lib64/libc.so.6
#2 0x000000000048b0c0 in _ast_malloc (size=160) at /usr/src/asterisk-1.6.2.14/include/asterisk/utils.h:439
#3 ast_smoother_new (size=160) at frame.c:181
#4 0x00000000004cf04f in ast_rtp_write (rtp=0x2aaac819d9d0, _f=0x2aaac80151b8) at rtp.c:3898
#5 0x00002aaabba23653 in ?? () from /usr/lib/asterisk/modules/chan_ooh323.so
#6 0x000000000045273b in ast_write (chan=0x2aaaac14f9a0, fr=0x2aaac80ca840) at channel.c:3835
#7 0x0000000000487b86 in ast_readaudio_callback (s=0x2aaac8069de8) at file.c:751
#8 0x0000000000487d46 in ast_playstream (s=0x2aaac8000020) at file.c:858
#9 0x00002aaaab529319 in handle_streamfile (chan=0x2aaaac14f9a0, agi=0x42e11870, argc=, argv=)
at res_agi.c:1222
#10 0x00002aaaab527ab7 in agi_handle_command (chan=0x2aaaac14f9a0, agi=0x42e11870, buf=, dead=0) at res_agi.c:2803
#11 0x00002aaaab52e200 in run_agi (chan=0x2aaaac14f9a0, data=, enhanced=, dead=0) at res_agi.c:2963
#12 agi_exec_full (chan=0x2aaaac14f9a0, data=, enhanced=, dead=0) at res_agi.c:3249
#13 0x00000000004b308d in pbx_exec (c=0x2aaaac14f9a0, app=0x12343370, data=0x42e139c0) at pbx.c:1352
#14 0x00000000004bf39c in pbx_extension_helper (c=0x2aaaac14f9a0, con=, context=0x2aaaac14fd60 “sp-ops-welcome”,
exten=0x2aaaac14fdb0 “s”, priority=2, label=0x0, callerid=0x2aaaac1d8690 “2395434884”, action=E_SPAWN, found=0x42e1603c,
combined_find_spawn=1) at pbx.c:3715
#15 0x00000000004c2122 in ast_spawn_extension (c=0x2aaaac14f9a0, args=0x0) at pbx.c:4193
#16 __ast_pbx_run (c=0x2aaaac14f9a0, args=0x0) at pbx.c:4287
#17 0x00000000004c36fb in pbx_thread (data=0x2aaac8000020) at pbx.c:4574
#18 0x00000000004fddbc in dummy_start (data=) at utils.c:968
#19 0x0000003cc400673d in start_thread () from /lib64/libpthread.so.0
#20 0x0000003cc38d3f6d in clone () from /lib64/libc.so.6
[/code]

david55 · July 17, 2012, 6:52am

Segmentation faults in malloc are generally secondary faults with the primary fault being memory corruption. The memory corruption can, in turn, often be the result of locking problems, so have a look for any error messages relating to locks.

The other thing to do is to make sure that you use the final end of life version, not an earlier version, even if you are locked into 1.6.2.x versions.