Asterisk 14.5.0 occasional crashes

Asterisk Asterisk Support
1

Hello,

We’ve been running Asterisk 11 for a while now and it’s been running flawlessly for years.
But because of our need to implement WebRTC to our services we needed to upgrade Asterisk to a later version. So we upgraded to Asterisk 14.5.0 on Ubuntu 12.04.2 LTS

However, after this upgrade we experience occasional crashes on our Asterisk servers. It crashes more or less once per week. So we’ve checked logs to try and find a reason why this happens, but we cannot see a pattern.

I have now recompiled Asterisk 14.5.0 with the BETTER_BACKTRACES and DONT_OPTIMIZE flags in hope of getting some help from the coredumps.

To be honest, I have no idea what too look for in the coredumps, but we do get messages about problems accessing memory. However we are running Asterisk on different physical machines and we experience the same crashes. So it’s highly unlikely that this is due to faulty memory. Also, the Asterisk 11-installations were on the same hardware without any problems.

Asterisk 14.5.0 is compiled with pjsip, however only chan_sip is actually used. we also have one third party-module installed called AppKonference - apart from that everything is Vanilla.

I have also tried installing Asterisk 15 on an Ubuntu 16-machine. But we experienced the same issue.

https://ufile.io/30jy2 - Here’s the coredump generates from ast_coredumper. if anyone would like the whole coredump (100mb) just let me know.

Does anyone have any idea why this would happen? Just let me know if you need any more information!

thanks!

best regards

Timmy

2

It is not clear what is in your file and how big it is. As such I don’t want to risk downloading it on a metered connection.

What we need, initially is backtraces, and the full backtrace of the failing thread, together with the kernel fault code, should easily fit inline in a posting.

From that, it should be possible either to get a good idea of what is happening, or determine that the problem is memory corruption, which will be much more difficult to debug, and will require you to work out what triggered the crash, and find early problem log entries.

3

Hello.

Thank you for your reply.
It’s a zip file with all the output from the ast_corepumper script. 9kb total. i followed the steps in this article:
https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace#GettingaBacktrace-ast_coredumper

I seem to only be allowed to write 32000 characters in the forum post. the whole coredump info in the file is about 80000 characters. i can only fit the file called brief.txt, but maybe that can help out some? :slight_smile:

!@!@!@! brief.txt !@!@!@!


Thread 94 (LWP 23480):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000000005a0430c1 in ?? ()
#2  0x0000000000000028 in ?? ()
#3  0x0000000000000004 in ?? ()
#4  0x00007fdf47d0e080 in ?? ()
#5  0x00007fdf47d0e230 in ?? ()
#6  0x00000000004c16a4 in tonepair_generator (chan=0x7fdf7804b2e8, data=0x4, len=32735, samples=-1810835344) at channel.c:7379
Backtrace stopped: Cannot access memory at address 0x8ffffffd8

Thread 93 (LWP 23976):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000000005a0430c1 in ?? ()
#2  0x0000000000000028 in ?? ()
#3  0x0000000000000004 in ?? ()
#4  0x00007fdf47794080 in ?? ()
#5  0x00007fdf47794230 in ?? ()
#6  0x00000000004c16a4 in tonepair_generator (chan=0xd7000000d8, data=0x4, len=0, samples=-1810739632) at channel.c:7379
Backtrace stopped: Cannot access memory at address 0x8ffffffd8

Thread 92 (LWP 27832):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000000005a0430c1 in ?? ()
#2  0x0000000000000028 in ?? ()
#3  0x0000000000000004 in ?? ()
#4  0x00007fdf4788c080 in ?? ()
#5  0x00007fdf4788c230 in ?? ()
#6  0x00000000004c16a4 in tonepair_generator (chan=0x7fdf780700a8, data=0x4, len=32735, samples=-1811416864) at channel.c:7379
Backtrace stopped: Cannot access memory at address 0x8ffffffd8

Thread 91 (LWP 44208):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000000005a0430c1 in ?? ()
#2  0x0000000000000028 in ?? ()
#3  0x0000000000000004 in ?? ()
#4  0x00007fdf48072080 in ?? ()
#5  0x00007fdf48072230 in ?? ()
#6  0x00000000004c16a4 in tonepair_generator (chan=0xd7000000d8, data=0x4, len=0, samples=-1811749440) at channel.c:7379
Backtrace stopped: Cannot access memory at address 0x8ffffffd8

Thread 90 (LWP 53127):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000000005a0430c1 in ?? ()
#2  0x0000000000000028 in ?? ()
#3  0x0000000000000004 in ?? ()
#4  0x00007fdf481e6080 in ?? ()
#5  0x00007fdf481e6230 in ?? ()
#6  0x00000000004c16a4 in tonepair_generator (chan=0xd7000000d8, data=0x4, len=0, samples=-1811456528) at channel.c:7379
Backtrace stopped: Cannot access memory at address 0x8ffffffd8

Thread 89 (LWP 29036):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x0000002000000000 in ?? ()
#2  0x00000000ffffffff in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x00007fff103749a0 in ?? ()
#5  0x000006f6103749b0 in ?? ()
#6  0x000000000045ee18 in __ao2_link (self=0x925a40, obj_new=0x1, flags=-1, tag=0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, file=0x0, line=1, func=0x0) at astobj2_container.c:103
#7  0x0000000000000000 in ?? ()

Thread 88 (LWP 59915):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000000005a0430c1 in ?? ()
#2  0x0000000000000028 in ?? ()
#3  0x0000000000000004 in ?? ()
#4  0x00007fdf47718080 in ?? ()
#5  0x00007fdf47718230 in ?? ()
#6  0x00000000004c16a4 in tonepair_generator (chan=0x7fdf780793f8, data=0x4, len=32735, samples=-1810855952) at channel.c:7379
Backtrace stopped: Cannot access memory at address 0x8ffffffd8

Thread 87 (LWP 60616):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000000005a0430c1 in ?? ()
#2  0x0000000000000028 in ?? ()
#3  0x0000000000000004 in ?? ()
#4  0x00007fdf46f57fe0 in ?? ()
#5  0x00007fdf46f58190 in ?? ()
#6  0x00000000004c16a4 in tonepair_generator (chan=0x7fdf9418c708, data=0x4, len=32735, samples=-1811749440) at channel.c:7379
Backtrace stopped: Cannot access memory at address 0x8ffffffd8

Thread 86 (LWP 29040):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x0000000002ae1fb0 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdfa878add0 "240256x250337177", fin=0x0, fout=0x7fdfa878b9c0, ferr=0x7fff10374700) at el.c:74
#3  0x00007fdfa878b700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 85 (LWP 29049):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x0000000002df88a0 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdfa4aacdd0 "240Ϊ244337177", fin=0x0, fout=0x7fdfa4aad9c0, ferr=0x7fff10374720) at el.c:74
#3  0x00007fdfa4aad700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 84 (LWP 29050):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x0000000002df9320 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdfa4a30dd0 "240016243244337177", fin=0x0, fout=0x7fdfa4a319c0, ferr=0x7fff103747e0) at el.c:74
#3  0x00007fdfa4a31700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 83 (LWP 29051):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 82 (LWP 29052):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 81 (LWP 29053):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 80 (LWP 29054):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x0000000002e127b0 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdfa4840dd0 "240016204244337177", fin=0x0, fout=0x7fdfa48419c0, ferr=0x7fff10374800) at el.c:74
#3  0x00007fdfa4841700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 79 (LWP 29055):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x0000000002e41180 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdfa47c4dd0 "240N|244337177", fin=0x0, fout=0x7fdfa47c59c0, ferr=0x7fff10374680) at el.c:74
#3  0x00007fdfa47c5700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 78 (LWP 29056):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x0000000002e53f20 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdfa4748dd0 "240216t244337177", fin=0x0, fout=0x7fdfa47499c0, ferr=0x7fff10374720) at el.c:74
#3  0x00007fdfa4749700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 77 (LWP 29057):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x0000000002e6cb80 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdfa46ccdd0 "240316l244337177", fin=0x0, fout=0x7fdfa46cd9c0, ferr=0x7fff10374720) at el.c:74
#3  0x00007fdfa46cd700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 76 (LWP 29058):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x0000000002eec030 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdfa4650dd0 "240016e244337177", fin=0x0, fout=0x7fdfa46519c0, ferr=0x7fff103740e0) at el.c:74
#3  0x00007fdfa4651700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 75 (LWP 29061):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x0000000002ecc3b0 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdfa44dcdd0 "240316M244337177", fin=0x0, fout=0x7fdfa44dd9c0, ferr=0x7fff103747e0) at el.c:74
#3  0x00007fdfa44dd700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 74 (LWP 29062):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 73 (LWP 29064):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 72 (LWP 29065):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x00000000030ba310 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdfa40e0dd0 "240016016244337177", fin=0x0, fout=0x7fdfa40e19c0, ferr=0x7fff103744e0) at el.c:74
#3  0x00007fdfa40e1700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 71 (LWP 29067):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x0000000003141f20 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdf681fedd0 "240356037h337177", fin=0x0, fout=0x7fdf681ff9c0, ferr=0x7fff103743e0) at el.c:74
#3  0x00007fdf681ff700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 70 (LWP 29068):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 69 (LWP 29070):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x000000000314ab20 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdf49450dd0 "240016EI337177", fin=0x0, fout=0x7fdf494519c0, ferr=0x7fff10374480) at el.c:74
#3  0x00007fdf49451700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 68 (LWP 29071):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 67 (LWP 29072):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x000000000315d1f0 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdf49358dd0 "240216065I337177", fin=0x0, fout=0x7fdf493599c0, ferr=0x7fff10373fe0) at el.c:74
#3  0x00007fdf49359700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 66 (LWP 29073):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000001 in ?? ()
#2  0x0000000003165c60 in ?? ()
#3  0x0000000003165c28 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 65 (LWP 29074):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000001 in ?? ()
#2  0x0000000003166f20 in ?? ()
#3  0x0000000003166ee8 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 64 (LWP 29075):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000001 in ?? ()
#2  0x00000000031681e0 in ?? ()
#3  0x00000000031681a8 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 63 (LWP 29076):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000001 in ?? ()
#2  0x00000000031694a0 in ?? ()
#3  0x0000000003169468 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 62 (LWP 29077):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000001 in ?? ()
#2  0x000000000316a760 in ?? ()
#3  0x000000000316a728 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 61 (LWP 29078):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000001 in ?? ()
#2  0x000000000316ba20 in ?? ()
#3  0x000000000316b9e8 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 60 (LWP 29079):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000001 in ?? ()
#2  0x000000000316cce0 in ?? ()
#3  0x000000000316cca8 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 59 (LWP 29085):
#0  0x00007fdfa673304d in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 58 (LWP 29080):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000001 in ?? ()
#2  0x000000000316dfa0 in ?? ()
#3  0x000000000316df68 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 57 (LWP 29081):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000001 in ?? ()
#2  0x000000000316f390 in ?? ()
#3  0x000000000316f358 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 56 (LWP 29082):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000001 in ?? ()
#2  0x0000000003170780 in ?? ()
#3  0x0000000003170748 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 55 (LWP 29086):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 54 (LWP 29087):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x000000000318ca20 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdf48c14dd0 "240N301H337177", fin=0x0, fout=0x7fdf48c159c0, ferr=0x7fff10373e80) at el.c:74
#3  0x00007fdf48c15700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 53 (LWP 29088):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x00007fdf62189258 in ?? ()
#2  0x00007fdf0000003a in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x0000000003188b00 in ?? ()
#5  0x00007fdf62173fa0 in ?? ()
#6  0x000000000054e4b1 in ast_skip_blanks (str=0x3a <error: Cannot access memory at address 0x3a>) at /root/asterisk-14.5.0/include/asterisk/strings.h:146
#7  __astman_get_header (mode=0, var=0x7fdf48b98d20 "301060004Z", m=0x7fff103744c0) at manager.c:2727
#8  astman_get_header (var=0x7fdf48b98d20 "301060004Z", m=0x7fff103744c0) at manager.c:2753
#9  handle_updates (m=<optimized out>, cfg=0x630c50 <__PRETTY_FUNCTION__.19126>, dfn=0x7fff103744c0 "310=005003", s=<optimized out>) at manager.c:3694
#10 0x00007fdf62149f43 in ?? ()
#11 0x00000000000073f5 in ?? ()
#12 0x00007fdf6218de00 in ?? ()
#13 0x00007fdf48b98dd0 in ?? ()
#14 0x0000000000000000 in ?? ()

Thread 52 (LWP 29093):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x0000034600000000 in ?? ()
#2  0x00007fdfffffffff in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x00007fdf4892ad50 in ?? ()
#5  0x00007fdf4892cdd0 in ?? ()
#6  0x00007fdf54472afe in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 51 (LWP 29089):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x0000000002f62a90 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdf48b1cdd0 "240αH337177", fin=0x0, fout=0x7fdf48b1d9c0, ferr=0x7fff10374380) at el.c:74
#3  0x00007fdf48b1d700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 50 (LWP 29090):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x00000000030270f0 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdf48aa0dd0 "240016252H337177", fin=0x0, fout=0x7fdf48aa19c0, ferr=0x7fff10374380) at el.c:74
#3  0x00007fdf48aa1700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 49 (LWP 29091):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x0000000003033650 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdf48a24dd0 "240N242H337177", fin=0x0, fout=0x7fdf48a259c0, ferr=0x7fff103744e0) at el.c:74
#3  0x00007fdf48a25700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 48 (LWP 29092):
#0  0x00007fdfa673352d in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 47 (LWP 31741):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000019530068bf16 in ?? ()
#2  0x00000000ffffffff in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x00007fdf480f0180 in ?? ()
#5  0x00007fdf94116db8 in ?? ()
#6  0x000000000063300b in state_properties ()
#7  0x00007fdf94116d80 in ?? ()
#8  0x0000001bffffffff in ?? ()
#9  0x000000030000001b in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 46 (LWP 11914):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000000005a0430c1 in ?? ()
#2  0x0000000000000028 in ?? ()
#3  0x0000000000000004 in ?? ()
#4  0x00007fdf482de080 in ?? ()
#5  0x00007fdf482de230 in ?? ()
#6  0x00000000004c16a4 in tonepair_generator (chan=0x7fdf7801fcc8, data=0x4, len=32735, samples=-1810750208) at channel.c:7379
Backtrace stopped: Cannot access memory at address 0x8ffffffd8

Thread 45 (LWP 20074):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000000005a0430c1 in ?? ()
#2  0x0000000000000028 in ?? ()
#3  0x0000000000000004 in ?? ()
#4  0x00007fdf47a00080 in ?? ()
#5  0x00007fdf47a00230 in ?? ()
#6  0x00000000004c16a4 in tonepair_generator (chan=0x7fdf780428d8, data=0x4, len=32735, samples=-1810719216) at channel.c:7379
Backtrace stopped: Cannot access memory at address 0x8ffffffd8

Thread 44 (LWP 29084):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000000005a0430c0 in ?? ()
#2  0x00000000000003e8 in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x0000000003171d80 in ?? ()
#5  0x00007fdf5b5fd1a4 in ?? ()
#6  0x000000000054e4b1 in ast_skip_blanks (str=0x3e8 <error: Cannot access memory at address 0x3e8>) at /root/asterisk-14.5.0/include/asterisk/strings.h:146
#7  __astman_get_header (mode=0, var=0x7fdf48d88d30 "300231330H337177", m=0x7fff103745c0) at manager.c:2727
#8  astman_get_header (var=0x7fdf48d88d30 "300231330H337177", m=0x7fff103745c0) at manager.c:2753
#9  handle_updates (m=<optimized out>, cfg=0x557401 <auth_http_callback+3185>, dfn=0x3170f40 "", s=<optimized out>) at manager.c:3694
#10 0x00007fdf5b5f6256 in ?? ()
#11 0x00007fdf48d88dd0 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 43 (LWP 29103):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x0000000002f956a0 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdf48454dd0 "240NEH337177", fin=0x0, fout=0x7fdf484559c0, ferr=0x7fff10374360) at el.c:74
#3  0x00007fdf48455700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 42 (LWP 29083):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x00007fdf48e04db0 in ?? ()
#2  0x00000000000003e8 in ?? ()
#3  0x0000000000000002 in ?? ()
#4  0x0000000003159cd0 in ?? ()
#5  0x00007fdf48e04db0 in ?? ()
#6  0x000000000054e4b1 in ast_skip_blanks (str=0x3e8 <error: Cannot access memory at address 0x3e8>) at /root/asterisk-14.5.0/include/asterisk/strings.h:146
#7  __astman_get_header (mode=0, var=0x7fdf48e04d50 "", m=0x7fff10374560) at manager.c:2727
#8  astman_get_header (var=0x7fdf48e04d50 "", m=0x7fff10374560) at manager.c:2753
#9  handle_updates (m=<optimized out>, cfg=0x0, dfn=0x0, s=<optimized out>) at manager.c:3694
#10 0x00007fdf5ecf7133 in ?? ()
#11 0x00007fdf48e059c0 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 41 (LWP 31592):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000019530068bf16 in ?? ()
#2  0x00000000ffffffff in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x00007fdfa4e10180 in ?? ()
#5  0x00007fdf9410a508 in ?? ()
#6  0x000000000063300b in state_properties ()
#7  0x00007fdf9410a4d0 in ?? ()
#8  0x00000023ffffffff in ?? ()
#9  0x0000000300000023 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 40 (LWP 31593):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000019530068bf16 in ?? ()
#2  0x00000000ffffffff in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x00007fdfa4d18180 in ?? ()
#5  0x00007fdf9410ab18 in ?? ()
#6  0x000000000063300b in state_properties ()
#7  0x00007fdf9410aae0 in ?? ()
#8  0x00000024ffffffff in ?? ()
#9  0x0000000300000024 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 39 (LWP 29069):
#0  0x00007fdfa59d5363 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 38 (LWP 31594):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000019530068bf16 in ?? ()
#2  0x00000000ffffffff in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x00007fdfa4d94180 in ?? ()
#5  0x00007fdf9410b128 in ?? ()
#6  0x000000000063300b in state_properties ()
#7  0x00007fdf9410b0f0 in ?? ()
#8  0x00000025ffffffff in ?? ()
#9  0x0000000300000025 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 37 (LWP 29066):
#0  0x00007fdfa67300fe in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 36 (LWP 31632):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000019530068bf16 in ?? ()
#2  0x00000000ffffffff in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x00007fdfa4ba4180 in ?? ()
#5  0x00007fdf9401ef78 in ?? ()
#6  0x000000000063300b in state_properties ()
#7  0x00007fdf9401ef40 in ?? ()
#8  0x00000026ffffffff in ?? ()
#9  0x0000000300000026 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 35 (LWP 29063):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x0000000002eeda30 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdfa43e4dd0 "240N>244337177", fin=0x0, fout=0x7fdfa43e59c0, ferr=0x7fff103746a0) at el.c:74
#3  0x00007fdfa43e5700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 34 (LWP 31634):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000019530068bf16 in ?? ()
#2  0x00000000ffffffff in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x00007fdfa4c20180 in ?? ()
#5  0x00007fdf9404b038 in ?? ()
#6  0x000000000063300b in state_properties ()
#7  0x00007fdf9404b000 in ?? ()
#8  0x00000027ffffffff in ?? ()
#9  0x0000000300000027 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 33 (LWP 31635):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000019530068bf16 in ?? ()
#2  0x00000000ffffffff in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x00007fdf47e84180 in ?? ()
#5  0x00007fdf9404b648 in ?? ()
#6  0x000000000063300b in state_properties ()
#7  0x00007fdf9404b610 in ?? ()
#8  0x00000028ffffffff in ?? ()
#9  0x0000000300000028 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 32 (LWP 29060):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 31 (LWP 31636):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000019530068bf16 in ?? ()
#2  0x00000000ffffffff in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x00007fdf47e08180 in ?? ()
#5  0x00007fdf943ad1f8 in ?? ()
#6  0x000000000063300b in state_properties ()
#7  0x00007fdf943ad1c0 in ?? ()
#8  0x00000029ffffffff in ?? ()
#9  0x0000000300000029 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 30 (LWP 29059):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x0000000002ef68b0 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdfa45d4dd0 "240N]244337177", fin=0x0, fout=0x7fdfa45d59c0, ferr=0x7fff10374100) at el.c:74
#3  0x00007fdfa45d5700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 29 (LWP 31715):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000019530068bf16 in ?? ()
#2  0x00000000ffffffff in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x00007fdf483d8180 in ?? ()
#5  0x00007fdf78002238 in ?? ()
#6  0x000000000063300b in state_properties ()
#7  0x00007fdf78002200 in ?? ()
#8  0x00000016ffffffff in ?? ()
#9  0x0000000300000016 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 28 (LWP 31716):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000019530068bf16 in ?? ()
#2  0x00000000ffffffff in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x00007fdf47f00180 in ?? ()
#5  0x00007fdf98000a78 in ?? ()
#6  0x000000000063300b in state_properties ()
#7  0x00007fdf98000a40 in ?? ()
#8  0x00000019ffffffff in ?? ()
#9  0x0000000300000019 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 27 (LWP 29048):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 26 (LWP 31717):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000019530068bf16 in ?? ()
#2  0x00000000ffffffff in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x00007fdf47f7c180 in ?? ()
#5  0x00007fdf9c000a78 in ?? ()
#6  0x000000000063300b in state_properties ()
#7  0x00007fdf9c000a40 in ?? ()
#8  0x0000001affffffff in ?? ()
#9  0x000000030000001a in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 25 (LWP 29045):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x0000000000000001 in ?? ()
#2  0x01000000ffffffff in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x00007fdfa4c9cd20 in ?? ()
#5  0x00007fdfa4c9cdd0 in ?? ()
#6  0x000000000045813b in really_quit (niceness=<optimized out>, restart=0, num=0) at asterisk.c:2115
#7  0x0000000000654460 in ?? ()
#8  0x0000000000000000 in ?? ()

Thread 24 (LWP 29095):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x00000440a5c99720 in ?? ()
#2  0x00007fdfffffffff in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 23 (LWP 29041):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x0000000002dbd690 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdfa4e8cdd0 "240316350244337177", fin=0x0, fout=0x7fdfa4e8d9c0, ferr=0x7fff103746c0) at el.c:74
#3  0x00007fdfa4e8d700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 22 (LWP 29096):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x0000000002f704f0 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdf487b8dd0 "240216{H337177", fin=0x0, fout=0x7fdf487b99c0, ferr=0x7fff10374440) at el.c:74
#3  0x00007fdf487b9700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 21 (LWP 29039):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 20 (LWP 29097):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x3d3d41585a374d63 in ?? ()
#2  0x00000000000003e8 in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x0000000003045310 in ?? ()
#5  0x00007fdf6063a0fd in ?? ()
#6  0x000000000054e4b1 in ast_skip_blanks (str=0x3e8 <error: Cannot access memory at address 0x3e8>) at /root/asterisk-14.5.0/include/asterisk/strings.h:146
#7  __astman_get_header (mode=0, var=0x7fdf4873cd40 "vYqifEK7370222234245337177", m=0x7fff10374580) at manager.c:2727
#8  astman_get_header (var=0x7fdf4873cd40 "vYqifEK7370222234245337177", m=0x7fff10374580) at manager.c:2753
#9  handle_updates (m=<optimized out>, cfg=0x2fa9600, dfn=0x557401 <auth_http_callback+3185> "211205x366377377H213205 366377377I211207200005", s=<optimized out>) at manager.c:3694
#10 0x00007fdf60629acc in ?? ()
#11 0x00007fdf4873cea0 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 19 (LWP 29038):
#0  0x00007fdfa59a084d in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 18 (LWP 29098):
#0  0x00007fdfa59a084d in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 17 (LWP 60769):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000000005a0430c1 in ?? ()
#2  0x000000000000441f in ?? ()
#3  0x0000000000000004 in ?? ()
#4  0x00007fdf47c941c0 in ?? ()
#5  0x00007fdf47c94370 in ?? ()
#6  0x00000000004c16a4 in tonepair_generator (chan=0x7fdf940af1a8, data=0x4, len=32735, samples=1204378048) at channel.c:7379
Backtrace stopped: Cannot access memory at address 0x8ffffffd8

Thread 16 (LWP 29099):
#0  0x00007fdfa59a084d in ?? ()
#1  0x0000000000000002 in ?? ()
#2  0x00007fdfa59a06ec in ?? ()
#3  0x0000000000458e13 in netconsole (vconsole=<error reading variable: Cannot access memory at address 0xfffffb47>) at asterisk.c:1531
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 15 (LWP 60768):
#0  0x00007fdfa672fd84 in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 14 (LWP 29100):
#0  0x00007fdfa673352d in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 13 (LWP 60767):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000000005a0430c1 in ?? ()
#2  0x0000000000000028 in ?? ()
#3  0x0000000000000004 in ?? ()
#4  0x00007fdf47527fe0 in ?? ()
#5  0x00007fdf47528190 in ?? ()
#6  0x00000000004c16a4 in tonepair_generator (chan=0x7fdf9410ecb8, data=0x4, len=32735, samples=-1810739632) at channel.c:7379
Backtrace stopped: Cannot access memory at address 0x8ffffffd8

Thread 12 (LWP 29101):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x00007fdf4854cd80 in ?? ()
#2  0x00000000000003e8 in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x000000000325ef30 in ?? ()
#5  0x00007fdf602106ff in ?? ()
#6  0x000000000054e4b1 in ast_skip_blanks (str=0x3e8 <error: Cannot access memory at address 0x3e8>) at /root/asterisk-14.5.0/include/asterisk/strings.h:146
#7  __astman_get_header (mode=0, var=0x7fdf4854cd20 "240315TH337177", m=0x7fff10374650) at manager.c:2727
#8  astman_get_header (var=0x7fdf4854cd20 "240315TH337177", m=0x7fff10374650) at manager.c:2753
#9  handle_updates (m=<optimized out>, cfg=0x7fdfa5963fc5, dfn=0x0, s=<optimized out>) at manager.c:3694
#10 0x00007fdf6020a20e in ?? ()
#11 0x0000000000000000 in ?? ()

Thread 11 (LWP 60515):
#0  0x00007fdfa67300fe in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 10 (LWP 60199):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000000005a0430c1 in ?? ()
#2  0x0000000000000028 in ?? ()
#3  0x0000000000000004 in ?? ()
#4  0x00007fdf46a7ffe0 in ?? ()
#5  0x00007fdf46a80190 in ?? ()
#6  0x00000000004c16a4 in tonepair_generator (chan=0x7fdf9414c238, data=0x4, len=32735, samples=-1811456528) at channel.c:7379
Backtrace stopped: Cannot access memory at address 0x8ffffffd8

Thread 9 (LWP 29102):
#0  0x00007fdfa6731fd0 in ?? ()
#1  0x0000000003014aa0 in ?? ()
#2  0x000000000061a120 in el_init (prog=0x7fdf484d0dd0 "240016MH337177", fin=0x0, fout=0x7fdf484d19c0, ferr=0x7fff10374400) at el.c:74
#3  0x00007fdf484d1700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 8 (LWP 59970):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000000005a0430c1 in ?? ()
#2  0x0000000000000028 in ?? ()
#3  0x0000000000000004 in ?? ()
#4  0x00007fdf46e5ffe0 in ?? ()
#5  0x00007fdf46e60190 in ?? ()
#6  0x00000000004c16a4 in tonepair_generator (chan=0x7fdf943f9908, data=0x4, len=32735, samples=-1810857536) at channel.c:7379
Backtrace stopped: Cannot access memory at address 0x8ffffffd8

Thread 7 (LWP 10057):
#0  0x00007fdfa59a084d in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 6 (LWP 58940):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000000005a0430c1 in ?? ()
#2  0x0000000000000028 in ?? ()
#3  0x0000000000000004 in ?? ()
#4  0x00007fdf46d68080 in ?? ()
#5  0x00007fdf46d68230 in ?? ()
#6  0x00000000004c16a4 in tonepair_generator (chan=0x7fdf78075458, data=0x4, len=32735, samples=-1811302896) at channel.c:7379
Backtrace stopped: Cannot access memory at address 0x8ffffffd8

Thread 5 (LWP 42496):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000000005a0430c1 in ?? ()
#2  0x0000000000000028 in ?? ()
#3  0x0000000000000004 in ?? ()
#4  0x00007fdf473b4080 in ?? ()
#5  0x00007fdf473b4230 in ?? ()
#6  0x00000000004c16a4 in tonepair_generator (chan=0x7fdf7803f238, data=0x4, len=32735, samples=-1810209984) at channel.c:7379
Backtrace stopped: Cannot access memory at address 0x8ffffffd8

Thread 4 (LWP 38911):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000000005a0430c1 in ?? ()
#2  0x0000000000000028 in ?? ()
#3  0x0000000000000004 in ?? ()
#4  0x00007fdf4835a080 in ?? ()
#5  0x00007fdf4835a230 in ?? ()
#6  0x00000000004c16a4 in tonepair_generator (chan=0xd7000000d8, data=0x4, len=0, samples=-1810857536) at channel.c:7379
Backtrace stopped: Cannot access memory at address 0x8ffffffd8

Thread 3 (LWP 19279):
#0  0x00007fdfa59c9313 in ?? ()
#1  0x000000005a0430c1 in ?? ()
#2  0x0000000000000028 in ?? ()
#3  0x0000000000000004 in ?? ()
#4  0x00007fdf47984080 in ?? ()
#5  0x00007fdf47984230 in ?? ()
#6  0x00000000004c16a4 in tonepair_generator (chan=0x7fdf7803d468, data=0x4, len=32735, samples=-1810750848) at channel.c:7379
Backtrace stopped: Cannot access memory at address 0x8ffffffd8

Thread 2 (LWP 11090):
#0  0x00007fdfa6732d2d in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 1 (LWP 29094):
#0  0x00007fdfa5917425 in ?? ()
#1  0x00007fdfa591ab8b in ?? ()
#2  0x0000000000000004 in ?? ()
#3  0x0000000000000005 in ?? ()
#4  0x00007fdfa88899a8 in ?? ()
#5  0x000000000000001f in ?? ()
#6  0x00007fdfa5a5abb7 in ?? ()
#7  0x0000000000000001 in ?? ()
#8  0x00007fdfa58f30d1 in ?? ()
#9  0x0000000000000005 in ?? ()
#10 0x00007fdfa5a5bc4b in ?? ()
#11 0x0000000000000003 in ?? ()
#12 0x00007fdf488afbbe in ?? ()
#13 0x0000000000000002 in ?? ()
#14 0x00007fdfa5a5ab82 in ?? ()
#15 0x0000000000000001 in ?? ()
#16 0x00007fdfa5a5cea7 in ?? ()
#17 0x0000000000000003 in ?? ()
#18 0x00007fdf488afba4 in ?? ()
#19 0x000000000000000c in ?? ()
#20 0x00007fdfa5a5ceab in ?? ()
#21 0x0000000000000002 in ?? ()
#22 0x0000000000000020 in ?? ()
#23 0x0000000000000000 in ?? ()

best regards

Timmy

4

It looks like the stack is severely corrupted, assuming you ran gdb on he same machine.

That could be difficult to debug. You need to try and work out what triggered the failure.

5

hmm… actually, i did’t run the ast_coredumper on the same machine. I had issues getting the gdb ubuntu package installed on it. so i copied the coredump to another machine and ran it from there. i didn’t know it made any difference. sorry about that.

i’ll see if i can manage to get gdb installed on the crashing machines and run ast_coredumper from there

6

Yes, the environment (binaries, etc) have to be the same as to what the core dump was generated from or else the backtrace will be completely useless.

7

Hello again!.

Thank you for your help. I finally managed to get gdb running on the crashing machines, and just got a brand new crash. this is the information i get when running ast_coredumper. any thoughts? it doesn’t seem to say very much to me…

Full.txt

!@!@!@! full.txt !@!@!@!


Thread 14 (LWP 2955):
#0  0x00007fb971bafd84 in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 13 (LWP 2956):
#0  0x00007fb971bafd84 in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 12 (LWP 2981):
#0  0x00007fb972ce5313 in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 11 (LWP 2982):
#0  0x00007fb971bafd84 in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 10 (LWP 2984):
#0  0x00007fb971bafd84 in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 9 (LWP 2999):
#0  0x00007fb972ce5313 in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 8 (LWP 3000):
#0  0x00007fb971bafd84 in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 7 (LWP 3001):
#0  0x00007fb971bafd84 in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 6 (LWP 2915):
#0  0x00007fb972cbc84d in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 5 (LWP 2922):
#0  0x00007fb972ce5313 in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 4 (LWP 2921):
#0  0x00007fb971bafd84 in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 3 (LWP 2923):
#0  0x00007fb972c79062 in ?? ()
No symbol table info available.
#1  0x000000000000002a in ?? ()
No symbol table info available.
#2  0x00007fb972c47f39 in ?? ()
No symbol table info available.
#3  0xffffffff00000000 in ?? ()
No symbol table info available.
#4  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 2 (LWP 2983):
#0  0x00007fb971bafd84 in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 1 (LWP 2913):
#0  0x00007fb972c7c6d7 in ?? ()
No symbol table info available.
#1  0x00007fffd08cbf20 in ?? ()
No symbol table info available.
#2  0x0000000100000000 in ?? ()
No symbol table info available.
#3  0x00000000016ee6c0 in ?? ()
No symbol table info available.
#4  0x00000000016f27b0 in ?? ()
No symbol table info available.
#5  0x0000000000000000 in ?? ()
No symbol table info available.

Thread.txt

!@!@!@! thread1.txt !@!@!@!


Thread 1 (LWP 2913):
#0  0x00007fb972c7c6d7 in ?? ()
No symbol table info available.
#1  0x00007fffd08cbf20 in ?? ()
No symbol table info available.
#2  0x0000000100000000 in ?? ()
No symbol table info available.
#3  0x00000000016ee6c0 in ?? ()
No symbol table info available.
#4  0x00000000016f27b0 in ?? ()
No symbol table info available.
#5  0x0000000000000000 in ?? ()
No symbol table info available.
8

There is absolutely nothing present in the backtrace. Can’t comment at all on it.

9

i was afraid of that. Isn’t it quite strange that the backtrace gives no information of interest what so ever?
is there anything i can do to get better information on why this happens?

Asterisk is already compiled with DONT_OPTIMIZE and BETTER_BACKTRACES

10

What are the complete steps you are doing to build Asterisk?

11

I have made upgrades from Asterisk 11, the reason is because of some bugs when using WebRTC in that version.
As mentioned we never had any issues like this with Asterisk 11.

the changes that have been made for installing Asterisk 14:

Installed ubuntu packages:
libjansson4
libjansson-dev
uuid-dev

Asterisk install:

./configure --with-pjproject-bundled
make menuconfig
(Ticking DONT_OPTIMIZE and BETTER_BACKTRACES. Unticking COMPILE_DOUBLE)
make
make install
12

Somehow you have ended up with completely stripped binaries, and no debug symbol tables, so there is nothing for the debugger to work on.

13

If you add “–enable-dev-mode” to confgure does that alter things?

14

I’ll recompile and await a new crash. Thanks for the tip!

15

One more thing: Ubuntu 12.04 LTS is not supported anymore. Furthermore, Asterisk 14 does not get any bug fixes anymore. I recommend to go for Ubuntu 16.04 and Asterisk 13, because both are LTS versions. If you find a bug, and somebody fixes that bug, you would have to change versions anyway.

16

traud: thanks for the tip. I’ll look into it.

however, i just had a crash again (after a month). this time compiled also wtih AST_DEVMODE, DONT_OPTIMIZE, BETTER_BACKTRACES. i can’t make much out of these, but maybe you can?

here are the files genereated from ast_coredumper. They were too large to paste into the forum, so i’ve made pastebin-links

Brief:
https://pastebin.com/Yu4VJ3ia

Full:
https://pastebin.com/Ks9vPDid

Thread1:
https://pastebin.com/Nm5gqY4n

17

You have memory corruption. Unfortunately that means the crash is secondary to the original fault. You should look for errors and warnings, particularly about locking or something having gone away, in the few minutes before the crash.

18

The crash also appears to be during the use of a third party module, app_konference.

19

Thank you for your replies!

David551: What in the logs tells you that i have memory corruption? We’re running the Ast14 on the exact same server as the previous flawlessly working Ast11-install. we’ve upgraded 50% of our asterisks and only the Ast14-installations ones crash like this.

Jcolp: Yes, we’re using a 3rd party module called app_konference quite frequently in our setup. is there anything that suggests that it would be the cause of the crash?

20

The thread which things crashed in was in app_konference:

Thread 1 (Thread 0x7f2582ad2700 (LWP 1082)):
#0  0x00007f25c1d7c425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f25c1d7fb8b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f25c1dba39e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f25c1dc4b96 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007f25c1dc6d0b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x00007f25c1dca388 in calloc () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x000000000065c388 in _ast_calloc (num=1, len=530, file=0x6b0140 "frame.c", lineno=333, func=0x6b0714 "ast_frdup") at /home/s2/asterisk-14.7.2/include/asterisk/utils.h:573
#7  0x00000000005539ba in ast_frdup (f=0x7f25bc05c488) at frame.c:333
#8  0x00000000005533ae in ast_frisolate (fr=0x7f25bc05c488) at frame.c:196
#9  0x0000000000652c7e in ast_trans_frameout (pvt=0x7f25bc05c480, datalen=0, samples=0) at translate.c:458
#10 0x0000000000652ca2 in default_frameout (pvt=0x7f25bc05c480) at translate.c:463
#11 0x0000000000653320 in ast_translate (path=0x7f25bc05c480, f=0x7f25bc0571d0, consume=0) at translate.c:579
#12 0x00007f257b5cb967 in mix_frames () from /usr/lib/asterisk/modules/app_konference.so
#13 0x00007f257b5c400d in ?? () from /usr/lib/asterisk/modules/app_konference.so
#14 0x000000000065e2b4 in dummy_start (data=0x7f25bc00f150) at utils.c:1233
#15 0x00007f25c2b4de9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00007f25c1e39ccd in clone () from /lib/x86_64-linux-gnu/libc.so.6

I don’t know how they are using Asterisk frames and if something may have changed that they are not taking into account causing corruption or other problems.