Asterisk Crash - MoH module

[it_manager]

I have an Asterisk Box that is crashing periodically when playing MoH in the queue. Below you can see the backtrace:

#0  0x00007f18cfcaa387 in raise () from /lib64/libc.so.6
#1  0x00007f18cfcaba78 in abort () from /lib64/libc.so.6
#2  0x00007f18cfcecf67 in __libc_message () from /lib64/libc.so.6
#3  0x00007f18cfcf5329 in _int_free () from /lib64/libc.so.6
#4  0x00007f1867916777 in moh_scan_files (class=class@entry=0x7f17f4003f68) at res_musiconhold.c:1109
#5  0x00007f1867918073 in local_ast_moh_start (chan=0x7f185ee96838, mclass=<optimized out>, interpclass=0x0) at res_musiconhold.c:1560
#6  0x00007f18c81d017e in say_position (qe=qe@entry=0x7f18c9d54230, ringing=ringing@entry=0) at app_queue.c:3354
#7  0x00007f18c81d7669 in wait_our_turn (reason=<optimized out>, ringing=<optimized out>, qe=<optimized out>) at app_queue.c:4899
#8  queue_exec (chan=<optimized out>, data=<optimized out>) at app_queue.c:7306
#9  0x00000000005341fa in pbx_exec (c=c@entry=0x7f185ee96838, app=app@entry=0x2efe910, data=data@entry=0x7f18c9d55a00 "queue1,t,,,3600,,,handler") at pbx.c:1686
#10 0x000000000053f050 in pbx_extension_helper (c=c@entry=0x7f185ee96838, context=<optimized out>, exten=exten@entry=0x7f185ee976d8 "s", priority=priority@entry=23, label=label@entry=0x0,
    callerid=callerid@entry=0x7f17e8021e20 "012345678", action=action@entry=E_SPAWN, found=0x7f18c9d59b70, combined_find_spawn=combined_find_spawn@entry=1, con=0x0) at pbx.c:5037
#11 0x000000000053f713 in ast_spawn_extension (c=c@entry=0x7f185ee96838, context=<optimized out>, exten=exten@entry=0x7f185ee976d8 "s", priority=priority@entry=23,
    callerid=callerid@entry=0x7f17e8021e20 "012345678", found=<optimized out>, combined_find_spawn=combined_find_spawn@entry=1) at pbx.c:6254
#12 0x00007f18a4881c37 in _macro_exec (chan=<optimized out>, data=<optimized out>, exclusive=<optimized out>) at app_macro.c:433
#13 0x00000000005341fa in pbx_exec (c=c@entry=0x7f185ee96838, app=app@entry=0x2f8c710, data=data@entry=0x7f18c9d5aac0 "toqueue,123,1") at pbx.c:1686
#14 0x000000000053f050 in pbx_extension_helper (c=c@entry=0x7f185ee96838, context=<optimized out>, exten=exten@entry=0x7f185ee976d8 "s", priority=priority@entry=725, label=label@entry=0x0,
    callerid=callerid@entry=0x7f17e8021e20 "012345678", action=action@entry=E_SPAWN, found=0x7f18c9d5ec40, combined_find_spawn=combined_find_spawn@entry=1, con=0x0) at pbx.c:5037
#15 0x000000000053f713 in ast_spawn_extension (c=c@entry=0x7f185ee96838, context=<optimized out>, exten=exten@entry=0x7f185ee976d8 "s", priority=priority@entry=725,
    callerid=callerid@entry=0x7f17e8021e20 "012345678", found=<optimized out>, combined_find_spawn=combined_find_spawn@entry=1) at pbx.c:6254
#16 0x00007f18a4881c37 in _macro_exec (chan=<optimized out>, data=<optimized out>, exclusive=<optimized out>) at app_macro.c:433
#17 0x00000000005341fa in pbx_exec (c=c@entry=0x7f185ee96838, app=app@entry=0x2f8c710, data=data@entry=0x7f18c9d5fba0 "Inbound,101,101,2,101") at pbx.c:1686
#18 0x000000000053f050 in pbx_extension_helper (c=c@entry=0x7f185ee96838, context=<optimized out>, exten=exten@entry=0x7f185ee976d8 "s", priority=priority@entry=33, label=label@entry=0x0,
    callerid=callerid@entry=0x7f17e8021e20 "012345678", action=action@entry=E_SPAWN, found=found@entry=0x7f18c9d63c20, combined_find_spawn=combined_find_spawn@entry=1, con=0x0) at pbx.c:5037
#19 0x0000000000544c7b in ast_spawn_extension (combined_find_spawn=1, found=0x7f18c9d63c20, callerid=0x7f17e8021e20 "012345678", priority=33, exten=0x7f185ee976d8 "s", context=<optimized out>,
    c=0x7f185ee96838) at pbx.c:6254
#20 __ast_pbx_run (c=c@entry=0x7f185ee96838, args=args@entry=0x0) at pbx.c:6729
#21 0x000000000054627b in pbx_thread (data=data@entry=0x7f185ee96838) at pbx.c:7059
#22 0x000000000058d7aa in dummy_start (data=<optimized out>) at utils.c:1223
#23 0x00007f18ce64fea5 in start_thread () from /lib64/libpthread.so.0
#24 0x00007f18cfd72b0d in clone () from /lib64/libc.so.6

Music class seems to be fine, and there is one single file in that directory:

(gdb) f 4
#4  0x00007f1867916777 in moh_scan_files (class=class@entry=0x7f17f4003f68) at res_musiconhold.c:1109
1109    res_musiconhold.c: No such file or directory.
(gdb) p *class
$5 = {name = "1656065184", '\000' <repeats 69 times>, dir = "/var/lib/sounds/moh/1656065184", '\000' <repeats 196 times>, args = '\000' <repeats 255 times>,
  announcement = '\000' <repeats 255 times>, mode = "files", '\000' <repeats 74 times>, digit = 0 '\000', filearray = 0x7f17f401bd80, allowed_files = 8, total_files = 1, flags = 0, format = {
    id = AST_FORMAT_SLINEAR, fattr = {format_attr = {0 <repeats 64 times>}, rtp_marker_bit = 0 '\000'}}, pid = 0, start = 1669625461, thread = 0, srcfd = -1, timer = 0x0, realtime = 1,
  delete = 0, members = {first = 0x0, last = 0x0}, list = {next = 0x0}}

This is happening randomly and can’t be replicated.

Any ideas about what could be the issue?

[jcolp]

You’d need to specify the version of Asterisk in use.

[it_manager]

Sorry, it’s a bit old - Asterisk 11.25.3

[jcolp]

walks away

[david551]

From the backtrace it is most likely to be freeing memory that has not been allocated, or has already been freed. The original fault could well be somewhat removed from the crash. If you can access the parameters of __libc_message, you will probably find a message saying is it is an invalid free.

However, as Joshua implies, the only possible response is to upgrade to a supported version.

Also, if reporting a crash on a supported version, you need to use an un-optimised build, as optimised backtraces loose too much information.

[it_manager]

Thanks, @david551. I will try to upgrade and see if it still persists.