Asterisk crashes on realtime with libc-2.17.so

Asterisk Asterisk Support
#1

Hello,

we have an Asterisk 16.9.0 in a Centos 7.
After setting up realtime for ps_endpoint, ps_contacts, etc. and restart the asterisk service, it crashes.
The odbc driver is → mysql-connector-odbc-5.3.14-linux-el7-x86-64bit

The configuration of the odbc.ini, etc is ok but as soon as res_odbc.so is loaded it crashes.

This is the log:

 asterisk[26472]: segfault at 1010070 ip 00007f25729a166e sp 00007ffdd2731980 error 4 in libmyodbc5w.so

We’re not sure what would be the proper version of the odbc driver to get this working or at least not crashing Asterisk.

Can you help us with that please?.

Many thanks !!!
Richi

#2

https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace

Also, for crashes, you are likely to need the logs that Asterisk, itself, produces, leading up to the crash. The initial cause of the crash may happen some time before the crash.

#3

I got more data.

We can start Asterisk with realtime setup as long as we don’t load the res_odbc module. As soon as we load that module Asterisk crashes:

a2*CLI> module load res_odbc
a2*CLI>
Disconnected from Asterisk server
Asterisk cleanly ending (0).
Executing last minute cleanups

And this is the dmesg log

[195056.829810] asterisk[12391]: segfault at 0 ip 00007f265b9e1016 sp 00007f25a3fc27f8 error 4 in libc-2.17.so[7f265b8a3000+1c2000]

It doesn’t look like a problem with ODBC it self because we are able to do a:

isql sipodbc -v
+---------------------------------------+
| Connected!                            |
|                                       |
| sql-statement                         |
| help [tablename]                      |
| quit                                  |
|                                       |
+---------------------------------------+
SQL>

Looks like smth is wrong with the res_odbc module.

Thanks for your help with this :wink:

#4

You can get crashes on Load if the Asterisk core was compiled with different versions or options, from those used for the module.

#5

David … we’re trying to get the Backtrace using

/var/lib/asterisk/scripts/ast_coredumper --latest

but it takes so long that we don’t even know if it’s doing smth.
The core is 109M … how long it would normally take to process it?

In the directory we just can see a .ast_coredumper.gdbinit, not any .txt yet

#6

I’ve always done it manually, directly issuing gdb commands, and the actual processing time is quite short. I wouldn’t expect the script to take more than a few seconds to run, but I’ve run it on systems where all the files are probably still in the cache.

#7

Got it, thanks David.
Here is the backtrace:

#0  0x00007f61f3482016 in __strcmp_sse42 () from /usr/lib64/libc.so.6
No symbol table info available.
#1  0x00007f61f4d203b9 in getrn () from /usr/lib64/libcrypto.so.10
No symbol table info available.
#2  0x00007f61f4d206a0 in lh_insert () from /usr/lib64/libcrypto.so.10
No symbol table info available.
#3  0x00007f61f4c6b45f in OBJ_NAME_add () from /usr/lib64/libcrypto.so.10
No symbol table info available.
#4  0x00007f613b01a432 in openssl_add_all_ciphers_int () at crypto/evp/c_allc.c:83
No locals.
#5  0x00007f613b00bc4b in ossl_init_add_all_ciphers () at crypto/init.c:252
No locals.
#6  ossl_init_add_all_ciphers_ossl_ () at crypto/init.c:241
No locals.
#7  0x00007f61f43a6e40 in pthread_once () from /usr/lib64/libpthread.so.0
No symbol table info available.
#8  0x00007f613afed9e0 in CRYPTO_THREAD_run_once (once=<optimized out>, init=<optimized out>) at crypto/threads_pthread.c:118
No locals.
#9  0x00007f613b00b966 in OPENSSL_init_crypto (opts=76, settings=0x0) at crypto/init.c:677
No locals.
#10 0x00007f613afbc46c in OPENSSL_init_ssl (opts=76, settings=<optimized out>) at ssl/ssl_init.c:205
        stoperrset = 0
#11 0x00007f613afd6bb0 in ssl_start () at /export/home/pb2/build/sb_0-36131509-1569573142.98/mysql-5.7.28/vio/viosslfactories.c:440
No locals.
#12 0x00007f613af5a865 in mysql_server_init (argc=<optimized out>, argv=<optimized out>, groups=<optimized out>) at /export/home/pb2/build/sb_0-36131509-1569573142.98/mysql-5.7.28/libmysql/libmysql.c:132
        result = 0
#13 0x00007f613af4e1d7 in mysql_init (mysql=0x7f614401cd48) at /export/home/pb2/build/sb_0-36131509-1569573142.98/mysql-5.7.28/sql-common/client.c:2359
No locals.
#14 0x00007f613af0af04 in myodbc_do_connect (dbc=dbc@entry=0x7f614401cd40, ds=ds@entry=0x7f614401db40) at /export/home2/pb2/test/sb_3-36517154-1572285605.31/mysql-connector-odbc-5.3.14-src/driver/connect.cc:180
        flags = <optimized out>
        off_int = 0
        rc = 0
        mysql = 0x7f614401cd48
        opt_ssl_verify_server_cert = 4294967295
        on = 1 '\001'
        on_int = 1
        max_long = 18446744073709551615
#15 0x00007f613af0b8f6 in MySQLConnect (hdbc=0x7f614401cd40, szDSN=0x7f6144018590, cbDSN=-3, szUID=0x7f6144018460, cbUID=-3, szAuth=0x7f614401da40, cbAuth=-3)
    at /export/home2/pb2/test/sb_3-36517154-1572285605.31/mysql-connector-odbc-5.3.14-src/driver/connect.cc:605
        rc = <optimized out>
        dbc = 0x7f614401cd40
        ds = 0x7f614401db40
#16 0x00007f613af24f66 in SQLConnectW (hdbc=<optimized out>, dsn=<optimized out>, dsn_len=<optimized out>, user=<optimized out>, user_len=<optimized out>, auth=<optimized out>, auth_len=-3)
    at /export/home2/pb2/test/sb_3-36517154-1572285605.31/mysql-connector-odbc-5.3.14-src/driver/unicode.cc:240
No locals.
#17 0x00007f613bb153b3 in SQLConnect () from /usr/lib64/libodbc.so.2
No symbol table info available.
#18 0x00007f613bd72e8a in odbc_obj_connect (obj=0x7f6144005270) at res_odbc.c:1059
        res = 0
        err = 0
        mlen = 0
        msg = "P\336\006<a\177\000\000]\363E", '\000' <repeats 13 times>, "\240\023\000Da\177\000\000\000\335\006<a\177\000\000P\305\001\000\000\000\000\000\340\064\327;a\177\000\000\353\064\327;a\177\000\000\000\000\000\000\000\000\000\000\217Ld\000\000\000\000\000u\370\326;a\177\000\000 \000\000\000\000\000\000\000\311!\327;a\177\000\000 5\000Da\177\000\000\000\000\000\000\000\000\000\000\060R\000Da\177\000\000@\000\000\000\000\000\000\000XR\000Da\177\000\000\030", '\000' <repeats 15 times>, "\240\336\006<a\177\000\000\302\366E\000\000\000\000\000\004\000\000\000\003\000\000\000\b5\000Da\177\000\000\354\064\327;a\177\000"
        state = "\257\003\000\000\001\000\000\000 5"
        con = 0x7f61440052a0
        negative_cache_expiration = 0
        __PRETTY_FUNCTION__ = "odbc_obj_connect"
#19 0x00007f613bd72642 in _ast_odbc_request_obj2 (name=0x7f6144003528 "asterisk-cdr", flags=..., file=0x7f613bd734e0 "res_odbc.c", function=0x7f613bd74340 <__PRETTY_FUNCTION__.16963> "odbc_register_class", lineno=805) at res_odbc.c:944
        obj = 0x7f6144005270
        class = 0x7f6144003520
        __PRETTY_FUNCTION__ = "_ast_odbc_request_obj2"
#20 0x00007f613bd72a50 in _ast_odbc_request_obj (name=0x7f6144003528 "asterisk-cdr", check=0, file=0x7f613bd734e0 "res_odbc.c", function=0x7f613bd74340 <__PRETTY_FUNCTION__.16963> "odbc_register_class", lineno=805) at res_odbc.c:992
        flags = {flags = 0}
#21 0x00007f613bd71fc3 in odbc_register_class (class=0x7f6144003520, preconnect=-1) at res_odbc.c:805
 obj = 0x7f613bd734e0
        __PRETTY_FUNCTION__ = "odbc_register_class"
#22 0x00007f613bd71877 in load_odbc_config () at res_odbc.c:705
        cfg = 0x7f613bd742e6 "res_odbc.conf"
        config = 0x7f6144001ca0
        v = 0x0
        cat = 0x7f61440041c0 "asterisk-cdr"
        dsn = 0x7f61440043a4 "asterisk-cdr"
        username = 0x7f6144004439 "root"
        password = 0x7f61440044c9 "HJD873sdf*oiu3rjklsdf"
        sanitysql = 0x0
        enabled = -1
        bse = 1
        conntimeout = 10
        forcecommit = 0
        isolation = 2
        maxconnections = 1
        logging = 0
        slowquerylimit = 5000
        ncache = {tv_sec = 0, tv_usec = 0}
        preconnect = -1
        res = 0
        config_flags = {flags = 0}
        new = 0x7f6144003520
        __PRETTY_FUNCTION__ = "load_odbc_config"
#23 0x00007f613bd733ea in load_module () at res_odbc.c:1130
        __PRETTY_FUNCTION__ = "load_module"
#24 0x000000000050a63c in start_resource (mod=0x7f6144001b30) at loader.c:1711
        tmp = "\020\341\006<a\177\000\000P\020F\000\000\000\000\000\250\341\006<a\177\000\000\220\341\006<a\177", '\000' <repeats 19 times>, "\320\a\000\000\000\000\000\004\000\000\000\000\000\000\000`\341\006<a\177\000\000\313(Q\000\000\000\000\000\220\341\006<a\177\000\000\200\341\006<a\177\000\000\200\341\006<a\177\000\000p-f\000\000\000\000\000\256\341\006<a\177\000\000\220\341\006<\020\000\000\000\360\341\006<a\177\000\000\246\204P\000\000\000\000\000\030\000\000\000\060\000\000\000h?\327;a\177\000\000\207vy5#\352:\323%*\273\065\207\344\"$", '\000' <repeats 79 times>
        res = 32609
        __PRETTY_FUNCTION__ = "start_resource"
#25 0x000000000050ac7e in load_resource (resource_name=0x7f6144000a4c "res_odbc.so", suppress_logging=0, module_priorities=0x0, required=0, preload=0) at loader.c:1798
        mod = 0x7f6144001b30
        res = AST_MODULE_LOAD_SUCCESS
        __PRETTY_FUNCTION__ = "load_resource"
#26 0x000000000050ad80 in ast_load_resource (resource_name=0x7f6144000a4c "res_odbc.so") at loader.c:1821
        res = 0
        __PRETTY_FUNCTION__ = "ast_load_resource"
#27 0x00000000004baf93 in handle_load (e=0x8e7820 <cli_cli+2400>, cmd=-4, a=0x7f613c06e320) at cli.c:241
No locals.
#28 0x00000000004c3aae in ast_cli_command_full (uid=0, gid=0, fd=37, s=0x7f613c06e600 "module load res_odbc.so") at cli.c:2833
        args = {0x8e7820 <cli_cli+2400> "\240\226*\001", 0x7f6144000a40 "module", 0x7f6144000a47 "load", 0x7f6144000a4c "res_odbc.so", 0x0, 0x7f61440022d9 "", 0x7f61440022dc "", 0x0, 0x25 <Address 0x25 out of bounds>,
          0x7f613c06e7ec "\f", 0x10 <Address 0x10 out of bounds>, 0x0, 0x0, 0x0, 0x0, 0xffff00001fa0 <Address 0xffff00001fa0 out of bounds>, 0x0 <repeats 16 times>, 0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>,
          0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x4040404040404040 <Address 0x4040404040404040 out of bounds>, 0x4040404040404040 <Address 0x4040404040404040 out of bounds>,
          0x5a5a5a5a5a5a5a5a <Address 0x5a5a5a5a5a5a5a5a out of bounds>, 0x5a5a5a5a5a5a5a5a <Address 0x5a5a5a5a5a5a5a5a out of bounds>, 0x2020202020202020 <Address 0x2020202020202020 out of bounds>,
          0x2020202020202020 <Address 0x2020202020202020 out of bounds>, 0xffffffff <Address 0xffffffff out of bounds>, 0xffffffffffff0000 <Address 0xffffffffffff0000 out of bounds>, 0x0, 0x0,
          0xff00ffffffffffff <Address 0xff00ffffffffffff out of bounds>, 0xffff00ffffffffff <Address 0xffff00ffffffffff out of bounds>, 0x2000000000000000 <Address 0x2000000000000000 out of bounds>,
          0x20000000000000 <Address 0x20000000000000 out of bounds>, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
          0x7f613c06e850 "\374Channel PJSIP/NSG-00000000 left 'simple_bridge' stasis-bridge <bridge_A2-1615017473.1>\n1>\n\373Bridge bridge_A2-1615017473.1: switching from native_rtp technology to simple_bridge\n"}
        e = 0x8e7820 <cli_cli+2400>
        x = 3
        duplicate = 0x7f6144000a40 "module"
        tmp = "module load res_odbc.so\000 22 silent\000\000\000\000\000\000\000\367\006<a\177\000\000\020\350\006<a\177\000\000\036=4\366a\177\000\000\005"
        retval = 0x2 <Address 0x2 out of bounds>
        a = {fd = 37, argc = 3, argv = 0x7f613c06e3a8, line = 0x0, word = 0x0, pos = 0, n = 0}
        __PRETTY_FUNCTION__ = "ast_cli_command_full"
#29 0x00000000004c3c30 in ast_cli_command_multiple_full (uid=0, gid=0, fd=37, size=24, s=0x7f613c06ea50 "module load res_odbc.so") at cli.c:2860
        cmd = "module load res_odbc.so\000 22 silent", '\000' <repeats 54 times>, "*\216C\363a\177\000\000\000\000nux", '\000' <repeats 59 times>, "\377\377\377\377", '\000' <repeats 20 times>, " \000\000Da\177\000\000P\347\006<a\177\000\000"...
        x = 23
        y = 24
        count = 0
#30 0x0000000000456879 in netconsole (vconsole=0x913f40 <consoles>) at asterisk.c:1425
_PRETTY_FUNCTION__ = "load_odbc_config"
#23 0x00007f613bd733ea in load_module () at res_odbc.c:1130
        __PRETTY_FUNCTION__ = "load_module"
#24 0x000000000050a63c in start_resource (mod=0x7f6144001b30) at loader.c:1711
        tmp = "\020\341\006<a\177\000\000P\020F\000\000\000\000\000\250\341\006<a\177\000\000\220\341\006<a\177", '\000' <repeats 19 times>, "\320\a\000\000\000\000\000\004\000\000\000\000\000\000\000`\341\006<a\177\000\000\313(Q\000\000\000\000\000\220\341\006<a\177\000\000\200\341\006<a\177\000\000\200\341\006<a\177\000\000p-f\000\000\000\000\000\256\341\006<a\177\000\000\220\341\006<\020\000\000\000\360\341\006<a\177\000\000\246\204P\000\000\000\000\000\030\000\000\000\060\000\000\000h?\327;a\177\000\000\207vy5#\352:\323%*\273\065\207\344\"$", '\000' <repeats 79 times>
        res = 32609
        __PRETTY_FUNCTION__ = "start_resource"
#25 0x000000000050ac7e in load_resource (resource_name=0x7f6144000a4c "res_odbc.so", suppress_logging=0, module_priorities=0x0, required=0, preload=0) at loader.c:1798
        mod = 0x7f6144001b30
        res = AST_MODULE_LOAD_SUCCESS
        __PRETTY_FUNCTION__ = "load_resource"
#26 0x000000000050ad80 in ast_load_resource (resource_name=0x7f6144000a4c "res_odbc.so") at loader.c:1821
        res = 0
        __PRETTY_FUNCTION__ = "ast_load_resource"
#27 0x00000000004baf93 in handle_load (e=0x8e7820 <cli_cli+2400>, cmd=-4, a=0x7f613c06e320) at cli.c:241
No locals.
#28 0x00000000004c3aae in ast_cli_command_full (uid=0, gid=0, fd=37, s=0x7f613c06e600 "module load res_odbc.so") at cli.c:2833
        args = {0x8e7820 <cli_cli+2400> "\240\226*\001", 0x7f6144000a40 "module", 0x7f6144000a47 "load", 0x7f6144000a4c "res_odbc.so", 0x0, 0x7f61440022d9 "", 0x7f61440022dc "", 0x0, 0x25 <Address 0x25 out of bounds>,
          0x7f613c06e7ec "\f", 0x10 <Address 0x10 out of bounds>, 0x0, 0x0, 0x0, 0x0, 0xffff00001fa0 <Address 0xffff00001fa0 out of bounds>, 0x0 <repeats 16 times>, 0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>,
          0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x4040404040404040 <Address 0x4040404040404040 out of bounds>, 0x4040404040404040 <Address 0x4040404040404040 out of bounds>,
          0x5a5a5a5a5a5a5a5a <Address 0x5a5a5a5a5a5a5a5a out of bounds>, 0x5a5a5a5a5a5a5a5a <Address 0x5a5a5a5a5a5a5a5a out of bounds>, 0x2020202020202020 <Address 0x2020202020202020 out of bounds>,
          0x2020202020202020 <Address 0x2020202020202020 out of bounds>, 0xffffffff <Address 0xffffffff out of bounds>, 0xffffffffffff0000 <Address 0xffffffffffff0000 out of bounds>, 0x0, 0x0,
          0xff00ffffffffffff <Address 0xff00ffffffffffff out of bounds>, 0xffff00ffffffffff <Address 0xffff00ffffffffff out of bounds>, 0x2000000000000000 <Address 0x2000000000000000 out of bounds>,
          0x20000000000000 <Address 0x20000000000000 out of bounds>, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
          0x7f613c06e850 "\374Channel PJSIP/NSG-00000000 left 'simple_bridge' stasis-bridge <bridge_A2-1615017473.1>\n1>\n\373Bridge bridge_A2-1615017473.1: switching from native_rtp technology to simple_bridge\n"}
        e = 0x8e7820 <cli_cli+2400>
        x = 3
        duplicate = 0x7f6144000a40 "module"
        tmp = "module load res_odbc.so\000 22 silent\000\000\000\000\000\000\000\367\006<a\177\000\000\020\350\006<a\177\000\000\036=4\366a\177\000\000\005"
        retval = 0x2 <Address 0x2 out of bounds>
        a = {fd = 37, argc = 3, argv = 0x7f613c06e3a8, line = 0x0, word = 0x0, pos = 0, n = 0}
        __PRETTY_FUNCTION__ = "ast_cli_command_full"
#29 0x00000000004c3c30 in ast_cli_command_multiple_full (uid=0, gid=0, fd=37, size=24, s=0x7f613c06ea50 "module load res_odbc.so") at cli.c:2860
        cmd = "module load res_odbc.so\000 22 silent", '\000' <repeats 54 times>, "*\216C\363a\177\000\000\000\000nux", '\000' <repeats 59 times>, "\377\377\377\377", '\000' <repeats 20 times>, " \000\000Da\177\000\000P\347\006<a\177\000\000"...
        x = 23
        y = 24
        count = 0
#30 0x0000000000456879 in netconsole (vconsole=0x913f40 <consoles>) at asterisk.c:1425
        cmds_read = 1
        bytes_read = 24
        con = 0x913f40 <consoles>
        hostname = "a2", '\000' <repeats 61 times>
        inbuf = "module load res_odbc.so\000 22 silent\000logger mute silent", '\000' <repeats 107 times>, "\221\024\371Þ\200\377\377\003", '\000' <repeats 23 times>, "\060\000\000\000[\000\000\000p"...
        outbuf = "\374Channel PJSIP/NSG-00000000 left 'simple_bridge' stasis-bridge <bridge_A2-1615017473.1>\n1>\n\373Bridge bridge_A2-1615017473.1: switching from native_rtp technology to simple_bridge\n", '\000' <repeats 334 times>
        end_buf = 0x7f613c06ec50 "a2"
        start_read = 0x7f613c06ea50 "module load res_odbc.so"
        res = 1
        fds = {{fd = 37, events = 1, revents = 1}, {fd = 38, events = 1, revents = 0}}
        __PRETTY_FUNCTION__ = "netconsole"
#31 0x00000000005be46e in dummy_start (data=0x7f61d0000950) at utils.c:1249
        __cancel_buf = {__cancel_jmp_buf = {{__cancel_jmp_buf = {0, 3032146111578610150, 0, 512000, 4, 140055595644672, 3032146111536667110, -3110262857899730458}, __mask_was_saved = 0}}, __pad = {0x7f613c06edb0, 0x0, 0x0, 0x0}}
        __cancel_routine = 0x453ff5 <ast_unregister_thread>
        __cancel_arg = 0x7f613c06f700
        __not_first_call = 0
        ret = 0x0
        a = {start_routine = 0x45660a <netconsole>, data = 0x913f40 <consoles>, name = 0x7f61d0000970 "netconsole", ' ' <repeats 11 times>, "started at [ 1530] asterisk.c listener()"}
        __PRETTY_FUNCTION__ = "dummy_start"
#32 0x00007f61f43a1dd5 in start_thread () from /usr/lib64/libpthread.so.0
No symbol table info available.
#33 0x00007f61f3441ead in clone () from /usr/lib64/libc.so.6
No symbol table info available.

Thanks for your help

#8

It’s solved !!

I’m leaving here the solution.

The problem was the mysql-odbc-connector. The version really installed was the mysql-connector-odbc-5.2.7 one.
We uninstall that one and install mysql-connector-odbc-8.0.23-1.el7.x86_64.rpm with:

rpm -Uvh mysql-connector-odbc-8.0.23-1.el7.x86_64.rpm

Then the res_odbc.so was loaded properly.

Thanks for your feedback !!

#9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.