Hello Asterisk Community,
I’m encountering a recurring issue with Asterisk crashing in our production environment due to segmentation faults. We’re running Asterisk version 20.9.2 and FreePBX version:17.0.19.16 on a Debian server (AWS EC2 instance). The crashes occur randomly, and after reviewing the logs, I’ve noticed segmentation faults in the syslogs. Below are the relevant details from the syslogs:
2024-10-24T01:46:48.274010+00:00 ip-172-31-6-245 kernel: [71053.553732] show_signal_msg: 2 callbacks suppressed
2024-10-24T01:46:48.274356+00:00 ip-172-31-6-245 kernel: [71053.553736] asterisk[161524]: segfault at 7fc9003e689f ip 00007fc9003e689f sp 00007fc9003e5920 error 15 likely on CPU 0 (core 0, socket 0)
2024-10-24T01:46:48.274434+00:00 ip-172-31-6-245 kernel: [71053.559214] Code: 00 00 2c 00 00 00 00 00 2c 00 00 00 00 2c 00 00 00 2c 00 00 00 00 2c 00 00 00 2c 00 00 00 00 2c 00 00 00 00 2c 00 00 00 00 00 <2c> 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d
2024-10-24T03:58:34.937872+00:00 ip-172-31-6-245 kernel: [78960.160920] asterisk[195830]: segfault at ffffffffffffef38 ip 00007f4c5c8d2840 sp 00007f4c16c19920 error 5 in res_agi.so[7f4c5c8ca000+b000] likely on CPU 1 (core 0, socket 0)
2024-10-24T03:58:34.937890+00:00 ip-172-31-6-245 kernel: [78960.165829] Code: b0 ef ff ff 4d 89 e9 bb 03 00 00 00 f7 d1 c1 e9 1f e8 c4 dd ff ff c7 85 30 ef ff ff 00 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 <44> 8b bd 38 ef ff ff c7 85 90 ef ff ff ff ff ff ff 45 09 f7 0f 85
2024-10-24T03:58:35.016473+00:00 ip-172-31-6-245 mariadbd[615]: 2024-10-24 3:58:35 1942 [Warning] Aborted connection 1942 to db: ‘asterisk’ user: ‘root’ host: ‘localhost’ (Got an error reading communication packets)
2024-10-24T04:45:23.346337+00:00 ip-172-31-6-245 kernel: [81768.549478] asterisk[211517]: segfault at 7f5bfe9c48a9 ip 00007f5bfe9c48a9 sp 00007f5bfe9c3920 error 15 likely on CPU 1 (core 0, socket 0)
2024-10-24T04:45:23.346352+00:00 ip-172-31-6-245 kernel: [81768.556467] Code: 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 2d 30 30 3a 30 33 2d 2d 2d 2d 2c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <2c> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 2a 2a 2a
2024-10-24T04:45:23.384583+00:00 ip-172-31-6-245 mariadbd[615]: 2024-10-24 4:45:23 2366 [Warning] Aborted connection 2366 to db: ‘asterisk’ user: ‘root’ host: ‘localhost’ (Got an error reading communication packets)
2024-10-24T06:27:29.933943+00:00 ip-172-31-6-245 kernel: [87895.083966] asterisk[239618]: segfault at 0 ip 0000000000000000 sp 00007fe058bb9920 error 14 in asterisk[562d9cb84000+45000] likely on CPU 0 (core 0, socket 0)
2024-10-24T06:27:29.933958+00:00 ip-172-31-6-245 kernel: [87895.093148] Code: Unable to access opcode bytes at 0xffffffffffffffd6.
2024-10-24T06:27:29.955075+00:00 ip-172-31-6-245 mariadbd[615]: 2024-10-24 6:27:29 2862 [Warning] Aborted connection 2862 to db: ‘asterisk’ user: ‘root’ host: ‘localhost’ (Got an error reading communication packets)
2024-10-24T08:11:43.131711+00:00 ip-172-31-6-245 kernel: [94148.197681] asterisk[264613]: segfault at 0 ip 0000000000000000 sp 00007fc92547e920 error 14 in asterisk[564e41a65000+45000] likely on CPU 1 (core 0, socket 0)
2024-10-24T08:11:43.131725+00:00 ip-172-31-6-245 kernel: [94148.206781] Code: Unable to access opcode bytes at 0xffffffffffffffd6.
2024-10-24T08:11:43.135357+00:00 ip-172-31-6-245 mariadbd[615]: 2024-10-24 8:11:43 3364 [Warning] Aborted connection 3364 to db: ‘asterisk’ user: ‘root’ host: ‘localhost’ (Got an error reading communication packets)
2024-10-24T08:24:25.473153+00:00 ip-172-31-6-245 kernel: [94910.576035] asterisk[269933]: segfault at 0 ip 0000000000000000 sp 00007f79e4db4920 error 14 in asterisk[559162d5f000+45000] likely on CPU 0 (core 0, socket 0)
2024-10-24T08:24:25.473168+00:00 ip-172-31-6-245 kernel: [94910.585141] Code: Unable to access opcode bytes at 0xffffffffffffffd6.
2024-10-24T08:24:25.499542+00:00 ip-172-31-6-245 mariadbd[615]: 2024-10-24 8:24:25 4291 [Warning] Aborted connection 4291 to db: ‘asterisk’ user: ‘root’ host: ‘localhost’ (Got an error reading communication packets)
2024-10-24T09:19:28.866974+00:00 ip-172-31-6-245 kernel: [98213.949888] asterisk[285456]: segfault at 0 ip 00007f35bec95063 sp 00007f3573d5e8f0 error 4 in libc.so.6[7f35bec45000+155000] likely on CPU 1 (core 0, socket 0)
2024-10-24T09:19:28.872882+00:00 ip-172-31-6-245 kernel: [98213.958729] Code: c5 e9 b7 05 fb ff 90 41 55 41 54 55 53 48 83 ec 08 85 f6 0f 8e fe 00 00 00 49 89 fc 89 f3 83 fe 01 0f 84 00 01 00 00 48 89 d5 <8b> 12 89 d0 25 00 80 00 00 75 37 64 4c 8b 2c 25 10 00 00 00 48 8b
2024-10-24T09:19:28.934938+00:00 ip-172-31-6-245 mariadbd[615]: 2024-10-24 9:19:28 5118 [Warning] Aborted connection 5118 to db: ‘asterisk’ user: ‘root’ host: ‘localhost’ (Got an error reading communication packets)
these logs where written just before asterisk stopped working.
At the same time these logs are written in syslogs ,this is what is written in the asterisk full log.
[2024-10-24 01:46:47] WARNING[162069][C-00000437] ast_expr2.fl: ast_yyerror(): syntax error: syntax error, unexpected ‘=’, expecting $end; Input:
= 1
^
[2024-10-24 01:46:47] WARNING[162069][C-00000437] ast_expr2.fl: If you have questions, please refer to Overview - Asterisk Documentation
[2024-10-24 01:46:48] WARNING[161524][C-00000428] res_agi.c: Too many arguments, truncating
[2024-10-24 01:46:48] WARNING[161524][C-00000428] res_agi.c: Too many arguments, truncating
[2024-10-24 01:46:48] WARNING[161524][C-00000428] res_agi.c: Too many arguments, truncating
[2024-10-24 01:46:48] WARNING[161524][C-00000428] res_agi.c: Too many arguments, truncating
[2024-10-24 01:46:48] WARNING[161524][C-00000428] res_agi.c: Too many arguments, truncating
[2024-10-24 01:46:48] WARNING[161524][C-00000428] res_agi.c: Too many arguments, truncating
[2024-10-24 01:46:48] WARNING[161524][C-00000428] res_agi.c: Too many arguments, truncating
[2024-10-24 01:46:48] WARNING[161524][C-00000428] res_agi.c: Too many arguments, truncating
Has anyone encountered similar segmentation fault issues?
Any tips for debugging segmentation faults in Asterisk further, or specific modules we should focus on?
Any guidance or suggestions would be greatly appreciated!
Thank you for your time and support!
Best regards,