Asterisk 22.8.2 hangs during ast_hangup() - AO2 channel container lock contention causes new calls to stop

Asterisk Asterisk Support
1

Asterisk Version : 22.8.2
OS : RHEL 9.7
Kernel : 5.14.0-611.54.1.el9_7.x86_64
Deployment : Podman Container
CPU : 256 cores
Workload : Incoming call platform
ARI : Used only by one internal application (myqueue.js)
PJSIP : Yes

Problem Summary

Asterisk hangs after only running 9 calls.

Observed symptoms:

  • Existing calls continue.
  • New calls eventually stop getting processed.
  • Only about 9 active calls remain.
  • Channels remain stuck in the h extension.
  • Restarting Asterisk immediately resolves the problem.

Example:



15 active channels
9 active calls
601 calls processed

Dialplan

The h extension contains an AGI, but the AGI is not running when the issue occurs.

exten => h,n,AGI(tff/hangup.agi)

Verified:

ps -ef | grep hangup.agi

returns no running AGI process.

Host Health

Historical sar data collected during the incident shows the host is healthy.

CPU:

User    : 3.41%
System  : 0.42%
Idle    : 96.16%
IOWait  : 0.00%

Run queue:

runq-sz : 10
blocked : 0

Disk:

await : 0.2 ms
%util : 15%

No evidence of CPU, storage, scheduler, or network bottlenecks.

GDB Findings

Hundreds of threads are blocked in:

pthread_mutex_lock()

↓

__ast_pthread_mutex_lock()

↓

__ao2_lock()

↓

internal_ao2_traverse()

↓

channelstorage_ao2_legacy.c

Example stack:



ast_hangup()
    ↓
ast_channel_unlink()
    ↓
delete_channel()
    ↓
__ao2_unlink()
    ↓
internal_ao2_traverse()
    ↓
__ao2_lock()

This occurs in PBX threads as well as bridge threads.

Additional Observations

Approximately 650 threads are blocked in:



pthread_mutex_lock()

Many threads are waiting inside:



get_by_name_prefix()

or

delete_channel()

which ultimately wait for the same AO2 channel container lock.

Bridge Threads

Several bridge worker threads are also blocked:



bridge_channel_ind_thread()

↓

ast_hangup()

↓

ast_channel_unlink()

ARI

ARI is enabled, but only one internal application (myqueue.js) uses it.

The Node.js application maintains approximately six persistent ARI connections.

There is no evidence of repeated reconnects or an ARI connection leak.

Observed Behaviour

After approximately 68 hours:

  • channels stop being destroyed
  • ast_hangup() never completes
  • CLI commands like:


channel request hangup

also block

  • new calls eventually stop processing

Request

Could this be a known issue related to:

  • AO2 channel container locking
  • channel unlink
  • bridge teardown
  • ast_hangup()
  • Asterisk 22.8.2

Are there fixes in later Asterisk 22.x releases related to this behavior?

Attachments:-

asterisk -rx “core show taskprocessors” |grep distributor
pjsip/distributor-00000023 7488 312 16 450 500 0 1992865
pjsip/distributor-00000024 8131 392 27 450 500 0 1956137
pjsip/distributor-00000025 7948 392 17 450 500 0 2049094
pjsip/distributor-00000026 7662 427 17 450 500 0 1985859
pjsip/distributor-00000027 8211 407 26 450 500 0 1949071
pjsip/distributor-00000028 8177 393 39 450 500 0 2196972
pjsip/distributor-00000029 7660 397 24 450 500 0 2216711
pjsip/distributor-0000002a 8015 328 16 450 500 0 2354207
pjsip/distributor-0000002b 8059 381 21 450 500 0 2022512
pjsip/distributor-0000002c 7809 425 21 450 500 0 1913229
pjsip/distributor-0000002d 7986 373 39 450 500 0 2060471
pjsip/distributor-0000002e 8107 399 21 450 500 0 1914665
pjsip/distributor-0000002f 8180 434 20 450 500 0 2066118
pjsip/distributor-00000030 8257 260 23 450 500 0 2044701
pjsip/distributor-00000031 7881 467 16 450 500 0 2106263
pjsip/distributor-00000032 7524 452 18 450 500 0 2142821
pjsip/distributor-00000033 7976 326 21 450 500 0 2063411
pjsip/distributor-00000034 7931 372 24 450 500 0 1925992
pjsip/distributor-00000035 8128 325 31 450 500 0 2366070
pjsip/distributor-00000036 7832 500 21 450 500 0 1972425
pjsip/distributor-00000037 7653 315 22 450 500 0 9808598
pjsip/distributor-00000038 7905 278 27 450 500 0 2226834
pjsip/distributor-00000039 7645 425 20 450 500 0 1957006
pjsip/distributor-0000003a 7836 322 26 450 500 0 2010606
pjsip/distributor-0000003b 7762 486 27 450 500 0 2007021
pjsip/distributor-0000003c 7937 405 18 450 500 0 2200482
pjsip/distributor-0000003d 7922 401 22 450 500 0 2138054
pjsip/distributor-0000003e 7896 298 21 450 500 0 5336361
pjsip/distributor-0000003f 7814 298 23 450 500 0 2097988
pjsip/distributor-00000040 7966 374 31 450 500 0 2109570
pjsip/distributor-00000041 7956 214 19 450 500 0 3190315

asterisk -rx “core show taskprocessors” |grep taskpool
taskpool/s:pjsip-00000013 140136 3 4 450 500 0 35343885
taskpool/s:pjsip-00000014 50203 3 3 450 500 0 16687928
taskpool/s:pjsip-00000015 9359 3 3 450 500 0 20850589
taskpool/s:pjsip-00000016 6380 3 4 450 500 0 20581982
taskpool/s:pjsip-00000017 5008 3 2 450 500 0 19192402
taskpool/s:pjsip-00000018 4042 3 3 450 500 0 20308419
taskpool/s:pjsip-00000019 2897 3 3 450 500 0 14693195
taskpool/s:pjsip-0000001a 1463 3 2 450 500 0 20520280
taskpool/s:pjsip-0000001b 570 2 2 450 500 0 9277592
taskpool/s:pjsip-0000001c 180 2 2 450 500 0 6867003
taskpool/s:sorcery-00000005 416 0 2 450 500 0 177
taskpool/s:stasis-00000000 14 0 2 450 500 0 655
taskpool/s:stasis-00000001 2 0 1 450 500 0 1759
taskpool/s:stasis-00000002 2 0 1 450 500 0 2627
taskpool/s:stasis-00000003 1 0 1 450 500 0 0
taskpool/s:stasis-00000004 1 0 1 450 500 0 1

asterisk -rx “core show taskprocessors” |grep manager
stasis/m:manager:core-0000000c 1059572 0 5514 2700 3000 0 32099

2

When reporting a deadlock you need to always provide the backtrace, otherwise we’re relying on your own analysis. In this case it’s never the container itself.

Yes, there have been fixes for deadlocks like this in recent releases. I would suggest upgrading, or you can look through the reported issues/changes yourself to try to isolate things that would fix it and backport them.

3

Hi ,

Can we use 22.10.* version to solve the problem.

4

try and report back

5

@benphone @jcolp we upgraded to asterisk 22.10.* but our problem not solved.

Than we change the Channel storage backend: ao2_legacy to cpp_map_name_id
now my asterisk stop hanging and working fine.
Kindly share your observation on this change this will help us in future or not so we can move changes to the production.

6

@benphone @jcolp we upgraded to asterisk 22.10.* but our problem not solved.

Than we change the Channel storage backend: ao2_legacy to cpp_map_name_id
now my asterisk stop hanging and working fine.
Kindly share your observation on this change this will help us in future or not so we can move changes to the production.

7

i use cpp_map_name_id did not go in to any bugs recently

but if it not solved on ao2_legacy , i would open up a issue on the tracker [1]

get 2 back traces one after the other

sudo /var/lib/asterisk/scripts/ast_coredumper --RUNNING --tarball-coredumps

see the docs

[1] Issues · asterisk/asterisk · GitHub

8

@benphone

Kindly find the coredump link.

9

To solve a deadlock, you need the identity of the thread that holds the lock not those of the ones waiting for it. There is (or at least was) a compile time option, thread debugging, that caused Asterisk to record this information, and added a CLI command to show all the held and awaited locks (although it doesn’t show cases where a conditional lock request is being looped on. The awaited locks are mainly of use where the deadlock is the result of circular chains of locks rather than waiting for external events, or simply forgetting to release a lock.

To catch loops on conditional locks as the second lock in the chain, you definitely need the backtrace information for all the threads.

10

This appears to be:

Which was solved in 22.10.0

You stated that upgrading did not work, but these backtraces were taken from 22.8.0 which would not have the change.