Excessive refcount 100000 reached on ao2 object

[azharkhn]

I am working with Asterisk 16.9 version and at some point during peak call load, I have seen following errors logs in Asterisk and with sound quality degradation but asterisk doesn’t crash and I have no clue regarding this issue:

ERROR[4634][C-0000059c]: frame.c:345 ast_frdup: Excessive refcount 100000 reached on ao2 object 0x19fffb8
ERROR[4634][C-0000059c]: frame.c:345 ast_frdup: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x19fffb8 (0)
ERROR[8694][C-000008f4]: frame.c:345 ast_frdup: Excessive refcount 100000 reached on ao2 object 0x19fffb8
ERROR[8694][C-000008f4]: frame.c:345 ast_frdup: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x19fffb8 (0)
ERROR[11130][C-00000af8]: frame.c:345 ast_frdup: Excessive refcount 100000 reached on ao2 object 0x19fffb8
ERROR[11130][C-00000af8]: frame.c:345 ast_frdup: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x19fffb8 (0)
ERROR[2566][C-000003e4]: frame.c:345 ast_frdup: Excessive refcount 100000 reached on ao2 object 0x19fffb8
ERROR[2566][C-000003e4]: frame.c:345 ast_frdup: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x19fffb8 (0)
ERROR[31461][C-00000100]: frame.c:345 ast_frdup: Excessive refcount 100000 reached on ao2 object 0x19fffb8
ERROR[31461][C-00000100]: frame.c:345 ast_frdup: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x19fffb8 (0)
ERROR[3813][C-000004dc]: frame.c:345 ast_frdup: Excessive refcount 100000 reached on ao2 object 0x19fffb8
ERROR[3813][C-000004dc]: frame.c:345 ast_frdup: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x19fffb8 (0)
ERROR[13956][C-00000d38]: frame.c:345 ast_frdup: Excessive refcount 100000 reached on ao2 object 0x19fffb8
ERROR[13956][C-00000d38]: frame.c:345 ast_frdup: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x19fffb8 (0)
ERROR[10236][C-00000a3a]: frame.c:345 ast_frdup: Excessive refcount 100000 reached on ao2 object 0x19fffb8
ERROR[9339][C-0000097e]: frame.c:345 ast_frdup: Excessive refcount 100000 reached on ao2 object 0x19fffb8
ERROR[10236][C-00000a3a]: frame.c:345 ast_frdup: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x19fffb8 (0)
ERROR[9339][C-0000097e]: frame.c:345 ast_frdup: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x19fffb8 (0)
ERROR[3813][C-000004dc]:   Got 12 backtrace records
# 0: asterisk(__ao2_ref+0x240) [0x45c290]
# 1: asterisk(ast_frdup+0x1ed) [0x4e66cd]
# 2: asterisk(ast_bridge_channel_queue_frame+0x88) [0x47cea8]
# 3: asterisk(ast_bridge_queue_everyone_else+0x40) [0x47d190]
# 4: /usr/lib/asterisk/modules/bridge_simple.so(+0x1089) [0x7fab867b2089]
# 5: asterisk() [0x47c3a1]
# 6: asterisk() [0x48122e]
# 7: asterisk(bridge_channel_internal_join+0x360) [0x4825d0]
# 8: asterisk() [0x47012e]
# 9: asterisk() [0x5933cd]
#10: /lib64/libpthread.so.0(+0x7aa1) [0x7fabd0d54aa1]
#11: /lib64/libc.so.6(clone+0x6d) [0x7fabd0130c4d]

[jcolp]

The message itself means that something is getting used heavily, in this case it’s usually a format (such as ulaw) which is internally referenced a lot. The more load and usage, the higher the chance - so you’d need to specify what exactly “peak call load” means and usage patterns. It may be that you are pushing things too far.

[azharkhn]

@jcolp thanks for your reply!

Actually, I am stress testing Asterisk server with 16 vCPUs with 4GB RAM and 65536 open file limit. I am just using an IVR mostly with playback() and background() applications. This issue happens after around 2000 calls. So just wondering if it can be fixed and Asterisk will allow more load.

What do you suggest?

[jcolp]

You can increase the count in the source code before the message is output, but alone it is not a problem. If you are having any issues then they are elsewhere, and you would have to investigate/isolate.

[azharkhn]

@jcolp Do you have idea of any Max. capacity that single Asterisk node can handle for IVR or Queues Setups?
So far, we have increased below variable values in source code of Asterisk 16 for increasing the Asterisk node capacity:

# file main/astobj2.c
EXCESSIVE_REF_COUNT 900000
# file include/asterisk/taskprocessor.h 
AST_TASKPROCESSOR_HIGH_WATER_LEVEL 20000

[jcolp]

No, I don’t. Any number would be for your environment, on your system, with your usage patterns.