Suddenly Extensions registration drop

Asterisk Asterisk SIP
1

Hi Team,

We observed that all SIP extensions suddenly dropped and became unavailable at the same time. The Asterisk server itself appears to be running, and port 5060 is listening correctly, but endpoint registrations were lost unexpectedly.

Could you please investigate the possible cause, including:

  • Asterisk service restart/crash

  • Network or firewall issues

  • SIP transport/NAT problems

  • Realtime database or PJSIP configuration issues

  • Any recent changes on the server

This issue affected multiple extensions simultaneously, so it does not appear to be related to individual user devices.

Please check the Asterisk logs and let us know the root cause.

Also Throw this error :
[2026-06-02 12:12:34] ERROR[3880931]: format_cap.c:368 ast_format_cap_update_by_allow_disallow: Excessive refcount 100000 reached on ao2 object 0x598b98ca4b38
[2026-06-02 12:12:34] ERROR[3880931]: format_cap.c:368 ast_format_cap_update_by_allow_disallow: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x598b98ca4b38 (0)
[2026-06-02 12:12:34] ERROR[3880931]: Got 18 backtrace records

Thanks.

2

You know this is a free forum for an open-source project, and there’s no one to “please investigate”, “please check” and “let [you] know the root cause”, right?

3

get these “Got 18 backtrace records”

will hep to debug

4

On Tuesday 02 June 2026 at 18:18:41, darsh720 wrote:

Hi Team,

Could you please investigate the possible cause, including:

This is not a commercial support service.

We are volunteers who know about Asterisk and are willing to help people who
have problems with Asterisk, and who provide sufficient information about those
problems to enable us either to reproduce them or at least to understand what
happened on the affected system at the time the problem occurred.

Please provide sufficient information, and someone may be able to help you.

Given what you have said so far about the problem, I suspect a network packet
capture of SIP registration attempts and responses would be a good start, but
please do not restrict yourself only to that.

Anything which you think could be helpful might be helpful…

Also, please let us know things like:

  • which version of Asterisk?
  • which Linux system is it running on?
  • what type/s of telephones are you using?
  • how many are affected?
  • how often does this happen?
  • are the telephones on a local network (no intervening routers) to the
    Asterisk server or are they remote (in which case, give us an overview of the
    network configuration)?

Antony.

–
“Remember: the S in IoT stands for Security.”

  • Jan-Piet Mens
5

You’re right, and I apologize for the wording.

I’m looking for guidance from the community to help identify the cause of the issue, not expecting anyone to provide formal support.

The problem I observed is that all SIP extensions suddenly became unavailable at the same time, while Asterisk was still running. I’m gathering logs, packet captures, and additional diagnostic information to better understand what happened.

I appreciate any suggestions on what information would be most useful to provide for troubleshooting.

Thank you.

6

grep -i “backtrace” /var/log/asterisk/full
grep: /var/log/asterisk/full: No such file or directory

7

Thank you for the clarification.

Here are some additional details:

  • Asterisk Version: 22.8.2

  • OS: Ubuntu 24.04

  • Phones: SIP softphones and desk phones

  • Number of affected extensions: All registered extensions

  • Network: Extensions are remote and register over the Internet

What happened:

  • All extensions were registered and working normally.

  • Suddenly all endpoints became unavailable.

  • Asterisk service was still running.

  • SIP port 5060 was listening correctly.

  • pjsip show contacts continued to show contacts, but pjsip show endpoints returned no endpoints after the issue occurred.

  • Some log messages included “write() failed: Broken pipe”.

Thank you.

8

[2026-06-02 11:53:35] ERROR[3870485]: format_cap.c:279 ast_format_cap_append_from_cap: Excessive refcount 100000 reached on ao2 object 0x598b98ca4b38 [2026-06-02 11:53:35] ERROR[3870485]: format_cap.c:279 ast_format_cap_append_from_cap: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x598b98ca4b38 (0)

Due to these ERROR possible to all extensions registration drop suddenly ???

and pjsip show endpoints is empty where as pjsip show contacts show all Avail status

9

@darsh720 Please do not open new threads for the same thing. I’ve merged your new one into here.

10

There is no assertion on that line in version 22.8.2! This output is inconsistent with the claimed version unless format_cache_get is a macro.

11

[2026-06-03 12:04:21] ERROR[89424] format_cap.c: Excessive refcount 100000 reached on ao2 object 0x598b98ba15d8 [2026-06-03 12:04:21] ERROR[89424] format_cap.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x598b98ba15d8 (0) [2026-06-03 12:04:21] ERROR[89424] : Got 18 backtrace records # 0: /usr/sbin/asterisk(__ao2_ref+0x2df) [0x598b949ca56f] # 1: /usr/sbin/asterisk(+0x75716) [0x598b949cd716] # 2: /usr/sbin/asterisk(__ao2_find+0x2a) [0x598b949cd85a] # 3: /usr/sbin/asterisk(__ast_format_cache_get+0x3b) [0x598b94a65e1b] # 4: /usr/sbin/asterisk(ast_format_cap_update_by_allow_disallow+0x2ce) [0x598b94a67bee] # 5: /usr/sbin/asterisk(aco_process_var+0x6a) [0x598b94a3cc9a] # 6: /usr/sbin/asterisk(ast_sorcery_objectset_apply+0x9d) [0x598b94afc43d] # 7: /usr/lib/asterisk/modules/res_sorcery_realtime.so(+0x2e6a) [0x715ed90a6e6a] # 8: /usr/lib/asterisk/modules/res_sorcery_realtime.so(+0x3391) [0x715ed90a7391] # 9: /usr/sbin/asterisk(ast_sorcery_retrieve_by_regex+0x104) [0x598b94afcfe4] #10: /usr/lib/asterisk/modules/res_pjsip.so(+0x32ad4) [0x715eca0a9ad4] #11: /usr/lib/asterisk/modules/res_pjsip.so(ast_sip_cli_traverse_objects+0x49e) [0x715eca0a887e] #12: /usr/sbin/asterisk(ast_cli_command_full+0x1f9) [0x598b94a36d39] #13: /usr/sbin/asterisk(ast_cli_command_multiple_full+0x7f) [0x598b94a3705f] #14: /usr/sbin/asterisk(+0x6ea7c) [0x598b949c6a7c] #15: /usr/sbin/asterisk(+0x1def55) [0x598b94b36f55] #16: /lib/x86_64-linux-gnu/libc.so.6(+0x9caa4) [0x715edd29caa4] #17: /lib/x86_64-linux-gnu/libc.so.6(+0x129c6c) [0x715edd329c6c] [2026-06-03 12:04:21] ERROR[89424] format_cap.c: Excessive refcount 100000 reached on ao2 object 0x598b98bc0c28 [2026-06-03 12:04:21] ERROR[89424] format_cap.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x598b98bc0c28 (0) [2026-06-03 12:04:21] ERROR[89424] : Got 18 backtrace records # 0: /usr/sbin/asterisk(__ao2_ref+0x2df) [0x598b949ca56f] # 1: /usr/sbin/asterisk(+0x75716) [0x598b949cd716] # 2: /usr/sbin/asterisk(__ao2_find+0x2a) [0x598b949cd85a] # 3: /usr/sbin/asterisk(__ast_format_cache_get+0x3b) [0x598b94a65e1b] # 4: /usr/sbin/asterisk(ast_format_cap_update_by_allow_disallow+0x2ce) [0x598b94a67bee] # 5: /usr/sbin/asterisk(aco_process_var+0x6a) [0x598b94a3cc9a] # 6: /usr/sbin/asterisk(ast_sorcery_objectset_apply+0x9d) [0x598b94afc43d] # 7: /usr/lib/asterisk/modules/res_sorcery_realtime.so(+0x2e6a) [0x715ed90a6e6a] # 8: /usr/lib/asterisk/modules/res_sorcery_realtime.so(+0x3391) [0x715ed90a7391] # 9: /usr/sbin/asterisk(ast_sorcery_retrieve_by_regex+0x104) [0x598b94afcfe4] #10: /usr/lib/asterisk/modules/res_pjsip.so(+0x32ad4) [0x715eca0a9ad4] #11: /usr/lib/asterisk/modules/res_pjsip.so(ast_sip_cli_traverse_objects+0x49e) [0x715eca0a887e] #12: /usr/sbin/asterisk(ast_cli_command_full+0x1f9) [0x598b94a36d39] #13: /usr/sbin/asterisk(ast_cli_command_multiple_full+0x7f) [0x598b94a3705f] #14: /usr/sbin/asterisk(+0x6ea7c) [0x598b949c6a7c] #15: /usr/sbin/asterisk(+0x1def55) [0x598b94b36f55] #16: /lib/x86_64-linux-gnu/libc.so.6(+0x9caa4) [0x715edd29caa4] #17: /lib/x86_64-linux-gnu/libc.so.6(+0x129c6c) [0x715edd329c6c]

What is the error ? Can explain something about this

We are running:
asterisk 22.8.2
Ubuntu 24.04

When these errors are come so all extensions registrations are drop so what is solution to up it again ? and what is the reason behind that ??

Help me out for this ?

12

@netygeek1 @darsh720 You are both pasting the same thing. If you are the same person or work for the same company, don’t keep creating new threads. Use this same one. Also if you’re the same person, don’t create multiple accounts.

13

You either have a resource leak, in which case it is strange as to why others haven’t reported it (are you using third party code, or one of the APIs?) or you have an exceptionally large number of some entity in your system which needs to hold a reference to a media format. I’m not sure exactly what would hold such references, but without going through the code, I guess that active channels and endpoint definitions might do so.

I’m assuming that the round number of 10,000 means it has changed that it is exactly one over the limit, so this isn’t the result of random memory corruption.

Does this start happening at approximately the same amount of time after Asterisk has been restarted?

14

After Restart Asterisk this thing happens about after 16 hours this problem arise again

15

We not use any third party code or api just laravel method to generate dialplan file in /etc/asterisk/

16

You’re using realtime. Is it caching? How many endpoints are there? How many calls per second and concurrent calls?

17

Here right now on 3563 total extensions and online extensions are 1885 extensions are register well.

18

If i upgrade asterisk version so will possible solve my problem ?

19

possible ? for sure worth the try !

20

You were asked what type of telephones were in use, and I have not seen a reply to that yet.

Also, have you even tried googling for an answer at all?

From co-pile-o-p:

"In Asterisk, a refcount (reference count) is a counter used to track how many parts of the system are currently holding a reference to an object.
Asterisk uses the astobj2 (ao2) API for reference-counted objects.

If the refcount is wrong, you can get two main classes of bugs:

  • Refcount too high → Object never freed (memory leak).

  • Refcount too low → Object freed too early, leading to bad magic number errors or crashes.

1. How Refcounting Works in Asterisk

  • Every ao2 object starts with a refcount of 1 when created.

  • Functions like ao2_ref(obj, +1) increase the count, and ao2_ref(obj, -1) decrease it.

  • When the count reaches 0, the object’s destructor is called and memory is freed.

  • If code keeps a reference without releasing it, the count never reaches 0 → leak.

  • If code releases too many times, the object is freed while still in use → crash.

2. Debugging Refcount Issues

Asterisk has Reference Count Debugging mode 1:

Enabling in Asterisk 14+

  1. Edit /etc/asterisk/asterisk.conf and set:
    Ini

    Copy code

    refdebug=yes

  2. Restart Asterisk.

  3. Reproduce the problem.

  4. Stop Asterisk gracefully:
    Bash

    Copy code

    asterisk -rx "core stop gracefully"

  5. Process the log:
    Bash

    Copy code

    /var/lib/asterisk/scripts/refcounter.py -f /var/log/asterisk/refs -n > /tmp/refs.txt

  6. Review /tmp/refs.txt or attach it to a bug report.

Note: The -n flag suppresses normal lifetimes so you only see leaks, invalid refs, or skewed lifetimes.

3. Example of a Raw Refcount Log

From /var/log/asterisk/refs:

Copy code

0x7f9dbc002048,+1,19256,chan_sip.c,8651,sip_alloc,**constructor**,allocate a dialog(pvt) struct
0x7f9dbc002048,-1,19256,chan_sip.c,28226,handle_request_do,4,throw away dialog ptr

  • +1 / -1 → refcount change

  • constructor / destructor → object creation/destruction

  • File, line, and function show where the change happened.

4. Common Problems

  • Autodestruct warnings: A channel still has a reference preventing cleanup.

  • Bad magic number: Accessing an object after it’s freed.

  • FRACK! Excessive refcount: Refcount reached absurd values (e.g., 100,000) — often due to loops or repeated ao2_ref() without matching ao2_unref() 2.

Example FRACK error:

Copy code

ERROR: Excessive refcount 100000 reached on ao2 object 0x563b78882c30
FRACK!, Failed assertion Excessive refcount 100000 reached

5. Tips for Developers

  • Always pair ao2_ref(obj, +1) with a matching ao2_ref(obj, -1).

  • Use refdebug mode in development to catch leaks early.

  • Avoid holding references longer than necessary.

  • When debugging, search the refs log for the object address to see its entire lifetime.

If you want, I can give you a step-by-step guide to trace a specific leaked or over-referenced object in Asterisk using refcounter.py so you can pinpoint the exact function causing the problem.

Do you want me to prepare that tracing workflow?"

I’m not going to bother verifying if any of that is even correct since it’s an AI spew - but - if even half of it is correct, you can run the commands to log this. According to AI spew increasing refcounts mean a memory leak.

In My Humble Opinion anyone running that many calls and extensions needs to be doing on a version of Asterisk they have compiled themselves, NOT an apt install thingie. If you are doing this - then it would be very simple to add some extra logging code into asterisk in a few key areas to monitor refcounts - and if you see them increasing forever - you have a leak. Then you try finding this with valgrind.

There’s not a lot of asterisk systems with as many extensions as yours out there so a memory leak that you have crop up every 16 hours would likely take months to show on most other systems. In addition, you have said nothing about the underlying network infrastructure. Is your system behind a firewall/address translator or some such that is mucking with SIP? Are your extensions behind something that is mucking with SIP?

In general your going to get more help here if you try some rudimentary things to fix it.