Asterisk 22.10.1 / bundled PJSIP 2.17 TCP transport stops responding with ioqueue_epol Assert failed

Asterisk Asterisk SIP
1

Hi,

I am seeing a regression after upgrading from Asterisk 22.5.2 to 22.9/22.10.1.

Environment:

Asterisk/derived build: 22.10.1

PJPROJECT bundled: 2.17

Previous working version: 22.5.2

Previous PJPROJECT bundled: 2.15.1

Transport affected: PJSIP TCP and WSS

The system has many TCP/WSS endpoints. After upgrading, SIP TCP registrations work for a few minutes, then the TCP transport stops responding. Packets are visible with sngrep/tcpdump, but no SIP packet appears in the Asterisk PJSIP logger and no response is sent.

The log repeatedly shows:

pjproject: ../src/pj/ioqueue_epol Assert failed: !pj_list_empty(&ioqueue->free_list)

There are also many messages like:

res_pjsip/pjsip_transport_management.c: Shutting down transport ‘TCP to x.x.x.x:port’ since no request was received in 6 seconds

res_pjsip/pjsip_transport_management.c: Shutting down transport ‘WSS to x.x.x.x:port’ since no request was received in 6 seconds

res_http_websocket.c: Error reading from web socket: Success

iostream.c: SSL_shutdown() failed … Broken pipe

I confirmed this is not an OS file descriptor limit:

Max open files: 300000

lsof -p $(pidof asterisk): around 1900-2200

ss -tanpn | grep asterisk: around 1000-1400

I also tested increasing PJPROJECT limits:

#define PJ_IOQUEUE_MAX_HANDLES 20000

#define PJSIP_TCP_TRANSPORT_BACKLOG 4096

#define PJSIP_TLS_TRANSPORT_BACKLOG 4096

The listen backlog changed correctly:

LISTEN 0 4096 0.0.0.0:7048 users:((“asterisk”…))

but the issue still occurred on 22.10.1.

Rolling back to Asterisk 22.5.2 with the same configuration files immediately fixes the issue. With 22.5.2, the same traffic pattern and even more active TCP sockets do not trigger the ioqueue_epol assertion.

I compared the source trees and noticed that 22.10.1 uses bundled PJPROJECT 2.17 while 22.5.2 uses 2.15.1. The suspicious change seems to be in PJPROJECT ioqueue behavior. In 2.17, PJ_IOQUEUE_CALLBACK_NO_LOCK defaults to 1, and the comment says this may introduce a race between key unregistration and read/write callbacks. There is also new ioqueue_drain_pending_writes() logic during unregister.

This looks relevant because the issue is triggered by many TCP/WSS connections opening and closing quickly.

Questions:

Is this a known regression in PJPROJECT 2.17 with epoll and TCP/WSS transport churn?

Is it safe/recommended for Asterisk to build bundled PJPROJECT 2.17 with:

#define PJ_IOQUEUE_CALLBACK_NO_LOCK 0

#define PJSIP_SAFE_MODULE 1

Are there any known patches or workarounds for ioqueue_epol Assert failed: !pj_list_empty(&ioqueue->free_list) in Asterisk 22.9/22.10.1?

For now I have rolled back to 22.5.2 because it is stable under the same load.

Thanks.

2

We’ve had no reports of issues and did do some testing of TCP and TLS before release. The bundled config_site.h is how it is expected to be built. As well, WSS would not be causing this - as that doesn’t use any of said functionality.

If you can provide specific details of the usage pattern to reproduce it, then please file an issue[1].

[1] Issues · asterisk/asterisk · GitHub

3

Chiming in since I’ve been down this road.

That assert just means pjproject ran out of free ioqueue keys. The keys cycle free → active → closing (held ~500ms) → free, so the free list only empties if you genuinely have that many sockets open, or closed keys aren’t making it back to the free list.

Your own numbers point straight at the second one. You bumped MAX_HANDLES to 20000 but you’re only sitting at ~2000 fds and ~1000-1400 live sockets. If the assert still fires that far under the limit, keys are leaking between closing and free, not maxing out. That’s the thing to hammer on, otherwise it reads like a capacity complaint and gets waved off.

Also worth checking how your pjproject was actually built. The stock Asterisk bundled build sets NDEBUG, which compiles that assert out completely, so on a normal bundled build you’d never see “Assert failed” at all, it’d just quietly fail the registration. The fact you’re seeing it makes me think you’re not on the bundled build (distro package? custom compile?). Could be why nobody else has hit it quite this way.

On testing your theory, don’t flip PJ_IOQUEUE_CALLBACK_NO_LOCK and the epoll settings at the same time or you won’t know which one fixed it. The bigger change in 2.16/2.17 was the epoll refactor, so I’d start there: force plain epoll (no EXCLUSIVE/ONESHOT) via PJ_IOQUEUE_DEFAULT_EPOLL_FLAGS and see if the leak stops. If it doesn’t, then try PJ_IOQUEUE_CALLBACK_NO_LOCK 0. And put your kernel version in the report, EPOLLEXCLUSIVE behaves differently across kernels.

I’d also drop the WSS stuff from this one. WebSockets go through res_http_websocket, not the PJSIP TCP transport, so it’s a separate path at this point. I think keeping this pure PJSIP TCP troubleshooting at first is the best way forward.

Last thing: if you can throw together a small SIPp scenario that opens and closes a bunch of TCP registrations fast, and show the free list draining while live fds stay flat, that plus your config_site.h, kernel, and pjsip show version is a solid bug report.

4

Update: I found the actual limiting point.

The PJSIP endpoint ioqueue is created with PJSIP_MAX_TRANSPORTS, not directly with PJ_IOQUEUE_MAX_HANDLES:

pj_ioqueue_create(endpt->pool, PJSIP_MAX_TRANSPORTS, &endpt->ioqueue);

In my build, raising only PJ_IOQUEUE_MAX_HANDLES was not enough. The failure kept happening around 1025 Asterisk TCP sockets. After explicitly defining both:

PJ_IOQUEUE_MAX_HANDLES=20000
PJSIP_MAX_TRANSPORTS=20000

the system passed the previous failure point and has been stable. Current state is 1270 Asterisk TCP sockets, 1265 ESTAB, 1732 open FDs, and no ioqueue_epoll free_list assert.

So this was not caused by ARI, WSS, backlog alone, EPOLLEXCLUSIVE, or PJ_IOQUEUE_CALLBACK_NO_LOCK. The effective PJSIP transport/ioqueue limit was still around 1024 until PJSIP_MAX_TRANSPORTS was raised explicitly.