Bug: asterisk didn't cleaning up AMI connection properly

Asterisk Asterisk APIs
1

Due to an overload of the system, I observed the new warning in Asterisk CLI:

WARNING[664169][C-0000010e]: taskprocessor.c:1225 taskprocessor_push: The 'stasis/pool-control' task processor queue reached 500 scheduled tasks again.

after my analysis, I found out there are some inconsistencies.

When I used the command :

core show taskprocessors

I saw multiple manager core connections are popped out on the CLI. So stasis/pool-control it hits max depth 3800

But to check the number of subscriptions in stasis using the command

 stasis show topic manager:core

I saw the output having

Name: manager:core
Detail:
Subscribers count: 319
Forwarding topic count: 9
Duration time: 156:39:03

then, to check the connected AMI, I hit the command:

manager show connected

but none of the manager connections show. Then how will the subscription not be cleaned up? if there are no connections, it should be cleaned right. A little knowledge will be helpful

2

Connections don’t individually subscribe. Manager subscribes as a whole when it loads and it is enabled. Even with no connections there will still be subscriptions and the topic will still exist.

Building with DEVMODE enabled would add statistics CLI commands, which show more information. In this case “stasis statistics show topic manager:core” would include a list of subscriptions.

3

As in reality, I can’t do that because the asterisk is in production, but something weird I observed is that there are two managers during the core show task processors in the site

stasis/p:manager:core-00000203 ,stasis/m:manager:core-00000006

Like I saw stasis/p:manager-core - 30 with a different last number and stasis/m:manager-core in only one, but in my test environment, I just witnessed only

stasis/m:manager:core-00000006

What’s the difference between these two i just confused. I get a larger number of stasis/p:manager:core- but stasis/m:manager:core- only one

Sorry for the inconvenience I caused. This slows down the call process in this warning.

4

They are subscriptions from something. A prefix “m” means mailbox, that is it has a dedicated thread for handling messages. A prefix of “p” means pooled, which means it uses the stasis threadpool for handling messages.

5

Oh, thanks for that. Also, is there any way to add stasis/p :manager: core like this

stasis/p:manager:core-000001e1 157525 0 189 450 500
stasis/p:manager:core-000001e2 157477 0 189 450 500 
stasis/p:manager:core-000001e3 157466 0 189 450 500 
stasis/p:manager:core-000001e4 157455 0 189 450 500 
stasis/p:manager:core-000001e5 157343 0 189 450 500

any reason why it’s happening, or any way to replicate this to make manager: core in the pool thread in core show taskprocessors

6

I’m not sure what you’re asking. I don’t know where they are coming from, or what is subscribing. You have no control over how subscriptions occur.

The CLI command I mentioned before in dev mode would potentially include the source information of what is subscribing.

7

I mean, I want to know when the mailbox that is a dedicated thread is used for handling messages, and when the thread pool is used for handling messages

8

If it starts with “stasis/p:” the subscription is using the threadpool. If it starts with “stasis/m:” it is a dedicated thread.

9

so the subscription takes place randomly within threadpool or dedicated thread ?

10

It’s not random. It’s up to the developer who implemented the code that does the subscription. Each has pros/cons, and depending on the usage one can be a better fit than the other.

11

What version of Asterisk is this?

12

Asterisk 22.3.0

13

Okay, after analyzing the Asterisk repo code, I found out the manager.c subscribing to the mailbox dedicated thread during manager subscription init instead of the thread pool, but in production, manager: core is subscribing to the thread pool, where I am seeing prefix “p” there, as shown in the above. so somehow the manager.c core subscribing to both the thread pool and the mailbox. which stasis/pool-control overloaded

14

There can be multiple subscriptions, which you’ve shown. There IS a a mailbox based subscription which was likely the manager.c subscription itself. There are also OTHER subscriptions from somewhere else using pool. This does not mean that manager.c itself is subscribing using the pool, other things can. I don’t know what those things are.

15

I saw many manager:core in pooled-based subscriptions in core show taskprocessors. So manager:core can be subscribed to by others using the pool-based subscriptions instead of manager.c?

managercore-copy
managercore-copy926×390 264 KB

16

It can.

17

I do actually have a PR up[1] which improves the performance of stasis, and eliminates “stasis/pool-control” which would lessen the impact, though wouldn’t solve the fact the subscriptions exist in the first place. It’ll be released in ~1.5-2 months.

[1] taskpool: Add taskpool API, switch Stasis to using it. by jcolp · Pull Request #1339 · asterisk/asterisk · GitHub

18

Oh great, Thanks for letting me know. Eliminating the “stasis/pool-control” will give much more clarity for this warning message

19

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.