I’ve been poking at manager performance for the last two months or so experimenting with things and have put up a pull request:
Which changes how event queueing in manager works.
For developers, or even users, who are curious how it worked it used a single list of events. Each session would know its current position in the list, wake up, move through the list (either sending or not sending), stop, rinse and repeat. Another thread would wake up every so often and clean up any events that were no longer needed.
This new change makes it so that each session has its own queue. Some nice effects is that it reduces thread contention, allows filtering to occur a lot earlier, reduces session wakeups, and simplifies how waking up sessions works.
I’ve done some preliminary artificial testing myself and it has reduced the manager taskprocessor max size by 30-40%. The reason I’m posting on here is to get anyone with heavy AMI usage to give it a try if they can and let me know if they see any difference in general, and for the manager taskprocessor as well.
I do have the ability to get rid of the taskprocessor entirely but I’d like to see the actual impact of this change first with the taskprocessor in use for others.
Does it mean one managers usage cannot be seen by another? That was the first thing I noticed and maybe that’s good but maybe it should have a master manager option that allows them see everything? However that would probably reduce the performance back to the way it was (everyone receives everything)?
what data points can i get ? asterisk -rx " core show taskprocessors"|grep manager stasis/m:manager:core-0000000e 1590891970 145 99302 2700 3000 0 1260378
Your post previously never had an “in queue” value. Max Depth is the maximum depth the queue ever got.
As for why, events are being published to the manager topic faster than they can be turned into AMI events and queued to applicable manager sessions. As long as the subscription uses a taskprocessor, that’s always going to be a possibility. The alternative is to not use a taskprocessor and instead block every thread when they publish - which I may end up doing in the end, as the tradeoff could be worth it but it’s too early.
There’s nothing inherently in it that would block for long periods of time, aside from the thread itself just being starved - which can happen. Being more efficient though normal operation would probably use less CPU, and when it does hit high water mark it would be able to clear it faster.
