CPU load too high after upgrade from 1.8 to 11.23.1

Asterisk
1

Hello

Feel free to shoot me down in case there is more data that i can provide.

I want to track/fix cause of high cpu load. As call on this server almost double in next 6 months

CPU load has been quite high since we upgraded system.

  • This is bare metal setup, no virtulization

  • 8 cores

  • 32 G Ram

  • handles 600 calls during peak time

  • Load average can go up to 50’s

  • noticing this ONLY since upgraded from Asterisk 1.8 to 11.23.1

  • Linux version - Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org)

  • Another machine using Asterisk 1.8 with similar load is fine.

top - 12:52:25 up 487 days, 15:12, 4 users, load average: 49.90, 28.53, 20.71
Tasks: 192 total, 1 running, 191 sleeping, 0 stopped, 0 zombie
%Cpu(s): 14.1 us, 11.0 sy, 0.1 ni, 72.0 id, 0.6 wa, 0.0 hi, 2.1 si, 0.0 st
KiB Mem: 32774572 total, 31901268 used, 873304 free, 233072 buffers
KiB Swap: 15624188 total, 27732 used, 15596456 free. 28175700 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
73940 asterisk 20 0 6520900 1.524g 19264 S 198.4 4.9 574:50.48 asterisk
84532 root 20 0 490524 234432 14120 S 2.0 0.7 538:14.15 captagent
73955 root 20 0 56912 11300 3040 S 1.7 0.0 13:57.47 mysql-proxy
25 root 20 0 0 0 0 S 1.0 0.0 73:40.51 ksoftirqd/3
3 root 20 0 0 0 0 S 0.3 0.0 1662:43 ksoftirqd/0
8 root 20 0 0 0 0 S 0.3 0.0 1115:01 rcu_sched
5674 root 20 0 23636 2828 2296 S 0.3 0.0 10:04.49 top

++++++++

top - 11:27:42 up 487 days, 13:47, 4 users, load average: 40.79, 34.43, 31.62
Tasks: 189 total, 1 running, 188 sleeping, 0 stopped, 0 zombie
%Cpu0 : 9.7 us, 8.7 sy, 0.0 ni, 75.0 id, 0.7 wa, 0.0 hi, 6.0 si, 0.0 st
%Cpu1 : 16.0 us, 14.3 sy, 12.9 ni, 54.7 id, 0.7 wa, 0.0 hi, 1.4 si, 0.0 st
%Cpu2 : 11.5 us, 9.8 sy, 0.3 ni, 75.7 id, 0.0 wa, 0.0 hi, 2.7 si, 0.0 st
%Cpu3 : 20.1 us, 15.0 sy, 0.0 ni, 55.1 id, 0.7 wa, 0.0 hi, 9.1 si, 0.0 st
%Cpu4 : 9.3 us, 10.3 sy, 0.0 ni, 80.1 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu5 : 21.3 us, 16.0 sy, 0.0 ni, 62.3 id, 0.3 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 7.1 us, 9.1 sy, 0.0 ni, 83.4 id, 0.3 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 : 18.1 us, 15.4 sy, 0.0 ni, 66.2 id, 0.3 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 32774572 total, 31925508 used, 849064 free, 232344 buffers
KiB Swap: 15624188 total, 26352 used, 15597836 free. 28303972 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
73940 asterisk 20 0 6463216 1.427g 19264 S 210.1 4.6 382:14.33 asterisk
73955 root 20 0 55400 9716 3040 S 1.7 0.0 12:11.70 mysql-proxy
84532 root 20 0 490524 233396 14120 S 1.7 0.7 536:42.82 captagent
25 root 20 0 0 0 0 S 1.0 0.0 73:02.67 ksoftirqd/3

/proc/interrupts

110: 1669403939 252693 0 0 0 0 0 0 PCI-MSI-edge eth0-tx-0
111: 4070210232 466049458 128053 0 0 0 0 0 PCI-MSI-edge eth0-rx-1
112: 3977477299 0 256234622 47225 0 0 0 0 PCI-MSI-edge eth0-rx-2
113: 2078000933 295945 0 1528295501 0 0 0 0 PCI-MSI-edge eth0-rx-3

2

  1. Both Asterisk versions you refer to are no longer supported. 1.8 is completely EOL; 11 is now in security fix mode. Even if someone were to propose some form of change to the software, it would not go into either branch.

  2. There is no guarantee of comparable performance between two major versions. While the Development team tries hard to maintain performance between major versions, there’s no guarantee that between one major version and the next that performance is going to be completely identical. We do try to maintain performance within a major version.

  3. In software - as in life - there is no free lunch. Features cost generally cost something - CPU, memory, etc. Most likely - although I’m completely guessing here - Asterisk 11, supporting wide band audio, is consuming more resources due to the format overhaul that went into that effort. Generally, manipulating audio got a bit more expensive due to how formats had to be reworked. At 600 calls, you’ll definitely see more resources being consumed than in 1.8.

If you’re interested in improving the performance of Asterisk, patches are always welcome against the supported major versions, although major performance improvements requiring architectural changes would only be allowed in master to avoid regressions.

1 Like
3

@mjordan

Thanks for your feed back. I understand your point.

What I am planning to grab here is to break down where resources are going “with in Asterisk”. At least then I have direction or point to compare.

  • is there break down on where Asterisk is consuming more resources ?
  • Is this in codec conversions ?
  • Is it in call setup
  • is it running out of IO ?
4

There’s no way to tell without profiling the system. Doing that typically involves using something like gprof or oprofile, which obviously isn’t something you’d want to do on a production system.