Full title should’ve been “CentOS 6.4 running Asterisk 11.5 with high load, low CPU/memory/disk IO”
Hello everyone,
I have an Asterisk server doing automated calls and I’m noticing an unexplained high load in it. The server is only running Asterisk. Database and other support applications run in a different machine.
What can be causing this high load?
If the load gets too high for too long (over the maximum of 8) there is a drop in calls and general unresponsiveness.
The hardware is an 8 core “Intel Xeon E3-1230 V2 @ 3.30GHz” with 16GB of RAM
I read the other posts about similar problems so I’ll post all the information requested in them.
Following are the results of the monitoring tools I have been using. They system is currently handling about 200 channels.
It doesn’t scale linearly, at 400 channels the load gets to 8 and things go downhill from there.
“ps auxf” shows nothing in D state.
[code]top - 10:50:28 up 7 days, 23:20, 3 users, load average: 2.16, 2.44, 1.93
Tasks: 341 total, 1 running, 340 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.3%us, 5.9%sy, 0.0%ni, 89.4%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 16303732k total, 7179980k used, 9123752k free, 264836k buffers
Swap: 8224760k total, 0k used, 8224760k free, 5759716k cached
2512 root 20 0 4744m 173m 46m S 50.8 1.1 396:55.01 asterisk[/code]
A typical iostat -x. sda and sdb are in raid 1 and sdc is an ssd storing some sound files used very often.
[code]avg-cpu: %user %nice %system %iowait %steal %idle
4.30 0.00 5.94 0.38 0.00 89.38
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 6.00 0.00 9.00 0.00 112.00 12.44 0.04 4.33 3.67 3.30
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 7.00 0.00 8.00 0.00 112.00 14.00 0.04 5.25 4.62 3.70
md127 0.00 0.00 0.00 15.00 0.00 112.00 7.47 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 14.00 0.00 112.00 8.00 0.05 3.50 2.64 3.70
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00[/code]
cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
0: 127 0 0 0 0 0 0 0 IO-APIC-edge timer
1: 2 0 0 0 0 0 0 0 IO-APIC-edge i8042
3: 2 0 0 0 0 0 0 0 IO-APIC-edge
4: 2 0 0 0 0 0 0 0 IO-APIC-edge
8: 1 0 0 0 0 0 0 0 IO-APIC-edge rtc0
9: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi acpi
10: 2 0 0 0 0 0 0 0 IO-APIC-edge
12: 4 0 0 0 0 0 0 0 IO-APIC-edge i8042
16: 59 0 0 0 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb1
23: 97 0 0 0 0 0 0 40 IO-APIC-fasteoi ehci_hcd:usb2
24: 2298 0 0 0 0 0 0 0 HPET_MSI-edge hpet2
25: 0 0 0 0 0 0 0 0 HPET_MSI-edge hpet3
26: 0 0 0 0 0 0 0 0 HPET_MSI-edge hpet4
27: 0 0 0 0 0 0 0 0 HPET_MSI-edge hpet5
28: 0 0 0 0 0 0 0 0 HPET_MSI-edge hpet6
29: 412442 0 0 36 1175680 0 0 208 PCI-MSI-edge ahci
30: 74 335176384 0 0 0 0 0 0 PCI-MSI-edge eth0
31: 5 10 344792 0 0 0 0 0 PCI-MSI-edge eth1-rx-0
32: 0 0 0 0 0 0 0 0 PCI-MSI-edge eth1-tx-0
33: 3 0 0 0 0 0 0 0 PCI-MSI-edge eth1
NMI: 7784 14329 4689 5198 7033 7387 6069 6332 Non-maskable interrupts
LOC: 46833697 44931615 30462128 37088906 47922986 44201942 27616867 37813275 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 7784 14329 4689 5198 7033 7387 6069 6332 Performance monitoring interrupts
IWI: 0 0 0 0 0 0 0 0 IRQ work interrupts
RES: 897464 372249 589429 570768 646428 605601 478042 484381 Rescheduling interrupts
CAL: 92 292 281 289 267 292 288 291 Function call interrupts
TLB: 206630086 265955778 173872460 156528749 143771724 221909392 129664286 115580760 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 2300 2300 2300 2300 2300 2300 2300 2300 Machine check polls
ERR: 0
MIS: 0
vmstat 1 20
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 9103556 265224 5776272 0 0 1 1 2 8 0 1 99 0 0
0 0 0 9103356 265224 5776268 0 0 0 24 101813 25156 5 9 86 0 0
1 0 0 9103224 265224 5776324 0 0 0 2 139955 25205 5 12 83 0 0
3 0 0 9103968 265224 5776280 0 0 0 54 102550 24442 5 9 85 1 0
2 0 0 9103084 265224 5776304 0 0 0 0 84384 22729 3 7 90 0 0
2 0 0 9102116 265224 5776412 0 0 0 0 84072 24705 6 8 87 0 0
1 0 0 9103432 265224 5776328 0 0 0 0 108438 24144 5 9 86 0 0
0 0 0 9102924 265224 5776340 0 0 0 2 41961 23168 3 5 92 0 0
0 0 0 9102608 265224 5776364 0 0 0 90 76298 26135 5 7 87 1 0
2 0 0 9078068 265224 5776444 0 0 0 0 83315 24891 5 8 87 0 0
0 0 0 9103344 265224 5776436 0 0 0 0 67256 26539 6 7 87 0 0
1 0 0 9094300 265224 5776444 0 0 0 0 54944 24834 3 6 91 0 0
0 0 0 9103352 265224 5776460 0 0 0 2 92988 26388 5 9 86 0 0
2 0 0 9103592 265224 5776440 0 0 0 46 76231 27186 5 7 87 1 0
1 0 0 9103744 265224 5776500 0 0 0 0 67153 26006 5 7 88 0 0
3 0 0 9103056 265224 5776520 0 0 0 76 86165 26895 5 8 87 0 0
1 0 0 9094384 265224 5776568 0 0 0 84 59498 26179 4 6 90 0 0
1 0 0 9088632 265224 5776556 0 0 0 0 103184 27236 6 9 85 0 0
1 0 0 9102532 265224 5776608 0 0 0 40 94010 27663 6 9 85 0 0
1 0 0 9091052 265224 5776648 0 0 0 0 93813 26675 9 9 82 0 0
mpstat 1
11:05:18 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
11:05:19 AM all 8.22 0.00 11.88 0.00 0.00 0.63 0.00 0.00 79.27
11:05:20 AM all 3.05 0.00 4.96 0.00 0.00 0.38 0.00 0.00 91.60
11:05:21 AM all 5.64 0.00 7.27 0.63 0.00 0.38 0.00 0.00 86.09
11:05:22 AM all 5.44 0.00 6.96 0.00 0.00 0.25 0.00 0.00 87.34
11:05:23 AM all 3.76 0.00 7.14 0.00 0.00 0.25 0.00 0.00 88.85
11:05:24 AM all 4.80 0.00 9.86 0.00 0.00 0.51 0.00 0.00 84.83
11:05:25 AM all 3.80 0.00 5.58 0.00 0.00 0.38 0.00 0.00 90.24
11:05:26 AM all 6.58 0.00 7.72 0.51 0.00 0.51 0.00 0.00 84.68
11:05:27 AM all 6.67 0.00 8.43 0.00 0.00 0.50 0.00 0.00 84.40
11:05:28 AM all 4.32 0.00 5.97 0.00 0.00 0.25 0.00 0.00 89.45
11:05:29 AM all 5.04 0.00 7.06 0.00 0.00 0.50 0.00 0.00 87.39
11:05:30 AM all 3.93 0.00 6.34 0.13 0.00 0.51 0.00 0.00 89.10
11:05:31 AM all 4.07 0.00 5.60 0.38 0.00 0.25 0.00 0.00 89.69
11:05:32 AM all 7.08 0.00 9.48 0.00 0.00 0.51 0.00 0.00 82.93
11:05:33 AM all 4.19 0.00 8.51 0.00 0.00 0.51 0.00 0.00 86.79
11:05:34 AM all 2.67 0.00 4.45 0.00 0.00 0.25 0.00 0.00 92.63