IRQ affinity in SMP servers

an upgrade of a multi-TDM Asterisk (1xAFT200+1xAFT101) server at 5am today prompted this :

after installing the AFT101 i decided to check the interrupts using cat /proc/interrupts and was more than a bit surprised to see that in a Dual Xeon (4 CPUs) apart from a few timer interrupts to CPU0, everything else seemed to be generating interrupts on CPU3 only.

the wanpipe interfaces and the timer generate the most interrupts, so is there anything to be gained (or lost ) from spreading these out. i googled and found a how-to : bcr2.uwaterloo.ca/~brecht/server … finity.txt

on the box in question, the smp-affinity on the IRQ’s for the interfaces i’m interested in are all 15 (ffffffff) so i would have expected them to be fairly evenly spread out.

anyone with any insight ? i only have a single processor box here to play with and i don’t want to break a client’s box until after COB :smiley:

edit : should have googled some more … asteriskguru has a page that’s pretty good. asteriskguru.com/tutorials/p … noise.html

need to go check IRQ balancing in the kernel config !

yeah, we have the same problem with minimal installs of fedora - the irqbalance daemon doesn’t get installed, so for some reason, the IRQs all interrupt on the highest numbered CPU.

here is what i use at startup to adjust the interrupts (this is using APIC, so the IRQ numbers are expanded):

### move digium card affinity to CPU0 echo 1 > /proc/irq/233/smp_affinity

that will force whatever is on IRQ 223 to interrupt on CPU0 only.

echoing a 2 will use CPU1, a 4=CPU2, an 8=CPU3. there is also a way to force interrupts on multiple CPUs (so your card could interrupt on 0 and 1, while everything else interrupted on 2 and 3) but i haven’t gotten it mastered yet…to my understanding, you would sum the values from above for what you want to interrupt on…

so CPU0 and CPU1 would be 1 and 4 = 1+4 = 5.

or you could do a yum install irqbalance and let that handle your interrupting - i didn’t notice a bit of difference when interrupting on a single CPU. what would probably be more beneficial would be to have your NIC have it’s own IRQ, as that would provide uber-low latency for your network traffic…

anyways, hope this helps.

thanks wes.

i thought everything was going so well today, new E1 installed, the telco called me at home and we made it live in < 5 mins, great quality, no echo, very pleased.

then at 16:30 i get a phone call to say that everything has stopped. no 'net, no phones … scream over there to find the 24-port PoE switch is providing PoE but not switching … dead in the water. and you think i can find one locally ? most places don’t even know what PoE is !

when i get a chance i’ll see about playing with the affinities, although i read somewhere that irqbalance wasn’t needed on 2.6 kernels (hence me needing to revisit make menuconfig)

let me know what you find…even though it didn’t seem to make any difference, it’s nice to know HOW to do it if the need should arise…