How rebalance irq to all NUMA nodes
Jianlin-lv opened this issue · comments
In ubuntu, I found IRQ be rebalance to the cpus on the same node0, node1 is free and not loading.
irqbalance version: 1.6.0
OS: Ubuntu 20.04.4 LTS
lscpu |grep -i node
NUMA node(s): 2
NUMA node0 CPU(s): 0-19,40-59
NUMA node1 CPU(s): 20-39,60-79
# lshw -c network -businfo
Bus info Device Class Description
=============================================================
pci@0000:60:00.0 eth0 network Ethernet Connection X722 for 10GbE SFP+
pci@0000:60:00.1 eth1 network Ethernet Connection X722 for 10GbE SFP+
# cat /sys/class/net/eth0/device/numa_node
0
cat /proc/interrupts |grep -i eth0 |grep 122:
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23 CPU24 CPU25 CPU26 CPU27 CPU28 CPU29 CPU30 CPU31 CPU32 CPU33 CPU34 CPU35 CPU36 CPU37 CPU38 CPU39 CPU40 CPU41 CPU42 CPU43 CPU44 CPU45 CPU46 CPU47 CPU48 CPU49 CPU50 CPU51 CPU52 CPU53 CPU54 CPU55 CPU56 CPU57 CPU58 CPU59 CPU60 CPU61 CPU62 CPU63 CPU64 CPU65 CPU66 CPU67 CPU68 CPU69 CPU70 CPU71 CPU72 CPU73 CPU74 CPU75 CPU76 CPU77 CPU78 CPU79
122: 8185965 1741959 8684240 1932606 5203567 1539261 1611056 1400491 5143847 1602230 3616648 1743565 1858552 256816 2327054 4894293 2524546 8104170 2361994 4441960 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 37579860 8071362 2738317 45420420 1843680 8782652 1225839 1491943 1594332 2820421 2790455 11429964 1863750 1557335 2174569 2266504 2241046 6238079 2755234 3310145 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 323 0 0 0 0 IR-PCI-MSI 50331724-edge i40e-eth0-TxRx-75
but in Centos, irq can be mapping to all cpus in system.
bash-4.2# rpm -qf /sbin/irqbalance
irqbalance-1.0.7-11.el7.x86_64
CentOS Linux release 7.6.1810 (Core)
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23 CPU24 CPU25 CPU26 CPU27 CPU28 CPU29 CPU30 CPU31 CPU32 CPU33 CPU34 CPU35 CPU36 CPU37 CPU38 CPU39 CPU40 CPU41 CPU42 CPU43 CPU44 CPU45 CPU46 CPU47
33: 218627757 172356836 103007440 160780761 101933035 184787094 219885867 149650893 149321741 176588524 141278320 161084081 230241370 173881068 266418864 250606000 316421803 371311144 270139257 232641899 253326757 160960114 234300530 283026081 214489328 98440963 149915713 109589063 155623400 106427641 203805993 294270923 217743264 219312258 205627433 115205059 251429289 232995613 197807673 378423567 239863582 138385094 262305883 269859660 263040757 318029820 306930987 346309968 IR-PCI-MSI 31981569-edge i40e-eth0-TxRx-0
Is this a difference between versions?
Is there an option to change the behavior of irqbalance on ubuntu so that irq is mapped to all cpus?
Becasue irqbalance will get pci devices and irqs' numa node from /sys/bus/pci/devices/0000:60:00.0/numa_node or /proc/irq//node in the high version. According to this, irqbalance will rebalance irqs to the specified NUMA.
You can add parameters "--policyscript=policyscript.sh", and echo numa_node=-1 in policyscript.sh. For example:
if [ $2 --eq 122 ] then echo "numa_node=-1" fi
and
https://github.com/Irqbalance/irqbalance/blob/master/misc/irqbalance.policy.d/follow-affinity-hint.example
@liuchao173 "--policyscript" worked for me. Thanks!