I have a Dell 710 server using two 82575GB Intel Quad cards. I am using two interfaces on each card and using the bonding driver within linux to "hopefully" increase throughput. But I have run into a problem that around 180Mbits/s and about 103,000Packets/s the system is having problems Unfortunately I have not been about to be on the box during this time, but I have been on it when it is handling about 100Mbits/s It does this seemingly without issue. %si in Top shows at highest about 4% When I use vmstat to show interrupts though it is reporting 80,000 per interval. This seems pretty high I would think. Probably matching the Packets per second across the Device.
So it is my assumption that the box is just getting overloaded with interrupts and stops working properly. It was also my understanding that the igb driver in linux for the card already implemented the NAPI interface and that Polling was automatically enabled. I am wondering if either someone can point me to some more information on this, or if you have any Ideas what I might be able to do to resolve this issue? I read that the bonding module also had NAPI support enabled.
Other information this server is a dual - Quad core server 2.4Ghz
This server generaly is just routing traffic.
I have done some tweaks in proc fs eg:
net.core.optmem_max = 20480
# Increase number of incoming connections backlog
net.core.netdev_max_backlog = 4000
net.core.dev_weight = 64
# # Bump up default r/wmem to max
net.core.rmem_default = 262141
net.core.wmem_default = 262141
# # Bump up max r/wmem
net.core.rmem_max = 262141
net.core.wmem_max = 262141
Kernel messages for one of the interfaces:
Code:
[ 5.404672] igb 0000:0d:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 5.404677] igb 0000:0d:00.0: eth6: (PCIe:2.5Gb/s:Width x4) 00:1b:21:14:34:54
[ 5.404762] igb 0000:0d:00.0: eth6: PBA No: E34573-001
[ 5.404765] igb 0000:0d:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[ 5.404796] igb 0000:0d:00.1: PCI INT B -> GSI 50 (level, low) -> IRQ 50
[ 5.404807] igb 0000:0d:00.1: setting latency timer to 64
[ 5.405636] igb 0000:0d:00.1: irq 155 for MSI/MSI-X
[ 5.405645] igb 0000:0d:00.1: irq 156 for MSI/MSI-X
[ 5.405655] igb 0000:0d:00.1: irq 157 for MSI/MSI-X
[ 5.405665] igb 0000:0d:00.1: irq 158 for MSI/MSI-X
[ 5.405675] igb 0000:0d:00.1: irq 159 for MSI/MSI-X
[ 5.405684] igb 0000:0d:00.1: irq 160 for MSI/MSI-X
[ 5.405694] igb 0000:0d:00.1: irq 161 for MSI/MSI-X
[ 5.405703] igb 0000:0d:00.1: irq 162 for MSI/MSI-X
[ 5.405720] igb 0000:0d:00.1: irq 163 for MSI/MSI-X
vmstat just around the problem time
Code:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 0 628 1818196 520596 4522348 0 0 0 0 79053 709 2 3 95 0
1 0 628 1817940 520596 4522392 0 0 0 316 70700 691 1 5 93 0
0 0 628 1817576 520596 4522424 0 0 0 0 58692 622 0 3 96 0
0 0 628 1817448 520596 4522444 0 0 0 0 66368 599 0 3 97 0
0 0 628 1817588 520596 4522464 0 0 0 0 69631 604 0 2 98 0
0 0 628 1817720 520596 4522488 0 0 0 4 70053 599 0 2 97 0
0 0 628 1817600 520596 4522504 0 0 0 824 71409 630 0 1 99 0
0 0 628 1817360 520596 4522524 0 0 0 0 71755 573 0 2 98 0
0 0 628 1817380 520596 4522540 0 0 0 296 72128 678 0 1 99 0
0 0 628 1817380 520596 4522564 0 0 0 0 72177 671 0 3 97 0
0 0 628 1817636 520596 4522584 0 0 0 0 72491 737 0 4 96 0
0 0 628 1817604 520596 4522620 0 0 0 32 72096 784 0 2 97 0
0 0 628 1817744 520596 4522640 0 0 0 0 71995 550 0 3 97 0
/proc/interrupts
Code:
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15
0: 7194 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge timer
1: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge i8042
8: 103 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge rtc0
9: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi acpi
12: 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge i8042
17: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi uhci_hcd:usb3
18: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi uhci_hcd:usb4
19: 241 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi ehci_hcd:usb1
20: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi uhci_hcd:usb6
21: 42 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb5
23: 134 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi ata_piix
33: 2608103 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi megasas
72: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar0
73: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
74: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
75: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
76: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
77: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
78: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
79: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME
84: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
85: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
86: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
87: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
88: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
89: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
90: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
91: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge ioat-msix
92: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0
93: 553265553 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-rx-0
94: 2555272088 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-rx-1
95: 1815362598 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-rx-2
96: 3143009180 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-rx-3
97: 1053073265 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-tx-0
98: 1417931335 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-tx-1
99: 996211862 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-tx-2
100: 1069388305 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-tx-3
101: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1
102: 111956122 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-rx-0
103: 538684647 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-rx-1
104: 13874403 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-rx-2
105: 127882302 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-rx-3
106: 44266363 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-tx-0
107: 36119100 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-tx-1
108: 108671636 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-tx-2
109: 197984544 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth1-tx-3
146: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6
147: 592524988 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6-rx-0
148: 893886304 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6-rx-1
149: 746695293 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6-rx-2
150: 862866827 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6-rx-3
151: 1143351193 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6-tx-0
152: 1410125452 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6-tx-1
153: 1306880638 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6-tx-2
154: 2466459304 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth6-tx-3
155: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7
156: 733099223 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7-rx-0
157: 1046509942 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7-rx-1
158: 470466786 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7-rx-2
159: 542359295 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7-rx-3
160: 158374623 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7-tx-0
161: 2175918532 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7-tx-1
162: 869227435 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7-tx-2
163: 1117879192 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth7-tx-3
NMI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Non-maskable interrupts
LOC: 1056030832 86087796 234514868 96735869 86043630 28210362 28846412 109973308 13700828 10983891 7920961 8784801 14741473 12492078 12649210 3273281 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IRQ work interrupts
RES: 20929896 10885186 42500387 8740187 9113669 1743275 2193457 1360138 612269 282409 85796 72974 89511 134537 57812 25426 Rescheduling interrupts
CAL: 31208 1051252 70627 113451 186683 384620 171638 1203257 142888 608432 112412 647982 148703 671860 214656 160391 Function call interrupts
TLB: 549746 114394 13502 10895 5400 4608 3228 3100 1791 2030 1207 1219 881 820 763 667 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 7064 7064 7064 7064 7064 7064 7064 7064 7064 7064 7064 7064 7064 7064 7064 7064 Machine check polls
ERR: 0
MIS: 0
FYI after seeing the interrupts here I started the irqbalance process. I am not sure what changed because before when I changed the interrupts they were balanced. Will see if this makes the difference.
Update again:
While this seemed to make a little bit of a change there is still a very weird limit of about 116,000 Packets per second per Physical interface is what I am finding. I can provide Graphs and data for the graphs, but basically when one of the physical interfaces in the bond group reaches that many Packets per second it seems to be Maxed out. Should I be contacting Intel for this issue?
Below is some information I captured while the system was having problems. What I did was run vmstat watch interrupts while it was hitting about 135,000 the system was having problems I started a download to see if it would be limited. It was limited. I restarted irqbalance process which seemed to help the IRQ load on the system. after that my Download was faster. and the system was doing about 200,000 irqs from vmstat. What it looks like what happened is that the two interfaces from one of the Bonding groups was using the Same CPU for IRQs when I changed this it allowed the other to work. But the other interface was still limited to it's 116,000 Packets/s
Interesting Top information that made me play with irqbalance a bit.
Code:
Cpu(s): 3.2%us, 0.2%sy, 0.0%ni, 86.8%id, 0.0%wa, 0.0%hi, 9.7%si, 0.0%stMem: 8250044k total, 6222824k used, 2027220k free, 474688k buffers
Swap: 2097148k total, 264k used, 2096884k free, 4336972k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3 root 20 0 0 0 0 R 76 0.0 114:04.02 ksoftirqd/0
vmstat output
Code:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
1 0 264 2018016 474692 4344560 0 0 0 0 180402 1209 0 4 95 0
1 0 264 2018296 474692 4344616 0 0 0 0 189968 1186 0 3 97 0
0 0 264 2017896 474692 4344668 0 0 0 0 183079 1345 0 3 97 0
0 0 264 2017892 474692 4344728 0 0 0 16 189246 1161 0 7 93 0
1 0 264 2018076 474692 4344772 0 0 0 0 177470 1013 0 6 94 0
0 0 264 2017972 474692 4344820 0 0 0 0 185231 1145 0 3 97 0
1 0 264 2018040 474692 4344864 0 0 0 16 165925 1577 0 7 93 0
1 0 264 2017820 474692 4344932 0 0 0 0 173961 1245 0 7 93 0
1 0 264 2017812 474692 4344988 0 0 0 20 184350 1282 0 3 97 0
0 0 264 2018040 474692 4345048 0 0 0 0 177809 1185 0 3 97 0
0 0 264 2017864 474692 4345108 0 0 0 0 181536 1217 0 3 97 0
0 0 264 2017860 474692 4345164 0 0 0 0 181901 1143 0 4 96 0
0 0 264 2017704 474692 4345216 0 0 0 0 170947 1233 0 9 90 0
1 0 264 2017660 474692 4345272 0 0 0 20 173678 1087 0 6 94 0
1 0 264 2016904 474692 4345928 0 0 0 1036 184045 1338 0 5 95 0
1 0 264 2010440 474692 4351944 0 0 0 0 179177 1356 0 7 92 0
1 0 264 2010300 474692 4352624 0 0 0 904 167765 1321 0 9 91 0
1 0 264 2010460 474692 4352068 0 0 0 148 195913 1262 0 4 96 0
0 0 264 2010464 474692 4352136 0 0 0 8 192341 1943 0 4 96 0
0 0 264 2010412 474692 4352220 0 0 0 0 179244 1673 0 4 95 0
0 0 264 2010516 474692 4352292 0 0 0 0 168384 1335 0 5 95 0
0 0 264 2010020 474692 4352364 0 0 0 16 188664 1350 0 3 97 0
If you think the graphs and data from the interface and load on the server would be helpful let me know I will figure out how to get them available to you.
Let me know what other information I can provide here.
Thanks
