LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 12-06-2011, 03:07 AM   #1
zooppoop
LQ Newbie
 
Registered: Aug 2005
Posts: 13

Rep: Reputation: 1
GigE Firewall/Router High Interrupts


I have a Dell 710 server using two 82575GB Intel Quad cards. I am using two interfaces on each card and using the bonding driver within linux to "hopefully" increase throughput. But I have run into a problem that around 180Mbits/s and about 103,000Packets/s the system is having problems Unfortunately I have not been about to be on the box during this time, but I have been on it when it is handling about 100Mbits/s It does this seemingly without issue. %si in Top shows at highest about 4% When I use vmstat to show interrupts though it is reporting 80,000 per interval. This seems pretty high I would think. Probably matching the Packets per second across the Device.

So it is my assumption that the box is just getting overloaded with interrupts and stops working properly. It was also my understanding that the igb driver in linux for the card already implemented the NAPI interface and that Polling was automatically enabled. I am wondering if either someone can point me to some more information on this, or if you have any Ideas what I might be able to do to resolve this issue? I read that the bonding module also had NAPI support enabled.

Other information this server is a dual - Quad core server 2.4Ghz
This server generaly is just routing traffic.
I have done some tweaks in proc fs eg:

net.core.optmem_max = 20480
# Increase number of incoming connections backlog
net.core.netdev_max_backlog = 4000
net.core.dev_weight = 64
# # Bump up default r/wmem to max
net.core.rmem_default = 262141
net.core.wmem_default = 262141
# # Bump up max r/wmem
net.core.rmem_max = 262141
net.core.wmem_max = 262141

Kernel messages for one of the interfaces:
Code:
[    5.404672] igb 0000:0d:00.0: Intel(R) Gigabit Ethernet Network Connection
[    5.404677] igb 0000:0d:00.0: eth6: (PCIe:2.5Gb/s:Width x4) 00:1b:21:14:34:54
[    5.404762] igb 0000:0d:00.0: eth6: PBA No: E34573-001
[    5.404765] igb 0000:0d:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[    5.404796] igb 0000:0d:00.1: PCI INT B -> GSI 50 (level, low) -> IRQ 50
[    5.404807] igb 0000:0d:00.1: setting latency timer to 64
[    5.405636] igb 0000:0d:00.1: irq 155 for MSI/MSI-X
[    5.405645] igb 0000:0d:00.1: irq 156 for MSI/MSI-X
[    5.405655] igb 0000:0d:00.1: irq 157 for MSI/MSI-X
[    5.405665] igb 0000:0d:00.1: irq 158 for MSI/MSI-X
[    5.405675] igb 0000:0d:00.1: irq 159 for MSI/MSI-X
[    5.405684] igb 0000:0d:00.1: irq 160 for MSI/MSI-X
[    5.405694] igb 0000:0d:00.1: irq 161 for MSI/MSI-X
[    5.405703] igb 0000:0d:00.1: irq 162 for MSI/MSI-X
[    5.405720] igb 0000:0d:00.1: irq 163 for MSI/MSI-X
vmstat just around the problem time
Code:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  0    628 1818196 520596 4522348    0    0     0     0 79053  709  2  3 95  0
 1  0    628 1817940 520596 4522392    0    0     0   316 70700  691  1  5 93  0
 0  0    628 1817576 520596 4522424    0    0     0     0 58692  622  0  3 96  0
 0  0    628 1817448 520596 4522444    0    0     0     0 66368  599  0  3 97  0
 0  0    628 1817588 520596 4522464    0    0     0     0 69631  604  0  2 98  0
 0  0    628 1817720 520596 4522488    0    0     0     4 70053  599  0  2 97  0
 0  0    628 1817600 520596 4522504    0    0     0   824 71409  630  0  1 99  0
 0  0    628 1817360 520596 4522524    0    0     0     0 71755  573  0  2 98  0
 0  0    628 1817380 520596 4522540    0    0     0   296 72128  678  0  1 99  0
 0  0    628 1817380 520596 4522564    0    0     0     0 72177  671  0  3 97  0
 0  0    628 1817636 520596 4522584    0    0     0     0 72491  737  0  4 96  0
 0  0    628 1817604 520596 4522620    0    0     0    32 72096  784  0  2 97  0
 0  0    628 1817744 520596 4522640    0    0     0     0 71995  550  0  3 97  0
/proc/interrupts
Code:
            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      CPU12      CPU13      CPU14      CPU15      
   0:       7194          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC-edge      timer
   1:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC-edge      i8042
   8:        103          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC-edge      rtc0
   9:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC-fasteoi   acpi
  12:          4          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC-edge      i8042
  17:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC-fasteoi   uhci_hcd:usb3
  18:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC-fasteoi   uhci_hcd:usb4
  19:        241          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC-fasteoi   ehci_hcd:usb1
  20:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC-fasteoi   uhci_hcd:usb6
  21:         42          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC-fasteoi   ehci_hcd:usb2, uhci_hcd:usb5
  23:        134          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC-fasteoi   ata_piix
  33:    2608103          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-IO-APIC-fasteoi   megasas
  72:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  DMAR_MSI-edge      dmar0
  73:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      PCIe PME
  74:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      PCIe PME
  75:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      PCIe PME
  76:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      PCIe PME
  77:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      PCIe PME
  78:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      PCIe PME
  79:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      PCIe PME
  84:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      ioat-msix
  85:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      ioat-msix
  86:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      ioat-msix
  87:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      ioat-msix
  88:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      ioat-msix
  89:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      ioat-msix
  90:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      ioat-msix
  91:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      ioat-msix
  92:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0
  93:  553265553          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-rx-0
  94: 2555272088          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-rx-1
  95: 1815362598          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-rx-2
  96: 3143009180          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-rx-3
  97: 1053073265          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-tx-0
  98: 1417931335          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-tx-1
  99:  996211862          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-tx-2
 100: 1069388305          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth0-tx-3
 101:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1
 102:  111956122          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-rx-0
 103:  538684647          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-rx-1
 104:   13874403          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-rx-2
 105:  127882302          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-rx-3
 106:   44266363          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-tx-0
 107:   36119100          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-tx-1
 108:  108671636          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-tx-2
 109:  197984544          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth1-tx-3
 146:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth6
 147:  592524988          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth6-rx-0
 148:  893886304          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth6-rx-1
 149:  746695293          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth6-rx-2
 150:  862866827          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth6-rx-3
 151: 1143351193          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth6-tx-0
 152: 1410125452          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth6-tx-1
 153: 1306880638          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth6-tx-2
 154: 2466459304          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth6-tx-3
 155:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth7
 156:  733099223          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth7-rx-0
 157: 1046509942          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth7-rx-1
 158:  470466786          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth7-rx-2
 159:  542359295          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth7-rx-3
 160:  158374623          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth7-tx-0
 161: 2175918532          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth7-tx-1
 162:  869227435          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth7-tx-2
 163: 1117879192          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0  IR-PCI-MSI-edge      eth7-tx-3
 NMI:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Non-maskable interrupts
 LOC: 1056030832   86087796  234514868   96735869   86043630   28210362   28846412  109973308   13700828   10983891    7920961    8784801   14741473   12492078   12649210    3273281   Local timer interrupts
 SPU:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Spurious interrupts
 PMI:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Performance monitoring interrupts
 IWI:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IRQ work interrupts
 RES:   20929896   10885186   42500387    8740187    9113669    1743275    2193457    1360138     612269     282409      85796      72974      89511     134537      57812      25426   Rescheduling interrupts
 CAL:      31208    1051252      70627     113451     186683     384620     171638    1203257     142888     608432     112412     647982     148703     671860     214656     160391   Function call interrupts
 TLB:     549746     114394      13502      10895       5400       4608       3228       3100       1791       2030       1207       1219        881        820        763        667   TLB shootdowns
 TRM:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Thermal event interrupts
 THR:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Threshold APIC interrupts
 MCE:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Machine check exceptions
 MCP:       7064       7064       7064       7064       7064       7064       7064       7064       7064       7064       7064       7064       7064       7064       7064       7064   Machine check polls
 ERR:          0
 MIS:          0
FYI after seeing the interrupts here I started the irqbalance process. I am not sure what changed because before when I changed the interrupts they were balanced. Will see if this makes the difference.

Update again:
While this seemed to make a little bit of a change there is still a very weird limit of about 116,000 Packets per second per Physical interface is what I am finding. I can provide Graphs and data for the graphs, but basically when one of the physical interfaces in the bond group reaches that many Packets per second it seems to be Maxed out. Should I be contacting Intel for this issue?

Below is some information I captured while the system was having problems. What I did was run vmstat watch interrupts while it was hitting about 135,000 the system was having problems I started a download to see if it would be limited. It was limited. I restarted irqbalance process which seemed to help the IRQ load on the system. after that my Download was faster. and the system was doing about 200,000 irqs from vmstat. What it looks like what happened is that the two interfaces from one of the Bonding groups was using the Same CPU for IRQs when I changed this it allowed the other to work. But the other interface was still limited to it's 116,000 Packets/s

Interesting Top information that made me play with irqbalance a bit.
Code:
Cpu(s):  3.2%us,  0.2%sy,  0.0%ni, 86.8%id,  0.0%wa,  0.0%hi,  9.7%si,  0.0%stMem:   8250044k total,  6222824k used,  2027220k free,   474688k buffers
Swap:  2097148k total,      264k used,  2096884k free,  4336972k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                         
 
    3 root      20   0     0    0    0 R   76  0.0 114:04.02 ksoftirqd/0
vmstat output
Code:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 1  0    264 2018016 474692 4344560    0    0     0     0 180402 1209  0  4 95  0
 1  0    264 2018296 474692 4344616    0    0     0     0 189968 1186  0  3 97  0
 0  0    264 2017896 474692 4344668    0    0     0     0 183079 1345  0  3 97  0
 0  0    264 2017892 474692 4344728    0    0     0    16 189246 1161  0  7 93  0
 1  0    264 2018076 474692 4344772    0    0     0     0 177470 1013  0  6 94  0
 0  0    264 2017972 474692 4344820    0    0     0     0 185231 1145  0  3 97  0
 1  0    264 2018040 474692 4344864    0    0     0    16 165925 1577  0  7 93  0
 1  0    264 2017820 474692 4344932    0    0     0     0 173961 1245  0  7 93  0
 1  0    264 2017812 474692 4344988    0    0     0    20 184350 1282  0  3 97  0
 0  0    264 2018040 474692 4345048    0    0     0     0 177809 1185  0  3 97  0
 0  0    264 2017864 474692 4345108    0    0     0     0 181536 1217  0  3 97  0
 0  0    264 2017860 474692 4345164    0    0     0     0 181901 1143  0  4 96  0
 0  0    264 2017704 474692 4345216    0    0     0     0 170947 1233  0  9 90  0
 1  0    264 2017660 474692 4345272    0    0     0    20 173678 1087  0  6 94  0
 1  0    264 2016904 474692 4345928    0    0     0  1036 184045 1338  0  5 95  0
 1  0    264 2010440 474692 4351944    0    0     0     0 179177 1356  0  7 92  0
 1  0    264 2010300 474692 4352624    0    0     0   904 167765 1321  0  9 91  0
 1  0    264 2010460 474692 4352068    0    0     0   148 195913 1262  0  4 96  0
 0  0    264 2010464 474692 4352136    0    0     0     8 192341 1943  0  4 96  0
 0  0    264 2010412 474692 4352220    0    0     0     0 179244 1673  0  4 95  0
 0  0    264 2010516 474692 4352292    0    0     0     0 168384 1335  0  5 95  0
 0  0    264 2010020 474692 4352364    0    0     0    16 188664 1350  0  3 97  0
If you think the graphs and data from the interface and load on the server would be helpful let me know I will figure out how to get them available to you.



Let me know what other information I can provide here.

Thanks

Last edited by zooppoop; 12-14-2011 at 06:32 PM. Reason: Adding information, corrections
 
Old 04-01-2013, 03:27 AM   #2
robel
Member
 
Registered: Oct 2008
Location: Norway
Distribution: Slackware
Posts: 77

Rep: Reputation: 19
Did you solve this problem?
 
Old 09-13-2015, 09:27 PM   #3
zooppoop
LQ Newbie
 
Registered: Aug 2005
Posts: 13

Original Poster
Rep: Reputation: 1
No I was not able to find a solution. I found various documents talking about the problem, but nothing I could find to actually solve it. Was really just talking about how IRQs worked so normally they are there and the kernel etc has to get every one as it happens. vs Polling where the kernel etc would just come buy that device and see if it has anything. Polling is what most other PCI firewalls etc use is my understanding. I thought I had done what it would take to get linux to do it. I believe it was NETPOLL but either way the server or my settings just could not do it. I ended up buying a commercial firewall to do the job.

FYI I did get it working somewhat better starting irqBalance just that just gave me another 80k of breathing room before it had problems again.

Here was a good document:
http://zeniv.linux.org.uk/pub/linux/.../lca/paper.pdf

Thanks

Last edited by zooppoop; 09-13-2015 at 09:34 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] High hardware interrupts from timer and parport0 causing issues scheidel21 Linux - Hardware 5 07-25-2011 10:39 AM
Slow performance due to high rate of software interrupts in application bluehive Linux - General 0 05-14-2009 06:06 AM
Adding new Linux firewall/router on network with pre-existing gateway/router grittyminder Linux - Networking 4 08-13-2008 02:17 AM
router billion 5102 has firewall and software firewall tests aus9 Linux - Security 6 12-31-2006 10:09 PM
High interrupts dunkyb Linux - General 4 12-29-2003 08:04 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 06:48 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration