CentOS5 bridge: very high CPU usage

EnigmaPsi · 05-31-2007, 03:27 PM

Hello all, I'm wondering if any of you encountered the following problem.

Recently installed a Centos5 machine on an old Athlon (2000+ or so), and placed it on a network that constantly has about 70 Mbps Internet Traffic (an ISP). The machine had less than 10% of processor active, all the time (monitored it with cacti for a few weeks). All the unwanted services were disabled (boot-time around 10-15 seconds).

Well, the ISP wanted a stronger machine, in order to place a traffic shaper with thousands of tc rules. And I installed Centos5 on a Intel dual-dore machine. The boot-time here is about 10 seconds, and also the unwanted services are disabled.

Now this is very interesting: on the second (much better) machine, the CPU usage is always above 10%, without any traffic shaping or anything, and this gets 20% at 70 Mbps. With 1500 tc rules, the CPU gets 60%. An unseen value, until now, on the servers i've setted up.

I had Centos4 installed on this machine (it was a ROUTER before), it had a traffic shaper with 1500+ rules and *never* had a CPU time bigger than 15%.

Is the bridging code eating these resources? Did any of you encountered this?

Code:

Specs:
Kernel: 2.6.18-8.1.4.el5
Bridging: bridge-utils-1.1-2

Bridge configured strictly redhat-stylish (ifcfg-br0, ifcfg-eth1, ifcfg-eth2)

Module                  Size  Used by
sch_sfq                 9665  1578
cls_u32                11845  4
sch_htb                19777  2
bridge                 53341  0
video                  19269  0
button                 10705  0
asus_acpi              19289  0
ac                      9157  0
sg                     35933  0
r8169                  31561  0
e100                   36809  0
pcspkr                  7105  0
ide_cd                 40033  0
mii                     9409  1 e100
cdrom                  36705  1 ide_cd
dm_snapshot            20581  0
dm_zero                 6209  0
dm_mirror              29713  0
dm_mod                 56537  14 dm_snapshot,dm_zero,dm_mirror
ahci                   21573  4
libata                 96857  1 ahci
sd_mod                 22977  6
scsi_mod              130637  4 sg,ahci,libata,sd_mod
raid1                  25153  2
ext3                  123081  5
jbd                    56553  1 ext3

Runge_Kutta · 05-31-2007, 03:39 PM

type in "top" so we can see the processes that are taking up the cpu on both machines. Also, what network cards are in both machines?

EnigmaPsi · 05-31-2007, 04:20 PM

Code:

Tasks:  83 total,   1 running,  82 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.3%us,  0.0%sy,  0.0%ni, 79.7%id,  0.0%wa,  2.4%hi, 17.6%si,  0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni, 78.3%id,  0.0%wa,  2.3%hi, 19.4%si,  0.0%st
Mem:   1025728k total,   663824k used,   361904k free,   127568k buffers
Swap:  2097144k total,        0k used,  2097144k free,   395052k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  P SWAP COMMAND
    1 root      15   0  2036  644  552 S    0  0.1   0:00.55 1 1392 init
    2 root      RT   0     0    0    0 S    0  0.0   0:00.00 0    0 migration/0
    3 root      34  19     0    0    0 S    0  0.0   0:00.65 0    0 ksoftirqd/0
    4 root      RT   0     0    0    0 S    0  0.0   0:00.00 0    0 watchdog/0
    5 root      RT   0     0    0    0 S    0  0.0   0:00.00 1    0 migration/1
    6 root      35  19     0    0    0 S    0  0.0   0:00.73 1    0 ksoftirqd/1
    7 root      RT   0     0    0    0 S    0  0.0   0:00.00 1    0 watchdog/1 
    ...some more

Well, i certainly issue "top", and did not find anything. I mean, no process that eats CPU. All of them (top, ps aux) show 0.0 CPU activity, for all the services. CPU is occupied by kernel, that is certain.

The NICs are Realtek RTL-8169 GigaBit (rev 10), as can be seen from lsmod output, in my previous post.

syg00 · 06-01-2007, 07:49 AM

I presume Centos comes with sysstat.
I'd be trawling through sar to find out what all that interrupt usage is.

EnigmaPsi · 06-02-2007, 09:35 AM

Thanks, syg00, this is the first time I hear that "si" means "System Interrupts". I could not find this in Top manpage and thought it means System usage or something like it.

I was wondering...how can I use sar command to monitor system interrupts? I have the irqbalance service started, might it have something to do with this?

syg00 · 06-02-2007, 08:44 PM

Yeah, you're right - sar probably won't help much. Time for some basic problem determination.
I'd be suspicious that the code (bridging whatever) isn't multiprocessor compliant. Try turning off the extra processor (in the BIOS preferably, or in the kernel itself), and see if the impact goes away.
Either way, you're going to have to find the culprit - start turning things off and see if you can isolate the effect. I was going to suggest vmstat to track it, but I'm now thinking that interrupt count is only the hardware interrupts. Maybe you'll be best running top in batchmode and pipe it to a file so you can correlate any change/effect.

JohnE1 · 06-03-2007, 02:05 PM

htop may be more useful than top. htop shows processes in real time AND it is scrollable on the screen. That scrolling feature alone makes it much more useful than top and it's a very small utility. Google htop or go search sourceforge.net, I'm sure it's there.

EnigmaPsi · 06-14-2007, 08:35 AM

Well, I have investigated further, and found an interrupts rate, shown by /proc/interrupt, to be very high: over a bilion (!). Here is the output:

Code:

[root@bridge rc.d]# cat /proc/interrupts
           CPU0       CPU1
  0:   40854905   40831410    IO-APIC-edge  timer
  1:          8          0    IO-APIC-edge  i8042
  7:          0          0    IO-APIC-edge  parport0
  8:          3          0    IO-APIC-edge  rtc
  9:          0          0   IO-APIC-level  acpi
 14:         25          0    IO-APIC-edge  ide0
 66:      67524      86608         PCI-MSI  libata
 74:  709610016          0   IO-APIC-level  eth1
 82:    2097293    7988505   IO-APIC-level  eth0
185:  698714869          0   IO-APIC-level  eth2
NMI:          0          0
LOC:   79372411   80880545
ERR:          0
MIS:          0

Interesting that all these concentrate on the first CPU (bridge has eth1 and eth2 attached, eth0 is a standard configured interface).

Does anyone know if this is a bug in the bridge code, or is anything I cand do to change this interrupts level?

Thanks