CPU utilization on a busy traffic shaping router

robel · 11-09-2008, 12:54 PM

I am the administrator of a Linux router (Slackware 11.0) which also does traffic shaping. The hardware is a Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz

When the traffic load is around 200Mbps (pps ~ 30000) and approx 2000 users, top looks like this:

Code:

top - 19:36:25 up 1 day,  3:46,  1 user,  load average: 0.04, 0.06, 0.01
Tasks:  84 total,   1 running,  83 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.0%us,  0.0%sy,  0.0%ni, 44.0%id,  0.0%wa,  0.5%hi, 55.5%si,  0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.0%us,  0.0%sy,  0.0%ni, 58.9%id,  0.0%wa,  0.6%hi, 40.4%si,  0.0%st
Cpu3  :  0.0%us,  0.0%sy,  0.0%ni, 58.2%id,  0.0%wa,  0.8%hi, 41.0%si,  0.0%st
Mem:   1033688k total,   216284k used,   817404k free,    67156k buffers
Swap:   979956k total,        0k used,   979956k free,    67608k cached

I think this is kinda strange. Why is one CPU (core) almost idle while the other three is pretty busy? Is it possible to do load balancing better?

The shaper is htb. Kernel is 2.6.19.2. iproute2-ss060323

I could probably do better if I upgrade to 12.1 and the latest tools, but upgrading this router is not something I like to do. Any hints?

By the way: I have a spare router and I will upgrade the software and OS soon. Do I have to take some actions in order to load balance better after upgrading? The short uptime is due to a hardware upgrade yesterday. Before shutdown the uptime was 374 days. I replaced a Intel(R) Core(TM)2 CPU 6700 @ 2.66GHz with the new quad core. Oh, and after the hardware upgrade I got an angry message from the BIOS: "Could not load CPU uCode". Guess I have to upgrade the BISO as well.

chort · 11-10-2008, 01:12 AM

Do you perhaps have only 3 network interfaces in the machine? All the CPU time is spent on interrupts, which would be the NICs alerting the kernel that there's network traffic to handle. If you only have three NICs, perhaps each one is vectored to a specific CPU (just a guess)?

robel · 11-10-2008, 03:26 AM

Quote:

Originally Posted by chort

Do you perhaps have only 3 network interfaces in the machine?

In fact, yes. But... Only two is in use. eth0 and eth2 is in use, eth1 is not in use (no link, no cable connected).

But I think you're one the track. It's all about IRQ-balancing between the four cores. Think I read something about that some days ago. I will upgrade to Slackware 12.1 soon, and (by then) use kernel 2.6.28 which is in rc4 right now.

estabroo · 11-10-2008, 08:51 AM

Processor affinity could be keeping the work on the same cpus.

robel · 11-10-2008, 09:19 AM

Quote:

Originally Posted by estabroo

Processor affinity could be keeping the work on the same cpus.

Yes, that's true. Now, the router has a quad core single CPU and the load is spread among three cores. I would assume the four cores are affined...? Or is "Cpu1" different in some strange way? The load (mainly the shaping process) is on CPU 0,2 and 3. I find this weird, but I am not a CPU expert.

robel · 09-27-2010, 12:31 PM

Quote:

Originally Posted by robel

Yes, that's true. Now, the router has a quad core single CPU and the load is spread among three cores. I would assume the four cores are affined...? Or is "Cpu1" different in some strange way? The load (mainly the shaping process) is on CPU 0,2 and 3. I find this weird, but I am not a CPU expert.

Two years since this question. Somehow I managed to solve the problem, but... Now I have a new Slackware 13.1 and the same problem! I can't remember how I solved it, but obviously I did. Hmm... :-(

Code:

top - 19:29:47 up  2:45,  1 user,  load average: 0.00, 0.00, 0.00
Tasks:  96 total,   1 running,  95 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.0%us,  0.1%sy,  0.0%ni, 38.9%id,  0.0%wa,  0.0%hi, 61.0%si,  0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
Cpu2  :  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
Cpu3  :  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   1032660k total,   167044k used,   865616k free,    38616k buffers
Swap:  3903788k total,        0k used,  3903788k free,    30368k cached

Oh, and now the problem is even worse... All load on one core...

jefro · 09-27-2010, 03:47 PM

See if you can offload checksums to the nic's or get enterprise level nic's.

Seems like your drivers are not fully smp or maybe a bios setting like wrong smp version or wrong something with the kernel that thinks you only have three cores.

syg00 · 09-27-2010, 08:16 PM

irqbalance ?.

robel · 09-28-2010, 03:25 AM

Quote:

Originally Posted by jefro

See if you can offload checksums to the nic's or get enterprise level nic's.

Seems like your drivers are not fully smp or maybe a bios setting like wrong smp version or wrong something with the kernel that thinks you only have three cores.

I have three Intel NIC's:
82572EI Gigabit Ethernet Controller (Copper) (rev 06)
82540EM Gigabit Ethernet Controller (rev 02)
82541PI Gigabit Ethernet Controller (rev 05)

This router is a traffic shaper on eth2 (82572EI PCI Express), using HTB to shape outbond traffic based on soure IP.

I agree with syg00 that this is an irq-issue. I'm not running irqbalance on the old router (which balances si's perfect).

Two years ago I solved the problem, and now I can't remember how I did it... It was a pretty simple and obvious solution.

eth2 is on irq 45 but ifconfig claims it on irq 18. I'm not sure if this is a problem, probably not.

The driver (e1000e) is not a module in the new router. Could that be the problem?