I suddenly run into network throughput problems on our servers. Servers acts as KVM hypervisors and uses bridges to create several local networks. Each bridge is connected to VLAN (inter-server network) which in turn is connected to two bounded NICs.

Network diagram is on picture (forum software somehow breaks ASCII images between CODE tags).

Bridges throughput (measured with iperf) fluctuates from 300 to 40 Mbits/s on hypervisor with highest load and from 600 to 400 Mbits/s on hypervisor with almost no load at all (2 idle virtual servers). It has negative impact on every application that use database (on separated virtual server). At first I thought that there was something wrong going on bond0 as everything basically depends on physical NICs. However it doesn't matter if bridge is connected to VLAN or disconnected. STP is enabled.

I guess there must be some theoretical maximum throughput based on cpu usage network traffic, etc. but given that hypervisors has 24 CPU and virtual servers aren't under significant load most of the time, I can't figure out what went wrong.

Few info about hypervisor:
Gentoo Linux with custom compiled kernel 3.7.5-hardened (I can provide kernel config if needed)
bridge-utils 1.4

Is there anything I can do to debug it more?

Attached Thumbnails

 

Looking at the diagram you don't need STP - try turning it off. I know we never use it on our XEN servers.

Thanks for suggestion, but it didn't change anything. I'll try newer kernel and report back.

If you are using ethernet bridging in linux and using bridge netfilter (aka firewalling on the bridge), expect EPIC LOSS of throughput. For example, if you are using 10/100 Mbit, expect MAX 30 MBit on 1400 byte frames. Even less on 128byte frames unless you have spent alot of time with in-driver/kernel performance optimizations.

PS. Testing network interface coalescence value tuning will help alot with the amount of interrupts and keeping the system responsive. However, the Netfilter bridge code is... it just needs alot of work

Do you mean ebtables? I use packet filtering with ipables with rules like:

By the way, bridge performance slightly increased with newer kernel (3.9.9-hardened) along with more stable throughput values, but it's still much less than I would expected.

Yes, ebtables is one part of bad performance. I assume you have all debug out of the kernel, branch prediction set, SLUG vs. Slab allocation set as well - if you haven't: don't expect a serious performance change.

One thing also is interrupt handling, we have been doing some profiling, but its not looking good for us. NIC drivers are also a large part of this (at least in Linux, can't say for your VM). Did you tweak ethtool settings etc...?

If I use iptables rules on bridges, does that mean I'm using ebtables indirectly? I really can't tell right now.

Well, actually I believe that main problem might be in linux kernel bonding, because many people told me that linux bonding really sucks. And as far as I know, it can create really weird problems, like almost unsable connection to certain HP disk array via iSCSI.

I haven't tweak NIC settings with ethtool or debug it extensively, because I don't have much experience with debugging network in Linux and only thing I can think of is tweaking TCP in linux kernel, which strikes me as last resort. Right now I'm trying open vSwitch instead (see http://www.linuxquestions.org/questi...an-4175499820/) as someone recommended it to me. Even for bonding.