Bridging a bond (802.3ad) only works when stp is enabled?

OggVorbis2 · 07-21-2009, 05:06 AM

Hi,

For redundancy I've created a 802.3ad bond on a server (Debian Lenny). This server will be a host for several virtual (KVM) servers, so I created a bridge on the bond. The strange thing (that took me hours to figure out) is that this bridge only works when I enable spanning tree protocol (stp) on the bridge. Is that normal?
Normally I don't use bonds, and I create the bridge on the interface directly. I never have to enable stp when I do that.

This is the relevant part op my /etc/network/interfaces:

Code:

auto bond0
iface bond0 inet manual
        slaves eth0 eth4
        bond_mode 802.3ad

auto br0
iface br0 inet static
        address x.x.x.48
        netmask 255.255.255.128
        network x.x.x.0
        broadcast x.x.x.127
        gateway x.x.x.126
        bridge_ports bond0
        bridge_fd 9
        bridge_hello 2
        bridge_maxage 12
        bridge_stp on

When I use "bridge_stp on" like I normally do everything seems to work, but no data is sent or received at all.
I hope somebody knows if and why this is correct behaviour. I cannot find much information about this subject (bridging a bond), although it seems very relevant to me with so many people using KVM and Xen.

SteveK1979 · 07-22-2009, 08:40 AM

Hi,

I've not used the combination that you are referring to (bridged + bonded) but I think I can help from a networking perspective. I think that the reason you will need STP enabled is that if not, there are two paths to the network that you have bridged to, which from a networking perspective has the potential to introduce loops within your network. This is a Very Bad Thing. With spanning tree enabled, one of these paths will be blocked, therefor restoring sanity to the mac address tables.

If the active path fails, spanning tree should reconverge and the previously blocked link should become the active path. I would guess the only way around this is to configure bonding/aggregation or whatever your network vendor calls it to bond the switch ports together.

I don't think I've explained this two well, I feel I'm a bit rusty or got sucked into Cisco specifics these days - I can't decide which!

Hope that gives some indications.

Cheers,
Steve

OggVorbis2 · 07-23-2009, 01:54 AM

Thanks for your answer Steve. But your explanation doesn't really match my observations. I already configured 'trunking' (my vendor calls it that way) on my switch, this is necessary for 802.3ad to work. And also, I see that both links are used, like defined in 802.3ad. It's not that STP disables one of the two links.

Another problem I have now is that after a live migration of a KVM it takes a while (30 seconds or so) before the route to the new location is found. Without using bonds this is much much faster (<1 sec). KVM sends a 'gratuitous Arp' after a migration and the switch knows the new route immediately. With my current setup this apparently doesn't work, but I don't know why.

Third problem: I've read that it's also possible to attach two (or more) interfaces to a bond directly. I did that to test it:

Code:

auto br0
iface br0 inet static
        address x.x.x.48
        netmask 255.255.255.128
        network x.x.x.0
        broadcast x.x.x.127
        gateway x.x.x.126
        bridge_ports eth0 eth4
        bridge_fd 9
        bridge_hello 2
        bridge_maxage 12
        bridge_stp on

But when I activated this my WHOLE network became unreachable, and I had to get to the server and plug it off to get the network working again. (Yes we have power switches, but they were also unreachable due to the setup above!) Can somebody explain why this could happen? It's quite scary that one server can just put down a complete network (over several switches).

OggVorbis2 · 07-24-2009, 03:44 AM

I fixed the stp problem by removing the trunks from my switch, and opting for bond_mode balance-tlb (5) instead of 802.3ad. This works great with bridge_stp set to off.
The only problem that remains is that after a live migration the switch doesn't know about the new location, so the KVM becomes unreachable for some time (until the arp cache expires, I think).