[SOLVED] Virtual bridged network with too high load

HasC · 04-20-2010, 10:54 AM

Not sure if this issue belongs in "Virtualization" or "Networking". A little of both, I assume. And this will be a little bit long, so thanks for your patience :-)

I'm trying to setup a little virtual network, using 2 bridges (using bridge-utils), and 2 virtual machines (using QEMU/KVM), on Debian Lenny (no unstable-testing-backports-whatever which is not stable). With the "virtual-bridged-network.png" included graphic you can see my idea.

I created the virtual network with this script (as root):

Code:

#!/bin/bash

brctl addbr switch0
brctl addbr switch1

for interface in $(seq 100 103); do
	tunctl -t tap"$interface" -u 1000
	brctl addif switch0 tap"$interface"
	ifconfig tap"$interface" promisc up
done
for interface in $(seq 104 107); do
	tunctl -t tap"$interface" -u 1000
	brctl addif switch1 tap"$interfaz"
	ifconfig tap"$interface" promisc up
done

ifconfig switch0 up
ifconfig switch1 up

and created the virtual machines with this script (as my normal user):

Code:

#!/bin/bash

kvm -cdrom systemrescuecd.iso -m 256 \
	-net nic,macaddr=aa:bb:cc:00:11:01 \
	-net tap,ifname=tap100,script=no,downscript=no \
	-net nic,macaddr=aa:bb:cc:00:11:02 \
	-net tap,ifname=tap104,script=no,downscript=no &

kvm -cdrom systemrescuecd.iso -m 256 \
	-net nic,macaddr=aa:bb:cc:00:11:03 \
	-net tap,ifname=tap103,script=no,downscript=no \
	-net nic,macaddr=aa:bb:cc:00:11:04 \
	-net tap,ifname=tap107,script=no,downscript=no &

As you can read, I started two instances of an "System Rescue CD" ISO (v1.5.1 BTW), but you can use any distro you want.

Now, let's see the symptoms:
When the VMs finish booting and I get the welcome screen, I try to configure eth0 or eth1, on any guest.

Code:

ifconfig eth0 up

(Note I don't assign any IP address - But I can, the result will be the same)
Then, I get a bunch of syslog output on my Debian host

Code:

[ 4000.536746] tap104: received packet with  own address as source address
[ 4000.536746] tap104: received packet with  own address as source address
[ 4005.573240] __ratelimit: 109501 messages suppressed
[ 4005.573240] tap103: received packet with  own address as source address
[ 4010.588751] __ratelimit: 108864 messages suppressed
[ 4010.588754] tap103: received packet with  own address as source address
[ 4015.627566] __ratelimit: 111212 messages suppressed
[ 4015.627566] tap107: received packet with  own address as source address
[ 4020.682822] __ratelimit: 110245 messages suppressed
[ 4020.682825] tap107: received packet with  own address as source address

and the bandwith usage goes from 0 to an average of 20MBps, max 40MBps and min 5MBps (checked with

Code:

# bwm-ng

).

On the guests, I got too syslog output:

Code:

 eth0: IPv6 duplicate address fe80::a8bb:ccff:fe00:1103 detected!
[   74.900262] eth0: IPv6 duplicate address fe80::a8bb:ccff:fe00:1103 detected!
[   74.900265] eth0: IPv6 duplicate address fe80::a8bb:ccff:fe00:1103 detected!
[   74.900268] eth0: IPv6 duplicate address fe80::a8bb:ccff:fe00:1103 detected!
[   74.900271] eth0: IPv6 duplicate address fe80::a8bb:ccff:fe00:1103 detected!
[   74.900274] eth0: IPv6 duplicate address fe80::a8bb:ccff:fe00:1103 detected!
[   74.900277] eth0: IPv6 duplicate address fe80::a8bb:ccff:fe00:1103 detected!
[   74.900280] eth0: IPv6 duplicate address fe80::a8bb:ccff:fe00:1103 detected!
[   74.900283] eth0: IPv6 duplicate address fe80::a8bb:ccff:fe00:1103 detected!
[   74.900287] eth0: IPv6 duplicate address fe80::a8bb:ccff:fe00:1103 detected!
[   79.923074] __ratelimit: 3754 callbacks suppressed

(IPv6? WTH? I didn't even configured IPv4 yet...)

Using some network-analyzers for debugging:

With Wireshark/Ethereal, I got output like this:

Code:

Frame 803 (90 bytes on wire, 90 bytes captured)
Ethernet II, Src: aa:bb:cc:00:11:03 (aa:bb:cc:00:11:03), Dst: IPv6mcast_00:00:00:16 (33:33:00:00:00:16)
Internet Protocol Version 6
Internet Control Message Protocol v6

No.     Time        Source                Destination           Protocol Info
    804 0.035921    ::                    ff02::16              ICMPv6   Multicast Listener Report Message v2

Frame 804 (90 bytes on wire, 90 bytes captured)
Ethernet II, Src: aa:bb:cc:00:11:03 (aa:bb:cc:00:11:03), Dst: IPv6mcast_00:00:00:16 (33:33:00:00:00:16)
Internet Protocol Version 6
Internet Control Message Protocol v6

No.     Time        Source                Destination           Protocol Info
    805 0.035977    ::                    ff02::16              ICMPv6   Multicast Listener Report Message v2

Frame 811 (90 bytes on wire, 90 bytes captured)
Ethernet II, Src: aa:bb:cc:00:11:03 (aa:bb:cc:00:11:03), Dst: IPv6mcast_00:00:00:16 (33:33:00:00:00:16)
Internet Protocol Version 6
Internet Control Message Protocol v6

No.     Time        Source                Destination           Protocol Info
    812 0.036157    ::                    ff02::16              ICMPv6   Multicast Listener Report Message v2

Frame 812 (90 bytes on wire, 90 bytes captured)
Ethernet II, Src: aa:bb:cc:00:11:03 (aa:bb:cc:00:11:03), Dst: IPv6mcast_00:00:00:16 (33:33:00:00:00:16)
Internet Protocol Version 6
Internet Control Message Protocol v6

No.     Time        Source                Destination           Protocol Info
    813 0.036213    ::                    ff02::16              ICMPv6   Multicast Listener Report Message v2

With Tcpdump, I got output like this:

Code:

09:25:38.288125 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
09:25:38.288126 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
09:25:38.288158 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28

(ICMP6? IPv6 again? Why? I didn't configure IPv6...)

For my guests, I used first System Rescue CD v1.5.1, then Wolvix 2.0.0, then Knoppix 6.2, and last Debian Lenny. Always the same results.

Well, here is the question: why I get the high load (and the reduced performance) with my virtual net?

I thought first on a "broadcast storm", but I can't understand why the guests will pass the broadcast packets on, like bridges/switches do (the guests aren't like switches, they are more like routers, right?)

Any pointer will be appreciated.

Robhogg · 04-20-2010, 12:54 PM

I've no experience with QEMU/KVM, but one thing I notice (from my scant knowledge of IPv6) is that the fe80 prefix of the duplicated address (fe80::a8bb:ccff:fe00:1103) indicates that it is a link-local address.

These are the IPv6 equivalent of the IPv4 APIPA 169.254.0.0/16 addresses, that tend to get assigned automatically, in the absence of any other network configuration. In the case of IPv6, though, they are based on the network card's NIC number.

My bet is that, as the two machines were created from identical images, their "virtual" MAC addresses are the same. Often, also, IPv6 is configured as enabled by default. The result: clashing addresses.

Hope this helps,
Rob.

HasC · 04-20-2010, 01:25 PM

Quote:

Originally Posted by Robhogg

My bet is that, as the two machines were created from identical images, their "virtual" MAC addresses are the same.

AFAIK, the virtual MACaddrs are not the same.
If you read the script which creates the VMs

Code:

-net nic,macaddr=aa:bb:cc:00:11:01
-net nic,macaddr=aa:bb:cc:00:11:02
-net nic,macaddr=aa:bb:cc:00:11:03
-net nic,macaddr=aa:bb:cc:00:11:04

they all have different macaddrs, at least the last byte is different on each mac address.

Robhogg · 04-20-2010, 04:32 PM

Quote:

Originally Posted by HasC

If you read the script which creates the VMs

Which I obviously failed to do

Well, I'm out of ideas. Sorry.

HasC · 04-20-2010, 09:02 PM

oh, don't worry. We'll solve this thing :-)

BTW, I've been doing some more tests, and the results seem to support the thesis of broadcast storm, though I can't understand why... I'll post more on that later.

HasC · 04-21-2010, 01:41 PM

Well, I solved it... Though I still can't understand how nor why

I modified the script which creates the VMs:

Code:

#!/bin/bash

kvm -cdrom systemrescuecd.iso -m 256 \
	-net nic,macaddr=aa:bb:cc:00:11:01 \
	-net tap,ifname=tap100,script=no,downscript=no \
	-net nic,macaddr=aa:bb:cc:00:11:02,vlan=1 \
	-net tap,ifname=tap104,script=no,downscript=no,vlan=1 &

kvm -cdrom systemrescuecd.iso -m 256 \
	-net nic,macaddr=aa:bb:cc:00:11:03 \
	-net tap,ifname=tap103,script=no,downscript=no \
	-net nic,macaddr=aa:bb:cc:00:11:04,vlan=1 \
	-net tap,ifname=tap107,script=no,downscript=no,vlan=1 &

Note the "vlan=X" pieces.

Apparently, the 4 virtual NICs (2 for each VM, you see) belonged to the same VLAN, vlan0. Because of that, the VMs worked as bridges too. And that's why I had so much network traffic: a broadcast storm.

Changing the VLAN value of one of the NIC pairs (to vlan1 in the example), I finally got separated subnets.

Man, I thought a PC couldn't work as a bridge without some fancy configuration... Here I needed the fancy configuration to achieve the opposite