[SOLVED] Trouble setting up bridged adapter

ssorbom · 06-25-2013, 05:31 PM

Hello,
I have had issues setting up bridged networking for qemu machines on my server.
I started by following the instructions here:
http://www.cyberciti.biz/faq/debian-...sical-network/
Weirdly, when I brought up the network interfaces again none of the physical machines I had connected on my server's subnet could connect to the server, or outside. However, the server itself could reach the internet just fine.

I also tried using the brctl utilities like so:

Code:

brctl addbr br0
brctl addif br0 eth0

Nothing went wrong on the first command, but the second messed up my network.

On my third attempt, I tried using the virsh utility to autoconfigure the bridge and avoid doing it myself, but I got the following error:

Code:

virsh # iface-bridge eth0 br0
error: failed to get interface 'eth0'
error: internal error couldn't find interface named 'eth0': required entry missing

here is my /etc/network/interfaces file:

Code:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo                                                                                                                                        
iface lo inet loopback                                                                                                                         
                                                                                                                                               
# The internal network interface                                                                                                               
auto eth0                                                                                                                                      
allow-hotplug eth0                                                                                                                             
iface eth0 inet static                                                                                                                         
address 192.168.200.1
network 192.168.200.0
netmask 255.255.255.0
broadcast 192.168.200.255
gateway 192.168.200.1

#external interface
auto eth1
allow-hotplug eth1
iface eth1 inet dhcp
#address 192.168.100.131
network 192.168.100.0
netmask 255.255.255.0
broadcast 192.168.100.255
gateway 192.168.100.1
eth1 up

#bridge connection – had to comment it out because it caused problems
#auto br0
#iface br0 inet static – tried with both static and dhcp settings
#        address 192.168.200.2 
#         netmask 255.255.255.0
#         network 192.168.200.0
#         broadcast 192.168.200.255
#       gateway 192.168.200.1
#         bridge_ports eth0
#        bridge_stp on
#         bridge_maxwait 0

What am I doing wrong in either case?
Thanks in advance.
PS I am running Debian Wheezy if that means anyt

Ser Olmy · 06-25-2013, 05:47 PM

Quote:

Originally Posted by ssorbom

I also tried using the brctl utilities like so:

Code:

brctl addbr br0
brctl addif br0 eth0

Nothing went wrong on the first command, but the second messed up my network.

How exactly did it "mess up" your network? Because the commands are correct. If you add them to a startup script of some sort, you may have to add ifconfig eth0 up before the 2nd brctl command.

Once eth0 is connected to the bridge, the IP address previously assigned to eth0 should be assigned to br0 instead. eth0 should be left unconfigured.

ssorbom · 06-25-2013, 06:35 PM

That may be part of the problem then. I used the two commands in sequence, and eth0 was already configured and running. the "mes up" I was referring to was that client machines on that subnet could no longer use eth0 as a gateway.

I'm a bit confused as to why eth0 needs to stay unconfigured, I thought the idea of a bridge was to have one network card serving 2 addresses (ie eth0 192.168.200.1 and br0 being 192.168.200.2). I am trying to have a vm that is on the same subnet as my internal network card

suicidaleggroll · 06-25-2013, 06:40 PM

Quote:

Originally Posted by ssorbom

I am trying to have a vm that is on the same subnet as my internal network card

What VM software? VirtualBox has an option in the network settings for "bridged adapter" that does exactly what you're looking for, with no further setup required.

ssorbom · 06-25-2013, 06:49 PM

virsh + qemu.. I considered using virtualbox, but I had heard somewhere that qemu was a better choice for servers (why though?). Virtualbox also brings in a lot of gui dependancies, which I guess I can live with if I have to. I may end up going that route, but I wanted to give this way one last try.

Ser Olmy · 06-25-2013, 08:06 PM

Quote:

Originally Posted by ssorbom

I'm a bit confused as to why eth0 needs to stay unconfigured, I thought the idea of a bridge was to have one network card serving 2 addresses (ie eth0 192.168.200.1 and br0 being 192.168.200.2). I am trying to have a vm that is on the same subnet as my internal network card

The point of the bridge device is to serve as an, er, bridge between your physical NIC and any virtual NICs that need access to the same physical network.

(I've set up a lot of KVM/Qemu hypervisors, and have many VMs with NICs connected to bridged adapters.)

Shadow_7 · 06-25-2013, 09:43 PM

The bridge "becomes" the configured device. So it might not make sense to have them setup before bridging.

# brctl addbr br0
# brctl addif eth0 wlan0
# ip link set dev br0 up

Along with ebtables rules (static on eth0 side / dhcp in wlan0 side to internet in my case).

All that being said, bridge-utils seems broke on kernels > 2.6.x. At least for ethernet to wireless. Not supported or other similar error.

# ifconfig br0 down
# brctl delif br0 eth0 wlan0
# brctl delbr br0

To undo the bridge. Make sure your interfaces are as you expect them to be.

# ifconfig -a

If you boot from a USB linux and switch machines, the interfaces will get re-assigned new names. So if you scripted it, you might need to modify the scripts depending on the machine of the moment. Or delete some udev rules and restart udev to have the ???0 variants assigned as names.

EDIT: I guess I should add that I have to setup my wireless device twice. I guess the first attempt powers it on, the second one actually sets up the parameters.

ssorbom · 06-26-2013, 11:21 AM

Here is what I have done so far in order on this attempt:
1. Brought down both server interfaces
2. commented out all entries for eth0 in /etc/network/interfaces (except I forgot to comment out: allow hotplug eth0)
3. used command brctl addbr br0
4. used command brctl addif br0 eth0
5. bring up eth0
6. ifconfig reveals eth0 STILL has ip address 192.168.200.1 (despite being commented out)
7. bring up interface eth1
8. bring up br0 with command: ifconfig br0 up(ip address field is a mixture of letters and numbers, [ie 08ef:ff8a:, etc])
9. can now ping outside world from server.
10. Start physical machine on subnet, machine properly reccieves ip address 192.168.200.20 from server's dhcp (as expected)
11. 192.168.200.20 CANNOT ping server address 192.168.200.1
12. cannot ping 192.168.200.20 from server address 192.168.200.1
13. bring down server interface br0
14. server interface still cannot ping 192.168.200.20
15. use command delif br0 eth0
16. server can sow successfully pings 192.168.200.20 over interface 192.168.200.1
17. 192.168.200.20 can now successfully ping server and outside world
What have I done wrong here? I probably will use virtualbox at this point, but I can't just run away from a problem like this forever... I would still like to know what is going on.

Shadow_7 · 06-26-2013, 06:38 PM

Did you setup the ebtables rules?

ssorbom · 06-27-2013, 12:35 AM

No, I used the sysctl method. I suppose I should have tried both. I will try it and get back to you.

Shadow_7 · 06-28-2013, 03:30 PM

I finally got bridge working (sort of) on a 2nd bootable linux (same machine). With quirks, it doesn't seem to happen at boot time, even though it's setup as far as I can tell the same as the other USB booted linux I have that does do it at boot. But a little manual coaxing and ifconfig br0 down && ifconfig br0 up and it worked. Most of which is the wireless driver b43 for my quite old gear.

For me as far as I can remember here's my edits. Maybe it'll help, or just useless banter. Or a reference for me should I lose all my data at a later date.

### /etc/sysctl.conf

Code:

### custom edit ###
vm.swappiness = 40
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-filter-pppoe-tagged = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0

Obviously you don't need to modify the swappiness. And you could set them manually prefixing with /proc/sys/ and replacing the "."'s with "/"'s. Like $(echo 1 > /proc/sys/net/ipv4/ip_forward)

### /etc/rc.local

Code:

### custom edit ###
/etc/init.d/procps restart

In theory that's needed which seemed to be most of my initial failings. Something about to sysctl or not to sysctl.

### /etc/network/interfaces

Code:

auto lo br0

iface lo inet loopback
iface eth0 inet manual
iface wlan0 inet manual
iface br0 inet dhcp
  pre-up /some/script/to/configure/firewall/and/wireless_card.sh
# use the MAC of the wireless card (which goes out to the internet in my case)
  bridge_hw 00:11:22:33:44:55
  bridge_ports eth0 wlan0

And if that works the bridge is there and I can browse the internet on this middleman of sorts. But I can't yet browse the internet on the behind the bridge machine. So ebtables seems to be what worked for me.

### ebtables rules

Code:

ebtables -t broute -F
# not sure why I have the previous one, but I copied it from somewhere.  Flushes the rules I hope.
#
ebtables -t nat -F
# flushes the rules for where I know they go in the style of iptables.
#

# The mac of the wireless to the internet, a magic bullet for me missing from the debian bridge page.
# but I have an iptables firewall on all my boxes.
#
ebtables -t nat -A POSTROUTING -o wlan0 \
         -j snat --to-src 00:11:22:33:44 \
         --snat-arp --snat-target ACCEPT

# The debian bridge basically copied.
IP_BEHIND_THE_BRIDGE=192.168.1.99
MAC_BEHIND_THE_BRIDGE=66:55:44:33:22:11
ebtables -t nat -A PREROUTING -p IPv4 -i wlan0 \
         --ip-dst $IP_BEHIND_THE_BRIDGE \
         -j dnat --to-dst $MAC_BEHIND_THE_BRIDGE \
         --dnat-target ACCEPT
ebtables -t nat -A PREROUTING -p ARP -i wlan0 \
         --arp-ip-dst $IP_BEHIND_THE_BRIDGE \
         -j dnat --to-dst $MAC_BEHIND_THE_BRIDGE \
         --dnat-target ACCEPT

I should note that this only works for 2.6.x kernels for me. brctl the manual way always comes back with cannot add wlan0 to bridge or something similar in 3.x series. It would be interesting to know if anyone found a way around this, but I'll probably just wait for it to magically work in 3.x someday.

And with a static ip and default gateway route, this works for me. To use a laptop as a redneck wireless card for my desktop that ran out of PCI slots with a cat5e crossover cable from its onboard ethernet. I mostly typed this while reading the files on the bridge device so appologies in advance for fat finger typos which I may or may not edit later. YMMV

# ifconfig eth0 192.168.1.99 netmask 255.255.255.0 broadcast 192.168.1.255 up
# route add default gw 192.168.1.1

Most of this is debian centric and derived from:
http://wiki.debian.org/BridgeNetworkConnections

ssorbom · 06-30-2013, 02:21 PM

Thank you,
I will not be able to try your solution for another couple of days (I am out of town, away from my server), but as soon as I can I will post results.

ssorbom · 07-02-2013, 06:32 PM

I got the following error when trying to to load your sysctl canges:

Code:

[....] Setting kernel variables ...sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-arptables: No such file or directory
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-filter-pppoe-tagged: No such file or directory
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-filter-vlan-tagged: No such file or directory
done.

I havent been able to set up the firewall yet. Are ebtables and iptables interchangable?

Shadow_7 · 07-06-2013, 04:17 PM

ebtables applies to the bridge afaik. It has a syntax that is similar to iptables, but it is different afaik.

I got that same error, but I'm not sure which thing that I did, that got rid of it. I think it had to do with the loaded kernel modules. So here's my /etc/modules.

FILE: /etc/modules

Code:

yenta_socket
xt_state
xt_tcpudp
xt_conntrack
lib80211_crypt_wep
lib80211
mac80211
cfg80211
input_polldev
led_class
rfkill
ssb
pcmcia
mmc_core
bcma
rng-core
b43

$ egrep -r -i "nf-filter" /lib/modules/$(uname -r)/kernel/net/*

Matches the bride-nf-filter for vlan and pppoe to the bridge.ko module. And lsmod on my old usb linux router matches modules "stp, bridge, ebtable_broute" to bridge. And it looks like ebtables and ebtables_nat are in play in lsmod output.

On the new one, I basically did a dpkg -i on a stable debian kernel version (2.6.x) into a sid install. And that works if I downgrade the firmware for the b43 wireless driver to v.410 vs. v.666 that comes with sid. The kernel is sufficiently out of sync with sid that I can't get an X session running under it. But I haven't been able to get bridging working under a 3.x series kernel.

HTH

RootMason · 07-06-2013, 05:55 PM

Just thought I'd jump in way down at the bottom of this conversation! Looking at you original config I would say one problem is that you have addressing configured for eth0 as well as br0. The second problem is that br0 has no "pointer" to use eth0. Maybe make a copy (so you can revert) of eth0 and try erasing the IP addressing and figure out how to add that pointer under br0... I believe your "pointer should look like this:

iface br0 inet static # Or DHCP, right?
bridge_ports eth0

My Debian (my assumption w/ apt-get...) is a bit rusty, but here are my test configs in Centos6:

vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=00:1B:78:09:20:FB
ONBOOT=yes
TYPE=Ethernet
IPV6INIT=no
USERCTL=no
BRIDGE=br0

vi /etc/sysconfig/network-scripts/ifcfg-br0
DEVICE=br0
TYPE=Bridge
BOOTPROTO=static
DNS1=192.168.1.5
DNS2=192.168.1.1
IPADDR=192.168.1.5
NETMASK=255.255.255.0
ONBOOT=yes
SEARCH="sddomain.local"

My first problem with this was to make a copy of my eth0 and just add the BRIDGE=br0 and DEVICE=br0. I tried commenting them out, but it just gave another error. Erasing the comment-out configs from eth0 did the trick though, but I did make a copy... always good to make backups!