How to get guest on same subnet as host

irreverentryan · 04-19-2013, 03:17 PM

Hi all,

I'm building what I assume is a relatively complex setup. I have a physical host (VMHOST01) running RHEL 6.4 x64. This machine is hosting a virtualized RHEVM guest. I cannot seem to (successfully) get the guest to operate on the same subnet as the host. If this is not possible, how would I set it up so that hosts on the production network are able to access the guest in its virtual network?

My host setup is as follows:

em1
>bond0-->192.168.0.0/22 network (production)
em2

em3
>bond1-->192.168.10.0/24 network (storage / RHEV)
em4

These bonds seem to be working correctly. I have no packet loss with pings on either network from the host, I am able to go online, and add SAN storage from the RHEV network.

The problem is, when I select "Host device bond0 : macvtap" I receive DHCP (which I don't want, but am using for testing) and I am able to ping SOME addresses on my production LAN.

Working addresses:
192.168.1.244 (our prod network gateway)
192.168.1.39 (DNS1)
192.168.1.150 (DNS2)
8.8.8.8 (Google's Public DNS)

Non-Working addresses:
192.168.1.16 (the host bond0)
192.168.1.17 (the guest's only nic)
173.194.43.8 (Google's resolved IP address)

If I select "Virtual network 'default' : NAT" then I receive proper DHCP, and am able to ping across the board, as well as access the internet (which I couldn't do with the bond0 interface selected.) The only reason I don't stay with this configuration is that my guest receives an address in the 192.168.122.0/24 network (the virtual network between the host and guest) and I am (seemingly) unable to access that address from my production network.

I am sure I could probably make some static route entries, but what I'd really like to accomplish is to have my Guest on the same subnet as my host.

Thanks!

dyasny · 04-20-2013, 07:14 AM

1. what does RHEV have to do with this configuration? You are obviously talking about libvirt and virt-manager?
2. Use a normal bridge, not a macvtap

irreverentryan · 04-22-2013, 07:36 AM

Hi, thanks for the reply. RHEV does not neccessarily have anything to do with this setup, orhter than the fact that it is the guest. I am indeed talking about libvirt and virt-manager. All of the options in the drop-down menu for virt-manager include "macvtap." Do I need to create a new confif file to create the bridge?

Thanks

irreverentryan · 04-22-2013, 11:21 AM

Alright, so I've created a bridge (br0) and set my virtual interface to Host Device bond0 (Bridge 'br0') and I was able to get DHCP (no change yet, I was able to do this before.)

I set the virtual nic to a static on my subnet (192.168.1.17)
I pinged google.com and got this result:

Code:

32 packets transmitted, 2 received (93% loss)

The traceroute was almost instant, though...

brctl show br0:

Code:

bridge name	bridge id		STP enabled	interfaces
br0		8000.00219ba7404c	no		bond0
							vnet0
virbr0		8000.525400da074f	yes		virbr0-nic

I did some additional testing, and it seems to get only 2 responses, no matter how many requests it sends out, and now matter which sequence the first answer is (sometimes immediately, sometimes later.) but it is always 2 responses.

I am very confused as to why it would connect, get dhcp, allow a static address, and traceroute all while also haveing horible ping stats.

dyasny · 04-22-2013, 12:59 PM

what bonding mode do you use? modes 0 and 6 are not compatible with a bridge, I always recommend mode 4 if the switches support it

irreverentryan · 04-22-2013, 02:47 PM

I switched it to mode=1 and removed the bridge configs on the host machine. I then selected the "Bond0" option from the virt-manager drop down menu, and all seems great now. 100% ping success from host and guest to google.com

nicolasdiogo · 04-22-2013, 04:50 PM

had similar problem.
http://www.linuxquestions.org/questi...tu-4175458587/

i will try your mode=1

however, i am sure that RedHat has a bug posted about problems of bonding and bridge with KVM.but i will try it..

thanks,

dyasny · 04-23-2013, 01:33 AM

There is a bugreport, somewhere, but the fact remains - bridging and bond modes 0 and 6 don't work together, because of the way these bonding modes operate. You can read up on what exactly they do, and think about that in conjunction with using a bridge and virtual machine NICs on top - will make sense after a few passes

irreverentryan · 04-24-2013, 08:46 AM

Although my previous solution worked to get my VM on the host network, I am unable to ping the host from the guest, and vice versa. My guest VM can ping my physical gateway, google's DNS servers, and other hosts on my physical LAN, however it cannot ping the host address. I am able to connect to RHEVM, which is running on my VM, but I am unable to run the spice console for the RHEV vms. I am pretty sure this is a relative issue with the odd ping results I'm experiencing.

irreverentryan · 04-24-2013, 02:35 PM

I actually cannot confirm that my host is properly forwarding traffic... I have enables ip forwarding in sysctl.conf, and I have disabled iptables. What could be stopping it at this point?

dyasny · 04-24-2013, 11:49 PM

when you use a bridge, there is no need to forward anything. can you post your ifcfg-* files?

irreverentryan · 04-25-2013, 08:08 AM

DEVICE=bond0
IPADDR=192.168.1.16
PREFIX=22
DNS1=192.168.1.39
DNS2=192.168.1.150
DEFROUTE=yes
GATEWAY=192.168.1.244
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
//////////////////////////////////////////////////////
DEVICE=bond1
IPADDR=192.168.10.100
PREFIX=24
USERCTL=no
BOOTPROTO=none
ONBOOT=yes
DEFROUTE=no
//////////////////////////////////////////////////////
DEVICE=em4
BOOTPROTO=none
ONBOOT=yes
MASTER=bond1
SLAVE=yes
USERCTL=no
//////////////////////////////////////////////////////
DEVICE=em3
BOOTPROTO=none
ONBOOT=yes
MASTER=bond1
SLAVE=yes
USERCTL=no
//////////////////////////////////////////////////////
DEVICE=em2
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no
/////////////////////////////////////////////////////
DEVICE=em1
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no
////////////////////////////////////////////////////

---------- Post added 04-25-13 at 09:09 AM ----------

I also swapped the bridge for the forwarding. I'll have to undo the forwarding and reconfigure the bridge.

irreverentryan · 04-25-2013, 08:12 AM

Also, I'd be bridging bond0 and bond1 for passthrough (I think) but would there be an issue with the two bonds being on different subnets, or will it route for me?

Or am I confused? I was also thinking I might have to create a bridge with only bond0 and point the VM at the bridge as the host device? and the same for bond1?

irreverentryan · 04-25-2013, 01:01 PM

Alright, so I've completely redone my bonds and bridges. I can confirm that, on the host, they are working properly. Here are the VMHOST config files:

////////////////////
DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
USERCTL=NO

### BRIDGING PARAMS ###
BRIDGE=br0

//////////////////////
DEVICE=bond1
BOOTPROTO=none
ONBOOT=yes
USERCTL=NO

### BRIDGING PARAMS ###
BRIDGE=br1

/////////////////////
TYPE=Bridge
ONBOOT=yes
DEVICE=br0
IPADDR=192.168.1.16
GATEWAY=192.168.1.244
DNS1=192.168.1.39
DNS2=192.168.1.150
BOOTPROTO=static
USERCTL=no
IPV6INIT=no
STP=no

////////////////////
TYPE=Bridge
ONBOOT=yes
DEVICE=br1
IPADDR=192.168.10.100
BOOTPROTO=static
USERCTL=no
IPV6INIT=no
STP=no

////////////////////
DEVICE=em1
BOOTPROTO=none
ONBOOT=yes

### Bonding ###
MASTER=bond0
SLAVE=yes

////////////////////
DEVICE=em2
BOOTPROTO=none
ONBOOT=yes

### Bonding ###
MASTER=bond0
SLAVE=yes

////////////////////
DEVICE=em3
BOOTPROTO=none
ONBOOT=yes

### BONDING PARAMS ###
MASTER=bond1
SLAVE=yes

//////////////////////
DEVICE=em4
BOOTPROTO=none
ONBOOT=yes

###BONDING PARAMS ###
MASTER=bond1
SLAVE=yes
////////////////////////////////////////////////////////////////////////////////////
I am able to ping the following from my VMHOST:
192.168.1.244 (default gateway on production LAN)
192.168.1.150 (dns server on production LAN)
8.8.8.8 (public Google DNS server)
173.194.43.4 (Google's public address)
192.168.10.110 (RHEVH01)
192.168.10.120 (RHEVH02)
192.168.10.50 (SAN VIP)
////////////////////////////////////////////////////////////////////////////////////
Where I am getting lost is how to properly configure the virtual machine to be able to do the same. I am using virt-manager and I was able to set up one connection (Host device bond0 {bridge 'br0'}. With this setup, I made the following config file on my VMGUEST:

###ifcfg-Auto_eth0###
HWADDR=52:54:00:B1:F9:2F
TYPE=Ethernet
BOOTPROTO=none
IPADDR=192.168.1.17
PREFIX=22
GATEWAY=192.168.1.16
DNS1=192.168.1.39
DNS2=192.168.1.150
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="Auto eth0"
UUUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
ONBOOT=yes
LAST_CONNECT=1366907507
///////////////////////////////////////////////////////
With this config, I am able to ping the following:
192.168.1.244 (GW)
192.168.1.16 (VMHOST IP)
192.168.1.150 (DNS)
8.8.8.8 (Google)
173.194.43.4 (Google)
192.168.10.100 VMHOST IP on br1

I am UNABLE to ping anything else on the 192.168.10.0 network from my VM. I tried adding a second virtual nic to br1, in the same fashion I did for br0, but it did not seem to fix anything.