[SOLVED] Centos 6 machine - strange behaviour when on intranet, consistent when not

rylan76 · 11-06-2012, 03:23 AM

Hi Guys

I've configured a KVM Centos 6 inside Centos 6 (e. g. Centos inside Centos) for a class B network environment. The VM is run bridged using openvpn. The VM starts automatically from rc.local once the physical machine boots.

The physical machine has an IP of 172.16.1.2, the VM then starts in bridged mode and the machine becomes 172.16.1.1

The problem is that the above works perfectly on a closed network (not physically connected to my main network). However, when I put this machine on my network along with other Windows Server 2008R2 machines, and about 30 WinXP and Win7 machines, I get very strange behaviour from it.

E. g. if I ssh to the machine's IP 172.16.1.1, I -SOMETIMES- get the physical hardware instance Centos SSHd instance coming up, SOMETIMES I get the VM's SSHd instance answering.

Same with VNC'ing to the VM - sometimes I will get the VM's VNC stream coming back, sometimes the machine is just unreachable. At such times as the VNC stream is not reachable, the SSH connections go to the physical machine - the moment the VNC stream quits, the SSH connections reach the VM.

What the heck is going on here? I've checked and no other machines are using 172.16.1.1 or 172.16.1.2 - yet the machine literally flip-flops back and forth - SSH decides to go to physical, VNC works to VM, SSH decides to go to VM, VNC stops working for VM.

The behaviour and timings in which it happens is completely random.

I cannot find anything relevant in /var/log/messages or dmesg for either the physical or virtual, depending on where SSH "decides" to land...

Any ideas at all?

Thanks!

rylan76 · 11-07-2012, 06:15 AM

Hi guys

To all interested parties, I think I have solved this issue. In the tap0 setup script that sets up a bridged connection so the KVM VM can access the network, I made two mistakes.

The first mistake was to setup the bridge based on the wrong IP address. !

The second mistake was to use the wrong netmask for the Class B network I was on.

Once I corrected both of the above, everything started working.

First, the rationale. What I wanted was this:

172.16.1.1 - static IP of physical Centos 6 machine (host)
172.16.1.2 - static IP of virtual Centos 6 KVM qemu machine (e. g. the guest which runs "inside" a KVM on 172.16.1.1)

So here is my now fully working and correct qemu-ifup.sh for the above:

Code:

#!/bin/sh 
# 
# script to bring up the tun device in QEMU in bridged mode 
# first parameter is name of tap device (e.g. tap0)
#
# some constants specific to the local host - change to suit your host
#
ETH0IP=172.16.1.2
GATEWAY=172.16.1.9
BROADCAST=172.16.255.255
#
# First take eth0 down, then bring it up with IP 0.0.0.0 
#
/sbin/ifdown eth0
/sbin/ifconfig eth0 0.0.0.0 promisc up
#
# Bring up the tap device (name specified as first argument, by QEMU)
#
/usr/local/sbin/openvpn --mktun --dev $1 --user `id -un`
/sbin/ifconfig $1 0.0.0.0 promisc up
#
# create the bridge between eth0 and the tap device
#
/usr/sbin/brctl addbr br0
/usr/sbin/brctl addif br0 eth0
/usr/sbin/brctl addif br0 $1
# 
# only a single bridge so loops are not possible, turn off spanning tree protocol
#
/usr/sbin/brctl stp br0 off 
# 
# Bring up the bridge with ETH0IP and add the default route 
#
/sbin/ifconfig br0 $ETH0IP netmask 255.255.0.0 broadcast $BROADCAST
/sbin/route add default gw $GATEWAY

The problem line at the top was

Code:

ETH0IP=172.16.1.2

which I had as

Code:

ETH0IP=172.16.1.1

which was WRONG, since the Centos host's IP was 172.16.1.2

The problem line at the bottom was

Code:

/sbin/ifconfig br0 $ETH0IP netmask 255.255.0.0 broadcast $BROADCAST

which I had as

Code:

/sbin/ifconfig br0 $ETH0IP netmask 255.255.255.0 broadcast $BROADCAST

Once I fixed both of these everything started working - I can now ssh to 172.16.1.1 to get to the physical hosts's SSH daemon, and I can ssh to 172.16.1.2 to get to the KVM qemu guest's SSH daemon.

Also, both accesses are stable and I no longer get the "random ssh daemon" problem I had earlier. Clearly it was all my fault, I had let IPs clash in the bridging step and I was using the incorrect netmask for a class B network.

Once the above ifup script is done, I start my KVM Qemu instance - something like so:

Code:

sh -f /home/verisharepdc/Desktop/qemu-ifup.sh tap0

/usr/local/kvm/bin/qemu-system-x86_64 /home/verisharepdc/Desktop/vdisk.img -m 2048 -smp 2 -vnc 172.16.1.2:1 -net nic -net tap,ifname=tap0,script=no

sh -f /home/verisharepdc/Desktop/qemu-ifdown.sh tap0

in order to get it to use the tap0 device to be visible at 172.16.1.1 on the network.

Note that in the above, the VNC feed for the running QEmu KVM instance is available at 172.16.1.2:1 - if I pass this as is to TightVNC (for example) it works fine and I can see the XWindows instance running inside the qemu KVM instance.

For reference, here is my qemu-ifdown.sh:

Code:

#!/bin/sh 
# 
# Script to bring down and delete bridge br0 when QEMU exits 
# 
# Bring down eth0 and br0 
#
/sbin/ifdown eth0
/sbin/ifdown br0
/sbin/ifconfig br0 down 
# 
# Delete the bridge
#
/usr/sbin/brctl delbr br0 
# 
# bring up eth0 in "normal" mode 
#
/sbin/ifconfig eth0 -promisc
/sbin/ifup eth0 
#
# delete the tap device
#
/usr/local/sbin/openvpn --rmtun --dev $1

For reference as well, here is my physical machine's /etc/rc.local:

Code:

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local
/sbin/ifconfig eth0 172.16.1.2 netmask 255.255.0.0 broadcast 172.16.255.255 up
route add default gw 172.16.1.9 eth0
#service dhcpd start
#service named start
cp /etc/resolv.conf.bak /etc/resolv.conf
#smbd -D
#nmbd -D
modprobe kvm-intel
modprobe kvm
modprobe tun
sh -f /home/verisharepdc/Desktop/run_vm_unattended.sh &

- As you can see I assign it 172.16.1.2. Correlate with:

Code:

ETH0IP=172.16.1.2

in my qemu-ifup,.sh

Here's my Qemu KVM instance's /etc/rc.local:

Code:

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local
/sbin/ifconfig eth0 172.16.1.1 netmask 255.255.0.0 broadcast 172.16.255.255
route add default gw 172.16.1.6 eth0
#service dhcpd start
#service named start
cp /etc/resolv.conf.bak /etc/resolv.conf
#service nmb start
#service smb start

- As you can see I assign the VM 172.16.1.1

Hope this helps somebody...