Hi,

I am trying to setup networking within a CentOS server that will be a single server frontend+VM host for OpenNebula. http://opennebula.org/

I’ve been allocated 1 fixed IP address for the main server and an additional /29 subnet (6 usable IPs) by my hosting provider hetzner.de. I’ve managed to get this configured to use a routed bridge device (br0) for the subnet, leaving the main eth0 device as-is and applying a simple route for the /29 subnet to br0. Despite the documentation for OpenNebula asking for a bridged set-up, this is not possible with my hosting provider network configuration.

The problem I have is that now I have installed OpenNebula, I have a weird problem where upon a reboot the ifcfg-eth0 script is replaced by *something* giving the device an IP of 122.0.52.90 (at least on this occasion) - I’m a little baffled by this to be honest, as I can’t see anything in the logs other than when the network service is started and it checks if 122.0.52.90 is in use (so the ifcfg-eth0 file has been modified before then). This has only started happening since I installed open nebula (and of course its dependencies..)

I was suspecting libvirt or dnsmasq or some such thing causing the problem, but have yet to find a configuration that drives this behaviour.

I would really appreciate if someone could shed some light on this and point me in the right direction.

The config file on your hypervisor NIC is being overwritten?

Post the before and after config file, including 'ip addr show', and 'ip route show'.

Yes, the host/hypervisor network scripts are being overwritten, both when opennebula (and dependencies like libvirt and dnsmasq) are first installed, and also when rebooted from then on (before this is installed, it does not get overwritten).

My script for initially configuring the network scripts into a working network configuration:

cat << EOT > /etc/sysconfig/network-scripts/ifcfg-eth0
GATEWAY=A.B.C.D
IPV6_DEFAULTDEV=eth0
IPV6INIT=yes
HWADDR=zz:zz:zz:zz:zz:zz
BROADCAST=A.B.C.E
SCOPE="peer A.B.C.D"
BOOTPROTO=none
NAME=""
MACADDR=""
DEVICE=eth0
NETMASK=255.255.255.255
MTU=""
IPADDR=A.B.C.E
IPV6ADDR=1:1:1:1::2/64
NETWORK=A.B.C.E
IPV6ADDR_SECONDARIES=""
ONBOOT=yes
EOT

cat << EOT > /etc/sysconfig/network-scripts/ifcfg-br0
BOOTPROTO=none
NAME=""
MACADDR=""
IPV6INIT=no
TYPE=Bridge
DEVICE=br0
NETMASK=255.255.255.248
MTU=""
BROADCAST=A.B.F.G
IPADDR=A.B.F.H
NETWORK=A.B.F.I
ONBOOT=yes
EOT

cat << EOT > /etc/sysconfig/network-scripts/route-eth0
ADDRESS0=0.0.0.0
NETMASK0=0.0.0.0
GATEWAY0=A.B.C.D
EOT

cat << EOT > /etc/sysconfig/network-scripts/route-br0
ADDRESS0=A.B.F.I
GATEWAY0=A.B.C.E
NETMASK0=255.255.255.248
EOT


This is what is overwritten when the system is rebooted:

ifcfg-eth0:

DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
NETMASK=255.255.255.0
IPADDR=122.0.52.90
GATEWAY=122.0.52.1

All other files are left untouched.

Obviously this means that I lose connectivity to the server. I can still get access to the server via a rescue-image provided by the hosting provider, which allows me to mount the server's filesystems for inspection (e.g. I can review logs etc).

Even a hint as where to look would be helpful - at the moment, I'm just not sure how to debug this given all the usual places seem to be devoid of info.

What I don't see in your initial script content is "NM_CONTROLLED=NO".

You can even go a step further and set NetworkManager to 'off' with chkconfig. That's usually what changes ifcfg scripts, so unless you are using the NetworkManager for something (DHCP or DNS management), just disable it, and try again.

A good idea, but I don't think it's the culprit: NetworkManager is not present on the install as far as I can tell - the default minimal install of CentOS 6.5 appears to use the standard 'network' service, rather than 'NetworkManager' (which neither chkconfig nor service can find), so it shouldn't be mangled by NM afaik. If it was NM I would also expect this behaviour to occur on every reboot, not just the ones after installing opennebula and dependencies (e.g. libvirt, dnsmasq, kvm etc).

To be sure though, I did add that to the configuration, and also removed persistent net rules:

rm -f /etc/udev/rules.d/70-persistent-net.rules
rm -f /lib/udev/rules.d/75-persistent-net-generator.rules

(As found to be advised in a blog post relating to fullly removing NM).

Still the same problem unfortunately.

:-( Any other things it could be?

The NetworkManager is the built-in process that messes with network configuration from time to time. Since this didn't happen before opennebula was installed, I'd suggest starting there.

Something in that program or install is doing the rewrites on your files...

Did you maybe install this package on your host by accident?

http://pkgs.org/debian-wheezy/debian...1_all.deb.html

:-)

Well, it looks like you might be right! Although I don't explicitly install that package (I only included those defined for the Front-End and Host Servers in the OpenNebula installation guide, it does seem to have crept in somewhere along the way, maybe as an erroneous dependency - it certainly isn't in the install scripts I've written and use consistently. I do use the contextualisation package in the VMs but that is explicitly done in that context. Odd.

Anyway I just rebooted the machine and my 'auto-fixer' script that was resetting the network scripts after a reboot claim not to have needed to do anything, so I think this might have been the culprit!

I'll do a few more tests and monitor what happens, but I am feeling hopeful.

Thanks for your time, patience and help GaWdLy, much appreciated.

You're very welcome. Sometimes we just need someone to help us put it all into perspective.