LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Virtualization and Cloud
User Name
Password
Linux - Virtualization and Cloud This forum is for the discussion of all topics relating to Linux Virtualization and Linux Cloud platforms. Xen, KVM, OpenVZ, VirtualBox, VMware, Linux-VServer and all other Linux Virtualization platforms are welcome. OpenStack, CloudStack, ownCloud, Cloud Foundry, Eucalyptus, Nimbus, OpenNebula and all other Linux Cloud platforms are welcome. Note that questions relating solely to non-Linux OS's should be asked in the General forum.

Notices

Reply
 
LinkBack Search this Thread
Old 11-27-2013, 04:43 PM   #1
Sniperm4n
LQ Newbie
 
Registered: May 2012
Posts: 27

Rep: Reputation: Disabled
Linux-KVM “broken” after a motherboard swap


I just had the motherboard replaced in a Dell PowerEdge R410 that functions as one of our virtual servers (running Ubuntu 10.04.3 LTS). I'm fairly new to Linux, and was quite surprised when networking was completely broken after the swap. Another disclaimer is that I didn't build our virtual servers to begin with, and have very limited understanding of how Linux-KVM works. Once the motherboard was swapped, I then ran the LifeCycle Controller application and applied a variety of upgrades (the most notable of which would be firmware upgrades for the NICs). After a lot of research, I finally managed to "fix" networking by editing the /etc/udev/70-persistent-net.rules file. Within that file, I removed the 2 old Broadcom (bnx2) entries for the prior motherboard, and then renamed the new bnx2 eth2 and eth3 to eth0 and eth1 respectively. I then moved eth0 and eth1 to the top of the document. The (igb) entries are for a PCI-based Intel Gigabit NIC that's currently unused. Here are the contents of my 70-persistent-net.rules file:

Code:
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x14e4:0x163b (bnx2)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="78:2b:cb:20:9d:71", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x14e4:0x163b (bnx2)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="78:2b:cb:20:9d:72", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x8086:0x10c9 (igb)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="90:e2:ba:0c:7e:f9", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

# PCI device 0x8086:0x10c9 (igb)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="90:e2:ba:0c:7e:f8", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"
This resolved the networking being completely broken (couldn't ping the gateway before), but a much more major issue has persisted where the server hardware randomly reboots. I can't easily reproduce the crash, but it entails bringing up the 5 guest OSes that are running on the machine, and then doing Splunk queries/pings/running X11 forwarding to puTTY, etc. The hardware itself passes all self-tests, and a Dell technician reviewed a DSET I collected and mentioned everything looks great hardware-wise.

Here's my /etc/network/interfaces file:

Code:
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

# 10.1.225.x network
auto br0
iface br0 inet static
  address 10.1.225.12
  netmask 255.255.255.0
  network 10.1.225.0
  broadcast 10.1.225.255
  gateway 10.1.225.1
        bridge_ports eth0
        bridge_fd 9
        bridge_hello 2
        bridge_maxage 12
        bridge_stp off

#vlan 231
auto eth1.231
iface eth1.231 inet manual
        up ifconfig eth1.231 up

#KVM bridge, vlan 231, via eth1
iface br231 inet static
        bridge_ports eth1.231
        bridge_fd 9
        bridge_hello 2
        bridge_maxage 12
        bridge_stp off

##vlan 229
#auto eth1.229
#iface eth1.229 inet manual
#       up ifconfig eth1.229 up

##KVM bridge, vlan 229, via eth1
#auto br229
#iface br229 inet manual
#       bridge_ports eth1.229
#       bridge_maxwait 5
#       bridge_fd 1
#       bridge_stp on

#
# !!!!! NOTE (MGRACE): This *is* used !!!!!
#
#No! Unused
auto br1
iface br1 inet manual
        bridge_ports eth1
        bridge_fd 9
        bridge_hello 2
        bridge_maxage 12
        bridge_stp off

#auto br2
#iface br2 inet manual
#        bridge_ports eth1
#        bridge_fd 9
#        bridge_hello 2
#        bridge_maxage 12
#        bridge_stp off

#auto br3
#iface br3 inet manual
#        bridge_ports eth1
#        bridge_fd 9
#        bridge_hello 2
#        bridge_maxage 12
#        bridge_stp off
I've scanned every log I can get my hands on, and have yet to find a bread crumb to follow =(. The Dell Technician mentioned this should potentially be as easy as changing the Hypervisor's MAC address, but I've been unable to figure out how to do that up until this point. Any help is greatly appreciated, and I'd be happy to provide any additional information that may prove beneficial.

Thanks,
-Snipe
 
Old 11-30-2013, 03:10 PM   #2
jefro
Guru
 
Registered: Mar 2008
Posts: 10,246

Rep: Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255
This has been here for a while but I don't know the answer really. Linux doesn't really fully swap to new hardware in all cases directly. In some cases it is quite difficult to get an image or hard drive to work correctly. Best practice is to do clean install still no matter how many people claim a swap should work and they do it all the time.

Now as to your issue. I only know qemu but kvm ought to be similar.

I have two concerns.

One is how kvm may have installed originally. So, I'd consider removing it and installing it again. Remember that the virtual machine virtual hard drives need to be identified and maybe saved along with the configuration file.

Second is how a client uses kvm. Bridges on a client are tied to the hosts hardware more directly than a nat connection. I'd consider looking at making your connection again in clients. Again I am not an expert, just trying to offer some clues.
 
Old 12-06-2013, 06:08 PM   #3
Sniperm4n
LQ Newbie
 
Registered: May 2012
Posts: 27

Original Poster
Rep: Reputation: Disabled
Thanks for the input Jefro. I may be speaking a little too soon, but it appears as though this was a horrible misdiagnosis on my part. After a significant amount of testing, a Dell technician and I have determined that a faulty power distribution block within the server was causing the unexpected reboots. The random rebooting worsened significantly on Monday (so much so it was getting stuck in reboot loops during POST, and I was able to cause the crashes by merely starting a 40GB SCP copy). Dell then decided to re-replace the motherboard, replace the PCI riser, replace the RAID controller, etc., and the problem still existed. We then noticed a bunch of PSU1 redundancy lost errors in the System Event Log, which was just enough of a bread crumb to follow. The system has been stable for 2 days with only 1 PSU installed, and I'm about to swap the other PSU in to see if the system remains stable. If it remains stable, the power distribution block is assuredly the source of all of my heartache. Dell will be replacing the PDB on Monday. I'll let everyone know if the PDB swap fixes the issue when I know more.

Thanks,
-Snipe
 
Old 12-07-2013, 08:18 PM   #4
jefro
Guru
 
Registered: Mar 2008
Posts: 10,246

Rep: Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255Reputation: 1255
Thanks for the update!
 
Old 12-10-2013, 07:01 PM   #5
Sniperm4n
LQ Newbie
 
Registered: May 2012
Posts: 27

Original Poster
Rep: Reputation: Disabled
A faulty power distribution block was indeed the culprit! The server is now happily running with 2 PSUs connected, and is no longer "randomly" rebooting =). My bad...
 
  


Reply

Tags
kvm, motherboard, networking, qemu, ubuntu


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
KVM/Xen HVM guests require SWAP ?? vzxen Linux - Virtualization and Cloud 5 11-02-2010 10:16 PM
[SOLVED] Qemu-kvm memory and swap problem Linux.tar.gz Linux - Virtualization and Cloud 4 09-17-2010 07:18 AM
Possibly broken Motherboard madsporkmurderer Linux - Hardware 2 05-06-2006 03:34 PM
motherboard sensors broken? jon_k Linux - Software 3 11-04-2004 07:06 PM
Ho hard to swap motherboard and keep the same kernel/build? bruce1271 Linux - Hardware 8 12-11-2003 04:30 PM


All times are GMT -5. The time now is 02:27 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration