Highly Available Xen Troubles

Raptor354 · 07-17-2011, 05:05 PM

Hello all,

I've spent some time now trying to get a highly available Xen cluster running. I keep running into problems everywhere.

Perhaps the most difficult part for me is knowing exactly what is needed to make the whole thing work. I started with two reasonably new machines and put Debian Squeeze on them. Next, I installed the Xen Dom0 kernel (I think, but I'd like a way to check) and rebooted into it, and now

Code:

uname -a

says

Code:

Linux <lookieme1> 2.6.32-5-xen-amd64 #1 SMP Tue Jun 14 12:46:30 UTC 2011 x86_64 GNU/Linux

just like I'd expect.

Then, I synced a 1.5T partition across the machines (they're still syncing, but at least it works). Now I go to create and start a VM on one computer. The create works great, but the start fails when using the command

Code:

xm create SystemHA.cfg

saying

Code:

WARNING!  Can't find hypervisor information in sysfs!
Using config file "/etc/xen/SystemHA.cfg".
Error: Unable to connect to xend: No such file or directory. Is xend running?

Hmm, xend isn't running? Well ok, I'll just start it using

Code:

xend start

and it returns

Code:

WARNING!  Can't find hypervisor information in sysfs!
ERROR Internal error: Could not obtain handle on privileged command interface (2 = No such file or directory)
Traceback (most recent call last):
  File "/usr/lib/xen-default/bin/xend", line 38, in <module>
    from xen.xend.server import SrvDaemon
  File "/usr/lib/xen-4.0/lib/python/xen/xend/server/SrvDaemon.py", line 26, in <module>
    import relocate
  File "/usr/lib/xen-4.0/lib/python/xen/xend/server/relocate.py", line 28, in <module>
    from xen.xend import XendDomain
  File "/usr/lib/xen-4.0/lib/python/xen/xend/XendDomain.py", line 36, in <module>
    from xen.xend import XendOptions, XendCheckpoint, XendDomainInfo
  File "/usr/lib/xen-4.0/lib/python/xen/xend/XendCheckpoint.py", line 20, in <module>
    from xen.xend import balloon, sxp, image
  File "/usr/lib/xen-4.0/lib/python/xen/xend/image.py", line 46, in <module>
    xc = xen.lowlevel.xc.xc()
xen.lowlevel.xc.Error: (1, 'Internal error', 'Could not obtain handle on privileged command interface (2 = No such file or directory)')

Yipe! That stumped me. I looked around online all the articles and threads I found recommended making sure I was booted into the Xen Dom0 kernel.

Next, I'd like to try Remus and get it working.
If anybody has any suggestions or ideas or criticisms or complaints, I'd like to hear from you.

Thanks.

Raptor354 · 07-17-2011, 08:20 PM

Here's another piece to the puzzle.

As seen near the bottom of the page here, both servers output

Code:

# dmesg | grep -i paravirtualized
[    0.000000] Booting paravirtualized kernel on bare hardware

What does that mean? Why would it prevent xend from starting?

Raptor354 · 07-17-2011, 11:59 PM

Sorry folks, but this is getting a little ridiculous. What could I do better to get more responses?

Personal issues aside, I've made progress and will document it here for all to see and hopefully illicit responses.

I skimmed this article and noticed something towards the bottom that caught my attention. Turns out grub isn't loading Xen properly. I'd encourage all who are having similar problems to check it out.

Ok, cool, dmesg now claims that I'm running a paravirtualized kernel on Xen. Great. Now I can't get xencommons to start. dmesg says things like

Code:

[  840.280458] INFO: task xend:2485 blocked for more than 120 seconds.
[  840.280473] xend          D 0000000000000000     0  2485   2483 0x00000000
[  840.280566]  [<ffffffff8100e629>] ? xen_force_evtchn_callback+0x9/0xa
[  840.280591]  [<ffffffff8100e579>] ? xen_set_pte_at+0xa2/0xc2
[  840.280641]  [<ffffffff811f573e>] ? xenbus_dev_request_and_reply+0x21/0x99
[  840.280654]  [<ffffffffa03bc5f4>] ? xenbus_file_write+0x196/0x515 [xenfs]
[  840.280698] INFO: task xenstore-read:2692 blocked for more than 120 seconds.
[  840.280712] xenstore-read D 0000000000000000     0  2692   2691 0x00000004
[  840.280791]  [<ffffffff8100e579>] ? xen_set_pte_at+0xa2/0xc2
[  840.280828]  [<ffffffff811f573e>] ? xenbus_dev_request_and_reply+0x21/0x99
[  840.280841]  [<ffffffffa03bc5f4>] ? xenbus_file_write+0x196/0x515 [xenfs]

but there are now things in /proc/xen. I'm thinking and hoping that there's some version incompatibility here.

I'm looking for ideas here, folks.

Blue_Ice · 07-18-2011, 04:13 AM

To me it sounds like you are booting the wrong kernel.

Which packages did you install to get Xen up and running? If I remember correctly you need to install the following packages:

Code:

apt-get install xen-hypervisor-4.0-amd64 linux-image-xen-amd64

The kernel that you installed is probably for the VM itself.

Raptor354 · 07-19-2011, 07:26 PM

Thank you, kind person, for taking the time to read and respond.

Yes, that was the original problem, but now it seems as though everything about xencommons doesn't do anything. There is nothing in the logs about it that I can see, but when I try and run "/etc/init.d/xencommons start", it just sits around and does nothing. FYI, I haven't found anything on Google to help.

Blue_Ice · 07-20-2011, 12:49 AM

You have to be a bit more specific. As far as I know xencommons just starts the processes required to use Xen. So is Xen working? And if not which parts aren't working?

Raptor354 · 07-20-2011, 01:49 AM

I apologize for my ignorance, but I'm not sure what to check when you ask "which parts aren't working". Perhaps a list of things to check would help.

Here's what I do know when I run a command such as "xm list", I get back this message:

Code:

Error: Unable to connect to xend: No such file or directory. Is xend running?

Then when I try and run "xend start", absolutely nothing happens. The console effectively locks up and I have to SSH in again.

Blue_Ice · 07-21-2011, 05:00 AM

Are you sure that you are booting into the Grub option: 'Debian GNU/Linux, with Linux 2.6.32-5-xen-amd64 and XEN 4.0-amd64'? This is the kernel that contains the hypervisor.

Adris · 05-09-2012, 02:11 PM

First, sorry about the crappy english, I speak spanish.

I've been trying to install a dom0 on a Phenom x6 1055T + Asus M5A97EVO, and have many of the problems you wrote about.

Adris · 05-09-2012, 02:12 PM

Some says is the board:

http://ubuntuforums.org/archive/inde...t-1818433.html

Some says is GRUB:

http://xen-orchestra.com/2010/08/xen...queeze-how-to/

The point is, I tried all and now I`m stuck. I know it´s been several months since you wrote this, but, any clues you can hook me up?

Thanks in advance,

Adri

Plam · 05-09-2012, 02:40 PM

Hi,

I'm the guy behind Xen orchestra. Can you put your Grub config here ? (/boot/grub/grub.cfg)

Adris · 05-09-2012, 08:59 PM

Of Course,

Code:

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
 load_env
fi
set default="0"
if [ "${prev_saved_entry}" ]; then
 set saved_entry="${prev_saved_entry}"
 save_env saved_entry
 set prev_saved_entry=
 save_env prev_saved_entry
 set boot_once=true
fi

function savedefault {
 if [ -z "${boot_once}" ]; then
   saved_entry="${chosen}"
   save_env saved_entry
 fi
}

function load_video {
 insmod vbe
 insmod vga
 insmod video_bochs
 insmod video_cirrus
}

insmod part_gpt
insmod ext2
set root='(hd0,gpt3)'
search --no-floppy --fs-uuid --set 50122b6f-a10f-4409-8d82-9ea093a91a7d
if loadfont /usr/share/grub/unicode.pf2 ; then
 set gfxmode=640x480
 load_video
 insmod gfxterm
fi
terminal_output gfxterm
insmod part_gpt
insmod ext2
set root='(hd0,gpt1)'
search --no-floppy --fs-uuid --set 7b4f33a6-71b5-4141-b6ac-5570e7917065
set locale_dir=($root)/grub/locale
set lang=es
insmod gettext
set timeout=5
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/05_debian_theme ###
insmod part_gpt
insmod ext2
set root='(hd0,gpt3)'
search --no-floppy --fs-uuid --set 50122b6f-a10f-4409-8d82-9ea093a91a7d
insmod png
if background_image /usr/share/images/desktop-base/spacefun-grub.png; then
 set color_normal=light-gray/black
 set color_highlight=white/black
else
 set menu_color_normal=cyan/blue
 set menu_color_highlight=white/blue
fi
### END /etc/grub.d/05_debian_theme ###

### BEGIN /etc/grub.d/20_linux_xen ###
menuentry 'Debian GNU/Linux, with Linux 2.6.32-5-xen-amd64 and XEN
4.0-amd64' --class debian --class gnu-linux --class gnu --class os
--class xen {
       insmod part_gpt
       insmod ext2
       set root='(hd0,gpt1)'
       search --no-floppy --fs-uuid --set 7b4f33a6-71b5-4141-b6ac-5570e7917065
       echo    'Loading Linux 2.6.32-5-xen-amd64 ...'
       multiboot       /xen-4.0-amd64.gz placeholder
       module  /vmlinuz-2.6.32-5-xen-amd64 placeholder
root=UUID=50122b6f-a10f-4409-8d82-9ea093a91a7d ro  quiet
       echo    'Loading initial ramdisk ...'
       module  /initrd.img-2.6.32-5-xen-amd64
}
menuentry 'Debian GNU/Linux, with Linux 2.6.32-5-xen-amd64 and XEN
4.0-amd64 (recovery mode)' --class debian --class gnu-linux --class
gnu --class os --class xen {
       insmod part_gpt
       insmod ext2
       set root='(hd0,gpt1)'
       search --no-floppy --fs-uuid --set 7b4f33a6-71b5-4141-b6ac-5570e7917065
       echo    'Loading Linux 2.6.32-5-xen-amd64 ...'
       multiboot       /xen-4.0-amd64.gz placeholder
       module  /vmlinuz-2.6.32-5-xen-amd64 placeholder
root=UUID=50122b6f-a10f-4409-8d82-9ea093a91a7d ro single
       echo    'Loading initial ramdisk ...'
       module  /initrd.img-2.6.32-5-xen-amd64
}
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  $prefix/custom.cfg ]; then
 source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###

### BEGIN /etc/grub.d/50_linux ###
menuentry 'Debian GNU/Linux, with Linux 2.6.32-5-xen-amd64' --class
debian --class gnu-linux --class gnu --class os {
       insmod part_gpt
       insmod ext2
       set root='(hd0,gpt1)'
       search --no-floppy --fs-uuid --set 7b4f33a6-71b5-4141-b6ac-5570e7917065
       echo    'Loading Linux 2.6.32-5-xen-amd64 ...'
       linux   /vmlinuz-2.6.32-5-xen-amd64
root=UUID=50122b6f-a10f-4409-8d82-9ea093a91a7d ro  quiet
       echo    'Loading initial ramdisk ...'
       initrd  /initrd.img-2.6.32-5-xen-amd64
}
menuentry 'Debian GNU/Linux, with Linux 2.6.32-5-xen-amd64 (recovery
mode)' --class debian --class gnu-linux --class gnu --class os {
       insmod part_gpt
       insmod ext2
       set root='(hd0,gpt1)'
       search --no-floppy --fs-uuid --set 7b4f33a6-71b5-4141-b6ac-5570e7917065
       echo    'Loading Linux 2.6.32-5-xen-amd64 ...'
       linux   /vmlinuz-2.6.32-5-xen-amd64
root=UUID=50122b6f-a10f-4409-8d82-9ea093a91a7d ro single
       echo    'Loading initial ramdisk ...'
       initrd  /initrd.img-2.6.32-5-xen-amd64
}
menuentry 'Debian GNU/Linux, with Linux 2.6.32-5-amd64' --class debian
--class gnu-linux --class gnu --class os {
       insmod part_gpt
       insmod ext2
       set root='(hd0,gpt1)'
       search --no-floppy --fs-uuid --set 7b4f33a6-71b5-4141-b6ac-5570e7917065
       echo    'Loading Linux 2.6.32-5-amd64 ...'
       linux   /vmlinuz-2.6.32-5-amd64
root=UUID=50122b6f-a10f-4409-8d82-9ea093a91a7d ro  quiet
       echo    'Loading initial ramdisk ...'
       initrd  /initrd.img-2.6.32-5-amd64
}
menuentry 'Debian GNU/Linux, with Linux 2.6.32-5-amd64 (recovery
mode)' --class debian --class gnu-linux --class gnu --class os {
       insmod part_gpt
       insmod ext2
       set root='(hd0,gpt1)'
       search --no-floppy --fs-uuid --set 7b4f33a6-71b5-4141-b6ac-5570e7917065
       echo    'Loading Linux 2.6.32-5-amd64 ...'
       linux   /vmlinuz-2.6.32-5-amd64
root=UUID=50122b6f-a10f-4409-8d82-9ea093a91a7d ro single
       echo    'Loading initial ramdisk ...'
       initrd  /initrd.img-2.6.32-5-amd64
}
### END /etc/grub.d/50_linux ###

It´s a new fresh install, since I play around to much with the last one. But it´s pretty much on the same place where I got stuck.

By the way... I mean no offense in the way I "quote" you.

Thanks for the fast answer.

Cheers!

gr0undzer0 · 05-12-2012, 12:51 AM

Do any of the xm commands work? If none of them do then this is not a .cfg problem. I had the same problem with Fedora. The most important thing is to be sure that your running the Multi Kernel. The second thing is check for your version of Ubuntu and make sure you don't need any special patches to that the kernel. Also make sure virtualization is turned on in your bios. vt-x must be turned on in the bios.

So in review.
1. Make sure your booting as a multi-kernel
2. Make sure you don't need any special patches for your linux kernel / distro I did for fedora.
3. Make sure virtualization is on in your BIOS and that your processor supports it.

Plam · 05-14-2012, 02:53 AM

Adris, your Grub config seems OK for me.

Does 'xm info' return anything ? if yes, please post it here.

Adris · 05-16-2012, 11:26 AM

Quote:

Originally Posted by gr0undzer0

So in review.
1. Make sure your booting as a multi-kernel
2. Make sure you don't need any special patches for your linux kernel / distro I did for fedora.
3. Make sure virtualization is on in your BIOS and that your processor supports it.

1. Not completely sure what you mean by multi-kernel
2. As for dom0 Debian squeeze should be ok.
3. I´m playing around with the BIOS limitation of my ASUS M5A97 EVO. Which means, while I have IOMMU ON, I can´t use Ethernet or USB. So, every time I have to download something, reboot turn IOMMU support OFF, download, reboot, turn IOMMU support ON, and install/build.

Quote:

Originally Posted by Plam

Adris, your Grub config seems OK for me.

Does 'xm info' return anything ? if yes, please post it here.

Yes, it did. Something like Raptor354 wrote about:

Code:

WARNING!  Can't find hypervisor information in sysfs!

But I don´t have that install anymore. I was kind of busy last days, i tried a fresh netinstall (the new 6.0.5 of debian) and build everything from source: XEN 4.2 (unstable) and kernel 3.3.6
following the wiki guidelines about required packages.

Shamefully it didn´t work out. Maybe because my lack of experience building kernels :P.
Ended up with several kernels to boot in grub, none of them finish the boot process and some even reboot the system.

That´s where i am now. If I can´t go further down this path I will start all over after some bios updating to the Beta Version. Other owners of Asus (and MSI) Mobos have similar problems related to IOMMU on desktop linux versions.

Thanks for you answers guys.