LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Virtualization and Cloud
User Name
Password
Linux - Virtualization and Cloud This forum is for the discussion of all topics relating to Linux Virtualization and Linux Cloud platforms. Xen, KVM, OpenVZ, VirtualBox, VMware, Linux-VServer and all other Linux Virtualization platforms are welcome. OpenStack, CloudStack, ownCloud, Cloud Foundry, Eucalyptus, Nimbus, OpenNebula and all other Linux Cloud platforms are welcome. Note that questions relating solely to non-Linux OS's should be asked in the General forum.

Notices

Reply
 
Search this Thread
Old 07-17-2011, 05:05 PM   #1
Raptor354
LQ Newbie
 
Registered: Jul 2008
Distribution: Ubuntu 11.04, Debian Squeeze, CentOS 5.6
Posts: 17

Rep: Reputation: 1
Highly Available Xen Troubles


Hello all,

I've spent some time now trying to get a highly available Xen cluster running. I keep running into problems everywhere.

Perhaps the most difficult part for me is knowing exactly what is needed to make the whole thing work. I started with two reasonably new machines and put Debian Squeeze on them. Next, I installed the Xen Dom0 kernel (I think, but I'd like a way to check) and rebooted into it, and now
Code:
uname -a
says
Code:
Linux <lookieme1> 2.6.32-5-xen-amd64 #1 SMP Tue Jun 14 12:46:30 UTC 2011 x86_64 GNU/Linux
just like I'd expect.

Then, I synced a 1.5T partition across the machines (they're still syncing, but at least it works). Now I go to create and start a VM on one computer. The create works great, but the start fails when using the command
Code:
xm create SystemHA.cfg
saying
Code:
WARNING!  Can't find hypervisor information in sysfs!
Using config file "/etc/xen/SystemHA.cfg".
Error: Unable to connect to xend: No such file or directory. Is xend running?
Hmm, xend isn't running? Well ok, I'll just start it using
Code:
xend start
and it returns
Code:
WARNING!  Can't find hypervisor information in sysfs!
ERROR Internal error: Could not obtain handle on privileged command interface (2 = No such file or directory)
Traceback (most recent call last):
  File "/usr/lib/xen-default/bin/xend", line 38, in <module>
    from xen.xend.server import SrvDaemon
  File "/usr/lib/xen-4.0/lib/python/xen/xend/server/SrvDaemon.py", line 26, in <module>
    import relocate
  File "/usr/lib/xen-4.0/lib/python/xen/xend/server/relocate.py", line 28, in <module>
    from xen.xend import XendDomain
  File "/usr/lib/xen-4.0/lib/python/xen/xend/XendDomain.py", line 36, in <module>
    from xen.xend import XendOptions, XendCheckpoint, XendDomainInfo
  File "/usr/lib/xen-4.0/lib/python/xen/xend/XendCheckpoint.py", line 20, in <module>
    from xen.xend import balloon, sxp, image
  File "/usr/lib/xen-4.0/lib/python/xen/xend/image.py", line 46, in <module>
    xc = xen.lowlevel.xc.xc()
xen.lowlevel.xc.Error: (1, 'Internal error', 'Could not obtain handle on privileged command interface (2 = No such file or directory)')
Yipe! That stumped me. I looked around online all the articles and threads I found recommended making sure I was booted into the Xen Dom0 kernel.

Next, I'd like to try Remus and get it working.
If anybody has any suggestions or ideas or criticisms or complaints, I'd like to hear from you.

Thanks.
 
Old 07-17-2011, 08:20 PM   #2
Raptor354
LQ Newbie
 
Registered: Jul 2008
Distribution: Ubuntu 11.04, Debian Squeeze, CentOS 5.6
Posts: 17

Original Poster
Rep: Reputation: 1
Here's another piece to the puzzle.

As seen near the bottom of the page here, both servers output
Code:
# dmesg | grep -i paravirtualized
[    0.000000] Booting paravirtualized kernel on bare hardware
What does that mean? Why would it prevent xend from starting?
 
1 members found this post helpful.
Old 07-17-2011, 11:59 PM   #3
Raptor354
LQ Newbie
 
Registered: Jul 2008
Distribution: Ubuntu 11.04, Debian Squeeze, CentOS 5.6
Posts: 17

Original Poster
Rep: Reputation: 1
Progress Achieved

Sorry folks, but this is getting a little ridiculous. What could I do better to get more responses?

Personal issues aside, I've made progress and will document it here for all to see and hopefully illicit responses.

I skimmed this article and noticed something towards the bottom that caught my attention. Turns out grub isn't loading Xen properly. I'd encourage all who are having similar problems to check it out.

Ok, cool, dmesg now claims that I'm running a paravirtualized kernel on Xen. Great. Now I can't get xencommons to start. dmesg says things like
Code:
[  840.280458] INFO: task xend:2485 blocked for more than 120 seconds.
[  840.280473] xend          D 0000000000000000     0  2485   2483 0x00000000
[  840.280566]  [<ffffffff8100e629>] ? xen_force_evtchn_callback+0x9/0xa
[  840.280591]  [<ffffffff8100e579>] ? xen_set_pte_at+0xa2/0xc2
[  840.280641]  [<ffffffff811f573e>] ? xenbus_dev_request_and_reply+0x21/0x99
[  840.280654]  [<ffffffffa03bc5f4>] ? xenbus_file_write+0x196/0x515 [xenfs]
[  840.280698] INFO: task xenstore-read:2692 blocked for more than 120 seconds.
[  840.280712] xenstore-read D 0000000000000000     0  2692   2691 0x00000004
[  840.280791]  [<ffffffff8100e579>] ? xen_set_pte_at+0xa2/0xc2
[  840.280828]  [<ffffffff811f573e>] ? xenbus_dev_request_and_reply+0x21/0x99
[  840.280841]  [<ffffffffa03bc5f4>] ? xenbus_file_write+0x196/0x515 [xenfs]
but there are now things in /proc/xen. I'm thinking and hoping that there's some version incompatibility here.

I'm looking for ideas here, folks.
 
Old 07-18-2011, 04:13 AM   #4
Blue_Ice
Member
 
Registered: Jul 2006
Location: Belgium
Distribution: Debian, Fedora, CentOS, Windows
Posts: 352

Rep: Reputation: Disabled
To me it sounds like you are booting the wrong kernel.

Which packages did you install to get Xen up and running? If I remember correctly you need to install the following packages:
Code:
apt-get install xen-hypervisor-4.0-amd64 linux-image-xen-amd64
The kernel that you installed is probably for the VM itself.
 
Old 07-19-2011, 07:26 PM   #5
Raptor354
LQ Newbie
 
Registered: Jul 2008
Distribution: Ubuntu 11.04, Debian Squeeze, CentOS 5.6
Posts: 17

Original Poster
Rep: Reputation: 1
Thank you, kind person, for taking the time to read and respond.

Yes, that was the original problem, but now it seems as though everything about xencommons doesn't do anything. There is nothing in the logs about it that I can see, but when I try and run "/etc/init.d/xencommons start", it just sits around and does nothing. FYI, I haven't found anything on Google to help.
 
Old 07-20-2011, 12:49 AM   #6
Blue_Ice
Member
 
Registered: Jul 2006
Location: Belgium
Distribution: Debian, Fedora, CentOS, Windows
Posts: 352

Rep: Reputation: Disabled
You have to be a bit more specific. As far as I know xencommons just starts the processes required to use Xen. So is Xen working? And if not which parts aren't working?
 
Old 07-20-2011, 01:49 AM   #7
Raptor354
LQ Newbie
 
Registered: Jul 2008
Distribution: Ubuntu 11.04, Debian Squeeze, CentOS 5.6
Posts: 17

Original Poster
Rep: Reputation: 1
I apologize for my ignorance, but I'm not sure what to check when you ask "which parts aren't working". Perhaps a list of things to check would help.

Here's what I do know when I run a command such as "xm list", I get back this message:
Code:
Error: Unable to connect to xend: No such file or directory. Is xend running?
Then when I try and run "xend start", absolutely nothing happens. The console effectively locks up and I have to SSH in again.
 
Old 07-21-2011, 05:00 AM   #8
Blue_Ice
Member
 
Registered: Jul 2006
Location: Belgium
Distribution: Debian, Fedora, CentOS, Windows
Posts: 352

Rep: Reputation: Disabled
Are you sure that you are booting into the Grub option: 'Debian GNU/Linux, with Linux 2.6.32-5-xen-amd64 and XEN 4.0-amd64'? This is the kernel that contains the hypervisor.
 
Old 05-09-2012, 02:11 PM   #9
Adris
LQ Newbie
 
Registered: May 2012
Posts: 5

Rep: Reputation: Disabled
Question Similar issue

First, sorry about the crappy english, I speak spanish.

I've been trying to install a dom0 on a Phenom x6 1055T + Asus M5A97EVO, and have many of the problems you wrote about.
 
Old 05-09-2012, 02:12 PM   #10
Adris
LQ Newbie
 
Registered: May 2012
Posts: 5

Rep: Reputation: Disabled
Question Similar issue part2

Some says is the board:

http://ubuntuforums.org/archive/inde...t-1818433.html

Some says is GRUB:

http://xen-orchestra.com/2010/08/xen...queeze-how-to/

The point is, I tried all and now I`m stuck. I know itīs been several months since you wrote this, but, any clues you can hook me up?


Thanks in advance,

Adri
 
Old 05-09-2012, 02:40 PM   #11
Plam
LQ Newbie
 
Registered: May 2012
Posts: 3

Rep: Reputation: Disabled
Hi,

I'm the guy behind Xen orchestra. Can you put your Grub config here ? (/boot/grub/grub.cfg)
 
Old 05-09-2012, 08:59 PM   #12
Adris
LQ Newbie
 
Registered: May 2012
Posts: 5

Rep: Reputation: Disabled
Smile

Of Course,

Code:
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
 load_env
fi
set default="0"
if [ "${prev_saved_entry}" ]; then
 set saved_entry="${prev_saved_entry}"
 save_env saved_entry
 set prev_saved_entry=
 save_env prev_saved_entry
 set boot_once=true
fi

function savedefault {
 if [ -z "${boot_once}" ]; then
   saved_entry="${chosen}"
   save_env saved_entry
 fi
}

function load_video {
 insmod vbe
 insmod vga
 insmod video_bochs
 insmod video_cirrus
}

insmod part_gpt
insmod ext2
set root='(hd0,gpt3)'
search --no-floppy --fs-uuid --set 50122b6f-a10f-4409-8d82-9ea093a91a7d
if loadfont /usr/share/grub/unicode.pf2 ; then
 set gfxmode=640x480
 load_video
 insmod gfxterm
fi
terminal_output gfxterm
insmod part_gpt
insmod ext2
set root='(hd0,gpt1)'
search --no-floppy --fs-uuid --set 7b4f33a6-71b5-4141-b6ac-5570e7917065
set locale_dir=($root)/grub/locale
set lang=es
insmod gettext
set timeout=5
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/05_debian_theme ###
insmod part_gpt
insmod ext2
set root='(hd0,gpt3)'
search --no-floppy --fs-uuid --set 50122b6f-a10f-4409-8d82-9ea093a91a7d
insmod png
if background_image /usr/share/images/desktop-base/spacefun-grub.png; then
 set color_normal=light-gray/black
 set color_highlight=white/black
else
 set menu_color_normal=cyan/blue
 set menu_color_highlight=white/blue
fi
### END /etc/grub.d/05_debian_theme ###

### BEGIN /etc/grub.d/20_linux_xen ###
menuentry 'Debian GNU/Linux, with Linux 2.6.32-5-xen-amd64 and XEN
4.0-amd64' --class debian --class gnu-linux --class gnu --class os
--class xen {
       insmod part_gpt
       insmod ext2
       set root='(hd0,gpt1)'
       search --no-floppy --fs-uuid --set 7b4f33a6-71b5-4141-b6ac-5570e7917065
       echo    'Loading Linux 2.6.32-5-xen-amd64 ...'
       multiboot       /xen-4.0-amd64.gz placeholder
       module  /vmlinuz-2.6.32-5-xen-amd64 placeholder
root=UUID=50122b6f-a10f-4409-8d82-9ea093a91a7d ro  quiet
       echo    'Loading initial ramdisk ...'
       module  /initrd.img-2.6.32-5-xen-amd64
}
menuentry 'Debian GNU/Linux, with Linux 2.6.32-5-xen-amd64 and XEN
4.0-amd64 (recovery mode)' --class debian --class gnu-linux --class
gnu --class os --class xen {
       insmod part_gpt
       insmod ext2
       set root='(hd0,gpt1)'
       search --no-floppy --fs-uuid --set 7b4f33a6-71b5-4141-b6ac-5570e7917065
       echo    'Loading Linux 2.6.32-5-xen-amd64 ...'
       multiboot       /xen-4.0-amd64.gz placeholder
       module  /vmlinuz-2.6.32-5-xen-amd64 placeholder
root=UUID=50122b6f-a10f-4409-8d82-9ea093a91a7d ro single
       echo    'Loading initial ramdisk ...'
       module  /initrd.img-2.6.32-5-xen-amd64
}
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  $prefix/custom.cfg ]; then
 source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###

### BEGIN /etc/grub.d/50_linux ###
menuentry 'Debian GNU/Linux, with Linux 2.6.32-5-xen-amd64' --class
debian --class gnu-linux --class gnu --class os {
       insmod part_gpt
       insmod ext2
       set root='(hd0,gpt1)'
       search --no-floppy --fs-uuid --set 7b4f33a6-71b5-4141-b6ac-5570e7917065
       echo    'Loading Linux 2.6.32-5-xen-amd64 ...'
       linux   /vmlinuz-2.6.32-5-xen-amd64
root=UUID=50122b6f-a10f-4409-8d82-9ea093a91a7d ro  quiet
       echo    'Loading initial ramdisk ...'
       initrd  /initrd.img-2.6.32-5-xen-amd64
}
menuentry 'Debian GNU/Linux, with Linux 2.6.32-5-xen-amd64 (recovery
mode)' --class debian --class gnu-linux --class gnu --class os {
       insmod part_gpt
       insmod ext2
       set root='(hd0,gpt1)'
       search --no-floppy --fs-uuid --set 7b4f33a6-71b5-4141-b6ac-5570e7917065
       echo    'Loading Linux 2.6.32-5-xen-amd64 ...'
       linux   /vmlinuz-2.6.32-5-xen-amd64
root=UUID=50122b6f-a10f-4409-8d82-9ea093a91a7d ro single
       echo    'Loading initial ramdisk ...'
       initrd  /initrd.img-2.6.32-5-xen-amd64
}
menuentry 'Debian GNU/Linux, with Linux 2.6.32-5-amd64' --class debian
--class gnu-linux --class gnu --class os {
       insmod part_gpt
       insmod ext2
       set root='(hd0,gpt1)'
       search --no-floppy --fs-uuid --set 7b4f33a6-71b5-4141-b6ac-5570e7917065
       echo    'Loading Linux 2.6.32-5-amd64 ...'
       linux   /vmlinuz-2.6.32-5-amd64
root=UUID=50122b6f-a10f-4409-8d82-9ea093a91a7d ro  quiet
       echo    'Loading initial ramdisk ...'
       initrd  /initrd.img-2.6.32-5-amd64
}
menuentry 'Debian GNU/Linux, with Linux 2.6.32-5-amd64 (recovery
mode)' --class debian --class gnu-linux --class gnu --class os {
       insmod part_gpt
       insmod ext2
       set root='(hd0,gpt1)'
       search --no-floppy --fs-uuid --set 7b4f33a6-71b5-4141-b6ac-5570e7917065
       echo    'Loading Linux 2.6.32-5-amd64 ...'
       linux   /vmlinuz-2.6.32-5-amd64
root=UUID=50122b6f-a10f-4409-8d82-9ea093a91a7d ro single
       echo    'Loading initial ramdisk ...'
       initrd  /initrd.img-2.6.32-5-amd64
}
### END /etc/grub.d/50_linux ###



Itīs a new fresh install, since I play around to much with the last one. But itīs pretty much on the same place where I got stuck.

By the way... I mean no offense in the way I "quote" you.

Thanks for the fast answer.


Cheers!
 
Old 05-12-2012, 12:51 AM   #13
gr0undzer0
Member
 
Registered: Apr 2012
Location: Alexandria, VA
Distribution: Fedora
Posts: 46

Rep: Reputation: 1
Do any of the xm commands work? If none of them do then this is not a .cfg problem. I had the same problem with Fedora. The most important thing is to be sure that your running the Multi Kernel. The second thing is check for your version of Ubuntu and make sure you don't need any special patches to that the kernel. Also make sure virtualization is turned on in your bios. vt-x must be turned on in the bios.

So in review.
1. Make sure your booting as a multi-kernel
2. Make sure you don't need any special patches for your linux kernel / distro I did for fedora.
3. Make sure virtualization is on in your BIOS and that your processor supports it.
 
Old 05-14-2012, 02:53 AM   #14
Plam
LQ Newbie
 
Registered: May 2012
Posts: 3

Rep: Reputation: Disabled
Adris, your Grub config seems OK for me.

Does 'xm info' return anything ? if yes, please post it here.
 
Old 05-16-2012, 11:26 AM   #15
Adris
LQ Newbie
 
Registered: May 2012
Posts: 5

Rep: Reputation: Disabled
Unhappy

Quote:
Originally Posted by gr0undzer0 View Post
So in review.
1. Make sure your booting as a multi-kernel
2. Make sure you don't need any special patches for your linux kernel / distro I did for fedora.
3. Make sure virtualization is on in your BIOS and that your processor supports it.

1. Not completely sure what you mean by multi-kernel
2. As for dom0 Debian squeeze should be ok.
3. Iīm playing around with the BIOS limitation of my ASUS M5A97 EVO. Which means, while I have IOMMU ON, I canīt use Ethernet or USB. So, every time I have to download something, reboot turn IOMMU support OFF, download, reboot, turn IOMMU support ON, and install/build.


Quote:
Originally Posted by Plam View Post
Adris, your Grub config seems OK for me.

Does 'xm info' return anything ? if yes, please post it here.
Yes, it did. Something like Raptor354 wrote about:

Code:
WARNING!  Can't find hypervisor information in sysfs!

But I donīt have that install anymore. I was kind of busy last days, i tried a fresh netinstall (the new 6.0.5 of debian) and build everything from source: XEN 4.2 (unstable) and kernel 3.3.6
following the wiki guidelines about required packages.

Shamefully it didnīt work out. Maybe because my lack of experience building kernels :P.
Ended up with several kernels to boot in grub, none of them finish the boot process and some even reboot the system.

Thatīs where i am now. If I canīt go further down this path I will start all over after some bios updating to the Beta Version. Other owners of Asus (and MSI) Mobos have similar problems related to IOMMU on desktop linux versions.



Thanks for you answers guys.

Last edited by Adris; 05-16-2012 at 01:42 PM. Reason: Quote
 
  


Reply

Tags
asus, drbd, failover, kernel, squeeze, xen


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Backport Intrepid Xen 3.3 Hypervisor to Ubuntu Hardy Dom0 (2.6.24-21-xen) LXer Syndicated Linux News 0 11-13-2008 06:11 PM
LXer: Install, Configure, Manage and Administer Xen Servers with New book on Xen LXer Syndicated Linux News 0 01-11-2008 11:41 PM
LXer: Managing Xen With Xen-Tools, Xen-Shell, And Argo LXer Syndicated Linux News 0 11-05-2006 12:21 PM
LXer: Managing Xen With Xen-Tools, Xen-Shell, And Argo LXer Syndicated Linux News 0 10-21-2006 11:33 PM


All times are GMT -5. The time now is 12:15 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration