LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Virtualization and Cloud
User Name
Password
Linux - Virtualization and Cloud This forum is for the discussion of all topics relating to Linux Virtualization and Linux Cloud platforms. Xen, KVM, OpenVZ, VirtualBox, VMware, Linux-VServer and all other Linux Virtualization platforms are welcome. OpenStack, CloudStack, ownCloud, Cloud Foundry, Eucalyptus, Nimbus, OpenNebula and all other Linux Cloud platforms are welcome. Note that questions relating solely to non-Linux OS's should be asked in the General forum.

Notices


Reply
  Search this Thread
Old 07-24-2010, 07:32 AM   #1
lxf
LQ Newbie
 
Registered: Jul 2010
Posts: 22

Rep: Reputation: 2
Run multiple Xen Dom0s on same machine


Hi,
is it possible to run multiple Xen Dom0s on a single physical machine, e.g. by stacking up Xen within a Xen DomU? Yes, I know you will doubt how this could be useful: It's not for a productive environment, but I wanted to use this to simulate a Xen cluster.

In particular I want to start a full Xen hypervisor + Dom0 kernel as DomU. When using pygrub I'm able to see a grub boot screen, however pygrub is not able to boot the hypervisor:

Code:
    pyGRUB  version 0.6
 ┌────────────────────────────────────────────────────────────────────────┐
 │ Debian GNU/Linux, with Linux 2.6.32-5-xen-amd64                        │
 │ Debian GNU/Linux, with Linux 2.6.32-5-amd64                            │
 │ Debian GNU/Linux, with Linux 2.6.32-5-xen-amd64 and XEN 4.0-amd64      │
 │                                                                        │
 │                                                                        │
 │                                                                        │
 │                                                                        │
 │                                                                        │
 └────────────────────────────────────────────────────────────────────────┘
     Use the ^ and v keys to select which entry is highlighted.
     Press enter to boot the selected OS, 'e' to edit the
     commands before booting, 'a' to modify the kernel arguments
     before booting, or 'c' for a command line.




     Will boot selected entry in  1 seconds


Traceback (most recent call last):
File "/usr/lib/xen-4.0/bin/pygrub", line 718, in <module>
chosencfg = run_grub(file, entry, fs, incfg["args"])
File "/usr/lib/xen-4.0/bin/pygrub", line 576, in run_grub
grubcfg["kernel"] = img.kernel[1]
TypeError: 'NoneType' object is unsubscriptable

debian:/etc/xen# Error: Boot loader didn't return any data!
Alternatively I also tried a PV configuration by providing the kernel manually.

Code:
 kernel = "/boot/xen-4.0-amd64.gz"
ramdisk = "/boot/initrd.img-2.6.32-5-xen-amd64"
extra = "/boot/vmlinuz-2.6.32-5-xen-amd64"
but this brings:

Code:
Error: (2, 'Invalid kernel', "elf_xen_note_check: ERROR: Not a Xen-ELF image: No ELF notes or '__xen_guest' section found.\n")
Is it simply not possible to boot a PV (HVM might possible I guess) hypervisor as DomU?


It would be ok if I could run such a cascaded virtualization by using another technique instead (e.g. KVM virtualizing Xen). Does anybody know what is possible here?

Last edited by lxf; 07-24-2010 at 08:14 AM.
 
Old 07-25-2010, 12:41 AM   #2
tkmsr
Member
 
Registered: Oct 2006
Distribution: Ubuntu,Open Suse,Debian,Mac OS X
Posts: 798

Rep: Reputation: 39
Quote:
Originally Posted by lxf View Post
Hi,
is it possible to run multiple Xen Dom0s on a single physical machine, e.g. by stacking up Xen within a Xen DomU?
Yes
Quote:
Originally Posted by lxf View Post
Yes, I know you will doubt
No
Quote:
Originally Posted by lxf View Post
Traceback (most recent call last):
File "/usr/lib/xen-4.0/bin/pygrub", line 718, in <module>
chosencfg = run_grub(file, entry, fs, incfg["args"])
File "/usr/lib/xen-4.0/bin/pygrub", line 576, in run_grub
grubcfg["kernel"] = img.kernel[1]
TypeError: 'NoneType' object is unsubscriptable

debian:/etc/xen# Error: Boot loader didn't return any data!
[/code]
This is a different error all together and could result because of corrupt xml files.

Quote:
Originally Posted by lxf View Post
Is it simply not possible to boot a PV (HVM might possible I guess) hypervisor as DomU?
It is possible.

Quote:
Originally Posted by lxf View Post
It would be ok if I could run such a cascaded virtualization by using another technique instead (e.g. KVM virtualizing Xen). Does anybody know what is possible here?
What exactly you want to do I am not getting.You want a Xen or KVM sort of thing running inside a Guest Operating System for your experiment.
If that is what you are trying.I will suggest you install CentOS.
CentOS has a nice GUI for installing Guest OS on Xen.
In fact if what ever OS you are using you install virt-manager
You will be able to do so
check
http://www.novell.com/communities/no...r-opensuse-103

http://www.centos.org/docs/5/html/5....zed_guest.html

The following video
has mentioned in case of OpenSuse but is applicable to other Linux also.Use a GUI your life will be simple.

Check the following page
Pygrub has its own issues.

Last edited by tkmsr; 07-25-2010 at 12:43 AM.
 
Old 07-26-2010, 03:11 AM   #3
lxf
LQ Newbie
 
Registered: Jul 2010
Posts: 22

Original Poster
Rep: Reputation: 2
First, thank you for your answer. You didn't help me though. See, I don't want to install any kind of a virtual machine, I know how to install them and I did that for many times.

The problem is, I don't want to install any given virtual machine that is supported by Xen, I want to install a Xen hypervisor as virtual machine. However, I can't specify the Xen hypervisor as kernel image to be booted by Xen - at least not that way as I tried, as Xen refuses to boot it (neither via pygrub, nor direct via "kernel=" configuration in a DomU configuration).
 
Old 07-26-2010, 04:00 AM   #4
tkmsr
Member
 
Registered: Oct 2006
Distribution: Ubuntu,Open Suse,Debian,Mac OS X
Posts: 798

Rep: Reputation: 39
You want to put more than one instance of Xen on Linux system.
 
Old 07-26-2010, 09:34 AM   #5
lxf
LQ Newbie
 
Registered: Jul 2010
Posts: 22

Original Poster
Rep: Reputation: 2
Right. Doesn't work as pure Xen solution, but I figured out to get this working by using VMware ESXi + Xen.
 
Old 07-26-2010, 12:39 PM   #6
tkmsr
Member
 
Registered: Oct 2006
Distribution: Ubuntu,Open Suse,Debian,Mac OS X
Posts: 798

Rep: Reputation: 39
can u post how u did ?
 
Old 07-26-2010, 01:30 PM   #7
lxf
LQ Newbie
 
Registered: Jul 2010
Posts: 22

Original Poster
Rep: Reputation: 2
Well, not that complicated. I installed an evaluation copy of VMWare ESXi (http://www.vmware.com/products/vsphere-hypervisor/) as baremetal hypervisor. Within that I installed two virtual machines (Debian Squeeze) and proceeded as usual by installing a Xen hypervisor (2.6.32 Dom0 + Xen 4.0). Boots up just fine, but sticks sometimes in the boot process though, so I wouldn't consider this as stable productive solution. It works for testing purposes though.
 
Old 07-26-2010, 08:29 PM   #8
tkmsr
Member
 
Registered: Oct 2006
Distribution: Ubuntu,Open Suse,Debian,Mac OS X
Posts: 798

Rep: Reputation: 39
But you wanted to use multiple instances of Xen in same Operating system.i.e in same Virtual Machine if you did in two virtual machine it is entirely different from what u asked.
 
Old 07-27-2010, 12:13 PM   #9
lxf
LQ Newbie
 
Registered: Jul 2010
Posts: 22

Original Poster
Rep: Reputation: 2
No, I want(ed) to run two hypervisors on the same physical machine, on different domain borders though (thus on different virtual machines).

When speaking about Xen, you should clarify what you mean by "operating system". To illustrate what I mean:




By the way: Using VMWare turned out to have some clues, works as said, but is very unstable.
 
Old 07-27-2010, 12:26 PM   #10
tkmsr
Member
 
Registered: Oct 2006
Distribution: Ubuntu,Open Suse,Debian,Mac OS X
Posts: 798

Rep: Reputation: 39
Ok now the things are clear.
As far as my understanding is then at the same instance of Dom0 you can not use multiple instance of Xen without rebooting.

In above figure you are running Dom0 in virtual machine which itself is running on VMware and then installing Guest which will be Dom0 then on that (Tier 2 Dom0 ) you have installed Xen
and if I understand correctly from the above figure you have been able to run multiple Dom0 (which are guest of Vmware) in which Xen is running and then On them you are running Guest.

You will not be able to do so the way you have done on Vmware.
What I mean to say is on a physical machine with nothing but a Linux installed you need to have different kernels from which you can boot.
say
vmlinuz-1
vmlinuz-2
vmlinuz-3

all the above
vmlinuz-1
vmlinuz-2
vmlinuz-3
can be of same version but you wont be able to have a file of same name in /boot

Then each of these kernel of Dom0 in your /boot needs to be xenified.Meaning it needs to be made aware that Xen will run on it.Some vendors who ship pre compiled binary have already done this for you so you do not know that.
If you do it yourself you need to add patches to be able to run Xen for any kernel above 2.6.18
(Novell has forward ported the patches for Xen in existing vanilla kernel)

Then you will be able to use different instances of hypervisor xen on Dom0
but then each time you will have to reboot Dom0 and use the different version(or same also you can use saying this is a and b and c and so on) of Xen and at one time only one hypervisor will be active in Dom0.

Say if you want to use Vmware more than one time in your Windows/Linux and apply the same logic to Xen.
At one time you can run only one instance of Vmware.Or you need to boot to a different OS.
Not that running a VMware instance inside a guest which itself is running on Vmware.

You may use same version of kernel on Dom0 more than one time and boot different Xen instances.
But you obviously do not want to do that and this is not what you did as in Vmware.So to answer your question what you asked is not possible the way you wanted to use.
If you feel I am wrong do post a solution here I would like to read.

By Xen cluster I think you are trying to have an environment for your experiment which will be equivalent to multiple physical machines running Xen on them (with Guest OS in each of them )and one powerful Xen machine as a front end to all of them as one cluster.
I will suggest you put this question here
http://news.gmane.org/gmane.comp.emulators.xen.user
with a reference to this thread on LQ.

What Linux OS you are using?

Last edited by tkmsr; 07-27-2010 at 12:40 PM.
 
Old 07-27-2010, 03:48 PM   #11
lxf
LQ Newbie
 
Registered: Jul 2010
Posts: 22

Original Poster
Rep: Reputation: 2
Thank you for your very detailed post

Quote:
Originally Posted by tkmsr View Post
Ok now the things are clear.
As far as my understanding is then at the same instance of Dom0 you can not use multiple instance of Xen without rebooting.
Right. Not the same instance in terms of a shared filesystem as this would require a concurrent access to the same hard disk image. This is not my intention though, as, referring to my figure, both Tier 2 Dom0s wouldn't share a filesystem, just the same host operating system.

Quote:
In above figure you are running Dom0 in virtual machine which itself is running on VMware and then installing Guest which will be Dom0 then on that (Tier 2 Dom0 ) you have installed Xen
and if I understand correctly from the above figure you have been able to run multiple Dom0 (which are guest of Vmware) in which Xen is running and then On them you are running Guest.
Somehow a very nested sentence, but as far as my understanding goes you are right. By now I managed to got the intended set-up from above up and running by using VMWare in place of the "Tier 1" hypervisor. I would like, however, replace this VMWare machine (which is - as said - just a evaluation copy limited in time) by using Xen. It could (didn't try though) be possible to do this by running the Tier 2 Dom0s as full virtualized HVM machines, but for performance reasons I would like to PV those machines.

Quote:
You will not be able to do so the way you have done on Vmware.
What I mean to say is on a physical machine with nothing but a Linux installed you need to have different kernels from which you can boot.
say
vmlinuz-1
vmlinuz-2
vmlinuz-3

all the above
vmlinuz-1
vmlinuz-2
vmlinuz-3
can be of same version but you wont be able to have a file of same name in /boot

Then each of these kernel of Dom0 in your /boot needs to be xenified.Meaning it needs to be made aware that Xen will run on it.Some vendors who ship pre compiled binary have already done this for you so you do not know that.
If you do it yourself you need to add patches to be able to run Xen for any kernel above 2.6.18
(Novell has forward ported the patches for Xen in existing vanilla kernel)
So far, everything is correct what you are saying. It doesn't matter for my purpose, however. What you mean by "xenifying" means to support DomU-PV operation. This is fully supported in every vanilla kernel from 2.6.27 and above. Note, that Dom0 kernel support is a totally different thing, to which you may be referring too, when speaking about the Novell patches (this is nowadays called old style Xen support). For Xen 4.0 (and 3.4 as backport) exist new pv_ops Dom0 patches, which are supposed to be included into mainline kernel sooner or later (or never - depending on the kernel maintainers).

Those new pv_ops Dom0 kernels are much less intrusive than older patches (e.g. the Novell patch/old style 2.6.18 Dom0 support) and boot up fine, even as PV kernel in a Xen Dom0, but a kernel with Dom0 support alone doesn't make a full hypervisor. This is, where my problem arises as Xen refuses to boot itself as hypervisor in a DomU.

Quote:
Then you will be able to use different instances of hypervisor xen on Dom0
but then each time you will have to reboot Dom0 and use the different version(or same also you can use saying this is a and b and c and so on) of Xen and at one time only one hypervisor will be active in Dom0.
Yes. Agreed.

Quote:
Say if you want to use Vmware more than one time in your Windows/Linux and apply the same logic to Xen.
At one time you can run only one instance of Vmware.Or you need to boot to a different OS.
Not that running a VMware instance inside a guest which itself is running on Vmware.
Sure you got the difference from VMWare ESXi and VMWare Workstation/Server products? One is a (Linux based by the way) baremetal hypervisor, which doesn't require a host operating system by itself. VMware is the only OS in that case. In other Workstation/Server products you still require a host operating system and you'll run VMware as an application on that host.

Quote:
You may use same version of kernel on Dom0 more than one time and boot different Xen instances.
But you obviously do not want to do that and this is not what you did as in Vmware.
I don't have to care at all by the way. The kernel, in Xen, is booted by the Dom0, not by the DomU (even when pygrub suggests that - but actually pygrub just copies the kernel image specified in the client grub/menu.lst to the Dom0 host). And: By the way, usually you boot up the very same physical kernel image on each virtual machine which can be indeed be the very same physical file as your Dom0 is running. This is the default behaviour for Debian's xen-tool script for example.


I am, to repeat myself, not talking about the Dom0 kernel. This boots up very fine within a virtual DomU machine. But if you look on the architecture of Xen you will notice that - for virtualization support - you actually boot a hypervisor (called xen - not to be confused with the technique having the same name this time). And afterwards this hypervisor kernel boots up a Linux kernel image with Dom0 support. If you don't believe me, look on your grub configuration. You will notice, that you boot a xen hypervisor kernel which then boots a Dom0 kernel as "module".
This Dom0 kernel alone (i.e. without this hypervisor) is nowadays able to behave like every other kernel image too, you wouldn't even notice, that you actually booted up a Dom0-supported kernel.

This is, where my problem comes up. As said, it is no problem to boot this Dom0 support kernel itself as DomU kernel for a virtual machine. But that alone doesn't make a full hypervisor - it's just one half. If I want to actually start virtual machines, I must not boot that Dom0 kernel, but the xen-hypervisor which then, by itself boots this Dom0 kernel as explained before. This is exactly where I failed.

Furthermore I'd like to point out that this architecture of two independent parts is the main reason why Linux kernel developers refuse to accept Dom0 patches into mainline vanilla kernels. This would mean, that parts of Xen are from that point and beyond developed and maintained within the kernel development process, the core component - the Xen hypervisor - on the other hand out of tree (although this hypervisor is again just a heavily modified and stripped down version of a Linux kernel). That's the reason why Linus & friends prefer KVM over Xen, which is a kernel based hypervisor. This means it eliminates the necessity of an external hypervisor, as Linux is able to cover every part by itself.

Quote:
So to answer your question what you asked is not possible the way you wanted to use.
If you feel I am wrong do post a solution here I would like to read.
I doubt too, that it is possible the way I want to achieve it. As said, it could be worth to give a full virtualized (HVM) virtual machine a try - don't know if that would work, and if, whether it were more stable than VMWare.

Quote:
By Xen cluster I think you are trying to have an environment for your experiment which will be equivalent to multiple physical machines running Xen on them (with Guest OS in each of them )and one powerful Xen machine as a front end to all of them as one cluster.
Exactly. I want to simulate a multiple phyisical machine, each running Xen on it for development/testing purposes.

Quote:
What Linux OS you are using?
As I already said: Debian (Lenny and/or Squeeze for my tests)

Last edited by lxf; 07-27-2010 at 03:59 PM.
 
Old 07-27-2010, 09:45 PM   #12
tkmsr
Member
 
Registered: Oct 2006
Distribution: Ubuntu,Open Suse,Debian,Mac OS X
Posts: 798

Rep: Reputation: 39
The hypervisor boots before any Dom0 kernel.I think this is where you got the idea of having multiple Dom0 and make the cluster as you said.
Subscribe
http://lists.xensource.com/mailman/listinfo/xen-users
and post this question here so that Xen users/developers can answer your question. I am also curious to know the answer.
May be things can work out as you want but we donot know the right way for that.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Vmware and Xen on same machine? linuxunix Linux - Newbie 1 04-20-2010 02:09 AM
not able to ping a machine but can do ssh to machine , wanna run vnc client amolgupta Linux - Software 4 07-28-2009 05:17 PM
How to see the IP of xen virtual machine from the mother machine lakshmi4linux Linux - Newbie 3 04-29-2009 03:27 AM
How to run RHEL 4 as guest in Xen(SLES 10 XEN) darshanbv Linux - Software 1 06-26-2006 11:21 AM
wan't to run "Virtual Machine Installation (XEN)" on SuSE 10? AQG Linux - Software 1 02-08-2006 10:48 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Virtualization and Cloud

All times are GMT -5. The time now is 11:20 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration