Run multiple Xen Dom0s on same machine
 

Hi,
is it possible to run multiple Xen Dom0s on a single physical machine, e.g. by stacking up Xen within a Xen DomU? Yes, I know you will doubt how this could be useful: It's not for a productive environment, but I wanted to use this to simulate a Xen cluster.

In particular I want to start a full Xen hypervisor + Dom0 kernel as DomU. When using pygrub I'm able to see a grub boot screen, however pygrub is not able to boot the hypervisor:

Alternatively I also tried a PV configuration by providing the kernel manually.

but this brings:

Is it simply not possible to boot a PV (HVM might possible I guess) hypervisor as DomU?


It would be ok if I could run such a cascaded virtualization by using another technique instead (e.g. KVM virtualizing Xen). Does anybody know what is possible here?

Yes
No
This is a different error all together and could result because of corrupt xml files.

It is possible.

What exactly you want to do I am not getting.You want a Xen or KVM sort of thing running inside a Guest Operating System for your experiment.
If that is what you are trying.I will suggest you install CentOS.
CentOS has a nice GUI for installing Guest OS on Xen.
In fact if what ever OS you are using you install virt-manager
You will be able to do so
check
http://www.novell.com/communities/no...r-opensuse-103

http://www.centos.org/docs/5/html/5....zed_guest.html

The following video
has mentioned in case of OpenSuse but is applicable to other Linux also.Use a GUI your life will be simple.

Check the following page
Pygrub has its own issues.

First, thank you for your answer. You didn't help me though. See, I don't want to install any kind of a virtual machine, I know how to install them and I did that for many times.

The problem is, I don't want to install any given virtual machine that is supported by Xen, I want to install a Xen hypervisor as virtual machine. However, I can't specify the Xen hypervisor as kernel image to be booted by Xen - at least not that way as I tried, as Xen refuses to boot it (neither via pygrub, nor direct via "kernel=" configuration in a DomU configuration).

You want to put more than one instance of Xen on Linux system.

Right. Doesn't work as pure Xen solution, but I figured out to get this working by using VMware ESXi + Xen.

can u post how u did ?

Well, not that complicated. I installed an evaluation copy of VMWare ESXi (http://www.vmware.com/products/vsphere-hypervisor/) as baremetal hypervisor. Within that I installed two virtual machines (Debian Squeeze) and proceeded as usual by installing a Xen hypervisor (2.6.32 Dom0 + Xen 4.0). Boots up just fine, but sticks sometimes in the boot process though, so I wouldn't consider this as stable productive solution. It works for testing purposes though.

But you wanted to use multiple instances of Xen in same Operating system.i.e in same Virtual Machine if you did in two virtual machine it is entirely different from what u asked.

No, I want(ed) to run two hypervisors on the same physical machine, on different domain borders though (thus on different virtual machines).

When speaking about Xen, you should clarify what you mean by "operating system". To illustrate what I mean:

http://img844.imageshack.us/img844/4481/50841722.png


By the way: Using VMWare turned out to have some clues, works as said, but is very unstable.

Ok now the things are clear.
As far as my understanding is then at the same instance of Dom0 you can not use multiple instance of Xen without rebooting.

In above figure you are running Dom0 in virtual machine which itself is running on VMware and then installing Guest which will be Dom0 then on that (Tier 2 Dom0 ) you have installed Xen
and if I understand correctly from the above figure you have been able to run multiple Dom0 (which are guest of Vmware) in which Xen is running and then On them you are running Guest.

You will not be able to do so the way you have done on Vmware.
What I mean to say is on a physical machine with nothing but a Linux installed you need to have different kernels from which you can boot.
say
vmlinuz-1
vmlinuz-2
vmlinuz-3

all the above
vmlinuz-1
vmlinuz-2
vmlinuz-3
can be of same version but you wont be able to have a file of same name in /boot

Then each of these kernel of Dom0 in your /boot needs to be xenified.Meaning it needs to be made aware that Xen will run on it.Some vendors who ship pre compiled binary have already done this for you so you do not know that.
If you do it yourself you need to add patches to be able to run Xen for any kernel above 2.6.18
(Novell has forward ported the patches for Xen in existing vanilla kernel)

Then you will be able to use different instances of hypervisor xen on Dom0
but then each time you will have to reboot Dom0 and use the different version(or same also you can use saying this is a and b and c and so on) of Xen and at one time only one hypervisor will be active in Dom0.

Say if you want to use Vmware more than one time in your Windows/Linux and apply the same logic to Xen.
At one time you can run only one instance of Vmware.Or you need to boot to a different OS.
Not that running a VMware instance inside a guest which itself is running on Vmware.

You may use same version of kernel on Dom0 more than one time and boot different Xen instances.
But you obviously do not want to do that and this is not what you did as in Vmware.So to answer your question what you asked is not possible the way you wanted to use.
If you feel I am wrong do post a solution here I would like to read.

By Xen cluster I think you are trying to have an environment for your experiment which will be equivalent to multiple physical machines running Xen on them (with Guest OS in each of them )and one powerful Xen machine as a front end to all of them as one cluster.
I will suggest you put this question here
http://news.gmane.org/gmane.comp.emulators.xen.user
with a reference to this thread on LQ.

What Linux OS you are using?

Thank you for your very detailed post

Right. Not the same instance in terms of a shared filesystem as this would require a concurrent access to the same hard disk image. This is not my intention though, as, referring to my figure, both Tier 2 Dom0s wouldn't share a filesystem, just the same host operating system.

Somehow a very nested sentence, but as far as my understanding goes you are right. By now I managed to got the intended set-up from above up and running by using VMWare in place of the "Tier 1" hypervisor. I would like, however, replace this VMWare machine (which is - as said - just a evaluation copy limited in time) by using Xen. It could (didn't try though) be possible to do this by running the Tier 2 Dom0s as full virtualized HVM machines, but for performance reasons I would like to PV those machines.

So far, everything is correct what you are saying. It doesn't matter for my purpose, however. What you mean by "xenifying" means to support DomU-PV operation. This is fully supported in every vanilla kernel from 2.6.27 and above. Note, that Dom0 kernel support is a totally different thing, to which you may be referring too, when speaking about the Novell patches (this is nowadays called old style Xen support). For Xen 4.0 (and 3.4 as backport) exist new pv_ops Dom0 patches, which are supposed to be included into mainline kernel sooner or later (or never - depending on the kernel maintainers).

Those new pv_ops Dom0 kernels are much less intrusive than older patches (e.g. the Novell patch/old style 2.6.18 Dom0 support) and boot up fine, even as PV kernel in a Xen Dom0, but a kernel with Dom0 support alone doesn't make a full hypervisor. This is, where my problem arises as Xen refuses to boot itself as hypervisor in a DomU.

Yes. Agreed.

Sure you got the difference from VMWare ESXi and VMWare Workstation/Server products? One is a (Linux based by the way) baremetal hypervisor, which doesn't require a host operating system by itself. VMware is the only OS in that case. In other Workstation/Server products you still require a host operating system and you'll run VMware as an application on that host.

I don't have to care at all by the way. The kernel, in Xen, is booted by the Dom0, not by the DomU (even when pygrub suggests that - but actually pygrub just copies the kernel image specified in the client grub/menu.lst to the Dom0 host). And: By the way, usually you boot up the very same physical kernel image on each virtual machine which can be indeed be the very same physical file as your Dom0 is running. This is the default behaviour for Debian's xen-tool script for example.


I am, to repeat myself, not talking about the Dom0 kernel. This boots up very fine within a virtual DomU machine. But if you look on the architecture of Xen you will notice that - for virtualization support - you actually boot a hypervisor (called xen - not to be confused with the technique having the same name this time). And afterwards this hypervisor kernel boots up a Linux kernel image with Dom0 support. If you don't believe me, look on your grub configuration. You will notice, that you boot a xen hypervisor kernel which then boots a Dom0 kernel as "module".
This Dom0 kernel alone (i.e. without this hypervisor) is nowadays able to behave like every other kernel image too, you wouldn't even notice, that you actually booted up a Dom0-supported kernel.

This is, where my problem comes up. As said, it is no problem to boot this Dom0 support kernel itself as DomU kernel for a virtual machine. But that alone doesn't make a full hypervisor - it's just one half. If I want to actually start virtual machines, I must not boot that Dom0 kernel, but the xen-hypervisor which then, by itself boots this Dom0 kernel as explained before. This is exactly where I failed.

Furthermore I'd like to point out that this architecture of two independent parts is the main reason why Linux kernel developers refuse to accept Dom0 patches into mainline vanilla kernels. This would mean, that parts of Xen are from that point and beyond developed and maintained within the kernel development process, the core component - the Xen hypervisor - on the other hand out of tree (although this hypervisor is again just a heavily modified and stripped down version of a Linux kernel). That's the reason why Linus & friends prefer KVM over Xen, which is a kernel based hypervisor. This means it eliminates the necessity of an external hypervisor, as Linux is able to cover every part by itself.

I doubt too, that it is possible the way I want to achieve it. As said, it could be worth to give a full virtualized (HVM) virtual machine a try - don't know if that would work, and if, whether it were more stable than VMWare.

Exactly. I want to simulate a multiple phyisical machine, each running Xen on it for development/testing purposes.

As I already said: Debian (Lenny and/or Squeeze for my tests)

The hypervisor boots before any Dom0 kernel.I think this is where you got the idea of having multiple Dom0 and make the cluster as you said.
Subscribe
http://lists.xensource.com/mailman/listinfo/xen-users
and post this question here so that Xen users/developers can answer your question. I am also curious to know the answer.
May be things can work out as you want but we donot know the right way for that.