Linux - Virtualization and CloudThis forum is for the discussion of all topics relating to Linux Virtualization and Linux Cloud platforms. Xen, KVM, OpenVZ, VirtualBox, VMware, Linux-VServer and all other Linux Virtualization platforms are welcome. OpenStack, CloudStack, ownCloud, Cloud Foundry, Eucalyptus, Nimbus, OpenNebula and all other Linux Cloud platforms are welcome. Note that questions relating solely to non-Linux OS's should be asked in the General forum.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Hi,
is it possible to run multiple Xen Dom0s on a single physical machine, e.g. by stacking up Xen within a Xen DomU? Yes, I know you will doubt how this could be useful: It's not for a productive environment, but I wanted to use this to simulate a Xen cluster.
In particular I want to start a full Xen hypervisor + Dom0 kernel as DomU. When using pygrub I'm able to see a grub boot screen, however pygrub is not able to boot the hypervisor:
Code:
pyGRUB version 0.6
┌────────────────────────────────────────────────────────────────────────┐
│ Debian GNU/Linux, with Linux 2.6.32-5-xen-amd64 │
│ Debian GNU/Linux, with Linux 2.6.32-5-amd64 │
│ Debian GNU/Linux, with Linux 2.6.32-5-xen-amd64 and XEN 4.0-amd64 │
│ │
│ │
│ │
│ │
│ │
└────────────────────────────────────────────────────────────────────────┘
Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS, 'e' to edit the
commands before booting, 'a' to modify the kernel arguments
before booting, or 'c' for a command line.
Will boot selected entry in 1 seconds
Traceback (most recent call last):
File "/usr/lib/xen-4.0/bin/pygrub", line 718, in <module>
chosencfg = run_grub(file, entry, fs, incfg["args"])
File "/usr/lib/xen-4.0/bin/pygrub", line 576, in run_grub
grubcfg["kernel"] = img.kernel[1]
TypeError: 'NoneType' object is unsubscriptable
debian:/etc/xen# Error: Boot loader didn't return any data!
Alternatively I also tried a PV configuration by providing the kernel manually.
Code:
kernel = "/boot/xen-4.0-amd64.gz"
ramdisk = "/boot/initrd.img-2.6.32-5-xen-amd64"
extra = "/boot/vmlinuz-2.6.32-5-xen-amd64"
but this brings:
Code:
Error: (2, 'Invalid kernel', "elf_xen_note_check: ERROR: Not a Xen-ELF image: No ELF notes or '__xen_guest' section found.\n")
Is it simply not possible to boot a PV (HVM might possible I guess) hypervisor as DomU?
It would be ok if I could run such a cascaded virtualization by using another technique instead (e.g. KVM virtualizing Xen). Does anybody know what is possible here?
Hi,
is it possible to run multiple Xen Dom0s on a single physical machine, e.g. by stacking up Xen within a Xen DomU?
Yes
Quote:
Originally Posted by lxf
Yes, I know you will doubt
No
Quote:
Originally Posted by lxf
Traceback (most recent call last):
File "/usr/lib/xen-4.0/bin/pygrub", line 718, in <module>
chosencfg = run_grub(file, entry, fs, incfg["args"])
File "/usr/lib/xen-4.0/bin/pygrub", line 576, in run_grub
grubcfg["kernel"] = img.kernel[1]
TypeError: 'NoneType' object is unsubscriptable
debian:/etc/xen# Error: Boot loader didn't return any data!
[/code]
This is a different error all together and could result because of corrupt xml files.
Quote:
Originally Posted by lxf
Is it simply not possible to boot a PV (HVM might possible I guess) hypervisor as DomU?
It is possible.
Quote:
Originally Posted by lxf
It would be ok if I could run such a cascaded virtualization by using another technique instead (e.g. KVM virtualizing Xen). Does anybody know what is possible here?
What exactly you want to do I am not getting.You want a Xen or KVM sort of thing running inside a Guest Operating System for your experiment.
If that is what you are trying.I will suggest you install CentOS.
CentOS has a nice GUI for installing Guest OS on Xen.
In fact if what ever OS you are using you install virt-manager
You will be able to do so
check http://www.novell.com/communities/no...r-opensuse-103
First, thank you for your answer. You didn't help me though. See, I don't want to install any kind of a virtual machine, I know how to install them and I did that for many times.
The problem is, I don't want to install any given virtual machine that is supported by Xen, I want to install a Xen hypervisor as virtual machine. However, I can't specify the Xen hypervisor as kernel image to be booted by Xen - at least not that way as I tried, as Xen refuses to boot it (neither via pygrub, nor direct via "kernel=" configuration in a DomU configuration).
Well, not that complicated. I installed an evaluation copy of VMWare ESXi (http://www.vmware.com/products/vsphere-hypervisor/) as baremetal hypervisor. Within that I installed two virtual machines (Debian Squeeze) and proceeded as usual by installing a Xen hypervisor (2.6.32 Dom0 + Xen 4.0). Boots up just fine, but sticks sometimes in the boot process though, so I wouldn't consider this as stable productive solution. It works for testing purposes though.
But you wanted to use multiple instances of Xen in same Operating system.i.e in same Virtual Machine if you did in two virtual machine it is entirely different from what u asked.
Ok now the things are clear.
As far as my understanding is then at the same instance of Dom0 you can not use multiple instance of Xen without rebooting.
In above figure you are running Dom0 in virtual machine which itself is running on VMware and then installing Guest which will be Dom0 then on that (Tier 2 Dom0 ) you have installed Xen
and if I understand correctly from the above figure you have been able to run multiple Dom0 (which are guest of Vmware) in which Xen is running and then On them you are running Guest.
You will not be able to do so the way you have done on Vmware.
What I mean to say is on a physical machine with nothing but a Linux installed you need to have different kernels from which you can boot.
say
vmlinuz-1
vmlinuz-2
vmlinuz-3
all the above
vmlinuz-1
vmlinuz-2
vmlinuz-3
can be of same version but you wont be able to have a file of same name in /boot
Then each of these kernel of Dom0 in your /boot needs to be xenified.Meaning it needs to be made aware that Xen will run on it.Some vendors who ship pre compiled binary have already done this for you so you do not know that.
If you do it yourself you need to add patches to be able to run Xen for any kernel above 2.6.18
(Novell has forward ported the patches for Xen in existing vanilla kernel)
Then you will be able to use different instances of hypervisor xen on Dom0
but then each time you will have to reboot Dom0 and use the different version(or same also you can use saying this is a and b and c and so on) of Xen and at one time only one hypervisor will be active in Dom0.
Say if you want to use Vmware more than one time in your Windows/Linux and apply the same logic to Xen.
At one time you can run only one instance of Vmware.Or you need to boot to a different OS.
Not that running a VMware instance inside a guest which itself is running on Vmware.
You may use same version of kernel on Dom0 more than one time and boot different Xen instances.
But you obviously do not want to do that and this is not what you did as in Vmware.So to answer your question what you asked is not possible the way you wanted to use.
If you feel I am wrong do post a solution here I would like to read.
By Xen cluster I think you are trying to have an environment for your experiment which will be equivalent to multiple physical machines running Xen on them (with Guest OS in each of them )and one powerful Xen machine as a front end to all of them as one cluster.
I will suggest you put this question here http://news.gmane.org/gmane.comp.emulators.xen.user
with a reference to this thread on LQ.
Ok now the things are clear.
As far as my understanding is then at the same instance of Dom0 you can not use multiple instance of Xen without rebooting.
Right. Not the same instance in terms of a shared filesystem as this would require a concurrent access to the same hard disk image. This is not my intention though, as, referring to my figure, both Tier 2 Dom0s wouldn't share a filesystem, just the same host operating system.
Quote:
In above figure you are running Dom0 in virtual machine which itself is running on VMware and then installing Guest which will be Dom0 then on that (Tier 2 Dom0 ) you have installed Xen
and if I understand correctly from the above figure you have been able to run multiple Dom0 (which are guest of Vmware) in which Xen is running and then On them you are running Guest.
Somehow a very nested sentence, but as far as my understanding goes you are right. By now I managed to got the intended set-up from above up and running by using VMWare in place of the "Tier 1" hypervisor. I would like, however, replace this VMWare machine (which is - as said - just a evaluation copy limited in time) by using Xen. It could (didn't try though) be possible to do this by running the Tier 2 Dom0s as full virtualized HVM machines, but for performance reasons I would like to PV those machines.
Quote:
You will not be able to do so the way you have done on Vmware.
What I mean to say is on a physical machine with nothing but a Linux installed you need to have different kernels from which you can boot.
say
vmlinuz-1
vmlinuz-2
vmlinuz-3
all the above
vmlinuz-1
vmlinuz-2
vmlinuz-3
can be of same version but you wont be able to have a file of same name in /boot
Then each of these kernel of Dom0 in your /boot needs to be xenified.Meaning it needs to be made aware that Xen will run on it.Some vendors who ship pre compiled binary have already done this for you so you do not know that.
If you do it yourself you need to add patches to be able to run Xen for any kernel above 2.6.18
(Novell has forward ported the patches for Xen in existing vanilla kernel)
So far, everything is correct what you are saying. It doesn't matter for my purpose, however. What you mean by "xenifying" means to support DomU-PV operation. This is fully supported in every vanilla kernel from 2.6.27 and above. Note, that Dom0 kernel support is a totally different thing, to which you may be referring too, when speaking about the Novell patches (this is nowadays called old style Xen support). For Xen 4.0 (and 3.4 as backport) exist new pv_ops Dom0 patches, which are supposed to be included into mainline kernel sooner or later (or never - depending on the kernel maintainers).
Those new pv_ops Dom0 kernels are much less intrusive than older patches (e.g. the Novell patch/old style 2.6.18 Dom0 support) and boot up fine, even as PV kernel in a Xen Dom0, but a kernel with Dom0 support alone doesn't make a full hypervisor. This is, where my problem arises as Xen refuses to boot itself as hypervisor in a DomU.
Quote:
Then you will be able to use different instances of hypervisor xen on Dom0
but then each time you will have to reboot Dom0 and use the different version(or same also you can use saying this is a and b and c and so on) of Xen and at one time only one hypervisor will be active in Dom0.
Yes. Agreed.
Quote:
Say if you want to use Vmware more than one time in your Windows/Linux and apply the same logic to Xen.
At one time you can run only one instance of Vmware.Or you need to boot to a different OS.
Not that running a VMware instance inside a guest which itself is running on Vmware.
Sure you got the difference from VMWare ESXi and VMWare Workstation/Server products? One is a (Linux based by the way) baremetal hypervisor, which doesn't require a host operating system by itself. VMware is the only OS in that case. In other Workstation/Server products you still require a host operating system and you'll run VMware as an application on that host.
Quote:
You may use same version of kernel on Dom0 more than one time and boot different Xen instances.
But you obviously do not want to do that and this is not what you did as in Vmware.
I don't have to care at all by the way. The kernel, in Xen, is booted by the Dom0, not by the DomU (even when pygrub suggests that - but actually pygrub just copies the kernel image specified in the client grub/menu.lst to the Dom0 host). And: By the way, usually you boot up the very same physical kernel image on each virtual machine which can be indeed be the very same physical file as your Dom0 is running. This is the default behaviour for Debian's xen-tool script for example.
I am, to repeat myself, not talking about the Dom0 kernel. This boots up very fine within a virtual DomU machine. But if you look on the architecture of Xen you will notice that - for virtualization support - you actually boot a hypervisor (called xen - not to be confused with the technique having the same name this time). And afterwards this hypervisor kernel boots up a Linux kernel image with Dom0 support. If you don't believe me, look on your grub configuration. You will notice, that you boot a xen hypervisor kernel which then boots a Dom0 kernel as "module".
This Dom0 kernel alone (i.e. without this hypervisor) is nowadays able to behave like every other kernel image too, you wouldn't even notice, that you actually booted up a Dom0-supported kernel.
This is, where my problem comes up. As said, it is no problem to boot this Dom0 support kernel itself as DomU kernel for a virtual machine. But that alone doesn't make a full hypervisor - it's just one half. If I want to actually start virtual machines, I must not boot that Dom0 kernel, but the xen-hypervisor which then, by itself boots this Dom0 kernel as explained before. This is exactly where I failed.
Furthermore I'd like to point out that this architecture of two independent parts is the main reason why Linux kernel developers refuse to accept Dom0 patches into mainline vanilla kernels. This would mean, that parts of Xen are from that point and beyond developed and maintained within the kernel development process, the core component - the Xen hypervisor - on the other hand out of tree (although this hypervisor is again just a heavily modified and stripped down version of a Linux kernel). That's the reason why Linus & friends prefer KVM over Xen, which is a kernel based hypervisor. This means it eliminates the necessity of an external hypervisor, as Linux is able to cover every part by itself.
Quote:
So to answer your question what you asked is not possible the way you wanted to use.
If you feel I am wrong do post a solution here I would like to read.
I doubt too, that it is possible the way I want to achieve it. As said, it could be worth to give a full virtualized (HVM) virtual machine a try - don't know if that would work, and if, whether it were more stable than VMWare.
Quote:
By Xen cluster I think you are trying to have an environment for your experiment which will be equivalent to multiple physical machines running Xen on them (with Guest OS in each of them )and one powerful Xen machine as a front end to all of them as one cluster.
Exactly. I want to simulate a multiple phyisical machine, each running Xen on it for development/testing purposes.
Quote:
What Linux OS you are using?
As I already said: Debian (Lenny and/or Squeeze for my tests)
The hypervisor boots before any Dom0 kernel.I think this is where you got the idea of having multiple Dom0 and make the cluster as you said.
Subscribe http://lists.xensource.com/mailman/listinfo/xen-users
and post this question here so that Xen users/developers can answer your question. I am also curious to know the answer.
May be things can work out as you want but we donot know the right way for that.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.