Hi,

We've been experiencing sudden host server crashes minutes after starting a fourth virtual machine.

Our setup looks like:

Dell Poweredge T300
1 x Intel Xeon X3323 Quad Core 2.5 ghz
16 GB Ram
CentOS 5.3 (64 bit)
(we've got two of them... same problem on both)

Server is running a stripped down version of CentOS 5.3 (64-bit), running only the built-in Xen Virtualization Environment. There is no other services running on the server (not samba, httpd, sendmail, cups... nothing except Xen)

We've created several virtual machines, and as long as we don't start a fourth virtual machine everything runs smoothly (impresive hardware).

Each virtual server is configured as:
PARAVIRTUALIZED
1 Virtual CPU
1 GB RAM

However, 5 minutes or so after starting a fourth virtual machine, the entire host server crashes and restarts itself.
---
Are we limited by the number of cores on the host machine CPU (4 cores)? 1 for the host and 3 for virtual machines?
---
We've read in forums about other Xen setups running up to 11 virtual machines on less powerful hardware? (a dual core server).
---
Should we be using FULLY VIRTUALIZED virtual machines instead?
---
Is the number of XEN virtual machines in fact limited by the number of cores? If so, how can someone run several virtual machines on a single core host?
---
Has anyone tried VirtualBox (Sun) or VMWare? (free alternatives)?
---

By the way, we were replacing a previous Dell Server (Poweredge 2600 with 512 MB Ram and a single Xeon single core processor running Open Virtuozzo). We were able to run up to 16 virtual machines at the same time. Of course none of the machines endured hard work (testing environments, etc). But hey, my point is that we expected to get a much higher number of virtual machines on this new hardware.

---

Anyone with a clue is welcome to comment.
Thanks,

Octavio Velasco

There is no limit at all, and never would be. With a paravirt system (which you do want, not fully virt - that would be a massive backwards step) you need physical ram to be assigned, but not CPU's. Would the behaviour you're experiencing make any sense if it was by design? Deliberately causing a system to crash?? Somehow I don't think so. I find my centos 5.3 Xen system spontaneously reboots at non-specific points in it's use, but as it's only a test lab it seldom matters enough to troubleshoot. Your's on the otherhand would preumably benefit with some exploration as to the reasons for the reboot. If it's as reproducible as you make out then there's plenty of scope to run more debugging and such.

No there's no limit, and if there was, do you really thing spontaneously nuking yourself is ever going to be a proper course of action??? if your scenario are as reproducible as you think, then you should have a good amount of scope to do some debugging and checking for known issues. Does this behvaiour vary across xen kernel versions, for example?

Exactly, that's what i thought. I'm not sure, but to my understanding CentOS has an unlimited virtual guest capability builtin, not unlike its twin brother RedHat Enterprise Server. RHES is limited to 4 virtual guests. Only RHES Advanced has unlimited virtual guests capability builtin. You can check their specs here:
http://www.redhat.com/rhel/server/compare/

Even so, we're talking about 4 virtual guests.. not three. So i don't believe this has anything to do with our crashes.


Obviously not. We have experienced this a few times just to confirm that the issue was reproducible. Virtual machines were holding no "real" apps or data. However we do not believe nuking ourselves is the proper course of action. Just explaining the situation.

Anyway, we've tried out a couple of versions on the xen-kernel. Same story.

We currently are using:

Logs aren't that insightfull; system instantly crashed and we haven't been able to find any useful info in them. Any suggestions on where to look for info that might be helpful?

Anyone else has used VirtualBox?

We need to run aprox 12 virtual machines on this server:

1 CentOS (SAMBA)
1 CentOS (DNS - Bind)
3 CentOS (MySQL)
6 CentOS (Apache)
1 WinXP (for VNC access to perform testing)

All of them will be used very "lightly". All of them are intended for web app development and testing among a group of 6 programers so no heavy duty use on any of them. (did we mention earlier that we are replacing a single core server with 512 ram running 10 virtual servers on open virtuozzo?)

Hi,

Are you able to fix this issue? If so please let me know. We are also facing the same problem.

Thanks and Regards,
Ram.

Sorry to tell you this. But we were unable to solve the issue. We tried on 5 different machines (from quad core xeons down to pentium 4). We also tried CentOS and Red Hat OS.

But the good news is, we ended up using VirtualBox. It turned out to be an even more robust solution.
We're even migrating everything from VMWare to Virtualbox.

- Great GUI, same one for Mac, Linux, Win, Solaris.
- We can move a VM one host to another without any trouble.
- Backups and Snapshots are so easy
- Free. (we would gladly pay for this... really)
- Supports a hell lot of guest OSs
- Takes care of all the different tweaks usually requiered to run a virtual machine.
- Easiest installation of all Virtualization solutions, and upgrades are very easy to perform (usually on a 4 week cycle)

You should really consider trying it.
http://www.virtualbox.org

Out of curiosity, why? Do you pay VMWare right now for support? I guess support is moot, you were/are on CentOS.

Our production environment runs on RedHat. Most of the development environment is using CentOS. VMWare was used on both prod and testing env.

We're using Virtualbox now in a lot of our infrastructure. We're even using it now on client's infrastructure.

VMWare support can solve a few things, but won't make things work if the techonology doesn't support it. Don't get me wrong, VMWare is great. However, we've found Virtualbox to be a better match to our needs. Community support is growing like hell and we've been able to solve everything.

Being free is just another plus.

Very interesting. I've been poking around with virtual box on my desktop, but had not considered it for what will become my (personal) virtual infrastructure.