LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 05-02-2009, 03:38 PM   #1
accend
LQ Newbie
 
Registered: Aug 2005
Location: Mexico
Distribution: RHES 5.2,5.3, CentOS 5.2,5.3
Posts: 7

Rep: Reputation: 0
Host crashes after 4th virtual machines is started on RHES / CentOS XEN?


Hi,

We've been experiencing sudden host server crashes minutes after starting a fourth virtual machine.

Our setup looks like:

Dell Poweredge T300
1 x Intel Xeon X3323 Quad Core 2.5 ghz
16 GB Ram
CentOS 5.3 (64 bit)
(we've got two of them... same problem on both)

Server is running a stripped down version of CentOS 5.3 (64-bit), running only the built-in Xen Virtualization Environment. There is no other services running on the server (not samba, httpd, sendmail, cups... nothing except Xen)

We've created several virtual machines, and as long as we don't start a fourth virtual machine everything runs smoothly (impresive hardware).

Each virtual server is configured as:
PARAVIRTUALIZED
1 Virtual CPU
1 GB RAM

However, 5 minutes or so after starting a fourth virtual machine, the entire host server crashes and restarts itself.
---
Are we limited by the number of cores on the host machine CPU (4 cores)? 1 for the host and 3 for virtual machines?
---
We've read in forums about other Xen setups running up to 11 virtual machines on less powerful hardware? (a dual core server).
---
Should we be using FULLY VIRTUALIZED virtual machines instead?
---
Is the number of XEN virtual machines in fact limited by the number of cores? If so, how can someone run several virtual machines on a single core host?
---
Has anyone tried VirtualBox (Sun) or VMWare? (free alternatives)?
---

By the way, we were replacing a previous Dell Server (Poweredge 2600 with 512 MB Ram and a single Xeon single core processor running Open Virtuozzo). We were able to run up to 16 virtual machines at the same time. Of course none of the machines endured hard work (testing environments, etc). But hey, my point is that we expected to get a much higher number of virtual machines on this new hardware.

---

Anyone with a clue is welcome to comment.
Thanks,

Octavio Velasco

Last edited by accend; 05-02-2009 at 09:21 PM. Reason: Better insight into real problem
 
Old 05-02-2009, 03:55 PM   #2
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,398

Rep: Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965
There is no limit at all, and never would be. With a paravirt system (which you do want, not fully virt - that would be a massive backwards step) you need physical ram to be assigned, but not CPU's. Would the behaviour you're experiencing make any sense if it was by design? Deliberately causing a system to crash?? Somehow I don't think so. I find my centos 5.3 Xen system spontaneously reboots at non-specific points in it's use, but as it's only a test lab it seldom matters enough to troubleshoot. Your's on the otherhand would preumably benefit with some exploration as to the reasons for the reboot. If it's as reproducible as you make out then there's plenty of scope to run more debugging and such.
 
Old 05-02-2009, 03:57 PM   #3
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,398

Rep: Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965Reputation: 1965
No there's no limit, and if there was, do you really thing spontaneously nuking yourself is ever going to be a proper course of action??? if your scenario are as reproducible as you think, then you should have a good amount of scope to do some debugging and checking for known issues. Does this behvaiour vary across xen kernel versions, for example?
 
Old 05-02-2009, 08:21 PM   #4
accend
LQ Newbie
 
Registered: Aug 2005
Location: Mexico
Distribution: RHES 5.2,5.3, CentOS 5.2,5.3
Posts: 7

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by acid_kewpie View Post
Would the behaviour you're experiencing make any sense if it was by design?
Exactly, that's what i thought. I'm not sure, but to my understanding CentOS has an unlimited virtual guest capability builtin, not unlike its twin brother RedHat Enterprise Server. RHES is limited to 4 virtual guests. Only RHES Advanced has unlimited virtual guests capability builtin. You can check their specs here:
http://www.redhat.com/rhel/server/compare/

Even so, we're talking about 4 virtual guests.. not three. So i don't believe this has anything to do with our crashes.


Quote:
Originally Posted by acid_kewpie View Post
do you really thing spontaneously nuking yourself is ever going to be a proper course of action???
Obviously not. We have experienced this a few times just to confirm that the issue was reproducible. Virtual machines were holding no "real" apps or data. However we do not believe nuking ourselves is the proper course of action. Just explaining the situation.

Anyway, we've tried out a couple of versions on the xen-kernel. Same story.

We currently are using:

Code:
[root@srv10 ~]# uname -a
Linux srv10.accend.com.mx 2.6.18-128.1.6.el5xen #1 SMP  Web Apr 1 09:53:14 EDT 2009 x86_64 x86_64 GNU/Linux
Logs aren't that insightfull; system instantly crashed and we haven't been able to find any useful info in them. Any suggestions on where to look for info that might be helpful?

Anyone else has used VirtualBox?

We need to run aprox 12 virtual machines on this server:

1 CentOS (SAMBA)
1 CentOS (DNS - Bind)
3 CentOS (MySQL)
6 CentOS (Apache)
1 WinXP (for VNC access to perform testing)

All of them will be used very "lightly". All of them are intended for web app development and testing among a group of 6 programers so no heavy duty use on any of them. (did we mention earlier that we are replacing a single core server with 512 ram running 10 virtual servers on open virtuozzo?)

Last edited by accend; 05-02-2009 at 08:42 PM.
 
Old 08-27-2009, 09:38 AM   #5
ramprakash
LQ Newbie
 
Registered: Dec 2008
Posts: 9

Rep: Reputation: 0
Hi,

Are you able to fix this issue? If so please let me know. We are also facing the same problem.

Thanks and Regards,
Ram.
 
Old 08-27-2009, 09:55 AM   #6
accend
LQ Newbie
 
Registered: Aug 2005
Location: Mexico
Distribution: RHES 5.2,5.3, CentOS 5.2,5.3
Posts: 7

Original Poster
Rep: Reputation: 0
Sorry to tell you this. But we were unable to solve the issue. We tried on 5 different machines (from quad core xeons down to pentium 4). We also tried CentOS and Red Hat OS.

But the good news is, we ended up using VirtualBox. It turned out to be an even more robust solution.
We're even migrating everything from VMWare to Virtualbox.

- Great GUI, same one for Mac, Linux, Win, Solaris.
- We can move a VM one host to another without any trouble.
- Backups and Snapshots are so easy
- Free. (we would gladly pay for this... really)
- Supports a hell lot of guest OSs
- Takes care of all the different tweaks usually requiered to run a virtual machine.
- Easiest installation of all Virtualization solutions, and upgrades are very easy to perform (usually on a 4 week cycle)

You should really consider trying it.
http://www.virtualbox.org
 
Old 08-27-2009, 10:17 AM   #7
TBC Cosmo
Member
 
Registered: Feb 2004
Location: NY
Distribution: Fedora 10, CentOS 5.4, Debian 5 Sparc64
Posts: 355

Rep: Reputation: 43
Quote:
We're even migrating everything from VMWare to Virtualbox.
Out of curiosity, why? Do you pay VMWare right now for support? I guess support is moot, you were/are on CentOS.

Last edited by TBC Cosmo; 08-27-2009 at 10:19 AM.
 
Old 08-27-2009, 10:24 AM   #8
accend
LQ Newbie
 
Registered: Aug 2005
Location: Mexico
Distribution: RHES 5.2,5.3, CentOS 5.2,5.3
Posts: 7

Original Poster
Rep: Reputation: 0
Our production environment runs on RedHat. Most of the development environment is using CentOS. VMWare was used on both prod and testing env.

We're using Virtualbox now in a lot of our infrastructure. We're even using it now on client's infrastructure.

VMWare support can solve a few things, but won't make things work if the techonology doesn't support it. Don't get me wrong, VMWare is great. However, we've found Virtualbox to be a better match to our needs. Community support is growing like hell and we've been able to solve everything.

Being free is just another plus.
 
Old 08-27-2009, 10:34 AM   #9
TBC Cosmo
Member
 
Registered: Feb 2004
Location: NY
Distribution: Fedora 10, CentOS 5.4, Debian 5 Sparc64
Posts: 355

Rep: Reputation: 43
Very interesting. I've been poking around with virtual box on my desktop, but had not considered it for what will become my (personal) virtual infrastructure.
 
  


Reply

Tags
machines, virtual, xen


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Virtual machines running their own KVM's bill.loney Linux - Newbie 2 05-01-2009 03:08 PM
Running out of space creating Virtual Machines OES2 sp1 on SUSE 10 SP2 Xen host serve tekygirl Suse/Novell 2 03-18-2009 01:43 PM
How to backup Xen Virtual machines jolly Linux - Software 0 01-07-2009 10:58 AM
What do people think of having virtual machines running in the kernelspace? wilsonsamm Linux - General 4 10-16-2008 02:17 PM
Too many virtual machines are running cbird Linux - Software 8 10-06-2007 11:45 AM


All times are GMT -5. The time now is 01:28 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration