Best distro to implement a High Performance Computing Cluster

iperetta · 09-05-2009, 01:40 PM

Hi, guys!

I'm new in this forum and in Linux universe. I'm trying to implement a HPC Cluster with 5 computers (Intel Core 2) in my lab at university to start GA development and I've been searching for information, but I couldn't reach any conclusion.

Has anybody heard of CHAOS (Clustered High Availability Operating System), a Linux distribution produced within the Livermore Computing center at the Lawrence Livermore National Laboratory? Is it available for downloading? To find more about it: https://computing.llnl.gov/linux/projects.html

I've also seen several Red Hat distro posts for Cluster implementation and I got in contact with some researchers who use Slackware distro for the same. I've also got a book where the writer uses CentOS. What I'm trying to say is: depending on the source, one particular and different distro cames up.

So, I'm here in this forum to try to understand what's the best distro to HPC Clusters or, at least, gain some knowledge to understand the specific differences among the distros I mentioned above regarding HPC Clusters. Is there somewhere a article or study comparing distros? Regarding clusters, is there any known study comparing distros and their perfomance?

markush · 09-05-2009, 04:30 PM

Hello iperetta,

wellcome to LQ.

In my opinion one can do everything with every distribution. Most people will take the distribution which they are most familiar with. I'd recommend to chose a distribution with an active and helpful community.
http://www.linuxquestions.org/questions/slackware-14/
http://www.slackware.org
Take a look at Gentoo as well.

Markus

syg00 · 09-05-2009, 08:00 PM

As a new Linux user, I would have thought a (pre-)packaged solution would be a better fit. A (very) quick google turned up some candidates - like this. The gentoo project referenced has some good info, but might be jumping in the "deep end" if you have no previous gentoo install experience. The doco looks a couple of years old as well. Also note the references/links to OSCAR and Rocks in the lwn article.
If you had to have absolute performance, gentoo would be worth considering with a bit of time/experience.

Redhat would be more interested in high availability clusters than high performance I would think, although I've never looked into it. Centos is merely the commercial Redhat repackaged as freeware - seems CHAOS was of similar lineage, but according to the Wikipedia article CHAOS may have died.

iperetta · 09-07-2009, 09:37 AM

Many thanks, markush, for your point of view, and syg00, for your advices.

I think Wikipedia's Chaos is a different one, while it's from Midnight Code (here) and the other is from LLNL (here), what makes me more confused :-P

Well, I'm start again from your points of view. Now, I need to know the essential differences between distros. I mean, why so many distros if the core system is the same? Is there some real difference that could make my application less effective? Or do I need only to run proper applications?

Thanks for all!

markush · 09-07-2009, 11:41 AM

Hi iperetta,

the big difference is what syg00 stated. One part of the distributions is highly preconfigured and almost everything will work out of the box, e.g. network, sound etc.
The other part of the distributions is less configured, one has to do much configuration by himself. Normaly I'd suggest that a linux-newbie gets started with one of the preconfigured distributions. But in your case it will be necessary to do much configuration on the system by yourself.
Slackware and Gentoo belong to the distributions which have to be manually configured. Slackware is very easy to install, the installation of Gentoo is more difficult (for a newbie).
But I'd recommend: just try it out and make your own experience.

Markus