LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - General (https://www.linuxquestions.org/questions/linux-general-1/)
-   -   Beowulf-like cluster (https://www.linuxquestions.org/questions/linux-general-1/beowulf-like-cluster-769457/)

luigi 11-16-2009 10:17 AM

Beowulf-like cluster
 
Hi all,
I would like to build a small beowulf like cluster with few machines that I have here around from the pc-farm. The problem is that up to now I have only been able to find docs and links to old/unupdated/dead projects...

Do you have some hints? I slighly prefere debian-based systems, but any (reasonably recent....) distro would be ok.

Thank you!
Luigi :D


PS: I hope this is the correct forum for such question...

MS3FGX 11-16-2009 07:39 PM

Well, "cluster" is a pretty general term. What exactly would you like to do with it? You need to have some kind of application in mind before you get started to figure out what sort of software you are looking at.

Beowulf is an advanced system for mathematical applications, but one could argue it is no more or less a cluster than two Debian boxes passing jobs between each other via SSH.

luigi 11-17-2009 03:50 AM

Hi MS3FGX,
I have a bunch of number-crunching applications that need to crunch few Tbytes of data in a "embarassing parallel" way (to say it in HPC jargon).

I did already the ssh trick between two machines but, as you know, there are lots of "little problems" like disk sharing, handling a machine/job failure, etc etc that must be handled and I would like not to reinvent the wheel...
So before extending it to more machines I would really be happy to try some beowulf clustering, i.e. the same game in pratice, but much simpler stuff for the end users.

What do you think? do you have some pointers to active projects?

Thanks!
Luigi

MS3FGX 11-17-2009 02:01 PM

So I assume you are going to be writing the software for this cluster then?

In that case, you might want to look at OSCAR. I haven't used it personally, but I have heard good things. Basically is a complete toolkit to create and maintain Beowulf-style clusters on common Linux distributions. It handles all the mundane details that you mentioned like inter-node communication and leaves you free to just write up your code.

As I understand it, OSCAR can even create disk images with your code and configuration that are suitable for booting your nodes.

luigi 11-18-2009 07:15 AM

Hi MS3FGX,
thank you for the link :)
I have just tried OSCAR on a VirtualBox image with ubuntu 9.04, but I am not able to have it running: is it possible that OSCAR supports only ubuntu up to 8.04, debian up to 4, etc and not newer distros?
Any hint to have it running on a newer system?

Regards!
Luigi

LinuxLover 11-18-2009 08:30 AM

Look at the video of Beowolf cluster

http://www.wbitt.com/hpcc-supercomputing-cbts.html


All times are GMT -5. The time now is 02:26 PM.