Clustering Linux Boxes for a Supercomputer

http://www.linuxquestions.org/questi...415367&page=10

This thread I've started to kind of get things into a slightly more suitable spot. The previous link is to the previous thread that this was on, and it has changed from its topic ENTIRELY, and figure that some more people may want this info.

I have migrated my last two post on the previously mentioned forum here to get things going.

I havn't found anything that seemed to be very helpful (most of it was people trying to SELL systems, and not how to build one).

If someone has the stuff to do it, I'd suggest trying it out. Grab some CAT5 cables, hook them up, and set them up on a network. Use a central PC (preferably one of the faster ones) as a hub that handles some of the bigger operations, then say you start to compile a kernel. Send that proccess over to PCA (your second-fastest) with a personalized proccess name (i.e. kernel_compile_2.6.16), and have a script that would, when a proccess completes, it checks the name to a list (in this case, we'll use RE_TRANS_LIST) to see to what PC that proccess should go. Lets say it transfers it back to the CPC (Central Personal Computer), and saves it into /boot/kernel-2.6.16. During this, you start a download of, say, all the Debian ISOs, and redirect that to PCD (one of oyur slower PCs that can handle the load), leaving CPC with 0% CPU usage. Write up a script to check personalize proccess IDs, add in a list (REPORT_COMPLETE in this case) so that once a proccess finishes, it sends a message to CPC to the effect of "Debian ISO #1 Finished. Moving on to ISO #2.

Ok, for a standard PC this isn't too big a task, but lets try this:

You're compiling the object files before linking them (I havn't dealt with this kind of thing before, so bear with me), and lets say total program src folder size is, say, 2GB (complex database in this case lets say). You use a command like:

compile_objects *.cc | split_operation PCA;PCB;PCC;PCD | return_operation | compile_linked_exe

Set up the configuration scripts to split these files by a pre-designed order (bigger files to faster PC, smaller to smaller PCs, for instance), and once the files are sent all sent, they all (in this case, possibly 2 at onces, depending on PC) start compiling. Once a file is finished, the files are then transfered back to CPC, and the message is echoed into as well as being redirected into a log. Upon file being received, it checkes the list of received files to a log (we'll call it ALL_LOG here), and see how many are missing. Lets say, the last one arrives, and then the CPC links and does the final operations, while possibly sending off a 2nd set of files to be compiled.

This is kind of what I think it is the way it could be designed, and for it to work effectivly like this, one may need a 1Gb Ethernet in CPC (which is still cheaper then a new CPU, more RAM, ect.) that can handle multiple lines (I don't know if I would suggest using a cental router or switch here, but it may not cause any problems).



Keep on hunting down a HOWTO ppl, though. I got a feeling something like this done on a massive scale could turn Linux into an OS thats not only multi-user, secure, but also easily built into a supercomputer (Imaging, this with a series of P4 Dual Cores, with 2GB ram each, a 100GB HDD, no extra devices like DVD/CDs (Well, maybe a DVD drive for system work if all else fails) to inhibit it, 1GB Ethernet cards, with a dual-proccessor CPC/Server to run it all... *drools* And with M$'s "new" OS, it will be cheaper to build then ever before) using old hardware and sometime (more likely a weekend if such a thing is ever released as a package, otherwise might take months, depending on skill level).

Personally, I'm thinking of raiding a used PC store near here for old PCs that still have old Pentiums (not even 2s) and 64MB RAM if anyone can ever figure out how to do this well.

BINGO (I think).

Its called a "cluster" (knew it had to have been done before), and the site I found on it is here:

--------------------------------------------------------------------

http://www.rgs.uky.edu/odyssey/sprin...computers.html

That covers the creation of Cluster (plus has a 9 monitor (I think they look like 17") image of Tux, which is BADASS).

--------------------------------------------------------------------

http://www.cns.nyu.edu/events/spf/SP..._etal_1999.pdf

This may be the PDF guide we're looking for on this, but I havn't read it yet

NVM, that last post is on the Brain :-P. I'll leave it, but its deep (and may interest someone, who knows)

--------------------------------------------------------------------

http://www.mcsr.olemiss.edu/bookshel...a_cluster.html

HOWTO build a Beowolf Linux CLuster

http://beowulf.org/

Thats the Beowolf website itself

[Googled Building a Cluster. Go figure]

-{----@ @----}--- ---{----@ @----}--- ---{----@ @----}--- ---{---@

http://www.clusterworld.com/article....5/06/29/123217

That site may or may not work. I got an error, but it may have been a temp thing

----====----====----====----====----====----====----====----====--

http://arrakis.ncsa.uiuc.edu/ps2/cluster.php

a PS2 Cluster

(I think I hit the motherload )

----====----====----====----====----====----====----====----====--

http://www.phptr.com/articles/articl...&seqNum=3&rl=1

This may help as well.

The links you have include the ..., so they don't work .

I am also in the same situation and I need to build a Linux cluster. I don't need anything fancy, but be able to combine 2 Linux boxes to share the load on some data analysis that people are doing.

Any good starts? I am mostly familiar with Windows based computers and network.

Google what you want, and see what they give.

As for the links not working, I gave what I used to Google the links, so you might want to look into researching, and looking for similar responces.

As for the links refering to people selling the software, thats what I found, but right now I'm a bit busy with life, so you're kind of on your own. If you find anything worthwhile, then please post it here

Anyone with a RSS feed to the BBC will likely have noticed this though I thought this would be applicable, if not interesting for someone who comes accross it...

http://news.bbc.co.uk/1/hi/technology/5287254.stm

10,000 machines (including PS3's) joined together to help tackle world ills.

Haha@ GOod stuff! Too bad PS3's are way overpriced.

By the way, I got my cluster and grid computing going! Woohoo!

hello there..i need some advice..
im built the openmosix cluster using redhat9 os.i've 12 slave nodes and a master.my prob here is my master cannot detect the rest of the slaves and the slaves also cannot detect the master.however, the master can ssh all the slave but the slave cannot ssh the master.its come out wif:

ssh:connect to the host master port 22:Connection refused

its also cannot reach the server X.
how?
is it switch prob?any posibility?

Originally Posted by Micro420 Haha@ GOod stuff! Too bad PS3's are way overpriced. By the way, I got my cluster and grid computing going! Woohoo!

I didn't think that really made sense myself :S

Good to hear you've succeeded I haven't been able to invest in buying some old bangers to experiment with myself just yet oh well some time soon I hope

BTW your signature made me laugh lol

Originally Posted by hussykoala hello there..i need some advice.. im built the openmosix cluster using redhat9 os.i've 12 slave nodes and a master.my prob here is my master cannot detect the rest of the slaves and the slaves also cannot detect the master.however, the master can ssh all the slave but the slave cannot ssh the master.its come out wif: ssh:connect to the host master port 22:Connection refused its also cannot reach the server X. how? is it switch prob?any posibility?

It sounds like
a) ssh is not installed
b) firewall is blocking port 22

Can you ping all the computers in the network?

Originally Posted by Gogul I didn't think that really made sense myself :S Good to hear you've succeeded I haven't been able to invest in buying some old bangers to experiment with myself just yet oh well some time soon I hope BTW your signature made me laugh lol

Save up ~800US and get some of those cheap-ass Dells.

Originally Posted by Dralnu Save up ~800US and get some of those cheap-ass Dells.

I was considering that though lookin at the spec of them I think I can get used ones roughly equivalent for half the price =)

Can't wait!

Originally Posted by Gogul I was considering that though I think I can get used ones roughly equivalent for half the price =) Can't wait!

Wait till 2G RAM is standard after Vista is released. Linux users will be very happy!

Cheap Server Farm anyone?

Originally Posted by Dralnu Save up ~800US and get some of those cheap-ass Dells.

Better get 'em soon. I read in Barron's this morning that Mikey Dell thinks they went a little too low. Probably gonna be coming up soon.