LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices

Reply
 
Search this Thread
Old 06-13-2006, 12:31 AM   #1
Dralnu
Member
 
Registered: Jul 2005
Distribution: Arch Linux, Gentoo Linux
Posts: 335

Rep: Reputation: Disabled
Clustering Linux Boxes for a Supercomputer


http://www.linuxquestions.org/questi...415367&page=10

This thread I've started to kind of get things into a slightly more suitable spot. The previous link is to the previous thread that this was on, and it has changed from its topic ENTIRELY, and figure that some more people may want this info.

I have migrated my last two post on the previously mentioned forum here to get things going.

Last edited by Dralnu; 06-13-2006 at 12:35 AM.
 
Old 06-13-2006, 12:33 AM   #2
Dralnu
Member
 
Registered: Jul 2005
Distribution: Arch Linux, Gentoo Linux
Posts: 335

Original Poster
Rep: Reputation: Disabled
I havn't found anything that seemed to be very helpful (most of it was people trying to SELL systems, and not how to build one).

If someone has the stuff to do it, I'd suggest trying it out. Grab some CAT5 cables, hook them up, and set them up on a network. Use a central PC (preferably one of the faster ones) as a hub that handles some of the bigger operations, then say you start to compile a kernel. Send that proccess over to PCA (your second-fastest) with a personalized proccess name (i.e. kernel_compile_2.6.16), and have a script that would, when a proccess completes, it checks the name to a list (in this case, we'll use RE_TRANS_LIST) to see to what PC that proccess should go. Lets say it transfers it back to the CPC (Central Personal Computer), and saves it into /boot/kernel-2.6.16. During this, you start a download of, say, all the Debian ISOs, and redirect that to PCD (one of oyur slower PCs that can handle the load), leaving CPC with 0% CPU usage. Write up a script to check personalize proccess IDs, add in a list (REPORT_COMPLETE in this case) so that once a proccess finishes, it sends a message to CPC to the effect of "Debian ISO #1 Finished. Moving on to ISO #2.

Ok, for a standard PC this isn't too big a task, but lets try this:

You're compiling the object files before linking them (I havn't dealt with this kind of thing before, so bear with me), and lets say total program src folder size is, say, 2GB (complex database in this case lets say). You use a command like:

compile_objects *.cc | split_operation PCA;PCB;PCC;PCD | return_operation | compile_linked_exe

Set up the configuration scripts to split these files by a pre-designed order (bigger files to faster PC, smaller to smaller PCs, for instance), and once the files are sent all sent, they all (in this case, possibly 2 at onces, depending on PC) start compiling. Once a file is finished, the files are then transfered back to CPC, and the message is echoed into as well as being redirected into a log. Upon file being received, it checkes the list of received files to a log (we'll call it ALL_LOG here), and see how many are missing. Lets say, the last one arrives, and then the CPC links and does the final operations, while possibly sending off a 2nd set of files to be compiled.

This is kind of what I think it is the way it could be designed, and for it to work effectivly like this, one may need a 1Gb Ethernet in CPC (which is still cheaper then a new CPU, more RAM, ect.) that can handle multiple lines (I don't know if I would suggest using a cental router or switch here, but it may not cause any problems).



Keep on hunting down a HOWTO ppl, though. I got a feeling something like this done on a massive scale could turn Linux into an OS thats not only multi-user, secure, but also easily built into a supercomputer (Imaging, this with a series of P4 Dual Cores, with 2GB ram each, a 100GB HDD, no extra devices like DVD/CDs (Well, maybe a DVD drive for system work if all else fails) to inhibit it, 1GB Ethernet cards, with a dual-proccessor CPC/Server to run it all... *drools* And with M$'s "new" OS, it will be cheaper to build then ever before) using old hardware and sometime (more likely a weekend if such a thing is ever released as a package, otherwise might take months, depending on skill level).

Personally, I'm thinking of raiding a used PC store near here for old PCs that still have old Pentiums (not even 2s) and 64MB RAM if anyone can ever figure out how to do this well.
 
Old 06-13-2006, 12:34 AM   #3
Dralnu
Member
 
Registered: Jul 2005
Distribution: Arch Linux, Gentoo Linux
Posts: 335

Original Poster
Rep: Reputation: Disabled
BINGO (I think).

Its called a "cluster" (knew it had to have been done before), and the site I found on it is here:

--------------------------------------------------------------------

http://www.rgs.uky.edu/odyssey/sprin...computers.html

That covers the creation of Cluster (plus has a 9 monitor (I think they look like 17") image of Tux, which is BADASS).

--------------------------------------------------------------------

http://www.cns.nyu.edu/events/spf/SP..._etal_1999.pdf

This may be the PDF guide we're looking for on this, but I havn't read it yet

NVM, that last post is on the Brain :-P. I'll leave it, but its deep (and may interest someone, who knows)

--------------------------------------------------------------------

http://www.mcsr.olemiss.edu/bookshel...a_cluster.html

HOWTO build a Beowolf Linux CLuster

http://beowulf.org/

Thats the Beowolf website itself

[Googled Building a Cluster. Go figure]

-{----@ @----}--- ---{----@ @----}--- ---{----@ @----}--- ---{---@

http://www.clusterworld.com/article....5/06/29/123217

That site may or may not work. I got an error, but it may have been a temp thing

----====----====----====----====----====----====----====----====--

http://arrakis.ncsa.uiuc.edu/ps2/cluster.php

a PS2 Cluster

(I think I hit the motherload )

----====----====----====----====----====----====----====----====--

http://www.phptr.com/articles/articl...&seqNum=3&rl=1

This may help as well.
 
Old 06-14-2006, 07:01 PM   #4
linux=future
Member
 
Registered: Apr 2005
Distribution: Debian
Posts: 154

Rep: Reputation: 30
The links you have include the ..., so they don't work .
 
Old 06-15-2006, 12:04 AM   #5
Micro420
Senior Member
 
Registered: Aug 2003
Location: Berkeley, CA
Distribution: Mac OS X Leopard 10.6.2, Windows 2003 Server/Vista/7/XP/2000/NT/98, Ubuntux64, CentOS4.8/5.4
Posts: 2,986

Rep: Reputation: 45
I am also in the same situation and I need to build a Linux cluster. I don't need anything fancy, but be able to combine 2 Linux boxes to share the load on some data analysis that people are doing.

Any good starts? I am mostly familiar with Windows based computers and network.
 
Old 06-16-2006, 12:15 AM   #6
Dralnu
Member
 
Registered: Jul 2005
Distribution: Arch Linux, Gentoo Linux
Posts: 335

Original Poster
Rep: Reputation: Disabled
Google what you want, and see what they give.

As for the links not working, I gave what I used to Google the links, so you might want to look into researching, and looking for similar responces.

As for the links refering to people selling the software, thats what I found, but right now I'm a bit busy with life, so you're kind of on your own. If you find anything worthwhile, then please post it here
 
Old 08-26-2006, 01:49 PM   #7
Gogul
Member
 
Registered: Nov 2005
Posts: 112

Rep: Reputation: 15
Anyone with a RSS feed to the BBC will likely have noticed this though I thought this would be applicable, if not interesting for someone who comes accross it...

http://news.bbc.co.uk/1/hi/technology/5287254.stm

10,000 machines (including PS3's) joined together to help tackle world ills.
 
Old 08-26-2006, 09:01 PM   #8
Micro420
Senior Member
 
Registered: Aug 2003
Location: Berkeley, CA
Distribution: Mac OS X Leopard 10.6.2, Windows 2003 Server/Vista/7/XP/2000/NT/98, Ubuntux64, CentOS4.8/5.4
Posts: 2,986

Rep: Reputation: 45
Haha@ GOod stuff! Too bad PS3's are way overpriced.

By the way, I got my cluster and grid computing going! Woohoo!
 
Old 08-27-2006, 02:58 AM   #9
hussykoala
LQ Newbie
 
Registered: Aug 2006
Posts: 7

Rep: Reputation: 0
hello there..i need some advice..
im built the openmosix cluster using redhat9 os.i've 12 slave nodes and a master.my prob here is my master cannot detect the rest of the slaves and the slaves also cannot detect the master.however, the master can ssh all the slave but the slave cannot ssh the master.its come out wif:

ssh:connect to the host master port 22:Connection refused

its also cannot reach the server X.
how?
is it switch prob?any posibility?
 
Old 08-27-2006, 01:03 PM   #10
Gogul
Member
 
Registered: Nov 2005
Posts: 112

Rep: Reputation: 15
Quote:
Originally Posted by Micro420
Haha@ GOod stuff! Too bad PS3's are way overpriced.

By the way, I got my cluster and grid computing going! Woohoo!

I didn't think that really made sense myself :S

Good to hear you've succeeded I haven't been able to invest in buying some old bangers to experiment with myself just yet oh well some time soon I hope

BTW your signature made me laugh lol
 
Old 08-27-2006, 01:08 PM   #11
Micro420
Senior Member
 
Registered: Aug 2003
Location: Berkeley, CA
Distribution: Mac OS X Leopard 10.6.2, Windows 2003 Server/Vista/7/XP/2000/NT/98, Ubuntux64, CentOS4.8/5.4
Posts: 2,986

Rep: Reputation: 45
Quote:
Originally Posted by hussykoala
hello there..i need some advice..
im built the openmosix cluster using redhat9 os.i've 12 slave nodes and a master.my prob here is my master cannot detect the rest of the slaves and the slaves also cannot detect the master.however, the master can ssh all the slave but the slave cannot ssh the master.its come out wif:

ssh:connect to the host master port 22:Connection refused

its also cannot reach the server X.
how?
is it switch prob?any posibility?
It sounds like
a) ssh is not installed
b) firewall is blocking port 22

Can you ping all the computers in the network?
 
Old 08-27-2006, 04:29 PM   #12
Dralnu
Member
 
Registered: Jul 2005
Distribution: Arch Linux, Gentoo Linux
Posts: 335

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Gogul
I didn't think that really made sense myself :S

Good to hear you've succeeded I haven't been able to invest in buying some old bangers to experiment with myself just yet oh well some time soon I hope

BTW your signature made me laugh lol
Save up ~800US and get some of those cheap-ass Dells.
 
Old 08-28-2006, 10:47 AM   #13
Gogul
Member
 
Registered: Nov 2005
Posts: 112

Rep: Reputation: 15
Quote:
Originally Posted by Dralnu
Save up ~800US and get some of those cheap-ass Dells.
I was considering that though lookin at the spec of them I think I can get used ones roughly equivalent for half the price =)

Can't wait!

Last edited by Gogul; 08-28-2006 at 10:54 AM.
 
Old 08-28-2006, 10:49 AM   #14
Dralnu
Member
 
Registered: Jul 2005
Distribution: Arch Linux, Gentoo Linux
Posts: 335

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Gogul
I was considering that though I think I can get used ones roughly equivalent for half the price =)

Can't wait!
Wait till 2G RAM is standard after Vista is released. Linux users will be very happy!

Cheap Server Farm anyone?
 
Old 08-28-2006, 11:26 AM   #15
weibullguy
ReliaFree Maintainer
 
Registered: Aug 2004
Location: Kalamazoo, Michigan
Distribution: Slackware-current, Cross Linux from Scratch, Gentoo
Posts: 2,731
Blog Entries: 1

Rep: Reputation: 224Reputation: 224Reputation: 224
Quote:
Originally Posted by Dralnu
Save up ~800US and get some of those cheap-ass Dells.
Better get 'em soon. I read in Barron's this morning that Mikey Dell thinks they went a little too low. Probably gonna be coming up soon.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Building a Linux supercomputer using SSH and PVM LXer Syndicated Linux News 0 04-10-2006 08:21 AM
Old ibm SP1 supercomputer & linux??? ronanio Linux - Hardware 1 09-15-2004 11:29 AM
NASA's next supercomputer = LINUX Lleb_KCir General 3 08-06-2004 10:42 PM
IBM to build Linux supercomputer jamaso General 6 08-04-2004 01:39 PM


All times are GMT -5. The time now is 03:18 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration