LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Articles > Jeremy's Magazine Articles
User Name
Password

Notices

By jeremy at 2006-01-08 20:52

Tech Support
Written by Jeremy Garcia
Tuesday, 15 November 2005

If you’ve run across Beowulf or another cluster implementation, but thought that assembling your own cluster was either too complicated or too resource-intensive, cheer up! Given five minutes, a specialized, live Linux distribution called ParallelKnoppix, and a handful of ordinary personal computers, you too can build your very own mini-mini-mini-supercomputer.

ParallelKnoppix, a remaster of the Knoppix (http://www.knoppix.org/) live CD distribution, allows you to construct a parallel processing cluster using off-the-shelf desktops, laptops, and servers, and the LAM-MPI and/or MPICH implementations of the Message Passing Interface (and PVM). Moreover, because ParallelKnoppix is a live CD, you can convert a room full of machines — even those running Windows — into a Linux cluster without affecting the natively-installed operating system. Getting a cluster up and running takes about five minutes if all of your machines have PXE network cards. Clusters from two to 200 machines are supported.

Download, Burn, Boot
The first thing to do is download the ParallelKnoppix ISO image from http://pareto.uab.es/mcreel/ParallelKnoppix/ and burn one CD for each computer you’d like to include in the cluster. Next, boot one of the machines you’ll be using with the CD. (Keep in mind that you’ll need at least one Linux accessible partition on this machine. If the machine in question only has NTFS partitions, you can use a USB drive formatted as FAT32 to gain the needed space.) The machine should follow the normal Knoppix boot sequence.

Once the machine is booted, select ParallelKnoppix –> Setup ParallelKnoppix from the KDE menu to start the configuration script. Once in the configuration script, click OK to start the Terminal Server. The next dialog box will ask you how many nodes will be in the cluster, including the master node you’re using at the moment. Next, you’ll be asked to select all of the network drivers needed for the cluster. To simplify things, ensure that each slave machine is setup to PXE boot. While it’s possible to work around this, it complicates the setup and is beyond the scope of this article.
The next screen gives you a couple of cluster options. Keep the default of textmode and do not check the secure box. (See the sidebar “ParallelKnoppix Precautions.”) Next, provide additional boot options, if any. (You can normally leave this blank.)

ParallelKnoppix Precautions
ParallelKnoppix is an extremely insecure distribution. It is not intended for desktop or server use; instead, ParallelKnoppix is designed to be easy-to-use in an environment that can be restored quickly if any disaster occurs.
It’s highly recommended that you run ParallelKnoppix and your entire cluster on a dedicated network that is disconnected from the Internet.

You’re now ready to start the terminal server and are at the point where you’ll need a read/write mountable partition. Select the partition you’d like to use and click OK. A working directory with the name parallel_knoppix_working is created and exported by NFS. Anything you want to be accessible to the cluster should be placed in this directory.

Slaving Away
Now it’s time to boot each of your slaves. Once you’re sure all of the slave node machines are booted, click OK to have them mount the working directory. You should now have a working Linux cluster.
The ParallelKnoppix ISO has some example cluster applications in /home/knoppix/Desktop/ParallelKnoppix/Examples. To run one of them, copy the entire subdirectory (for example, /Octave/) into your working directory. From there, each example should have a README that explains how to run the program on the cluster. One great thing about being a Knoppix derivative is that fact that you can further remaster ParallelKnoppix to suit your needs, which could include your own custom application and data.

With this article and a couple ParallelKnoppix discs, you should be able to have a Linux cluster up and running in no time. This is a great way to get your feet wet with clustering or to prototype your next custom clustering application. Make sure not to forget about the inherent insecurities in this setup. Have fun and enjoy the rocket science.


by Present on Mon, 2006-01-16 20:56
looks like a good article. i setup ClusterKnoppix yesterday on our home network, and it was a breeze!

1.boot one computer from CD and start the clustering server (in the icon menu, bottom left).
2.reboot all the computers in your network, changing boot sequence in bios to network (they will boot over the network).
3.party time!

it REALLY is that easy! parallelknoppix is probably just as simple. here's a big plug for anyone who's curious... download and burn the iso and experiment. you'll be glad you did .

ps you can click "configure and restart" for the cluster server and click the radio box to add support for non-standard network cards if needed (i had a couple in my house, and one computer that can't network boot...i used it as the master node).

anyway... clusters are easier than they seem, and great fun. here's to coding in parallel

edited: "clustering server" should read "terminal server" in the sequence above.

by mcreel on Tue, 2006-01-17 04:50
Thanks for the review. Two notes:
- you only need 1 CD to use PKPX. The master node is booted using the CD, then the slave nodes are booted over the network using the terminalserver script.
- as of the 2005-12-02 release, security is improving. The next release (soon) will add additional improvements.

by Present on Tue, 2006-01-17 10:13
Any suggestions on software the average user can experiment with on PXPX? It would be nice if some cpu intensive programs like dvdrip were supported. The implementation of clustering for dvdrip is heavily serial, and changing the code is way beyond my scope of abilities.

by mcreel on Wed, 2006-01-18 03:28
The CD has a number of examples, mostly toys like a hello.c program, or more serious things useful to statisticians. There is a parallel mp3 encoder. Maybe I'll get around to adding a parallel povray and mandelbrot set plotter. I agree that some impressive examples that make sense to the average user would be a nice thing to add. As always, contributed examples are welcome.

by Present on Wed, 2006-01-18 20:45
woot,

wish i had the abilities to contribute dvd:rip. gonna be needing it a lot shorts. the mp3 encoder sounds cool though. i'll experiment with that, does it support ogg?

cheers

by mcreel on Thu, 2006-01-19 03:17
The mp3 encoder is quite out of date, and doesn't encode to ogg. Using it would be a quick way to get a low quality collection of mp3 files. It's just an example.


  



All times are GMT -5. The time now is 10:45 AM.

Main Menu
Advertisement

Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration