Linux cluster question
 

Hi,

I need to run an application (written in C++) a few thousand times for a research project. The application takes a fair amount of data in the memory (currently connects to a local mySQL server) and does some CPU intensive calculations on it.

I work at a small university and in the holidays the computer rooms are completely at my disposal altough I am not allowed to touch the hard drive (windows machines) so I'm looking for a diskless solution. Obviously the OpenMosix project comes to mind but I fear it will not give me any performance gain. Reading the data (from file or mySQL) to the memory takes about as much time as doing the calculations so it will not be very efficient to let every process redo this work. It seams that threads cannot be moved around in an OpenMosix cluster so that's not a solution.

Is clustering an option in this case or will it not be very efficient given the large amount of data that is necessary? Should I look into other clustering solutions like Beowolf or would it suffer from the same problems?

thank you for your advice

Hi,

I need to run an application (written in C++) a few thousand times for a research project. The application takes a fair amount of data in the memory (currently connects to a local mySQL server) and does some CPU intensive calculations on it.

I work at a small university and in the holidays the computer rooms are completely at my disposal altough I am not allowed to touch the hard drive (windows machines) so I'm looking for a diskless solution. Obviously the OpenMosix project comes to mind but I fear it will not give me any performance gain. Reading the data (from file or mySQL) to the memory takes about as much time as doing the calculations so it will not be very efficient to let every process redo this work. It seams that threads cannot be moved around in an OpenMosix cluster so that's not a solution.

Is clustering an option in this case or will it not be very efficient given the large amount of data that is necessary? Should I look into other clustering solutions like Beowolf or would it suffer from the same problems?

thank you for your advice

Reply

In order to run cluster programs, be it pvm or mpi, your c++ program need to be able to
1) execute in parallel
2) if can execute in parallel, then u have to modify your codes and link to the appropriate pvm or mpi libraries.
3) finally, setup your pvm or mpi clusters and execute your programs.

Hope it help.

so would it be possible to create an instance of a data structure class (with several large STL containers) on the master node, communicate this instance to all child nodes and to let each child node perform some calculations (different parameter settings) on the same data structure? When the child node has finished it should send the results to the master node and request new parameters for the next calculation whilst keeping the data structure in memory. In this way the data structure should only be set up and communicated once.

What could I use as diskless distro to perform this kind of actions?

I do not know exactly what you are trying to accomplish with your C++ program but you can check out the mpi website to see if it suit your requirements. The mpi website is http://www-unix.mcs.anl.gov/mpi/mpich2/index.html ...

knoppix is a single disc linux distro that you can boot from cd-rom.

Indeed I'm already experimenting with parallelknoppix which is a diskless cluster solution that supports LAM/MPI. I must say it is fascinatingly easy to use

Good.

QLedge. http://www.qledge.com