Linux - NewbieThis Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place!
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I want to setup two desktops each one having 4 cores and combine them to have 8 cores with linux clustering. Is this how clustering works and what level of difficulty is it?
Also, any good howto sites on the subject is appreciated. Thanks
It depends on what you're trying to do. If you just want one heavy process to run in 8 threads on both machines, you should look into MPI. If you want to be able to run many simultaneous processes and have them automatically farmed out to the two machines based on load, memory usage, etc. then look into the TORQUE resource manager.
If you want your general purpose computer usage (opening spreadsheets, watching youtube, etc.) to use the processing power of both machines like it's one bigger machine, it's not going to happen, at least not in a way that would actually speed things up. Mostly because the CPU is rarely the bottleneck for those kinds of applications, and the latency when communicating between the two systems will slow things way down.
Last edited by suicidaleggroll; 02-23-2015 at 12:03 PM.
Only if you build your own video encoder and can program in all of the MPI hooks, or you use one with distributed encoding already built in, eg: x264farm, RipBot264, MediaEncodingCluster, etc. Note that I have no experience with any of these, it's just what I found with a quick google search.
Only if you build your own video encoder and can program in all of the MPI hooks, or you use one with distributed encoding already built in, eg: x264farm, RipBot264, MediaEncodingCluster, etc. Note that I have no experience with any of these, it's just what I found with a quick google search.
Same here. It's almost not worth doing a cluster then. I had the assumption that any application would utilize all resources from a cluster.
I thank you suicidaleggroll for the useful information about the types of clustering and other related info.
I read somewhere that video encoding isn't usually written to use no more than 4 cores.
From experience, it will definitely use at least 4 cores.
Perhaps, BUT...
Movies are rendered in clusters with thousands of cores, but as suicidaleggroll said earlier, they're probably programmed their own custom software to take advantage of the cores.
Movie rendering is a type of problem that is called "embarassingly parallel" because each frame is independent of any other. Therefore, if you have 5,000 quad core computers, you can use them to render 5,000 frames simultaneouslky. Usually these work in a master-worker paradigm. One computer is the master, which distributes work to the other computers. When a computer finioshes its frame, it goes back and asks the master for more work. I think DrQueue is popular as software to run on the master for workload distribution.
Most large-scale high performance computing clusters, by contrast, are designed to run a tightly-coupled parallel application written using MPI or a similar paradigm like PGAS. In otherwords, the calculations being performed on one computer are tightly coupled to those performed on other computers, unlike the movie rendering case. At my work, we have a large-ish cluster that is used to run molecular simulations, for example. If you look at top500.org - the list of the top 500 reported most powerful computer systems - you'll see that they're all clusters. Usually they have a specialized parallel interconnect (e.g. InfiniBand) that allows high bandwidth, low latency message passage. For workload management, these tend to use TORQUE, SLURM, or one of the grid engine derivatives.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.