LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 02-23-2015, 12:51 PM   #1
Roy.Geer
LQ Newbie
 
Registered: Feb 2015
Posts: 18

Rep: Reputation: Disabled
Is setting up a linux Cluster difficult?


hello

I want to setup two desktops each one having 4 cores and combine them to have 8 cores with linux clustering. Is this how clustering works and what level of difficulty is it?

Also, any good howto sites on the subject is appreciated. Thanks

Last edited by Roy.Geer; 02-23-2015 at 01:10 PM.
 
Old 02-23-2015, 01:00 PM   #2
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,561

Rep: Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127
It depends on what you're trying to do. If you just want one heavy process to run in 8 threads on both machines, you should look into MPI. If you want to be able to run many simultaneous processes and have them automatically farmed out to the two machines based on load, memory usage, etc. then look into the TORQUE resource manager.

If you want your general purpose computer usage (opening spreadsheets, watching youtube, etc.) to use the processing power of both machines like it's one bigger machine, it's not going to happen, at least not in a way that would actually speed things up. Mostly because the CPU is rarely the bottleneck for those kinds of applications, and the latency when communicating between the two systems will slow things way down.

Last edited by suicidaleggroll; 02-23-2015 at 01:03 PM.
 
Old 02-23-2015, 01:10 PM   #3
Roy.Geer
LQ Newbie
 
Registered: Feb 2015
Posts: 18

Original Poster
Rep: Reputation: Disabled
I would like to do video encoding, I guess this would fall to MPI clustering.
 
Old 02-23-2015, 01:19 PM   #4
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,561

Rep: Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127Reputation: 2127
Only if you build your own video encoder and can program in all of the MPI hooks, or you use one with distributed encoding already built in, eg: x264farm, RipBot264, MediaEncodingCluster, etc. Note that I have no experience with any of these, it's just what I found with a quick google search.
 
Old 02-23-2015, 01:31 PM   #5
Roy.Geer
LQ Newbie
 
Registered: Feb 2015
Posts: 18

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by suicidaleggroll View Post
Only if you build your own video encoder and can program in all of the MPI hooks, or you use one with distributed encoding already built in, eg: x264farm, RipBot264, MediaEncodingCluster, etc. Note that I have no experience with any of these, it's just what I found with a quick google search.
Same here. It's almost not worth doing a cluster then. I had the assumption that any application would utilize all resources from a cluster.

I thank you suicidaleggroll for the useful information about the types of clustering and other related info.
 
Old 02-23-2015, 01:37 PM   #6
JeremyBoden
Senior Member
 
Registered: Nov 2011
Distribution: Debian
Posts: 1,194

Rep: Reputation: 243Reputation: 243Reputation: 243
I read somewhere that video encoding isn't usually written to use no more than 4 cores.
From experience, it will definitely use at least 4 cores.
 
Old 02-23-2015, 01:46 PM   #7
Roy.Geer
LQ Newbie
 
Registered: Feb 2015
Posts: 18

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by JeremyBoden View Post
I read somewhere that video encoding isn't usually written to use no more than 4 cores.
From experience, it will definitely use at least 4 cores.
Perhaps, BUT...

Movies are rendered in clusters with thousands of cores, but as suicidaleggroll said earlier, they're probably programmed their own custom software to take advantage of the cores.

Last edited by Roy.Geer; 02-23-2015 at 01:48 PM.
 
Old 02-24-2015, 08:49 AM   #8
btmiller
Senior Member
 
Registered: May 2004
Location: In the DC 'burbs
Distribution: Arch, Scientific Linux, Debian, Ubuntu
Posts: 4,284

Rep: Reputation: 371Reputation: 371Reputation: 371Reputation: 371
Movie rendering is a type of problem that is called "embarassingly parallel" because each frame is independent of any other. Therefore, if you have 5,000 quad core computers, you can use them to render 5,000 frames simultaneouslky. Usually these work in a master-worker paradigm. One computer is the master, which distributes work to the other computers. When a computer finioshes its frame, it goes back and asks the master for more work. I think DrQueue is popular as software to run on the master for workload distribution.

Most large-scale high performance computing clusters, by contrast, are designed to run a tightly-coupled parallel application written using MPI or a similar paradigm like PGAS. In otherwords, the calculations being performed on one computer are tightly coupled to those performed on other computers, unlike the movie rendering case. At my work, we have a large-ish cluster that is used to run molecular simulations, for example. If you look at top500.org - the list of the top 500 reported most powerful computer systems - you'll see that they're all clusters. Usually they have a specialized parallel interconnect (e.g. InfiniBand) that allows high bandwidth, low latency message passage. For workload management, these tend to use TORQUE, SLURM, or one of the grid engine derivatives.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Setting up a mysql redundancy/cluster in Linux Vishal Koul Linux - Newbie 1 07-15-2013 03:15 PM
LXer: Setting up a MySQL Cluster for your Linux desktop LXer Syndicated Linux News 0 11-03-2009 11:50 AM
setting up a linux rendering cluster Jan Tanjo Linux - General 4 05-01-2006 03:48 PM
Setting up a cluster with pxe and cluster knoppix bucovaina78 Linux - Networking 1 05-16-2005 12:29 AM
difficult network setting, help me newbie44 Linux - Newbie 1 04-18-2004 03:14 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 11:06 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration