LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 10-21-2014, 12:25 PM   #1
ron7000
Member
 
Registered: Nov 2007
Location: CT
Posts: 248

Rep: Reputation: 26
creating a distributed memory cluster ?


can someone tell me what is needed to make such a system?
i'm considering getting some high end dell rack servers, each server having say 4 sockets with n core cpu's that i would install linux on.
i may get more of the same. if that happens, how then can i connect those separate servers to create a distributed system? I understand at least having a network card in each server in the rack that does fiber and i would network each server together. But from there, what software is needed to make it work? is 3rd party 'distributed memory cluster software' needed or are there ways in the linux operating system to set things up so there is a head node that users would see on the network and log in to?
Where i'm going is for various software either that i or coworkers write that use MPI or whatever, or purchased software that provides a gui interface showing a choice between shared memory system vs distributed memory, how do you set the cluster up to make that all work?
thanks.
 
Old 10-21-2014, 04:36 PM   #2
littleball
Member
 
Registered: Jan 2011
Distribution: Slackware, Red Hat Enterprise
Posts: 47

Rep: Reputation: 8
Well, I am not very deep about clustering, but I do know something about High-availability and load balancing services.

For example, there were memcached for MySQL distribute load balanced in read queries (but to make it work it was needed to develop using memcached API).
In RedHat once there was piranha, a software which let you decide wich services are going to load balance over several servers (I use it to load balance Apache web server and MySQL reads queries).

There,s also Drbd (but this one if only for High-availability active-passive cluster,like replicate blocks of your data accross other network disks) and since you,re looking more for load balancing, I think something like piranha will suit you more, if you find the way to make it work with your own software. Piranha is out of date, but there,s a fork of it around (but I don,t remember the name).

Last edited by littleball; 10-21-2014 at 04:37 PM.
 
Old 10-21-2014, 07:40 PM   #3
smallpond
Senior Member
 
Registered: Feb 2011
Location: Massachusetts, USA
Distribution: Fedora
Posts: 4,126

Rep: Reputation: 1260Reputation: 1260Reputation: 1260Reputation: 1260Reputation: 1260Reputation: 1260Reputation: 1260Reputation: 1260Reputation: 1260
Network isn't the only fabric. You could also look at a blade server with infiniband. That lets you do RDMA instead of packet-based transfers.
 
Old 10-22-2014, 09:53 PM   #4
btmiller
Senior Member
 
Registered: May 2004
Location: In the DC 'burbs
Distribution: Arch, Scientific Linux, Debian, Ubuntu
Posts: 4,290

Rep: Reputation: 378Reputation: 378Reputation: 378Reputation: 378
As someone who's built quite a few of things, let me just ask if you have an application in mind for this? Some types of applications parallelize better than others. Getting something to scale well using MPI or OpenMP (if a shared memory system) is not a trivial task; it requires substantial expertise. Also, some types of applications are bandwidth sensitive, some are latency sensitive, and some are both. Without knowing what yours is, you don't know the best way to invest your budget. InfiniBand really shines when high bandwidth and low latency are needed, but it's not cheap (BTW, most InfiniBand set ups I've seen run over thick copper cabling as the fiber optic cables are hideously expensive). BTW, you can do RDMA over 10 gig Ethernet with the right NICs and software stack (look up OpenFabrics). 10 gig is getting cheaper, to the point where it's comparible with InfiniBand.

Also don't neglect the environment! A high density set up with blades can easily draw 15 kW per rack plus. Most data centers (especially older ones) don't have the power and cooling capacity to handle such density. Be sure to check into this before you blow out a breaker or find that your expensive new kit melted into a pile of goo on the floor.
 
Old 10-22-2014, 11:33 PM   #5
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,573

Rep: Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142
As [hopefully] you're starting to realize, it's not just a matter of "I need it to run fast!" that controls a cluster layout. You need to know the intricate details of what, exactly, is going to be run and what its bottlenecks are. Are the processes you need to run single or multi-threaded? If they're multi-threaded, how well do they scale? Almost any multi-threaded process will run into a point of diminishing returns at some point, so where is yours? 2 procs? 4? 8? 12? 2000? Is the bottleneck the processor? memory bandwidth? I/O? Maybe a cluster is a waste of time and money, maybe you should focus your efforts on porting to a GPU?

I know that in my line of work, a cluster is a waste. The processes I deal with parallelize well up to 4-8 procs, and above that they saturate, their speed dominated by the single-threaded bottlenecks in their code. There is absolutely no reason to set up a cluster in my case when you can build a dual 12-core machine for <$10k which can run multiple instances of these models as fast as they'll go given their restrictions. A cluster, in that case, would just add unnecessary latency and overhead.

TBH - in this day and age, with co-processors (Xeon Phi), GPUs, and 30+ core regular systems available out of the box, the old-school "clusters" are becoming less of a requirement and more of an unnecessary overhead that actually slows things down.

Last edited by suicidaleggroll; 10-22-2014 at 11:34 PM.
 
Old 10-24-2014, 02:52 PM   #6
ron7000
Member
 
Registered: Nov 2007
Location: CT
Posts: 248

Original Poster
Rep: Reputation: 26
i'm currently playing with a shared mem ultraviolet 1000 to give you an idea, running cfd software. my room is wired for over 50kw of power.
i understand where your questions are coming from, they are all valid, but i don't want to go into detail about a lot of it.
the bottleneck is the cpu.
I cannot utilize GPU cores.
simply put, the higher the frequency of the cpu and the more cpu cores i have the better.
regarding multithreading/hyperthreading, i believe what software i will be running is better running without hyper-threading but that's getting outside the scope of my question.
I know there is infiniband, and that it is the more expensive solution for communication between servers which is why i didn't mention it.
but however the number of servers are connected within a rack, by whatever cable, i was looking to understand what is needed to make the system clustered and work... such that users see one (head) node on the network which they log in to. then there has to be some software on that head node that sets up the cluster- and knows all the other servers and cpu's in the rack to pawn jobs off on right?

i understand 30+ cores come out of the box now, a dell R920 comes with four 12-core cpu's. but i'm entertaining the thought of clustering which will give me the performance i'm looking for... if i get 96+ cores available. I haven't worked out the pricing yet, i'm assuming 4 and 6 core cpu's are the most economical so i'd be looking at numerous servers each having 2 or 4 sockets on each board... connected by infiniband or whatever. the connection between the servers is not important, because i know for some software i'm dealing with there's little communication back and forth, the software(s) have been written and optimized to run on clustered/distributed systems.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Distributed / Cluster computing in blender sunils1973 Linux - Networking 9 11-22-2012 07:27 AM
Distributed processing AMD chipset / motherboard / memory / network questions wevl Linux - Hardware 0 08-10-2011 02:55 PM
Creating Cluster - What do i do next? szboardstretcher Linux - Server 3 06-24-2011 01:00 PM
distributed posix shared memory hrmb Linux - Hardware 0 01-01-2010 04:27 PM
openMP? distributed shared memory? Hano Linux - General 0 05-11-2002 11:41 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 09:05 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration