LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 05-10-2014, 04:18 PM   #1
CCarper
LQ Newbie
 
Registered: May 2014
Posts: 1

Rep: Reputation: Disabled
Building a cluster and switching from windows 7 to Ubuntu 13,10


I am a pretty hardware savvy computer guy and would rate myself as an advanced computer user with virtually no programming knowledge. I built a 4 node cluster using Pelican HPC about a year ago that worked well but I couldn't get it to connect to the internet so I reverted back to Windows 7. Currently I've planned a build of a 10 node cluster. I have purchased the motherboard and chipset I am going to use for the build and the nodes will be pretty much spare parts I have Frankensteined together from time to time and used as PC's. Here are the question I have:

Is there a GUI based program compatible with Ubuntu that will handle the ENTIRE setup of the cluster?

If no GUI is available or only partially handles the setup is there a dummy guide with little to know programming?

Is there a way to power up the nodes without having to use a power button an PSU for each of the 10 motherboards?

Is it possible to utilize PCI, PCI-E or USB connections of the slave nodes for use with the master nodes?

I'm using the cluster for movies and music storage, video streaming, recording and conversion to DVD and virtual machines to create and test a much bigger cluster. I'm building it because some idiot (my brother) said it couldn't be done.

Master node motherboard: ASUS split power and dual-link PCI-E
server motherboard KFSN-DRE/SAS

Master Node CPU: Dual AMD Opteron 2350 Quad-core

Thanks for any guidance you can give me and feel free to ask if more info is needed.
 
Old 05-11-2014, 10:05 AM   #2
btmiller
Senior Member
 
Registered: May 2004
Location: In the DC 'burbs
Distribution: Arch, Scientific Linux, Debian, Ubuntu
Posts: 4,284

Rep: Reputation: 371Reputation: 371Reputation: 371Reputation: 371
As somebody who works with clusters for a libing, I'm a bit puzzled as to what you're trying to accomplish. In general, there are three types of clusters"

- Compute clusters (for high performance computing), e.g. Beowulf-style clusters. These clusters are designed to run specially-written application that do very heavy number crunching (think simulating entire galaxies or solving massive sets of coupled differential equations for fluid dynamics).
- Storage clusters. These are designed to aggregate the storage of many individual computers into one large pool for very high I/O access. These are usually built to host a parallel-fuilesystem such as Lustre, Ceph, or, with $$$ to spend, GPFS.
- High-availability clusters. These are designed to run mission critical services, so that if one node providing a service fails, another node takes over for it and users do not notice any interruption.

Recently there are things like map-reduce clusters (e.g. built using Hadoop) that are an interesting hybrid of compute and storage clusters.

Your requirements are that your cluster be used for "music storage, video streaming, recording and conversion to DVD and virtual machines", which fits into multiple paradigms listed here. You'll need to decide on what sort of software you want to use for all of these things. It certainly can be done, but it's a fairly advanced task. At minimum, you'll need the following:

- A common pool of storage. This can range from an NFS export from the master node or a dedicated storage node. However, this won't scale very well. You can look at Gluster or Ceph, but your network will be the bottleneck (see below).
- Some way to login from one machine to the other transparently without a password. This is easier; SSH with keys works really well. You'll want a signle user database (NIS or preferably LDAP) to keep things sane.
- If you want to run VMs, you'll need a hypervisor on your nodes. I suggest KVM because it comes baked in with most modern Linux distros and has good tool support, but really it's a matter of personal preference.
- Optionally, some sort of of job scheduler to schedule different workloads on various machines. There are lots of choices here: Slurm, Open Grid Engine, TORQUE, etc.
- Also optional, but if you plan to go the full ledged IaaS (Infrastructure as a Service) route and provision virtual machines that way, you can set up a full OpenStack deployment. I've never done this before, so I can't advise how difficult it is.

In answer to your more specific questions:

- There is no end-to-end GUI tool to do everything you want to do. There are some tools that may do part of what you want, but you have to realize that what you're trying to do is not exactly a newbie project. Environments like what you describe tend to be built my admins and systems engineers with years of experience building scale-up and scale-out environments. These folks work on the command line pretty much exclusively, for a wide variety of reasons.
- There are books and documents for most everything you want to do. If you Google around for beowulf clusters, storage clusters, media encoding, etc. using some of the specific technologies I listed above, you should find some useful resources. If you can't, let me know and I can post a few links that may help. Note: this is NOT a "for dummies" project. You're going to have to actually understand what you're doing.
- If your motherboards support Wake on LAN or IPMI (the latter is dubious on consumer-grade hardware) you can use it to remotely power up nodes.
- Most clusters have some sort of high-speed interconnect that connects to the CPU via PCI express. 10 gigabit Ethernet or InfiniBand are the preferred technologies, but if you're on a hobbyist budget you're probably stuck with gigabit Ethernet. Do yourself a favor and buy a decent switch, one preferably that is non-blocking. Flakey Ethernet switches are a frequent source of problems in clustered architectures.
- You really need to learn at least simple shell scripting and ideally the basics of Perl or Python. The secret to running large infrastructures is automation, and this means scripting. You don't need to become a hard core programmer, but basic scripting knowledge is pretty much required to do all aspects of a project like this. If you get more advanced, you can contemplate deploying Puppet, Chef, or similar for configuration management.
 
1 members found this post helpful.
Old 05-11-2014, 10:39 AM   #3
sycamorex
LQ Veteran
 
Registered: Nov 2005
Location: London
Distribution: Slackware64-current
Posts: 5,819
Blog Entries: 1

Rep: Reputation: 1209Reputation: 1209Reputation: 1209Reputation: 1209Reputation: 1209Reputation: 1209Reputation: 1209Reputation: 1209Reputation: 1209
I have no experience of clusters but if you want to use Ubuntu, you'd probably be better off with 14.04 which is a Long Term Support version. Ubuntu will stop supporting 13.10 in two months time.
 
1 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Building a simple Beowulf Like Cluster with Ubuntu 12.04 using MPICH 2 $u$hil_k$ Linux - Newbie 2 01-23-2013 03:32 AM
LXer: Building A 96-Core Ubuntu ARM Solar-Powered Cluster LXer Syndicated Linux News 0 06-20-2012 05:30 AM
Building some sort of cluster: slurm, pacemaker, cluster-glue or .... kaz2100 Linux - Software 2 07-21-2011 12:04 AM
Switching between users in Ubuntu like in Windows XP SuperDude123 Linux - Desktop 2 08-26-2009 01:40 PM


All times are GMT -5. The time now is 10:55 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration