Registered: May 2004
Location: In the DC 'burbs
Distribution: Arch, Scientific Linux, Debian, Ubuntu
As somebody who works with clusters for a libing, I'm a bit puzzled as to what you're trying to accomplish. In general, there are three types of clusters"
- Compute clusters (for high performance computing), e.g. Beowulf-style clusters. These clusters are designed to run specially-written application that do very heavy number crunching (think simulating entire galaxies or solving massive sets of coupled differential equations for fluid dynamics).
- Storage clusters. These are designed to aggregate the storage of many individual computers into one large pool for very high I/O access. These are usually built to host a parallel-fuilesystem such as Lustre, Ceph, or, with $$$ to spend, GPFS.
- High-availability clusters. These are designed to run mission critical services, so that if one node providing a service fails, another node takes over for it and users do not notice any interruption.
Recently there are things like map-reduce clusters (e.g. built using Hadoop) that are an interesting hybrid of compute and storage clusters.
Your requirements are that your cluster be used for "music storage, video streaming, recording and conversion to DVD and virtual machines", which fits into multiple paradigms listed here. You'll need to decide on what sort of software you want to use for all of these things. It certainly can be done, but it's a fairly advanced task. At minimum, you'll need the following:
- A common pool of storage. This can range from an NFS export from the master node or a dedicated storage node. However, this won't scale very well. You can look at Gluster or Ceph, but your network will be the bottleneck (see below).
- Some way to login from one machine to the other transparently without a password. This is easier; SSH with keys works really well. You'll want a signle user database (NIS or preferably LDAP) to keep things sane.
- If you want to run VMs, you'll need a hypervisor on your nodes. I suggest KVM because it comes baked in with most modern Linux distros and has good tool support, but really it's a matter of personal preference.
- Optionally, some sort of of job scheduler to schedule different workloads on various machines. There are lots of choices here: Slurm, Open Grid Engine, TORQUE, etc.
- Also optional, but if you plan to go the full ledged IaaS (Infrastructure as a Service) route and provision virtual machines that way, you can set up a full OpenStack deployment. I've never done this before, so I can't advise how difficult it is.
In answer to your more specific questions:
- There is no end-to-end GUI tool to do everything you want to do. There are some tools that may do part of what you want, but you have to realize that what you're trying to do is not exactly a newbie project. Environments like what you describe tend to be built my admins and systems engineers with years of experience building scale-up and scale-out environments. These folks work on the command line pretty much exclusively, for a wide variety of reasons.
- There are books and documents for most everything you want to do. If you Google around for beowulf clusters, storage clusters, media encoding, etc. using some of the specific technologies I listed above, you should find some useful resources. If you can't, let me know and I can post a few links that may help. Note: this is NOT a "for dummies" project. You're going to have to actually understand what you're doing.
- If your motherboards support Wake on LAN or IPMI (the latter is dubious on consumer-grade hardware) you can use it to remotely power up nodes.
- Most clusters have some sort of high-speed interconnect that connects to the CPU via PCI express. 10 gigabit Ethernet or InfiniBand are the preferred technologies, but if you're on a hobbyist budget you're probably stuck with gigabit Ethernet. Do yourself a favor and buy a decent switch, one preferably that is non-blocking. Flakey Ethernet switches are a frequent source of problems in clustered architectures.
- You really need to learn at least simple shell scripting and ideally the basics of Perl or Python. The secret to running large infrastructures is automation, and this means scripting. You don't need to become a hard core programmer, but basic scripting knowledge is pretty much required to do all aspects of a project like this. If you get more advanced, you can contemplate deploying Puppet, Chef, or similar for configuration management.