2020 Clustering question
 

hi everyone,

I am trying to learn how to connect servers together (minimum of 2, but it could be more in the near future) all with XEON CPUs, as I am planning to build a supercomputer with 12-16CPUs, or more.

I am trying to read and, hopefully, learn on clustering several nodes, HA, etc., however the most recent thread I found on this topic in the forum is dated back to 2012 and by googling it's like 'going down the rabbit hole'.

So, I thought it might be easier to ask for directions to experts.
I would like to learn:
1) what is the best clustering package to run on slackware_current?
2) info-related links or tutorials on:
a) how to connect each individual node,
b) what to install in each individual node,
c) specific requirements for each node (libraries, dependencies, etc.)
3) any other related details that could get me going.

Any inputs is appreciated.

Dear OP, did you really believe that this is proper place to ask about academic things like clustering?

I hope you noticed that the most complicated engineering talked around there is kinda like this:

How to install properly the multilib, then NVidia proprietary blob, then Wine64, then running STEAM games?

Seriously?
Why not?
The fact that such questions are rarely/never asked, doesn't mean noone might have any answers for them.
Not saying this might be the *best* place to ask such questions, but there are people around here doing quite a variety of stuff - either for fun or work.


Thanks
--
KarlMag

Well, these things happen when you are a total 'knob' on the subject.

I am reading on the topic at the same time while replying so, let's see if i can make more sense of that question:

Out of the 4 types of clusters, a) High Performance Clusters, b) High Availability Clusters, c) Load Balancing Clusters and d) Storage Clusters I would like to build a supercomputer to achieve 'a' where the processing power of many CPUs is combined.

Is there a package/software for slackware-current what would allow me to achieve it?

Amen to that.
Looks like LuckyCyborg's scoffing answer is born out of the limits of his own imagination.
I built a HA cluster on Slackware in 2002 using such things like nmbd, drbd and heartbeat, things have progressed quite a bit since then.

The scope of your question is really way to big to give any really meaningful (technical at least), general answer to.

First of all you need to figure out what type of clustering you want to do.
This is because the answer to "how to do that?" depends very much on what you want to achieve.

High availabily (e.g) webservers? A compile cluster? Number crunching? Parallel software development/execution? Rending? Cloud services? (Those are typically high availability/clustered in the back ends). Or about a gazillion other things...

There is no "one size fits all", I'm afraid.

I'm afraid I don't know too much about these things in-depth, but I have thought about and tried to research some of it a bit.

And I am also curious about what you are planning to build/set up (the hardware) :-)

Thanks
--
KarlMag

Hey, take a look at slurm: https://slurm.schedmd.com/documentation.html.

I worked with a cluster of computers (nodes) using slurm, and learning how to work with slurm should cover most of your questions.

OP, while posting in the Slackware forum (yay!), perhaps peek at Proxmox for HA clustering. Free/libre, AGPL, based on Debian.

Thank Alien bob,
your answer somehow explains my thread "Linux Clustering in 2020".

All the information I keep finding on the net are outdated (this for example, dated 2017, http://www.rocksclusters.org/).

I don't seem to find a reliable and recent tutorial/how-to page that helps users with clustering on linux, despite most world server systems run on linux.

All i was looking for was a link with updated info; I can read it myself!

Regards

Then, probably you want a Beowulf Cluster.

But, I never heard of someone building a Beowulf cluster with Slackware. However, a starting point could be there:

https://mitpress.mit.edu/books/beowu...omputing-linux

Yes, the suggested reading is a MIT published book.

High Performance Clusters

Parallel a few 4xoctacore XEOn processor to get something like 100+ core of processing power for 1) cpu mining and 2) 4K video editing/rendering

Aaah, Beowulf cluster.

I set up one back in the late 1990s.
Unfortunately not on Slackware, since the person ordering and paying for the cluster had different preferences (SuSE IIRC).
5 node dual CPU P3-<something>. Those where the days. Brought up my Seti score a good bit before it was put into production ;-)

Thanks
--
KarlMag

I think that this is exactly what i was looking for. )

Thanks

edit
I just noticed that all the installs are for centOS (

Yeah multiple dual or quad 6-8+ core (and maybe HT) CPU machines adds up the core/thread count quite quickly.
Sounds like you have rather nice hardware available to you. (Or plan to get.)

I'm guessing the software you need depends on those specific use cases.
Probably both CPU mining and video editing/rendering just need you to set up some quite general machines you can log into (presumably from a central node). But exact setup will be dictated by the specific software you want to run.

I did take a shallow look into how clustering with Blender would work. To me the specific setup of that seemed a bit backwards and non-intuitive (to me at least - might say more about me than anything else), and ended up never actually tried to set it up. At least not so far. Maybe I will try again at some point.

Thanks
--
KarlMag

https://slackbuilds.org/repository/14.2/network/slurm/

?

Thanks
--
KarlMag