LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware
User Name
Password
Slackware This Forum is for the discussion of Slackware Linux.

Notices


Reply
  Search this Thread
Old 11-21-2020, 04:27 PM   #1
however
Member
 
Registered: Jan 2019
Distribution: slackware current
Posts: 101

Rep: Reputation: Disabled
2020 Clustering question


hi everyone,

I am trying to learn how to connect servers together (minimum of 2, but it could be more in the near future) all with XEON CPUs, as I am planning to build a supercomputer with 12-16CPUs, or more.

I am trying to read and, hopefully, learn on clustering several nodes, HA, etc., however the most recent thread I found on this topic in the forum is dated back to 2012 and by googling it's like 'going down the rabbit hole'.

So, I thought it might be easier to ask for directions to experts.
I would like to learn:
1) what is the best clustering package to run on slackware_current?
2) info-related links or tutorials on:
a) how to connect each individual node,
b) what to install in each individual node,
c) specific requirements for each node (libraries, dependencies, etc.)
3) any other related details that could get me going.

Any inputs is appreciated.

Last edited by however; 11-21-2020 at 04:37 PM.
 
Old 11-21-2020, 04:47 PM   #2
LuckyCyborg
Member
 
Registered: Mar 2010
Posts: 821

Rep: Reputation: 590Reputation: 590Reputation: 590Reputation: 590Reputation: 590Reputation: 590
Dear OP, did you really believe that this is proper place to ask about academic things like clustering?

I hope you noticed that the most complicated engineering talked around there is kinda like this:

How to install properly the multilib, then NVidia proprietary blob, then Wine64, then running STEAM games?

Last edited by LuckyCyborg; 11-21-2020 at 04:51 PM.
 
Old 11-21-2020, 04:56 PM   #3
karlmag
Member
 
Registered: Apr 2014
Distribution: Slackware
Posts: 78

Rep: Reputation: Disabled
Quote:
Originally Posted by LuckyCyborg View Post
Dear OP, did you really believe that this is proper place to ask about academic things like clustering?

I hope you noticed that the most complicated engineering talked around there is kinda like this:

How to install properly the multilib, then NVidia proprietary blob, then Wine64, then running STEAM games?

Seriously?
Why not?
The fact that such questions are rarely/never asked, doesn't mean noone might have any answers for them.
Not saying this might be the *best* place to ask such questions, but there are people around here doing quite a variety of stuff - either for fun or work.


Thanks
--
KarlMag
 
2 members found this post helpful.
Old 11-21-2020, 04:59 PM   #4
however
Member
 
Registered: Jan 2019
Distribution: slackware current
Posts: 101

Original Poster
Rep: Reputation: Disabled
Well, these things happen when you are a total 'knob' on the subject.

I am reading on the topic at the same time while replying so, let's see if i can make more sense of that question:

Out of the 4 types of clusters, a) High Performance Clusters, b) High Availability Clusters, c) Load Balancing Clusters and d) Storage Clusters I would like to build a supercomputer to achieve 'a' where the processing power of many CPUs is combined.

Is there a package/software for slackware-current what would allow me to achieve it?
 
Old 11-21-2020, 05:01 PM   #5
Alien Bob
Slackware Contributor
 
Registered: Sep 2005
Location: Eindhoven, The Netherlands
Distribution: Slackware
Posts: 8,241

Rep: Reputation: 7253Reputation: 7253Reputation: 7253Reputation: 7253Reputation: 7253Reputation: 7253Reputation: 7253Reputation: 7253Reputation: 7253Reputation: 7253Reputation: 7253
Quote:
Originally Posted by karlmag View Post
Seriously?
Why not?
The fact that such questions are rarely/never asked, doesn't mean noone might have any answers for them.
Not saying this might be the *best* place to ask such questions, but there are people around here doing quite a variety of stuff - either for fun or work.


Thanks
--
KarlMag
Amen to that.
Looks like LuckyCyborg's scoffing answer is born out of the limits of his own imagination.
I built a HA cluster on Slackware in 2002 using such things like nmbd, drbd and heartbeat, things have progressed quite a bit since then.
 
1 members found this post helpful.
Old 11-21-2020, 05:06 PM   #6
karlmag
Member
 
Registered: Apr 2014
Distribution: Slackware
Posts: 78

Rep: Reputation: Disabled
Quote:
Originally Posted by however View Post
hi everyone,

I am trying to learn how to connect servers together (minimum of 2, but it could be more in the near future) all with XEON CPUs, as I am planning to build a supercomputer with 12-16CPUs, or more.

I am trying to read and, hopefully, learn on clustering several nodes, HA, etc., however the most recent thread I found on this topic in the forum is dated back to 2012 and by googling it's like 'going down the rabbit hole'.

So, I thought it might be easier to ask for directions to experts.
I would like to learn:
1) what is the best clustering package to run on slackware_current?
2) info-related links or tutorials on:
a) how to connect each individual node,
b) what to install in each individual node,
c) specific requirements for each node (libraries, dependencies, etc.)
3) any other related details that could get me going.

Any inputs is appreciated.

The scope of your question is really way to big to give any really meaningful (technical at least), general answer to.

First of all you need to figure out what type of clustering you want to do.
This is because the answer to "how to do that?" depends very much on what you want to achieve.

High availabily (e.g) webservers? A compile cluster? Number crunching? Parallel software development/execution? Rending? Cloud services? (Those are typically high availability/clustered in the back ends). Or about a gazillion other things...

There is no "one size fits all", I'm afraid.

I'm afraid I don't know too much about these things in-depth, but I have thought about and tried to research some of it a bit.

And I am also curious about what you are planning to build/set up (the hardware) :-)

Thanks
--
KarlMag

Last edited by karlmag; 11-21-2020 at 05:08 PM. Reason: Added a word
 
Old 11-21-2020, 05:07 PM   #7
Bindestreck
Member
 
Registered: Jul 2011
Location: Sweden
Distribution: Slackware
Posts: 713

Rep: Reputation: 346Reputation: 346Reputation: 346Reputation: 346
Quote:
Originally Posted by however View Post
hi everyone,

I am trying to learn how to connect servers together (minimum of 2, but it could be more in the near future) all with XEON CPUs, as I am planning to build a supercomputer with 12-16CPUs, or more.

I am trying to read and, hopefully, learn on clustering several nodes, HA, etc., however the most recent thread I found on this topic in the forum is dated back to 2012 and by googling it's like 'going down the rabbit hole'.

So, I thought it might be easier to ask for directions to experts.
I would like to learn:
1) what is the best clustering package to run on slackware_current?
2) info-related links or tutorials on:
a) how to connect each individual node,
b) what to install in each individual node,
c) specific requirements for each node (libraries, dependencies, etc.)
3) any other related details that could get me going.

Any inputs is appreciated.
Hey, take a look at slurm: https://slurm.schedmd.com/documentation.html.

I worked with a cluster of computers (nodes) using slurm, and learning how to work with slurm should cover most of your questions.
 
Old 11-21-2020, 05:08 PM   #8
upnort
Senior Member
 
Registered: Oct 2014
Distribution: Slackware, Proxmox, Debian, CentOS
Posts: 1,692

Rep: Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015Reputation: 1015
OP, while posting in the Slackware forum (yay!), perhaps peek at Proxmox for HA clustering. Free/libre, AGPL, based on Debian.
 
Old 11-21-2020, 05:11 PM   #9
however
Member
 
Registered: Jan 2019
Distribution: slackware current
Posts: 101

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Alien Bob View Post
Amen to that.
I built a HA cluster on Slackware in 2002 using such things like nmbd, drbd and heartbeat, things have progressed quite a bit since then.
Thank Alien bob,
your answer somehow explains my thread "Linux Clustering in 2020".

All the information I keep finding on the net are outdated (this for example, dated 2017, http://www.rocksclusters.org/).

I don't seem to find a reliable and recent tutorial/how-to page that helps users with clustering on linux, despite most world server systems run on linux.

All i was looking for was a link with updated info; I can read it myself!

Regards
 
Old 11-21-2020, 05:17 PM   #10
LuckyCyborg
Member
 
Registered: Mar 2010
Posts: 821

Rep: Reputation: 590Reputation: 590Reputation: 590Reputation: 590Reputation: 590Reputation: 590
Quote:
Originally Posted by however View Post
Well, these things happen when you are a total 'knob' on the subject.

I am reading on the topic at the same time while replying so, let's see if i can make more sense of that question:

Out of the 4 types of clusters, a) High Performance Clusters, b) High Availability Clusters, c) Load Balancing Clusters and d) Storage Clusters I would like to build a supercomputer to achieve 'a' where the processing power of many CPUs is combined.

Is there a package/software for slackware-current what would allow me to achieve it?
Then, probably you want a Beowulf Cluster.

But, I never heard of someone building a Beowulf cluster with Slackware. However, a starting point could be there:

https://mitpress.mit.edu/books/beowu...omputing-linux

Yes, the suggested reading is a MIT published book.

Last edited by LuckyCyborg; 11-21-2020 at 05:19 PM.
 
Old 11-21-2020, 05:19 PM   #11
however
Member
 
Registered: Jan 2019
Distribution: slackware current
Posts: 101

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by karlmag View Post
The scope of your question is really way to big to give any really meaningful (technical at least), general answer to.

First of all you need to figure out what type of clustering you want to do.
High Performance Clusters

Quote:

And I am also curious about what you are planning to build/set up (the hardware) :-)

Thanks
--
KarlMag
Parallel a few 4xoctacore XEOn processor to get something like 100+ core of processing power for 1) cpu mining and 2) 4K video editing/rendering
 
Old 11-21-2020, 05:25 PM   #12
karlmag
Member
 
Registered: Apr 2014
Distribution: Slackware
Posts: 78

Rep: Reputation: Disabled
Quote:
Originally Posted by LuckyCyborg View Post
Then, probably you want a Beowulf Cluster.

But, I never heard of someone building a Beowulf cluster with Slackware. However, a starting point could be there:

https://mitpress.mit.edu/books/beowu...omputing-linux

Yes, the suggested reading is a MIT published book.

Aaah, Beowulf cluster.

I set up one back in the late 1990s.
Unfortunately not on Slackware, since the person ordering and paying for the cluster had different preferences (SuSE IIRC).
5 node dual CPU P3-<something>. Those where the days. Brought up my Seti score a good bit before it was put into production ;-)

Thanks
--
KarlMag
 
Old 11-21-2020, 05:35 PM   #13
however
Member
 
Registered: Jan 2019
Distribution: slackware current
Posts: 101

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Bindestreck View Post
Hey, take a look at slurm: https://slurm.schedmd.com/documentation.html.

I worked with a cluster of computers (nodes) using slurm, and learning how to work with slurm should cover most of your questions.
I think that this is exactly what i was looking for. )

Thanks

edit
I just noticed that all the installs are for centOS (

Last edited by however; 11-21-2020 at 05:37 PM.
 
Old 11-21-2020, 05:40 PM   #14
karlmag
Member
 
Registered: Apr 2014
Distribution: Slackware
Posts: 78

Rep: Reputation: Disabled
Quote:
Originally Posted by however View Post
Parallel a few 4xoctacore XEOn processor to get something like 100+ core of processing power for 1) cpu mining and 2) 4K video editing/rendering

Yeah multiple dual or quad 6-8+ core (and maybe HT) CPU machines adds up the core/thread count quite quickly.
Sounds like you have rather nice hardware available to you. (Or plan to get.)

I'm guessing the software you need depends on those specific use cases.
Probably both CPU mining and video editing/rendering just need you to set up some quite general machines you can log into (presumably from a central node). But exact setup will be dictated by the specific software you want to run.

I did take a shallow look into how clustering with Blender would work. To me the specific setup of that seemed a bit backwards and non-intuitive (to me at least - might say more about me than anything else), and ended up never actually tried to set it up. At least not so far. Maybe I will try again at some point.

Thanks
--
KarlMag
 
Old 11-21-2020, 05:43 PM   #15
karlmag
Member
 
Registered: Apr 2014
Distribution: Slackware
Posts: 78

Rep: Reputation: Disabled
Quote:
Originally Posted by however View Post

edit
I just noticed that all the installs are for centOS (

https://slackbuilds.org/repository/14.2/network/slurm/

?

Thanks
--
KarlMag
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Setting up KVM Hypervisor on SparkyLinux 2020.9 rolled forward up to 11/08/2020 (VENV) LXer Syndicated Linux News 0 11-08-2020 05:21 PM
[SOLVED] Please help,i have a error while upgrading my kali linux 2020.1 to 2020.3 Deep2 Linux - Newbie 2 08-28-2020 10:24 AM
LXer: Akademy 2020 and GUADEC 2020 Linux Events Move to Online Conferences LXer Syndicated Linux News 0 04-18-2020 04:33 AM
Lvm clustering vrs NFS clustering call_krushna Linux - Server 4 06-06-2012 08:01 AM
Squid Clustering Question fivefive Linux - Networking 1 10-18-2003 01:13 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware

All times are GMT -5. The time now is 09:43 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration