LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware
User Name
Password
Slackware This Forum is for the discussion of Slackware Linux.

Notices


Reply
  Search this Thread
Old 01-30-2019, 07:55 PM   #1
1337_powerslacker
Member
 
Registered: Nov 2009
Distribution: Slackware64-current
Posts: 727
Blog Entries: 1

Rep: Reputation: 473Reputation: 473Reputation: 473Reputation: 473Reputation: 473
[HOWTO] Set up icecream


As some of you may have noticed, Pat V has added icecream to -current. What others may not know is what purpose for which icecream was created. In a nutshell, icecream is a centralized model for distributed compilation of packages; similar to distcc, but uses a central "server" to coordinate compiling.

Why would someone want to use this?
  1. A lot of users nowadays have lots of computers lying around their house with no purpose; just taking up space, but their owners are unwilling to throw away what still works.
  2. There are some packages which users may want to compile themselves, but one machine won't cut it (takes too long). Sharing the load among multiple machines cuts down on the load that one machine had to do in the past.
  3. We are a community-based distro; true to its roots, Slackware's strength lies in the willingness of its users to improve on deficiencies, such as compiling their own packages (Perhaps taking the load off of core members who need a break)?

How to set it up

Software

I would recommend a fresh install of -current, and exclude E,KDE,X,XAP,& XFCE as these won't be needed for the client machines.

Post-install, set your machine hostname to whatever your fancy is; icecream ignores this and uses the domain name (next prompt), so make sure that all of the machines you intend to use for icecream are in the same domain! Next, select static IP and assign it an IP of the form 192.168.x.x, where the first x is usually 0, and the second x can be any number between 1 and 254. (0 and 255 are used for special purposes).

On first boot, the client machines should have the rc.iceccd startup script made executable. Select the machine you intend to use as a scheduler machine and make the rc.icecc-scheduler script executable. Only one machine can have this script executable! This is usually the main machine. I usually don't need to set up an additional user account, as these machines are on a private network and totally inaccessible from the Internet;and, per the icecream wiki, root access is necessary for it to run. No additional setup for icecream is necessary.

From the icecream README.md:
Quote:
It is possible to run the scheduler and the daemon on one machine and only the daemon on another, thus forming a compile cluster with two nodes.
Quote:
On the scheduler machine, in your user's .bashrc file, add this line:
Code:
export CC=/usr/bin/icecc
This will tell the make command to use the icecc program (which is a wrapper program for gcc), because icecc and gcc reside in the same directory.

Execute
Code:
. ~/.bashrc
if you don't want to reboot. This will force bash to re-read its config file and incorporate the change you made.
EDIT: Via volkerdi:

Quote:
There's actually no need to define CC because if /etc/profile.d/icecream.sh sees either iceccd or icecc-scheduler running on the machine, it will add /usr/libexec/icecc/bin to the beginning of the $PATH, and this directory contains symlinks from all the usual compiler names to /usr/bin/icecc.

Really, all there is to do is make /etc/rc.d/rc.icecc-scheduler executable on one machine, and /etc/rc.d/rc.iceccd executable on all the machines that will be part of the compile cluster. And then set -j high enough.
Hardware

Obviously, several computers are involved. Just how many is actually immaterial to icecream, but can accommodate what spare computers you have lying around.

I highly recommend connecting all computers through a gigabit switch,as they are inexpensive. Make sure that the computer running the scheduler can see all of the client computers via ping.

I have all my machines hooked up to a keyboard and monitor for local control, but if you like to ssh into your machines "remotely", feel free to do so.

That's it! You can use any number of packages, but I used a kernel compile to demonstrate icecc's power. The number of jobs used will obviously be dependent on the total number of cores in your cluster. I have a total of 22 (1 octa-core machine, 3 quad-core machines and 1 dual-core). I add one extra job per machine, so I use a command like this:

Code:
make -j 27
Quote:
The load on my main machine dropped significantly, from ~115 degrees F to ~80 degrees, and compile time dropped appreciably as well.
After having also enabled the daemon on my scheduler machine, the temp did increase to near my previous temp (about 7-10 degrees cooler) for a much lesser period of time.

This is obviously a basic tutorial, and I don't claim to be an expert in icecream, so feel free to send constructive critique my way and/or suggestions for improvement.

Thanks for reading!

Last edited by 1337_powerslacker; 02-06-2019 at 12:38 PM. Reason: Clarification (per volkerdi & chris.willing)
 
Old 01-30-2019, 09:01 PM   #2
chris.willing
Member
 
Registered: Jun 2014
Location: Brisbane, Australia
Distribution: Slackware,LFS
Posts: 567

Rep: Reputation: 344Reputation: 344Reputation: 344Reputation: 344
Quote:
Originally Posted by 1337_powerslacker View Post
...
so make sure that all of the machines you intend to use for icecream have the same domain name!
...
This instruction doesn't make sense to me. Could you clarify please?

chris
 
Old 01-30-2019, 09:12 PM   #3
1337_powerslacker
Member
 
Registered: Nov 2009
Distribution: Slackware64-current
Posts: 727

Original Poster
Blog Entries: 1

Rep: Reputation: 473Reputation: 473Reputation: 473Reputation: 473Reputation: 473
Quote:
Originally Posted by chris.willing View Post
This instruction doesn't make sense to me. Could you clarify please?

chris
The hostname is actually made up of two parts, which is formally called the Fully Qualified Domain Name, or FQDN. The first part of the hostname is whatever you want to call your machine. As an example, my sig shows my icecream setup. The second part of the hostname is the domain name, which forms a specific group of networked computers. This is what icecream uses to address machines which also have iceccd running. I use the simple 'org' as a domain name. Not fancy, but it works.

Here's something you can read if you're still confused.

Last edited by 1337_powerslacker; 01-30-2019 at 09:14 PM.
 
Old 01-30-2019, 10:42 PM   #4
volkerdi
Slackware Maintainer
 
Registered: Dec 2002
Location: Minnesota
Distribution: Slackware! :-)
Posts: 1,698

Rep: Reputation: 5260Reputation: 5260Reputation: 5260Reputation: 5260Reputation: 5260Reputation: 5260Reputation: 5260Reputation: 5260Reputation: 5260Reputation: 5260Reputation: 5260
Quote:
Originally Posted by 1337_powerslacker View Post
On the scheduler machine, in your user's .bashrc file, add this line:
Code:
export CC=/usr/bin/icecc
This will tell the make command to use the icecc program (which is a wrapper program for gcc), because icecc and gcc reside in the same directory.
There's actually no need to define CC because if /etc/profile.d/icecream.sh sees either iceccd or icecc-scheduler running on the machine, it will add /usr/libexec/icecc/bin to the beginning of the $PATH, and this directory contains symlinks from all the usual compiler names to /usr/bin/icecc.

Really, all there is to do is make /etc/rc.d/rc.icecc-scheduler executable on one machine, and /etc/rc.d/rc.iceccd executable on all the machines that will be part of the compile cluster. And then set -j high enough. Per ppr:kut, it doesn't hurt to set the job number too high with this, because the scheduler won't start too many jobs.

Just in case you might find it useful, here's a script I use to return the number of jobs available from the scheduler:

Code:
#!/bin/sh

SCHEDULER="zap.slackware.lan"

cat << EOF | nc $SCHEDULER 8766 | grep jobs= | cut -f 3 -d = | cut -f 2 -d / | cut -f 1 -d ' ' | paste -sd+ - | bc
listcs
exit
EOF
 
4 members found this post helpful.
Old 01-30-2019, 10:54 PM   #5
1337_powerslacker
Member
 
Registered: Nov 2009
Distribution: Slackware64-current
Posts: 727

Original Poster
Blog Entries: 1

Rep: Reputation: 473Reputation: 473Reputation: 473Reputation: 473Reputation: 473
Quote:
Originally Posted by volkerdi View Post
There's actually no need to define CC because if /etc/profile.d/icecream.sh sees either iceccd or icecc-scheduler running on the machine, it will add /usr/libexec/icecc/bin to the beginning of the $PATH, and this directory contains symlinks from all the usual compiler names to /usr/bin/icecc.

Really, all there is to do is make /etc/rc.d/rc.icecc-scheduler executable on one machine, and /etc/rc.d/rc.iceccd executable on all the machines that will be part of the compile cluster. And then set -j high enough. Per ppr:kut, it doesn't hurt to set the job number too high with this, because the scheduler won't start too many jobs.

Just in case you might find it useful, here's a script I use to return the number of jobs available from the scheduler:

Code:
#!/bin/sh

SCHEDULER="zap.slackware.lan"

cat << EOF | nc $SCHEDULER 8766 | grep jobs= | cut -f 3 -d = | cut -f 2 -d / | cut -f 1 -d ' ' | paste -sd+ - | bc
listcs
exit
EOF
Thanks much, Pat! I stand corrected! And yes, the script was very helpful
 
Old 01-30-2019, 10:55 PM   #6
chris.willing
Member
 
Registered: Jun 2014
Location: Brisbane, Australia
Distribution: Slackware,LFS
Posts: 567

Rep: Reputation: 344Reputation: 344Reputation: 344Reputation: 344
Quote:
Originally Posted by 1337_powerslacker View Post
The hostname is actually made up of two parts, which is formally called the Fully Qualified Domain Name, or FQDN. The first part of the hostname is whatever you want to call your machine. As an example, my sig shows my icecream setup. The second part of the hostname is the domain name, which forms a specific group of networked computers. This is what icecream uses to address machines which also have iceccd running. I use the simple 'org' as a domain name. Not fancy, but it works.
Sorry, I misinterpreted your statement as saying that all machines must have the same FQDN. Saying that all machines must be in the same domain would have been clearer (to me).

chris
 
1 members found this post helpful.
Old 01-30-2019, 11:57 PM   #7
1337_powerslacker
Member
 
Registered: Nov 2009
Distribution: Slackware64-current
Posts: 727

Original Poster
Blog Entries: 1

Rep: Reputation: 473Reputation: 473Reputation: 473Reputation: 473Reputation: 473
Quote:
Originally Posted by chris.willing View Post
Sorry, I misinterpreted your statement as saying that all machines must have the same FQDN. Saying that all machines must be in the same domain would have been clearer (to me).

chris
Thanks for the post. I edited my OP to make that clearer.
 
Old 01-31-2019, 04:04 AM   #8
allend
LQ 5k Club
 
Registered: Oct 2003
Location: Melbourne
Distribution: Slackware-current
Posts: 5,192

Rep: Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853Reputation: 1853
I have iptables based firewalls and have these rules. (I found the icecream man page a little confusing on this.)
Code:
## Icecream distributed compiler
#   Daemon computers use TCP on port 10245
$IPTABLES -A INPUT -i $EXTIF -p tcp -s $LOCAL_LAN --dport 10245 -j ACCEPT
#   Scheduler computer uses UDP broadcast on port 8765 to locate daemon computers
$IPTABLES -A INPUT -i $EXTIF -p udp -s $LOCAL_LAN --sport 8765 -j ACCEPT
$IPTABLES -A INPUT -i $EXTIF -p udp -s $LOCAL_LAN --dport 8765 -j ACCEPT
 
1 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Icecream distrobuted cross compiler djsmiley2k Linux - Software 3 10-21-2018 02:10 PM
Icecream Slackbuild fail mlpa Slackware 1 09-14-2012 09:36 AM
Linux Multicasting HOWTO.HOWTO Join Multicast Groups ar24458 Linux - Networking 0 04-02-2007 01:55 AM
Howto update KDE and howto switch off kdm/gdm Canaris Linux - Software 1 06-15-2003 08:40 PM
Howto set up the ati Radeon64DDR for performance in 3D environments? Debianewb Linux - Hardware 7 05-05-2003 07:05 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware

All times are GMT -5. The time now is 02:41 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration