LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware
User Name
Password
Slackware This Forum is for the discussion of Slackware Linux.

Notices


Reply
  Search this Thread
Old 11-06-2020, 09:03 PM   #1
obobskivich
Member
 
Registered: Jun 2020
Distribution: Xubuntu / Slackware
Posts: 160

Rep: Reputation: Disabled
Multi-thread builds via SBOPKG/slackbuilds?


I'm playing around with Slackware64 14.2 Multilib on a 48-core system, and running some slackbuilds in sbopkg, and noting that its only using 1-2 threads when compiling. I've read some relatively old threads about setting makeflags in /etc/default and have added (based on the linked thread/post)
Code:
export MAKEFLAGS=" -j40"
It still seems to make no difference (based on watching htop while make is running).

Sources:
https://slackalaxy.com/2014/03/16/bu...ultiple-cores/
https://www.linuxquestions.org/quest...re-4175497383/

The package I'm testing with is barrier and its associated dependencies (avahi, qt5, etc).

Any ideas/tips on this? I know not all packages can build multi-thread but it'd be nice to see at least *some* gain from all the extra CPUs when building at least some of the things.

Last edited by obobskivich; 11-06-2020 at 11:07 PM.
 
Old 11-06-2020, 09:47 PM   #2
drgibbon
Member
 
Registered: Nov 2014
Distribution: Slackware64 -current
Posts: 846

Rep: Reputation: 589Reputation: 589Reputation: 589Reputation: 589Reputation: 589Reputation: 589
I don't know about sbopkg, but with sbotools it's JOBS=40 in /etc/sbotools/sbotools.conf

Last edited by drgibbon; 11-07-2020 at 12:02 AM.
 
1 members found this post helpful.
Old 11-06-2020, 10:02 PM   #3
Daedra
Senior Member
 
Registered: Dec 2005
Location: Springfield, MO
Distribution: Slackware64-14.2
Posts: 1,641

Rep: Reputation: 435Reputation: 435Reputation: 435Reputation: 435Reputation: 435
edit /etc/sbopkg/sbopkg.conf and add export MAKEFLAGS="j 40" here

Code:
# The following variables are required and can be tweaked if desired,
# although this is not recommended.  Note:  rsync already uses
# --archive, --delete, --no-owner, and --exclude in the main sbopkg
# script so there is no need to add those flags here.
DIFF=${DIFF:-diff}
DIFFOPTS=${DIFFOPTS:--u}
RSYNCFLAGS="${RSYNCFLAGS:---verbose --timeout=30}"
WGETFLAGS="${WGETFLAGS:--c --progress=bar:force --timeout=30 --tries=5}"
export MAKEFLAGS="j 40"
 
2 members found this post helpful.
Old 11-06-2020, 10:48 PM   #4
USUARIONUEVO
Senior Member
 
Registered: Apr 2015
Posts: 1,498

Rep: Reputation: 475Reputation: 475Reputation: 475Reputation: 475Reputation: 475
If you want all konsole package managers works with your number of cores , i suggest put the makeflag export under .bashrc

Then , no need edit config for sbopkg , sbotools ,or other manager.


Quote:
export MAKEFLAGS=-j$(nproc)
 
Old 11-06-2020, 10:51 PM   #5
obobskivich
Member
 
Registered: Jun 2020
Distribution: Xubuntu / Slackware
Posts: 160

Original Poster
Rep: Reputation: Disabled
drgibbon: I've never heard of that tool before, and I'll be sure to give it a look. Thanks.

Daedra: I was wondering about that with sbopkg.conf vs default, on adding that line to it, and I'll give that a shot. Do I need it as just 'j 40' or with the dash as '-j 40'?

Thanks for both ideas folks.

Quote:
Originally Posted by USUARIONUEVO View Post
If you want all konsole package managers works with your number of cores , i suggest put the makeflag export under .bashrc

Then , no need edit config for sbopkg , sbotools ,or other manager.

How does this differ from what I did? (Sorry if this is a 'dumb question')

UPDATE:
Did Daedra's suggestion - that one command line is an instant space heater, the second make restarted and all 4 sockets lit up and away we go! It's only using about 3x the power, but it's also moving through compilation substantially faster. Thanks for that tip, and it seems to be working.

Last edited by obobskivich; 11-06-2020 at 11:07 PM.
 
Old 11-06-2020, 11:01 PM   #6
Daedra
Senior Member
 
Registered: Dec 2005
Location: Springfield, MO
Distribution: Slackware64-14.2
Posts: 1,641

Rep: Reputation: 435Reputation: 435Reputation: 435Reputation: 435Reputation: 435
Quote:
Originally Posted by obobskivich View Post
drgibbon: I've never heard of that tool before, and I'll be sure to give it a look. Thanks.

Daedra: I was wondering about that with sbopkg.conf vs default, on adding that line to it, and I'll give that a shot. Do I need it as just 'j 40' or with the dash as '-j 40'?

Thanks for both ideas folks.




How does this differ from what I did? (Sorry if this is a 'dumb question')
I honestly don't know, that's just the syntax I use. Adding the line to sbopkg.conf will only affect sbopkg builds.
 
Old 11-06-2020, 11:02 PM   #7
Daedra
Senior Member
 
Registered: Dec 2005
Location: Springfield, MO
Distribution: Slackware64-14.2
Posts: 1,641

Rep: Reputation: 435Reputation: 435Reputation: 435Reputation: 435Reputation: 435
Quote:
Originally Posted by USUARIONUEVO View Post
If you want all konsole package managers works with your number of cores , i suggest put the makeflag export under .bashrc

Then , no need edit config for sbopkg , sbotools ,or other manager.
This works, but I have never been a fan of doing this, since you will occasionally get the random build that doesn't like parallel jobs.
 
Old 11-07-2020, 04:21 AM   #8
chrisretusn
Senior Member
 
Registered: Dec 2005
Location: Philippines
Distribution: Slackware64-current
Posts: 1,396

Rep: Reputation: 586Reputation: 586Reputation: 586Reputation: 586Reputation: 586Reputation: 586
I put this line in all of my SlackBuild's
Code:
NUMJOBS=-j$(( $(nproc --all) ))
 
Old 11-07-2020, 04:47 AM   #9
willysr
Senior Member
 
Registered: Jul 2004
Location: Jogja, Indonesia
Distribution: Slackware-Current
Posts: 4,228

Rep: Reputation: 1451Reputation: 1451Reputation: 1451Reputation: 1451Reputation: 1451Reputation: 1451Reputation: 1451Reputation: 1451Reputation: 1451Reputation: 1451
unfortunately it's not really recommended in SBo
 
1 members found this post helpful.
Old 11-07-2020, 05:08 AM   #10
OldHolborn
Member
 
Registered: Jul 2012
Distribution: Slackware
Posts: 191

Rep: Reputation: 156Reputation: 156
Quote:
Originally Posted by Daedra View Post
This works, but I have never been a fan of doing this, since you will occasionally get the random build that doesn't like parallel jobs.
from kde.SlackBuild
Code:
make $NUMJOBS || make || exit 1
 
Old 11-07-2020, 07:20 AM   #11
chrisretusn
Senior Member
 
Registered: Dec 2005
Location: Philippines
Distribution: Slackware64-current
Posts: 1,396

Rep: Reputation: 586Reputation: 586Reputation: 586Reputation: 586Reputation: 586Reputation: 586
This is what I put in most of my SlackBuild's
Code:
  make $NUMJOBS 2>&1 | tee $OUTPUT/makej-$PKGNAM.log \
    || make 2>&1 | tee $OUTPUT/make-$PKGNAM.log || exit 1
 
Old 11-07-2020, 07:25 PM   #12
bassmadrigal
LQ Guru
 
Registered: Nov 2003
Location: West Jordan, UT, USA
Distribution: Slackware
Posts: 7,517

Rep: Reputation: 5169Reputation: 5169Reputation: 5169Reputation: 5169Reputation: 5169Reputation: 5169Reputation: 5169Reputation: 5169Reputation: 5169Reputation: 5169Reputation: 5169
I've never used /etc/default/. I'm not sure what is required to make sure those things are included for users.

What I do use is /etc/profile.d/ and I created a new file called make-export.sh. In there, I set MAKEFLAGS to include setting the number of jobs to the number of cores +1 (on my system, it makes it -j17). This change will not affect any logged in users unless they either source the file or log out and then log back in.

Code:
jbhansen@craven-moorhead:~$ cat /etc/profile.d/make-export.sh
#!/bin/bash

# Set make to default to num cores + 1
export MAKEFLAGS="-j$(expr $(nproc) + 1)"
 
Old 11-08-2020, 07:33 PM   #13
karlmag
Member
 
Registered: Apr 2014
Distribution: Slackware
Posts: 78

Rep: Reputation: Disabled
Quote:
Originally Posted by chrisretusn View Post
I put this line in all of my SlackBuild's
Code:
NUMJOBS=-j$(( $(nproc --all) ))

As a kind of side note; in the slackware64-current tree (as of Nov 8th 2020, not including testing and extras), 737 of 901 Slackbuild scripts includes:
Code:
NUMJOBS=${NUMJOBS:-" -j$(expr $(nproc) + 1) "}

Edit:
Obviously this variable is specific to these scripts. Setting it as a part of MAKEFLAGS would be (I believe) the correct way to go about it more generally.

Thanks
--
KarlMag

Last edited by karlmag; 11-08-2020 at 07:36 PM.
 
Old 11-09-2020, 05:57 AM   #14
chrisretusn
Senior Member
 
Registered: Dec 2005
Location: Philippines
Distribution: Slackware64-current
Posts: 1,396

Rep: Reputation: 586Reputation: 586Reputation: 586Reputation: 586Reputation: 586Reputation: 586
Quote:
Originally Posted by karlmag View Post
As a kind of side note; in the slackware64-current tree (as of Nov 8th 2020, not including testing and extras), 737 of 901 Slackbuild scripts includes:
Code:
NUMJOBS=${NUMJOBS:-" -j$(expr $(nproc) + 1) "}

Edit:
Obviously this variable is specific to these scripts. Setting it as a part of MAKEFLAGS would be (I believe) the correct way to go about it more generally.

Thanks
--
KarlMag
I want the of number of CPU's passed as '$(nproc --all)', thus no +1. The plus one is a recommended thing anyway. Outside of a one shot test run, I see no reason to use parameter substitution ${var:-$DEFAULT} for NUMJOBS. On the rare occasion it needs to be something other than '$(nproc --all)' I'll modify the script.

I disagree regarding passing NUMJOBS via MAKEFLAGS being the correct way to go. Make adds command line options to MAKEFLAGS anyway. If I was to pass '--jobs' to MAKEFLAGS instead of via the command line, this would run both times with '--jobs', not what I want.
Code:
# Run make with NUMJOBS, if fails, attempt once without NUMJOBS.
make $NUMJOBS || make || exit 1
 
Old 11-09-2020, 06:17 AM   #15
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 15,606

Rep: Reputation: 5119Reputation: 5119Reputation: 5119Reputation: 5119Reputation: 5119Reputation: 5119Reputation: 5119Reputation: 5119Reputation: 5119Reputation: 5119Reputation: 5119
Yes, in general (not distro related) the parallel execution is driven by the [gnu]make engine itself, the make processes can (and will) communicate with each other.
You must tell the first make what you want (like -j 8) and everything else will be handled automatically by that make.
Using MAKEFLAGS and passing -j options to other [sub]make processes will just switch off this feature and will do something different (which is not what you want).
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Error compiling ffmpeg via Slackbuilds.org using sbopkg BoydRice Slackware 13 11-18-2013 01:12 AM
[SOLVED] what's the difference between a multi-cpu , multi-core and a multi-thread system ? entz Linux - Hardware 11 12-20-2011 05:49 PM
SlackBuilds.org and sbopkg suggestion vharishankar Slackware 2 11-02-2011 10:14 PM
[SOLVED] sbopkg doesn't find slackbuilds 13.1 repository samac Slackware 5 05-27-2010 06:38 PM
Fresh installation of Slackware 13 -64 and Slackbuilds/sbopkg arubin Slackware 6 11-09-2009 02:31 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware

All times are GMT -5. The time now is 09:38 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration