[SOLVED] Multi-thread builds via SBOPKG/slackbuilds?

Originally Posted by bassmadrigal I've never used /etc/default/. I'm not sure what is required to make sure those things are included for users.

I think the original /etc/default recommendation was misguided (at best incomplete), since /etc/default is a directory so the advice that OP saw "about setting makeflags in /etc/default" just doesn't make sense as it stands.

The /etc/default directory contains files which provide default values for the application they're (usually) named after e.g. /etc/default/grub contains default values for grub. It's up to any application which wants to set defaults like that to source the relevant file. For shell scripts, an example you can see is in /usr/sbin/grub-mkconfig which has the lines:

if test -f ${sysconfdir}/default/grub ; then
  . ${sysconfdir}/default/grub
fi

(where sysconfdir has earlier been set to /etc).

Another example is at /usr/libexec/lxc/lxc-net which has a line:

[ ! -f $distrosysconfdir/lxc ] || . $distrosysconfdir/lxc

(where distrosysconfdir has been set to /etc/default)

So, sbopkg could pick up default values if it sourced some file(s) in the /etc/default directory - probably /etc/default/sbopkg. However there's nothing I can see in sbopkg which sources any file in /etc/default.

chris

Originally Posted by chrisretusn I want the of number of CPU's passed as '$(nproc --all)', thus no +1. The plus one is a recommended thing anyway. Outside of a one shot test run, I see no reason to use parameter substitution ${var:-$DEFAULT} for NUMJOBS. On the rare occasion it needs to be something other than '$(nproc --all)' I'll modify the script. I disagree regarding passing NUMJOBS via MAKEFLAGS being the correct way to go. Make adds command line options to MAKEFLAGS anyway. If I was to pass '--jobs' to MAKEFLAGS instead of via the command line, this would run both times with '--jobs', not what I want. # Run make with NUMJOBS, if fails, attempt once without NUMJOBS. make $NUMJOBS || make || exit 1

I see that I was a bit unclear in my edit there. Sorry about that.
I did not mean to pass the NUMJOBS variable via MAKEFLAGS, but I agree it quite looks like that is what I was saying.
You would most likely edit the MAKEFLAGS directly.... Or whichever other way that may be more appropriate.

Anywho, I just intended to make aware of how it's done in the slackware proper tree (in case that was unfamiliar).

One thing though, $(nproc) ( or $(nproc --all) ) returns all threads on a computer, if SMT exists and is enabled.
$(nproc) ( or $(nproc --all) ) on my 6-core Intel and 8-core Amd Ryzen, both with SMT enabled report back 12 and 16 respectively.
It doesn't seem like it can just report back physical cores only (unless SMT is disabled).

Thanks
--
KarlMag

Originally Posted by karlmag I see that I was a bit unclear in my edit there. Sorry about that.

What was unclear with:

Originally Posted by karlmag NUMJOBS=${NUMJOBS:-" -j$(expr $(nproc) + 1) "} Edit: Obviously this variable is specific to these scripts. Setting it as a part of MAKEFLAGS would be (I believe) the correct way to go about it more generally.

I disagree with setting --jobs=N as part of MAKEFLAGS as the correct way to go about it. For my purposes and the purpose of those 735 SlackBuild scripts that set NUMJOBS in the slackware64-current tree. While there is nothing wrong with passing make options to MAKEFLAGS, the intent of MAKEFLAGS is to pass make command line options to child sub-makes. As I already explained, setting --jobs=N in MAKEFLAGS would not be appropriate my use case to run make again without --jobs=N if the first run fails as both runs would use --jobs=N.

This might be worth a read:
https://www.gnu.org/software/make/ma...Recursion.html

I did not mean to pass the NUMJOBS variable via MAKEFLAGS, but I agree it quite looks like that is what I was saying. You would most likely edit the MAKEFLAGS directly.... Or whichever other way that may be more appropriate.

No actually that's not how I read it. You are miss reading me.

Originally Posted by chrisretusn I disagree regarding passing NUMJOBS via MAKEFLAGS being the correct way to go. Make adds command line options to MAKEFLAGS anyway. If I was to pass '--jobs' to MAKEFLAGS instead of via the command line, this would run both times with '--jobs', not what I want.

I could have perhaps worded that differently. but the sentence in bold should have gave you a hint. I was referring to the contents of NUMJOBS. Not actually doing a MAKEFLAGS=$NUMJOBS.

Anywho, I just intended to make aware of how it's done in the slackware proper tree (in case that was unfamiliar).

LOL, okay.

One thing though, $(nproc) ( or $(nproc --all) ) returns all threads on a computer, if SMT exists and is enabled. $(nproc) ( or $(nproc --all) ) on my 6-core Intel and 8-core Amd Ryzen, both with SMT enabled report back 12 and 16 respectively. It doesn't seem like it can just report back physical cores only (unless SMT is disabled).

Because it reports cpu units available to the current process and with '--all' units available to the system. Your AMD Ryzen most likely has 8 cores per socket, 2 threads per core with 1 socket (8 x 2 x 1 = 16)
My AMD Phenom(tm) II X4 has 4 cores per socket, 1 thread per core with socket (4 x 1 x 1 = 4)

Wish I had your Ryzen
--

Originally Posted by chrisretusn What was unclear with: I disagree with setting --jobs=N as part of MAKEFLAGS as the correct way to go about it. For my purposes and the purpose of those 735 SlackBuild scripts that set NUMJOBS in the slackware64-current tree. While there is nothing wrong with passing make options to MAKEFLAGS, the intent of MAKEFLAGS is to pass make command line options to child sub-makes. As I already explained, setting --jobs=N in MAKEFLAGS would not be appropriate my use case to run make again without --jobs=N if the first run fails as both runs would use --jobs=N. This might be worth a read: https://www.gnu.org/software/make/ma...Recursion.html

That's what I planned to link too. You are right. Using -j XXX in MAKEFLAGS is just bad practice and discommended.

additionally the usage of: make -j XXX || make means incorrect makefiles

About parallel execution: in general you can use any number between 10xN and 2xN (N=numproc), that will not really affect the execution time.
If you wish you can use -l instead of -j and you can avoid overloading without calculating/knowing the optimal value for -j.

Originally Posted by chrisretusn What was unclear with: I disagree with setting --jobs=N as part of MAKEFLAGS as the correct way to go about it. For my purposes and the purpose of those 735 SlackBuild scripts that set NUMJOBS in the slackware64-current tree. While there is nothing wrong with passing make options to MAKEFLAGS, the intent of MAKEFLAGS is to pass make command line options to child sub-makes. As I already explained, setting --jobs=N in MAKEFLAGS would not be appropriate my use case to run make again without --jobs=N if the first run fails as both runs would use --jobs=N. This might be worth a read: https://www.gnu.org/software/make/ma...Recursion.html

Most of this went over my head, but if a make fails with my -j17 in my MAKEFLAGS, I can switch it to make -j1 and it will continue single threaded, or are you just talking about make -j17 || make, because that could be fixed by simply putting make -j17 || make -j1. I've not seen any issues in having this option set in MAKEFLAGS in the 3 years I've had the file in place.

Originally Posted by bassmadrigal Most of this went over my head, but if a make fails with my -j17 in my MAKEFLAGS, I can switch it to make -j1 and it will continue single threaded, or are you just talking about make -j17 || make, because that could be fixed by simply putting make -j17 || make -j1. I've not seen any issues in having this option set in MAKEFLAGS in the 3 years I've had the file in place.

There are two problems:
1. imagine a 6 level build system, where the first make will recursively go into 5 levels of [sub]directories running [sub]makes. When every make will be executed with -j 8 then theoretically it may mean 8^6 compile processes (>200 000) which cannot be handled, so the system will be overloaded. That's why putting -j into MAKEFLAGS is not suggested.
In general it will only slow down the build process, but also it may make the build server useless (unresponsive). It depends on the size of the build.

2. let we have a correct build system with correct makefiles and make -j 17 dies. That means make -j1 will (must) die too.

Originally Posted by pan64 1. imagine a 6 level build system, where the first make will recursively go into 5 levels of [sub]directories running [sub]makes. When every make will be executed with -j 8 then theoretically it may mean 8^6 compile processes (>200 000) which cannot be handled, so the system will be overloaded. That's why putting -j into MAKEFLAGS is not suggested. In general it will only slow down the build process, but also it may make the build server useless (unresponsive). It depends on the size of the build.

So, NUMJOBS makes the parent make aware of the total number of jobs and it will ensure the parent and any child make combined will never spawn more than the total number of jobs in NUMJOBS? Whereas if I set it in MAKEFLAGS, then every make process, regardless if it is the parent or child will attempt to spawn the total number of jobs, which could lead you to an exponential number of jobs running parallel?

In practice, I've never seen more than what I suspect is 17 cpp processes when building a package (never actually counted, but I don't think I've ever noticed an extreme number of processes in htop while building software). I've built a lot of software on this machine over the last 3 years (just checked and currently installed software that was installed after the file was in place is 841 -- there's been many more that were built, installed, and then upgraded with newer versions and programs that were built that didn't lead to a package (testing or my kernels) and I have never noticed any issues with slowing the build process or making the machine unresponsive while building.

Originally Posted by pan64 2. let we have a correct build system with correct makefiles and make -j 17 dies. That means make -j1 will (must) die too.

I understand this, but what does this have to do with setting the jobs in MAKEFLAGS? Whenever I attempt to build a program and get an error, I will almost always rerun make with -j1 to find the specific error (since many times additional errors are because of a previous failure and the rest of the make jobs eventually error out). If NUMJOBS was set separately, wouldn't you still need to specify make || make -j1?

Note: I'm not doing this to be argumentative, but trying to open up my understanding of the process. I could easily switch my make-export.sh to NUMJOBS instead of MAKEFLAGS without learning anything about it, just knowing others say it is better, but I prefer to actually understand the problem.

Actually, in doing a bit of research after my post, it seems that make is smart enough to use -j in MAKEFLAGS and not spawn more than the number of jobs specified between the parent and all child makes.

The ‘-j’ option is a special case (see Parallel Execution). If you set it to some numeric value ‘N’ and your operating system supports it (most any UNIX system will; others typically won’t), the parent make and all the sub-makes will communicate to ensure that there are only ‘N’ jobs running at the same time between them all. Note that any job that is marked recursive (see Instead of Executing Recipes) doesn’t count against the total jobs (otherwise we could get ‘N’ sub-makes running and have no slots left over for any real work!) SOURCE: https://www.gnu.org/software/make/ma..._002fRecursion

Am I misunderstanding this? I felt like I had done a lot of research 3 years back when I set this, but it's all a fuzz right now (like the Showtime Rotisserie, I "set it and forget it").

Originally Posted by bassmadrigal Most of this went over my head, but if a make fails with my -j17 in my MAKEFLAGS, I can switch it to make -j1 and it will continue single threaded, or are you just talking about make -j17 || make, because that could be fixed by simply putting make -j17 || make -j1. I've not seen any issues in having this option set in MAKEFLAGS in the 3 years I've had the file in place.

I would not expect you to have any ether. I see that a few post have been added since I started this reply. Much of what I was going to say has be said already. I will say this I disagree with pan64 on his reasoning for not using jobs in MAKEFLAGS.

The '-j17' option allows 17 make recipes or jobs to run simultaneously in parallel. The '-j1' restricts it to one and is the same as running make with no options. If you pass '-j' then make will try to run as many recipes simultaneously as possible. From the make manual:
- The ‘-j’ option is a special case. If you set it to some numeric value ‘N’ the parent make and all the sub-makes will communicate to ensure that there are only ‘N’ jobs running at the same time between them all. Note that any job that is marked recursive doesn’t count against the total jobs (otherwise we could get ‘N’ sub-makes running and have no slots left over for any real work!)

In other words make is smart enough to handle this.

My reasons for not using MAKEFLAGS (which have nothing to do with jobs).
A few snippets from the make manual:
- Flags are passed automatically to the sub-make through the variable MAKEFLAGS. This variable is set up automatically by make to contain the flag letters that make received.
- As a consequence, every sub-make gets a value for MAKEFLAGS in its environment. In response, it takes the flags from that value and processes them as if they had been given as arguments.
- Likewise variables defined on the command line are passed to the sub-make through MAKEFLAGS
- The MAKEFLAGS variable can also be useful if you want to have certain options, such as ‘-k’, set each time you run make. You simply put a value for MAKEFLAGS in your environment.
- If you do put MAKEFLAGS in your environment, you should be sure not to include any options that will drastically affect the actions of make and undermine the purpose of makefiles and of make itself.

It's not exactly encouraging the use of MAKEFLAGS, but you can if you want too. I don't want to set an environment variable for make. I don't see the logic of setting it in my SlackBuilds either. I normally set one make option, that is the jobs option. I assign that to a descriptive as to purpose variable NUMJOBS and pass it via the command line in the script. This is how it's done in the Slacware SlackBuilds. That is good enough for me.

If I did use MAKEFLAGS, then my script would be like this "make | make -j1". The first make with take the jobs from MAKEFLAGS, the second would override that with '-j1'

Run it with make -j5 and check (this is a sample makefile):

a:
	$(MAKE) b
	echo $(MAKEFLAGS)

b:
	$(MAKE) c
	echo $(MAKEFLAGS)

c:
	$(MAKE) -j6 d
	echo $(MAKEFLAGS)

d:
	export MAKEFLAGS=-j7 ; \
	$(MAKE) e
	echo $(MAKEFLAGS)

e:
	echo e
	echo $(MAKEFLAGS)

Is NUMJOBS supported with gnu's make without calling it as an argument? Because I can't find any documentation online talking about the NUMJOBS environment variable. Almost everything I've found online to set make to automatically use parallel jobs mentions to set it via MAKEFLAGS. Is NUMJOBS only used if it is passed to make (ie make $NUMJOBS)? The SBo templates don't contain the NUMJOBS variable, so I don't know if or how they would pick it up.

Based on a previous post, it seems most of Slackware's official scripts specifically call make with $NUMJOBS as an argument, but only 7 scripts on SBo call make with $NUMJOBS as an argument (it would be 8, but the draco script sets NUMJOBS further up, however, it doesn't actually specify it on the make line). Does this mean that only those 7 scripts (and the 700+ official scripts) would be able to use parallel jobs?

Originally Posted by bassmadrigal Is NUMJOBS supported with gnu's make without calling it as an argument?

if I remember well, this environment variable is not handled automatically by gnu make.

Originally Posted by pan64 Run it with make -j5 and check (this is a sample makefile): a: $(MAKE) b echo $(MAKEFLAGS) b: $(MAKE) c echo $(MAKEFLAGS) c: $(MAKE) -j6 d echo $(MAKEFLAGS) d: export MAKEFLAGS=-j7 ; \ $(MAKE) e echo $(MAKEFLAGS) e: echo e echo $(MAKEFLAGS)

I'm not at my home computer to test this, but is this contradicting what's stated by gnu.org? They state that -j is special and make will talk between the parent and child make processes to ensure they don't go over the total number of jobs specified with -j. I'll requote it here for simplicity:

The ‘-j’ option is a special case (see Parallel Execution). If you set it to some numeric value ‘N’ and your operating system supports it (most any UNIX system will; others typically won’t), the parent make and all the sub-makes will communicate to ensure that there are only ‘N’ jobs running at the same time between them all. Note that any job that is marked recursive (see Instead of Executing Recipes) doesn’t count against the total jobs (otherwise we could get ‘N’ sub-makes running and have no slots left over for any real work!) SOURCE: https://www.gnu.org/software/make/ma..._002fRecursion

Am I misunderstanding what's stated here or does it not work like is stated?

Originally Posted by pan64 if I remember well, this environment variable is not handled automatically by gnu make.

So, setting NUMJOBS does absolutely nothing if the SlackBuild doesn't include make $NUMJOBS? Other than manually modifying all SlackBuilds on SBo, is there a better way to automatically enable parallel builds if MAKEFLAGS isn't recommended? (I still haven't seen anything online not recommended MAKEFLAGS to set the job level.)

that is an example to demonstrate how does it really work. You will see what's happening when you specify -jN or MAKEFLAGS. I would say the result is:
you can specify either MAKEFLAGS or -jN before you invoke make, but must not override it (even in .bashrc or similar) and must not use -jN inside.
NUMJOBS is used only by scripts to calculate something, I would rather use MAKEFLAGS, but that is only my personal preference.

Originally Posted by pan64 Run it with make -j5 and check (this is a sample makefile): a: $(MAKE) b echo $(MAKEFLAGS) b: $(MAKE) c echo $(MAKEFLAGS) c: $(MAKE) -j6 d echo $(MAKEFLAGS) d: export MAKEFLAGS=-j7 ; \ $(MAKE) e echo $(MAKEFLAGS) e: echo e echo $(MAKEFLAGS)

Trying this at home didn't help clear up confusion at all. I'm not even sure what I'm supposed to be looking at, but there is no difference in the output when using the MAKEFLAGS containing -j17 and running make -j5 (make -j1 is unique). Was the point of this to say we shouldn't parallelize make jobs? There's only been a handful of times I've seen any issues with parallel make jobs, and that's easily cleared up with make -j1. And the savings in compile time with multiple make jobs is more than worth the occasional hiccup when I need to manually limit it to 1 job.

This is the output from testing that Makefile. I colored the lines containing my commands to make it easier to see the output.

jbhansen@craven-moorhead:/tmp/make-test$ echo $MAKEFLAGS
-j17
jbhansen@craven-moorhead:/tmp/make-test$ make
make b
make[1]: Entering directory '/tmp/make-test'
make c
make[2]: Entering directory '/tmp/make-test'
make -j6 d
make[3]: Entering directory '/tmp/make-test'
make[3]: warning: -jN forced in submake: disabling jobserver mode.
export MAKEFLAGS=-j7 ; \
make e
make[4]: Entering directory '/tmp/make-test'
echo e
e
echo w -j --jobserver-fds=5,6
w -j --jobserver-fds=5,6
make[4]: Leaving directory '/tmp/make-test'
echo w -j --jobserver-fds=3,4
w -j --jobserver-fds=3,4
make[3]: Leaving directory '/tmp/make-test'
echo w -j --jobserver-fds=3,4
w -j --jobserver-fds=3,4
make[2]: Leaving directory '/tmp/make-test'
echo w -j --jobserver-fds=3,4
w -j --jobserver-fds=3,4
make[1]: Leaving directory '/tmp/make-test'
echo  -j --jobserver-fds=3,4
-j --jobserver-fds=3,4
jbhansen@craven-moorhead:/tmp/make-test$ make -j5
make b
make[1]: Entering directory '/tmp/make-test'
make c
make[2]: Entering directory '/tmp/make-test'
make -j6 d
make[3]: Entering directory '/tmp/make-test'
make[3]: warning: -jN forced in submake: disabling jobserver mode.
export MAKEFLAGS=-j7 ; \
make e
make[4]: Entering directory '/tmp/make-test'
echo e
e
echo w -j --jobserver-fds=5,6
w -j --jobserver-fds=5,6
make[4]: Leaving directory '/tmp/make-test'
echo w -j --jobserver-fds=3,4
w -j --jobserver-fds=3,4
make[3]: Leaving directory '/tmp/make-test'
echo w -j --jobserver-fds=3,4
w -j --jobserver-fds=3,4
make[2]: Leaving directory '/tmp/make-test'
echo w -j --jobserver-fds=3,4
w -j --jobserver-fds=3,4
make[1]: Leaving directory '/tmp/make-test'
echo  -j --jobserver-fds=3,4
-j --jobserver-fds=3,4
jbhansen@craven-moorhead:/tmp/make-test$ make -j1
make b
make[1]: Entering directory '/tmp/make-test'
make c
make[2]: Entering directory '/tmp/make-test'
make -j6 d
make[3]: Entering directory '/tmp/make-test'
export MAKEFLAGS=-j7 ; \
make e
make[4]: Entering directory '/tmp/make-test'
echo e
e
echo w -j --jobserver-fds=5,6
w -j --jobserver-fds=5,6
make[4]: Leaving directory '/tmp/make-test'
echo w -j --jobserver-fds=3,4
w -j --jobserver-fds=3,4
make[3]: Leaving directory '/tmp/make-test'
echo w
w
make[2]: Leaving directory '/tmp/make-test'
echo w
w
make[1]: Leaving directory '/tmp/make-test'
echo

I'm still confused on how I should set make to use multiple jobs automatically (I'm not willing to go through and modify all SlackBuilds from SBo to benefit from that and I definitely want to use my 8C/16T CPU).