[SOLVED] Multi-thread builds via SBOPKG/slackbuilds?
SlackwareThis Forum is for the discussion of Slackware Linux.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
you can see --jobserver-fds=<whatever> lines. That would be ok, but only when <whatever> is exactly the same everywhere and every time.
As you can see it is changed in the example. That means a new jobserver was started and that means you have more than one jobserver for the same build. Which is obviously wrong.
Also you can see messages like: 'warning: -jN forced in submake: disabling jobserver mode' which means almost the same, the number of child processes is now unpredictable.
Last edited by pan64; 11-11-2020 at 03:00 AM.
Reason: typo
Is NUMJOBS supported with gnu's make without calling it as an argument? Because I can't find any documentation online talking about the NUMJOBS environment variable. Almost everything I've found online to set make to automatically use parallel jobs mentions to set it via MAKEFLAGS. Is NUMJOBS only used if it is passed to make (ie make $NUMJOBS)? The SBo templates don't contain the NUMJOBS variable, so I don't know if or how they would pick it up.
Based on a previous post, it seems most of Slackware's official scripts specifically call make with $NUMJOBS as an argument, but only 7 scripts on SBo call make with $NUMJOBS as an argument (it would be 8, but the draco script sets NUMJOBS further up, however, it doesn't actually specify it on the make line). Does this mean that only those 7 scripts (and the 700+ official scripts) would be able to use parallel jobs?
There is no documentation NUMJOBS. It is a made up variable used in Slackware SlackBuild scripts to store the value of the make NUMber of JOBS option for passing to make in the script. That where I got the idea for using it in my SlackBuild scripts. FYI, it's used in Alien Bob scripts too.
Last edited by chrisretusn; 11-11-2020 at 02:58 AM.
you can see --jobserver-fds=<whatever> lines. That would be ok, but only when <whatever> is exactly the same everywhere and every time.
As you can see it is changed in the example. That means a new jobserver was started and that means you have more than one jobserver for the same build. Which is obviously wrong.
Also you can see messages like: 'warning: -jN forced in submake: disabling jobserver mode' which means almost the same, the number of child processes is now unpredictable.
But this is likely due to a poorly made Makefile, right? (No offense meant, since I believe it was intentional.) You still see the --jobserver-fds=<whatever> line change even when forcing -j1.
And this is still leaving me confused. Are you advocating to not specify more than one job during a make process? If that isn't your intention, I saw the same output whether I used the MAKEFLAGS (which is -j17 on my computer) and manually specifying -j5. Does this mean that having a -j<whatever> specified in MAKEFLAGS is a decent method to ensure parallel jobs when running make?
Quote:
Originally Posted by chrisretusn
There is no documentation NUMJOBS. It is a made up variable used in Slackware SlackBuild scripts to store the value of the make NUMber of JOBS option for passing to make in the script. That where I got the idea for using it in my SlackBuild scripts. FYI, it's used in Alien Bob scripts too.
That is what I suspected. Setting NUMJOBS only works when SlackBuilds explicitly use it, so for most of SBo, it won't make a lick of difference.
But this is likely due to a poorly made Makefile, right? (No offense meant, since I believe it was intentional.) You still see the --jobserver-fds=<whatever> line change even when forcing -j1.
No, that is not the content of the makefile, but the [mis]usage of MAKEFLAGS and -jN. --jobserver is used internally by make to establish communication among the make and submakes. Different jobserver fds values mean the communication is broken, there are more than one make processes acting as jobserver (which makes the full job control useless).
The general rule is: you are allowed to specify -jN only once either as command line flag or via MAKEFLAGS (before invoking make). Specifying -j inside makefiles or overriding MAKEFLAGS will forcibly disable jobserver mode.
No, that is not the content of the makefile, but the [mis]usage of MAKEFLAGS and -jN. --jobserver is used internally by make to establish communication among the make and submakes. Different jobserver fds values mean the communication is broken, there are more than one make processes acting as jobserver (which makes the full job control useless).
The general rule is: you are allowed to specify -jN only once either as command line flag or via MAKEFLAGS (before invoking make). Specifying -j inside makefiles or overriding MAKEFLAGS will forcibly disable jobserver mode.
So again, I'm confused. The misuse of MAKEFLAGS and -jN seemed to occur in the Makefile, not with my environment MAKEFLAGS variable or passing -jN to the make command. Those things shouldn't be in a normal Makefile, right?
To try and finalize this, what is the recommendation to ensure multiple parallel jobs are able to be done? I'm never going to touch the inside of a Makefile to adjust how many jobs are going to be done and I have a systemwide environment variable of MAKEFLAGS="-j17". Is this ok or am I "overriding" MAKEFLAGS? It felt like I was being told I was wrong, but all subsequent posts seem to indicate what I'm doing is fine.
Sorry for this back and forth... I hate being confused.
in that case yes, you can say that is an incorrect makefile. But that was only a simplified example. A system wide env var can be ok, but there can be conflict if (for example) it was specified in ~/.bashrc.
A system wide env var can be ok, but there can be conflict if (for example) it was specified in ~/.bashrc.
I'm not seeing why it would matter where the MAKEFLAGS was specified as long as the jobs weren't adjusted in the Makefile. Even if you take the system MAKEFLAGS (that contains a -jN) and add a -jN to the variable in the .bashrc, it will still use the last -jN. The last -jN in MAKEFLAGS overrides all previous -jNs specified in MAKEFLAGS. In addition, the last -jN on the make line will override all the previous -jNs that were specified, whether in MAKEFLAGS or as an argument to make itself.
So, I'm still not seeing any reason to avoid specifying job count in $MAKEFLAGS wherever a user likes, whether systemwide as a sourced file in /etc/profile.d/ as I have, on a per user basis using a .bashrc, or for a single build by manually specifying MAKEFLAGS="-j12" before calling make or the running a SlackBuild. Users should (most likely) stay away from adjusting job options within a Makefile and it seems developers should as well, but if they are, they may have a good reason for it, probably understanding that many users from many distros use MAKEFLAGS to specify job count.
The below seems to indicate it doesn't matter how or where you specify MAKEFLAGS.
Code:
jbhansen@craven-moorhead:/tmp/make-test$ echo $MAKEFLAGS
-j17
jbhansen@craven-moorhead:/tmp/make-test$ MAKEFLAGS="${MAKEFLAGS} -j1"
jbhansen@craven-moorhead:/tmp/make-test$ echo $MAKEFLAGS
-j17 -j1
jbhansen@craven-moorhead:/tmp/make-test$ make
make b
make[1]: Entering directory '/tmp/make-test'
make c
make[2]: Entering directory '/tmp/make-test'
make -j6 d
make[3]: Entering directory '/tmp/make-test'
export MAKEFLAGS=-j7 ; \
make e
make[4]: Entering directory '/tmp/make-test'
echo e
e
echo w -j --jobserver-fds=5,6
w -j --jobserver-fds=5,6
make[4]: Leaving directory '/tmp/make-test'
echo w -j --jobserver-fds=3,4
w -j --jobserver-fds=3,4
make[3]: Leaving directory '/tmp/make-test'
echo w
w
make[2]: Leaving directory '/tmp/make-test'
echo w
w
make[1]: Leaving directory '/tmp/make-test'
echo
jbhansen@craven-moorhead:/tmp/make-test$ make -j5
make b
make[1]: Entering directory '/tmp/make-test'
make c
make[2]: Entering directory '/tmp/make-test'
make -j6 d
make[3]: Entering directory '/tmp/make-test'
make[3]: warning: -jN forced in submake: disabling jobserver mode.
export MAKEFLAGS=-j7 ; \
make e
make[4]: Entering directory '/tmp/make-test'
echo e
e
echo w -j --jobserver-fds=5,6
w -j --jobserver-fds=5,6
make[4]: Leaving directory '/tmp/make-test'
echo w -j --jobserver-fds=3,4
w -j --jobserver-fds=3,4
make[3]: Leaving directory '/tmp/make-test'
echo w -j --jobserver-fds=3,4
w -j --jobserver-fds=3,4
make[2]: Leaving directory '/tmp/make-test'
echo w -j --jobserver-fds=3,4
w -j --jobserver-fds=3,4
make[1]: Leaving directory '/tmp/make-test'
echo -j --jobserver-fds=3,4
-j --jobserver-fds=3,4
There is no documentation NUMJOBS. It is a made up variable used in Slackware SlackBuild scripts to store the value of the make NUMber of JOBS option for passing to make in the script. That where I got the idea for using it in my SlackBuild scripts. FYI, it's used in Alien Bob scripts too.
Yes, it's an "artifact", I guess, from Patrick's SlackBuild scripts (in the official slackware tree).
And as mentioned also adopted by others.
IIRC the variable originally (if used in a script at all) was set to a specific value. I think it was set '7' or something like that, which was "6 cores + 1" which was a value suitable for Pat's build machine (the value could have been something else, but I *think* he had a 6 core build machine at some point at least).
The NUMJOBS variable evolved over time to what it is today. I.e set to either "$(expr $(nproc) + 1)" or whatever value for it you pass to the SlackBuild script at invocation.
I can see that being useful since you might run the SlackBuild scripts on machines with various core/thread counts, or even fire up your favourite icecream cluster and build on all the cores you have available to you at once.
Because it reports cpu units available to the current process and with '--all' units available to the system. Your AMD Ryzen most likely has 8 cores per socket, 2 threads per core with 1 socket (8 x 2 x 1 = 16)
My AMD Phenom(tm) II X4 has 4 cores per socket, 1 thread per core with socket (4 x 1 x 1 = 4)
Wish I had your Ryzen
--
Yes, you're correct about my Ryzen's core count (1700x, 8c/16t).
And yes, it's quite nice, so I can't blame you for wanting one
But even with that (and even though I don't compile that much that often, usually), I do keep thinking of how nice it would be to have a cluster of machines to dispatch build jobs to, just to have them done a bit quicker. (Especially if the core/thread count would be in the dozen if not three digit range. - one can dream... )
I'm not seeing why it would matter where the MAKEFLAGS was specified as long as the jobs weren't adjusted in the Makefile.
This is an example again (how can you fool the make system without actually altering the makefiles:
Code:
echo "echo /tmp/bashrc" > /tmp/bashrc
echo 'export MAKEFLAGS="-j12"' >> /tmp/bashrc
export BASH_ENV=/tmp/bashrc
make -j8 a
with this makefile:
Code:
a:
$(MAKE) b
echo $(MAKEFLAGS)
b:
$(MAKE) c
echo $(MAKEFLAGS)
c:
$(MAKE) d
echo $(MAKEFLAGS)
d:
$(MAKE) e
echo $(MAKEFLAGS)
e:
@echo w
echo $(MAKEFLAGS)
will produce this result:
Code:
/tmp$ make -j8 a
make b
/tmp/bashrc
make[1]: Entering directory '/tmp'
make c
/tmp/bashrc
make[2]: Entering directory '/tmp'
make d
/tmp/bashrc
make[3]: Entering directory '/tmp'
make e
/tmp/bashrc
make[4]: Entering directory '/tmp'
/tmp/bashrc
w
echo w -j12 --jobserver-auth=11,12
/tmp/bashrc
w -j12 --jobserver-auth=11,12
make[4]: Leaving directory '/tmp'
echo w -j12 --jobserver-auth=9,10
/tmp/bashrc
w -j12 --jobserver-auth=9,10
make[3]: Leaving directory '/tmp'
echo w -j12 --jobserver-auth=7,8
/tmp/bashrc
w -j12 --jobserver-auth=7,8
make[2]: Leaving directory '/tmp'
echo w -j12 --jobserver-auth=5,6
/tmp/bashrc
w -j12 --jobserver-auth=5,6
make[1]: Leaving directory '/tmp'
echo -j8 --jobserver-auth=3,4
/tmp/bashrc
-j8 --jobserver-auth=3,4
which I think would be the generally expected result - command line overrides configuration file(s) which overrides environment variable. Anyway that has always been my expectation.
And further, it is unsurprising to end up with 5 jobserver processes if (as in the previous example) you call 'make' 5 separate times. Parallelism in make is surely based on the make system figuring out by itself what can be parallelised - based on the contents of its Makefile. The previous Makefile instructs serial processing i.e. 5 instances of make, one after the other, so of course there can be no parallelism.
chris
Last edited by chris.willing; 11-14-2020 at 05:39 AM.
This is an example again (how can you fool the make system without actually altering the makefiles:
Code:
echo "echo /tmp/bashrc" > /tmp/bashrc
echo 'export MAKEFLAGS="-j12"' >> /tmp/bashrc
export BASH_ENV=/tmp/bashrc
make -j8 a
with this makefile:
Code:
a:
$(MAKE) b
echo $(MAKEFLAGS)
b:
$(MAKE) c
echo $(MAKEFLAGS)
c:
$(MAKE) d
echo $(MAKEFLAGS)
d:
$(MAKE) e
echo $(MAKEFLAGS)
e:
@echo w
echo $(MAKEFLAGS)
will produce this result:
Code:
/tmp$ make -j8 a
make b
/tmp/bashrc
make[1]: Entering directory '/tmp'
make c
/tmp/bashrc
make[2]: Entering directory '/tmp'
make d
/tmp/bashrc
make[3]: Entering directory '/tmp'
make e
/tmp/bashrc
make[4]: Entering directory '/tmp'
/tmp/bashrc
w
echo w -j12 --jobserver-auth=11,12
/tmp/bashrc
w -j12 --jobserver-auth=11,12
make[4]: Leaving directory '/tmp'
echo w -j12 --jobserver-auth=9,10
/tmp/bashrc
w -j12 --jobserver-auth=9,10
make[3]: Leaving directory '/tmp'
echo w -j12 --jobserver-auth=7,8
/tmp/bashrc
w -j12 --jobserver-auth=7,8
make[2]: Leaving directory '/tmp'
echo w -j12 --jobserver-auth=5,6
/tmp/bashrc
w -j12 --jobserver-auth=5,6
make[1]: Leaving directory '/tmp'
echo -j8 --jobserver-auth=3,4
/tmp/bashrc
-j8 --jobserver-auth=3,4
That means again 5 jobserver processes.
All your examples seem to hinge on a poorly designed Makefile. While I'm sure they exist, they don't seem to be common. You still have yet to explain what your points are with these (or maybe I am too dense to understand your points). Are you encouraging users to not use parallelism with make?
Therefore, for both serial and parallel styled Makefiles (properly constituted), it's clear that a single jobserver is used.
If you want to run multiple instances of 'make', as the earlier example Makefile was contrived to do, then it's only logical that there will be multiple jobserver instances. I don't see how that "tricks" make into doing anything unexpected.
All your examples seem to hinge on a poorly designed Makefile. While I'm sure they exist, they don't seem to be common. You still have yet to explain what your points are with these (or maybe I am too dense to understand your points). Are you encouraging users to not use parallelism with make?
for simplicity I created only one makefile, but imagine all of these targets (a, b, c, d, e) are defined in different makefiles in different directories.
So for example:
Code:
$(MAKE) b
# would be
$(MAKE) -C <some dir> [other flags, whatever] b
That is the usual approach, but in this [my] example the real command is irrelevant.
for simplicity I created only one makefile, but imagine all of these targets (a, b, c, d, e) are defined in different makefiles in different directories.
So for example:
Code:
$(MAKE) b
# would be
$(MAKE) -C <some dir> [other flags, whatever] b
That is the usual approach, but in this [my] example the real command is irrelevant.
You missed answering the latter part of what you quoted.
You still have yet to explain what your points are with these (or maybe I am too dense to understand your points). Are you encouraging users to not use parallelism with make?
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.