LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware
User Name
Password
Slackware This Forum is for the discussion of Slackware Linux.

Notices


Reply
  Search this Thread
Old 01-07-2019, 12:46 PM   #1
slac-in-the-box
Member
 
Registered: Mar 2010
Location: oregon
Distribution: slackware64-14.2
Posts: 287

Rep: Reputation: 111Reputation: 111
Can makepkg use all available cpus?


Is there a way to get makepkg to use all available cpus like the way you can get make to do it with the -j $(nproc) option?
 
Old 01-07-2019, 01:55 PM   #2
ehartman
Member
 
Registered: Jul 2007
Location: Delft, The Netherlands
Distribution: Slackware
Posts: 301

Rep: Reputation: 133Reputation: 133
Quote:
Originally Posted by slac-in-the-box View Post
Is there a way to get makepkg to use all available cpus like the way you can get make to do it with the -j $(nproc) option?
No, makepkg is a shell script, thus runs in a shell (/bin/sh) and a single shell runs in a single cpu cq core. Make is a compiled program so can do a little bit more.
 
1 members found this post helpful.
Old 01-07-2019, 02:21 PM   #3
montagdude
Senior Member
 
Registered: Apr 2016
Distribution: Slackware
Posts: 1,328

Rep: Reputation: 1002Reputation: 1002Reputation: 1002Reputation: 1002Reputation: 1002Reputation: 1002Reputation: 1002Reputation: 1002
Quote:
Originally Posted by ehartman View Post
No, makepkg is a shell script, thus runs in a shell (/bin/sh) and a single shell runs in a single cpu cq core. Make is a compiled program so can do a little bit more.
While I believe you are correct that makepkg cannot use more than a single core, it is not correct that shell scripts cannot run commands in parallel.

https://www.gnu.org/software/bash/ma...-Parallel.html
 
1 members found this post helpful.
Old 01-07-2019, 02:43 PM   #4
Petri Kaukasoina
Member
 
Registered: Mar 2007
Posts: 364

Rep: Reputation: 213Reputation: 213Reputation: 213
In slackware current, there is a new makepkg option
Code:
--threads <number> For xz/plzip compressed packages, set the max
number of threads to be used for compression. Only has an
effect on large packages. For plzip, the default is equal to
the number of CPU threads available on the machine. For xz,
the default is equal to 2 (due to commonly occuring memory
related failures when using many threads with multi-threaded
xz compression).
 
1 members found this post helpful.
Old 01-07-2019, 04:10 PM   #5
ehartman
Member
 
Registered: Jul 2007
Location: Delft, The Netherlands
Distribution: Slackware
Posts: 301

Rep: Reputation: 133Reputation: 133
Quote:
Originally Posted by montagdude View Post
While I believe you are correct that makepkg cannot use more than a single core, it is not correct that shell scripts cannot run commands in parallel.
Of course a script CAN make use of parallel options to the programs it executes, but the script itself is pure single-threaded. Only external (non-bash ones) commands have the chance to run on another cpu/core or even multiple ones.
 
Old 01-07-2019, 04:12 PM   #6
slac-in-the-box
Member
 
Registered: Mar 2010
Location: oregon
Distribution: slackware64-14.2
Posts: 287

Original Poster
Rep: Reputation: 111Reputation: 111
So I found the /sbin/makepkg script and looked at it, and decided that I should not try to figure out if and where gnu parallel could speed it up--I would break it,lol; but I look forward to speeding up the compression with the --threads option! I wonder if that current /sbin/makepkg would work on the 14.2-stable branch... I upgraded the number of cpus of a cloud instance just to make a kernel and slackware package, and downgraded the moment it was through so I wouldn't pay to much, and noticed that, with all the cpus, the kernel was compiling faster than the slackware package was created... I have it scripted so I don't accidentally leave the upgraded instance on, draining the bank; but I'm gonna change the script to downgrade after the kernel compile, and then use makepkg back on 1 cpu. And I'll check out --threads. Thanks for the information. Slack On!
 
Old 01-07-2019, 04:18 PM   #7
montagdude
Senior Member
 
Registered: Apr 2016
Distribution: Slackware
Posts: 1,328

Rep: Reputation: 1002Reputation: 1002Reputation: 1002Reputation: 1002Reputation: 1002Reputation: 1002Reputation: 1002Reputation: 1002
Quote:
Originally Posted by ehartman View Post
Of course a script CAN make use of parallel options to the programs it executes, but the script itself is pure single-threaded. Only external (non-bash ones) commands have the chance to run on another cpu/core or even multiple ones.
Sure, but as far as I understand, the question was not about invoking makepkg in parallel in order to make many packages at once, but rather whether it can perform some of its internal operations in parallel. Per the comparison with Make, Make is invoked once but can compile many source files simultaneously. The fact that makepkg is a shell script does not prevent it from doing something analogous to that, which is what post #2 implied.

Last edited by montagdude; 01-07-2019 at 04:21 PM.
 
Old 01-07-2019, 04:44 PM   #8
ZhaoLin1457
Member
 
Registered: Jan 2018
Posts: 198

Rep: Reputation: 175Reputation: 175
Quote:
Originally Posted by slac-in-the-box View Post
So I found the /sbin/makepkg script and looked at it, and decided that I should not try to figure out if and where gnu parallel could speed it up--I would break it,lol; but I look forward to speeding up the compression with the --threads option! I wonder if that current /sbin/makepkg would work on the 14.2-stable branch... I upgraded the number of cpus of a cloud instance just to make a kernel and slackware package, and downgraded the moment it was through so I wouldn't pay to much, and noticed that, with all the cpus, the kernel was compiling faster than the slackware package was created... I have it scripted so I don't accidentally leave the upgraded instance on, draining the bank; but I'm gonna change the script to downgrade after the kernel compile, and then use makepkg back on 1 cpu. And I'll check out --threads. Thanks for the information. Slack On!
Excuse me, and the advantages of a parallelized makepkg will be what?

Just for records, my box spends 9 hours to compile the Qt5, then it needs only a couple of minutes to package it.

As if I care if it creates the package in 1 or 3 minutes, after spending 9 hours with the fans at max.

Last edited by ZhaoLin1457; 01-07-2019 at 04:51 PM.
 
Old 01-07-2019, 05:11 PM   #9
bassmadrigal
LQ Guru
 
Registered: Nov 2003
Location: West Jordan, UT, USA
Distribution: Slackware
Posts: 5,632

Rep: Reputation: 3389Reputation: 3389Reputation: 3389Reputation: 3389Reputation: 3389Reputation: 3389Reputation: 3389Reputation: 3389Reputation: 3389Reputation: 3389Reputation: 3389
Quote:
Originally Posted by ZhaoLin1457 View Post
Excuse me, and the advantages of a parallelized makepkg will be what?
I think it is more to use multiple threads when compressing the package. Sure, in the grand scheme of things it may only be a fraction of the time used to compile, but savings are savings. I wish the linker could use multiple threads when compiling... when you have 15 cores unused and 1 core at 100%, it's a little frustrating when it seems to go forever.
 
1 members found this post helpful.
Old 01-07-2019, 10:35 PM   #10
Richard Cranium
Senior Member
 
Registered: Apr 2009
Location: Carrollton, Texas
Distribution: Slackware64 14.2
Posts: 3,145

Rep: Reputation: 1514Reputation: 1514Reputation: 1514Reputation: 1514Reputation: 1514Reputation: 1514Reputation: 1514Reputation: 1514Reputation: 1514Reputation: 1514Reputation: 1514
Quote:
Originally Posted by ZhaoLin1457 View Post
As if I care if it creates the package in 1 or 3 minutes, after spending 9 hours with the fans at max.
If your laptop battery died at 9 hours and 2 minutes...
 
1 members found this post helpful.
Old 01-08-2019, 05:01 AM   #11
slac-in-the-box
Member
 
Registered: Mar 2010
Location: oregon
Distribution: slackware64-14.2
Posts: 287

Original Poster
Rep: Reputation: 111Reputation: 111
Quote:
Excuse me, and the advantages of a parallelized makepkg will be what?

Just for records, my box spends 9 hours to compile the Qt5, then it needs only a couple of minutes to package it.

As if I care if it creates the package in 1 or 3 minutes, after spending 9 hours with the fans at max.
With 64cpus and 58GB RAM, for an 8MB kernel:
Code:
Kernel compiled in 262 seconds! 
Slackware package built in 4015 seconds
When including the modules in the slackware package, makepkg becomes the bottleneck. I think the --threads option gonna help this later number!

I just connect to a virtual machine running slackware64-14.2, and can allocate cpu/ram resources accordingly for time I'm connected. In the end it was only a couple of bucks to save 9 hours of time (less than I would have spent on coffee in such a period).
 
Old 01-08-2019, 07:20 AM   #12
ehartman
Member
 
Registered: Jul 2007
Location: Delft, The Netherlands
Distribution: Slackware
Posts: 301

Rep: Reputation: 133Reputation: 133
Quote:
Originally Posted by slac-in-the-box View Post
When including the modules in the slackware package, makepkg becomes the bottleneck. I think the --threads option gonna help this later number!
It will only help on the compression phase and not even that much when you create .txz packages, xz isn't that good in multi-threading (it doesn't even support it when DEcompressing, see the man page). Creating .tlz files may be much faster, although less compressed as Pat has included plzip (Parallel Lzip) in -current.
The archiver tar, which does the bulk of the work, is not multi-threaded.

BTW: in my system I've built and installed lbzip2, a parallel bzip2 utility and that helps speeding up that kind of compression.
 
1 members found this post helpful.
Old 01-08-2019, 10:10 AM   #13
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 2,438

Rep: Reputation: 1487Reputation: 1487Reputation: 1487Reputation: 1487Reputation: 1487Reputation: 1487Reputation: 1487Reputation: 1487Reputation: 1487Reputation: 1487
Quote:
Originally Posted by ehartman View Post
in my system I've built and installed lbzip2, a parallel bzip2 utility and that helps speeding up that kind of compression.
I seem to recall Pat adding lbzip2 to current as well. Yes… yes he did

Code:
+--------------------------+
Wed Apr 11 20:38:06 UTC 2018
a/lbzip2-2.5-x86_64-1.txz:  Added.
a/pkgtools-15.0-noarch-7.txz:  Rebuilt.
  explodepkg: support parallel bzip2 (lbzip2). Thanks to ruario.
  installpkg: support parallel bzip2 (lbzip2). Thanks to ruario.
  makepkg: support parallel bzip2 (lbzip2). Thanks to ruario.
  pkgdiff: added tool to compare the file contents of two packages.
And now I remember why I remember it.
 
3 members found this post helpful.
Old 01-11-2019, 06:50 PM   #14
slac-in-the-box
Member
 
Registered: Mar 2010
Location: oregon
Distribution: slackware64-14.2
Posts: 287

Original Poster
Rep: Reputation: 111Reputation: 111
Thanks Everybody: I made an instance for current, and tried a build: lbzip2 squeezed some more seconds out of it!

32cpu / 28.8G ram:

Code:
Slackware package /tmp/kernel-slabc-4.20.0-x86_64-sib4.tbz created.

Kernel compiled in 500 seconds! 
Slackware package built in 2 seconds!
* (quit)
the resultant package was 55M

To be fair, I was making this thread while waiting on makepkg to finish; that particular 4015 second job created a 803MB package, that was a mistake, and unfair to makepkg. It finished in time for me to report the time. But when I looked into it, I had installed packages intended for another device onto the developer instance, and I think they were interfering. Just removed them with removepkg, and slackpkg clean-system, and subsequent makepkg was still a few hundred seconds longer than make. But not after building in current with the lbzip2. Wow. 2 seconds! It sped up my heart rate too.

I did notice that I had to pass to makepkg PKGTYPE="-.tbz" for it to succeed. Thanks Slackware.
 
2 members found this post helpful.
  


Reply

Tags
makepkg nproc


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Linux Apps Are Now Available on More Chromebooks Powered by Intel Braswell CPUs LXer Syndicated Linux News 0 08-12-2018 05:34 AM
can makepkg backup my entire system globaltree Slackware 3 09-15-2009 04:15 PM
How to make "make" use all cpus? kairen Slackware 12 09-07-2007 03:15 PM
create a slack-desc file also learning in prep for use of makepkg acummings Slackware 10 01-27-2007 01:01 PM
anybody use makepkg? servnov Slackware 20 12-23-2005 04:27 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware

All times are GMT -5. The time now is 04:49 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration