LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Linux From Scratch
User Name
Password
Linux From Scratch This Forum is for the discussion of LFS.
LFS is a project that provides you with the steps necessary to build your own custom Linux system.

Notices


Reply
  Search this Thread
Old 04-11-2013, 06:16 AM   #1
DeeGee
Member
 
Registered: Apr 2013
Distribution: Debian, Linux From Scratch, Mint
Posts: 118

Rep: Reputation: 24
Parallel Make on Pentium 4 HT?


I'm currently waiting for the second pass of gcc to finish its make. As the other packages left have less SBU's I'd like to run a parelle make but am a bit confused on whether it is possible with a Pentium 4 HT as there are two logical CPUs but essentially its just a single core.

Is there any way to check this out?
 
Old 04-11-2013, 07:45 AM   #2
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405
Check the output of the following command:
Code:
cat /proc/cpuinfo
If more then one processor is mentioned you can use export MAKEFLAGS='-j 2' or make -j2

One thing I should mention about using parallel make: Not all packages like this option, if a make (or make check) fails with the following message try removing the -j2 part and rerun the command.

Message pointing to possible -j2 conflict:
Quote:
make[1]: *** [some error] Error 2
make[1]: *** Waiting for unfinished jobs....
BTW: Do not use a number greater then the amount of cpu's (example: if you have 2 cpu's/cores, do _not_ use -j3)

Last edited by druuna; 04-11-2013 at 07:46 AM.
 
1 members found this post helpful.
Old 04-11-2013, 07:51 AM   #3
TobiSGD
Moderator
 
Registered: Dec 2009
Location: Germany
Distribution: Whatever fits the task best
Posts: 17,148
Blog Entries: 2

Rep: Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886
Quote:
Originally Posted by druuna View Post
BTW: Do not use a number greater then the amount of cpu's (example: if you have 2 cpu's/cores, do _not_ use -j3)
It depends: If you have a fast storage subsystem there is no problem with using higher numbers. I do all my compiles on a tmpfs in RAM and I can easily run my compiles with -j10 on my 6-core machine. On a machine with Hyperthreading, especially in its older versions (like the OP's Pentium 4) this is of course not really advisable.
 
1 members found this post helpful.
Old 04-11-2013, 07:58 AM   #4
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405
@TobiSGD: Definitely a truism

I was giving advise with the Pentium 4 HT in mind and the recollection of an earlier thread that came across this issue (the build failing miserably).

We could start a general discussion about the actual added value of assigning more core's/cpu's then actually present, but i'm sure this is not the place to do so
 
1 members found this post helpful.
Old 04-11-2013, 08:00 AM   #5
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,930

Rep: Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321
I do not think it depends on that. Just try -j2, -j3 or even more, and look at the speed. And yes, try to use tmpfs.
 
1 members found this post helpful.
Old 04-15-2013, 10:57 PM   #6
DeeGee
Member
 
Registered: Apr 2013
Distribution: Debian, Linux From Scratch, Mint
Posts: 118

Original Poster
Rep: Reputation: 24
[QUOTE=druuna;4929697]Check the output of the following command:
Code:
cat /proc/cpuinfo
If more then one processor is mentioned you can use export MAKEFLAGS='-j 2' or make -j2

{/QUOTE]

A really dumb question here, but what field shows the number of processors? Mine doesn't show any, so I assume its just a single processor?
 
Old 04-16-2013, 12:13 AM   #7
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,930

Rep: Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321
Quote:
Originally Posted by DeeGee View Post
A really dumb question here, but what field shows the number of processors? Mine doesn't show any, so I assume its just a single processor?
I have a result like this (of cat /proc/cpuinfo):
processor: 0
vendor_id ....
....

processor: 1
vendor ...
...
 
1 members found this post helpful.
Old 04-16-2013, 04:21 AM   #8
DeeGee
Member
 
Registered: Apr 2013
Distribution: Debian, Linux From Scratch, Mint
Posts: 118

Original Poster
Rep: Reputation: 24
Quote:
Originally Posted by pan64 View Post
I have a result like this (of cat /proc/cpuinfo):
processor: 0
vendor_id ....
....

processor: 1
vendor ...
...
Thank you.
Mine only gives info for processor: 0, which I take as meaning its just single core.

Adding to the discussion built up earlier, is it recommended if I use make -j2 ?
 
Old 04-16-2013, 04:30 AM   #9
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405
If you only have one core I wouldn't use the -j2 flag.

The above is with your hardware in mind, although I do think there isn't a point in doing so even if you would have a fast machine (fast storage subsystem / tmpfs in RAM / etc).

I haven't actually tested this but I do believe both threads are being handled by one core (assuming one core cpu and -j2) and this will not increase the build speed. I wouldn't be surprised if it will be slightly (??) slower when using -j2 compared to -j1 (or no -j flag).
 
Old 04-16-2013, 04:31 AM   #10
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,930

Rep: Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321Reputation: 7321
from my point of view you would need to check it, probably helps.
 
Old 04-16-2013, 04:57 AM   #11
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405
Quote:
Originally Posted by pan64 View Post
from my point of view you would need to check it, probably helps.
I just did a little searching and came up with 3 conflicting "rules":

1) amount of cores dictates the -j number (2 cores => -j2),
2) use 1.5 times the amount of cores (2 cores => -j3),
3) use 2 times the amount of cores (2 cores => -j4).

The GNU make page (this one) doesn't help.

I have to agree with pan64: You need to do some testing to figure out what is fastest/best for your scenario.

EDIT: Just quick tested the above rules using binutils and this is the result (I'm having an 8 core cpu):

make -j1 -> real 2m0.806s
make -j8 -> real 0m31.561s
make -j12 -> real 0m31.790s
make -j16 -> real 0m32.232s

Last edited by druuna; 04-16-2013 at 05:16 AM.
 
Old 04-16-2013, 05:33 AM   #12
DeeGee
Member
 
Registered: Apr 2013
Distribution: Debian, Linux From Scratch, Mint
Posts: 118

Original Poster
Rep: Reputation: 24
Well, considering as I have just around four more to go to complete Chapter 5 and they have low SBU values, will just give it a try, for better or for worse.

After searching a bit, I'm a bit confused on how to use tmpfs on RAM. What would be a suitable material to refer?

Thank You!
 
Old 04-16-2013, 05:38 AM   #13
DeeGee
Member
 
Registered: Apr 2013
Distribution: Debian, Linux From Scratch, Mint
Posts: 118

Original Poster
Rep: Reputation: 24
Quote:
Originally Posted by druuna View Post
I just did a little searching and came up with 3 conflicting "rules":

1) amount of cores dictates the -j number (2 cores => -j2),
2) use 1.5 times the amount of cores (2 cores => -j3),
3) use 2 times the amount of cores (2 cores => -j4).

The GNU make page (this one) doesn't help.

I have to agree with pan64: You need to do some testing to figure out what is fastest/best for your scenario.

EDIT: Just quick tested the above rules using binutils and this is the result (I'm having an 8 core cpu):

make -j1 -> real 2m0.806s
make -j8 -> real 0m31.561s
make -j12 -> real 0m31.790s
make -j16 -> real 0m32.232s
Sorry druuna, I didn't see your reply. This seems very interesting and promising. I'll do some checking and be back.

BTW adding a typical newbie question: would it be a problem if I run binutils or gcc again to test this? Because I guess its no point of running it for lesser SBU values.
 
Old 04-16-2013, 06:20 AM   #14
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405Reputation: 2405
Quote:
Originally Posted by DeeGee View Post
BTW adding a typical newbie question: would it be a problem if I run binutils or gcc again to test this? Because I guess its no point of running it for lesser SBU values.
If I where you I would not use your (partial) LFS build to test this: Use your host.

You do need to remove both the build and source directory before starting the next test. Here's how I did the quick test:
Code:
$ tar xf binutils-2.23.1.tar.bz2
$ cd binutils-2.23.1
$ mkdir -v ../binutils-build
$ cd ../binutils-build
$ ../binutils-2.23.1/configure --prefix=/usr --enable-shared
$ time make -jX       # timers are shown once the make command finishes
# get ready for next test:
$ cd ..
$ rm -rf binutils-2.23.1 binutils-build
# start from the top again
You cannot use fractions for X, so with having one core in mind I would use 1,2 and 3 for X.
 
Old 04-16-2013, 09:58 AM   #15
Lennie
Member
 
Registered: Aug 2012
Location: Sweden
Distribution: LFS, built with pacman
Posts: 374

Rep: Reputation: 85
I've learnt that one should use the amount of cpus +1. I have 4 cpus (2 physical and 2 virtual). This is what I got when I did the same test with binutils:

Code:
-j4
real	0m53.836s
user	2m34.011s
sys	0m12.939s

-j5
real	0m53.418s
user	2m38.521s
sys	0m13.489s

-j6
real	0m53.845s
user	2m36.961s
sys	0m13.354s
So for me -j5 was fastest.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Nvidia and ARM: It's a parallel, parallel, parallel world LXer Syndicated Linux News 0 03-21-2013 06:10 PM
[SOLVED] Slackware parallel make (make -j?) question. Daedra Slackware 4 02-19-2012 08:14 PM
Gnu make parallel problem Andrew Benton Programming 7 11-28-2010 09:56 AM
New Laptop Advice? Pentium-M, Brand/Make, Widescreen etc HaloinaHaystack Linux - Laptop and Netbook 10 10-28-2004 05:34 PM
How to make a parallel repository (e.g. with CVS) esben Linux - Software 0 01-18-2004 11:33 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Linux From Scratch

All times are GMT -5. The time now is 02:15 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration