[SOLVED] Distcc troubles between x86_64 host and arm client
SlackwareThis Forum is for the discussion of Slackware Linux.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Distcc troubles between x86_64 host and arm client
I am trying to use distcc to build the Slackware ARM kernel. The system that is building the kernel, and acting as a client to distccd, is a Rapsberry Pi 3 B running the latest SlackwareARM-current. The distccd host is my laptop and it has an i7 with 8 cores. I ran into some trouble with distcc on the client being able to access/find "arm-slackware-linux-gnueabihf-gcc" on the host. The steps I took to get where I am at are below.
I built and installed the ARM x-toolchain on my x86_64 laptop.
# This is a working example of a script that you may wish to add to your system's
# /etc/rc.d/rc.local in order to start distcc with the Cross compiler toolchain.
# Distcc:
case "$( uname -m )" in
i?86) export DISTCCARCH=i486 ;;
# Unless $ARCH is already set, use uname -m for all other archs:
*) export DISTCCARCH=$( uname -m ) ;;
esac
#DISTCC_SAVE_TEMPS=1 \
DISTCC_BASEDIR=/devel/build-servers
LD_LIBRARY_PATH=$DISTCC_BASEDIR/$DISTCCARCH/arm/$DISTCCARCH-slackware-linux/arm-slackware-linux-gnueabihf/lib/ \
DISTCCD_PATH=$DISTCC_BASEDIR/$DISTCCARCH/arm/bin \
/usr/bin/distccd \
--verbose \
--log-file /tmp/distcc.log \
-j 20 \
--daemon \
--allow 192.168.150.0/24
All of the variables point to the following locations on my laptop, which do indeed exist:
export DISTCC_HOSTS='localhost/4 mothership/8'
export NUMJOBS=-j$[$(sed 's?[^0-9]*/?+?g' <<<$DISTCC_HOSTS)]
PATH=/hd/DISTCC::$PATH # location where compiler symlinks to distcc were added
cd /hd/linux-4.17.19
make -j 12 zImage
Both machines are communitcating but I am seeing the following output in the distccd log on my laptop when I begin compiling the kernel on my client ARM device:
Code:
distccd[25777] (dcc_r_argv) got arguments: arm-slackware-linux-gnueabihf-gcc -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89 -c -o scripts/dtc/fstree.o scripts/dtc/fstree.c
distccd[25777] (dcc_scan_args) scanning arguments: arm-slackware-linux-gnueabihf-gcc -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89 -c -o scripts/dtc/fstree.o scripts/dtc/fstree.c
distccd[25777] (dcc_scan_args) found object/output file "scripts/dtc/fstree.o"
distccd[25777] (dcc_scan_args) found input file "scripts/dtc/fstree.c"
distccd[25777] compile from fstree.c to fstree.o
distccd[25777] (dcc_run_job) output file scripts/dtc/fstree.o
distccd[25777] (dcc_input_tmpnam) input file scripts/dtc/fstree.c
distccd[25777] (dcc_r_token_int) got DOTI0001af12
distccd[25777] (dcc_r_file) received 110354 bytes to file /tmp/distccd_955c4b3f.i
distccd[25777] (dcc_r_file_timed) 110354 bytes received in 0.008751s, rate 12315kB/s
distccd[25777] (dcc_set_input) changed input from "scripts/dtc/fstree.c" to "/tmp/distccd_955c4b3f.i"
distccd[25777] (dcc_set_input) command after: arm-slackware-linux-gnueabihf-gcc -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89 -c -o scripts/dtc/fstree.o /tmp/distccd_955c4b3f.i
distccd[25777] (dcc_set_output) changed output from "scripts/dtc/fstree.o" to "/tmp/distccd_95454b3f.o"
distccd[25777] (dcc_set_output) command after: arm-slackware-linux-gnueabihf-gcc -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89 -c -o /tmp/distccd_95454b3f.o /tmp/distccd_955c4b3f.i
distccd[25777] (dcc_check_compiler_masq) /devel/build-servers/x86_64/arm/bin/arm-slackware-linux-gnueabihf-gcc is not a symlink
distccd[25777] (dcc_check_compiler_whitelist) CRITICAL! arm-slackware-linux-gnueabihf-gcc not in /usr/lib64/distcc whitelist.
distccd[25777] (dcc_cleanup_tempfiles_inner) deleted 5 temporary files
distccd[25777] (dcc_job_summary) client: 192.168.150.200:37684 OTHER exit:0 sig:0 core:0 ret:0 time:9ms
I am sure the solution is simple I am just suffering from brain rot due to looking at it for too long. I searched the web for information about "dcc_check_compiler_whitelist" and did not find anything useful. All the results in Google refer to Gentoo, Arch Linux, and Raspbian. I've spent a few hours at this point troubleshooting. I need someone to point out where I went wrong. The full distccd log is attached below.
You don't need localhost there - the point of distcc is to outsource compilation to a more capable host, leaving the linking to the native machine. If distcc fails, it'll fall back to localhost anyway.
Quote:
Originally Posted by mralk3
Code:
distccd[25777] (dcc_check_compiler_masq) /devel/build-servers/x86_64/arm/bin/arm-slackware-linux-gnueabihf-gcc is not a symlink
I have the same in my logs. Indeed, it is not a symlink.
Quote:
Originally Posted by mralk3
Code:
Quote:
Originally Posted by mralk3
Code:
[B]distccd[25777] (dcc_check_compiler_whitelist) CRITICAL! arm-slackware-linux-gnueabihf-gcc not in /usr/lib64/distcc whitelist.
I don't run -current on my x86's and I have distcc 3.1. Distcc 3.3 as shipped presently in -current, has this new directory and 'whitelist' notion.
My guess is that doing this on the x86_64 will do it:
thanks I'll update that rc.local script on the FTP site next time.
I'm not sure what's going to happen if distcc is called as "gcc" or "cc" or any name not "arm-slackware.." since the whitelist entries for those point to the x86 toolchain. I'll have to dig in to that if that becomes an issue.
thanks I'll update that rc.local script on the FTP site next time.
I'm not sure what's going to happen if distcc is called as "gcc" or "cc" or any name not "arm-slackware.." since the whitelist entries for those point to the x86 toolchain. I'll have to dig in to that if that becomes an issue.
Since $PATH has the symlinks to distcc on my client and now the symlinks on the host, I don't think it will be an issue as long as PATH has cc and gcc in it prior to reaching the system gcc/cc.
The main problem was the documentation is very sparse in /usr/doc/distcc and on most places on the web. After I read the shell scripts it all made more sense. None of the Gentoo documentation was any help. The Arch Linux docs are too specific to Arch. There isn't much to go on for setup of distcc on Slackware either. Maybe a SlackDoc is in my future.
I tried to build the x-toolchain on my Pi at first and I had an error during the configure of binutils. Then I realized my mistake. Build it on the x86 instead.
With 8 cores on my laptop and 4 on the Pi I should be using make -j 14? So (8 cores + 1) + (4 cores + 1) = 14, right?
Everything is definitely compiling faster. Htop reports my laptop cores are barely doing much and distccmon-text (running on the Pi) shows localhost is being used for all the "preprocess" and even quite a few of the "compiles". I have distccd running with -j 20. I am curious if I can push it further. See screenshot. The bottom screen window panes are my laptop, top is my pi.
With 8 cores on my laptop and 4 on the Pi I should be using make -j 14? So (8 cores + 1) + (4 cores + 1) = 14, right?
Everything is definitely compiling faster. Htop reports my laptop cores are barely doing much and distccmon-text (running on the Pi) shows localhost is being used for all the "preprocess" and even quite a few of the "compiles". I have distccd running with -j 20. I am curious if I can push it further. See screenshot. The bottom screen window panes are my laptop, top is my pi.
The distcc host list I use comprises solely of x86_64, so the number of parallel jobs is equal to the aggregate of the available x86 cores. That said, I restrict the parallel builds to a max of 15 no matter how many x86s are available at build time, as the ARM machine cannot handle more without diminishing returns.
That's as far and as scientific as I went with tuning it.
Thanks. I figured there was some limitation due to the Pi hardware. So far I haven't come close to running out of RAM and swap is used very minimally. When I installed Slackware I mounted /tmp as tmpfs with 512MB RAM allocated, added a 1GB swap file on the external hard disk, and disabled the swap partition on board the SD card. I am usually around 150MB of RAM used when building the kernel. I am sure I could allocate more RAM to /tmp. I haven't played with the Pi's GPU memory allocation or CPU frequency scaling in /boot/config.txt.
Quote:
Originally Posted by drmozes
The distcc host list I use comprises solely of x86_64, so the number of parallel jobs is equal to the aggregate of the available x86 cores. That said, I restrict the parallel builds to a max of 15 no matter how many x86s are available at build time, as the ARM machine cannot handle more without diminishing returns.
That's as far and as scientific as I went with tuning it.
I will go with -j 12. It seems to be getting the job done fast enough. I do not want to burn up the Pi.
I am running into a strange problem. The last few months I haven't compiled anything using this setup because I have been either busy or ill. I am just now getting back to working on this. For some reason distcc on the x86_64 host is looking in /usr/lib/distcc instead of /usr/lib64/distcc for all of the whitelisted symlinks.
I tried to create the directory and symlinks in /usr/lib/distcc with no result. Maybe I am missing something very similar; I require another set of eyes.
The SlackBuild script creates the first set of symlinks within the /usr/lib64 (or /usr/lib for 32bit), and there's no /usr/lib/distcc :
Code:
prisere [slackware64-current] # tar tvvf slackware64/d/distcc-3.3.2-x86_64-1.txz|fgrep usr/lib/
prisere [slackware64-current] #
Which to me either implies a local problem with your setup, or that there's something wrong with the package itself. There's some sed in the SlackBuild to change /usr/lib to /usr/lib64 if necessary, so perhaps it's not working properly.
I don't have a -current machine to check if it works on a fresh installation - I may have a look some time later on.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.