LinuxQuestions.org - [SOLVED] distcc failing to compile

- Gentoo (http://www.linuxquestions.org/questions/gentoo-87/)

- - distcc failing to compile (http://www.linuxquestions.org/questions/gentoo-87/distcc-failing-to-compile-4175416543/)

distcc failing to compile

I've been making several attempts to get distcc working on my older Dell Latitude C800. It only has a 1ghz Pentium 3 Coppermine CPU and 512mb of ram. So I think it could really benefit from the efficiency of Gentoo, smaller and faster binaries, etc., but the compiling usually is taking too long. Especially when I'll need to rebuild the entire system or run revdamp-rebuild.

I have a desktop with a 2.8ghz quad core and 4GB ram, so I am attempting to use it to do the compiling. Last time I had Gentoo on the desktop, it compiled extremely fast, but I don't think the extra performance of Gentoo was so noticeable on it because it's already a fairly high end computer.

So on both machines, I've set:

Code:

export DISTCC_LOG=/var/log/distccd.log

export DISTCC_VERBOSE=1

Only the laptop produced a log.

/var/log/distccd.log:
http://vpaste.net/PWrNQ

Laptop LAN IP: 192.168.1.117
Desktop LAN IP: 192.168.1.127

On the laptop, I've

Code:

DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.0.0/24"

DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.1.117"

in the /etc/conf.d/distccd

I have done distcc-config --set-host "192.168.1.127" and the /etc/distcc/hosts shows that ip in there, that it had been set.

In the make.conf, I set

Code:

FEATURES="distcc"

MAKEOPTS="-J8" # 2X cpus, quad core, so 8

After setting all of that up on the laptop, I then did

Code:

/etc/init.d/distccd restart

==================

On the desktop, I've

Code:

DISTCCD_OPTS="${DISTCCD_OPTS} --allow 192.168.0.0/24"

DISTCCD_OPTS="${DISTCCD_OPTS} --listen 192.168.1.127"

in the /etc/conf.d/distcc

Code:

export DISTCC_LOG=/var/log/distccd.log

export DISTCC_VERBOSE=1

but no log was created.

set in the /etc/make.conf

Code:

FEATURES="distcc"

MAKEOPTS="-j8"

Code:

/etc/init.d/distcc start

This last attempt, I emerged epdfview on the laptop. I ran distccmon-gui on both machines. It showed no activity. I checked the processes on the desktop. I saw nothing that indicated compiling and the cpu usage was very low.

==================

Both computers have the same distcc version.
On the laptop, I had to upgrade the gcc version to

Code:

beastlinux ~ # gcc -v 2>&1 | tail -1

gcc version 4.6.2 (Gentoo Hardened 4.6.2 p1.4, pie-0.5.0) 

beastlinux ~ #

I then rebuilt libtool as described by the Gentoo Docs for upgrading gcc.

CHOST in make.conf on both machines is the same

Code:

CHOST="i686-pc-linux-gnu"

To avoid cross-compiling, I burned a 32-bit Sabayon/Gentoo disk and booted it up on the desktop. I had first tried doing all of this in virtual box. and the networking was all set up correctly. I could nmap port 3632 from either machine to the other and see that it was open and running distcc. I couldn't seem to figure out what was going wrong, so this last attempt was booting it up as a live cd, and then setting up distcc. If I can get distcc working first, then I'll consider using a 32-bit chroot or cross-compiling.

EDIT:
It seems be working now. According to logs, messages like these:

Code:

distcc[4938] (dcc_r_file_timed) 3148 bytes received in 0.000477s, rate 6445kB/s

distcc[4938] 4681009 bytes from main.cpp compiled on 192.168.1.127 in 9.5097s, rate 481kB/s

distcc[4938] (dcc_unlock) release lock fd4

distcc[4938] compile main.cpp on 192.168.1.127 completed ok

distcc[4938] elapsed compilation time 14.536347s

--

distcc[4868] (dcc_r_file_timed) 38936 bytes received in 0.001141s, rate 33325kB/s

distcc[4868] 4625570 bytes from notebooks/notebookmenuitem.cpp compiled on 192.168.1.127 in 9.8320s, rate 459kB/s

distcc[4868] (dcc_unlock) release lock fd4

distcc[4868] compile notebooks/notebookmenuitem.cpp on 192.168.1.127 completed ok

distcc[4868] elapsed compilation time 43.951518s

--

distcc[4907] (dcc_compile_remote) client finished sending request to server

distcc[4907] (dcc_note_state) note state 5, file "(NULL)", host "192.168.1.127"

distcc[4907] (dcc_select_for_read) select for read on fd6 for 300s

--

distcc[4907] (dcc_r_file_timed) 39656 bytes received in 0.001285s, rate 30137kB/s

distcc[4907] 5958039 bytes from dbus/remotecontrol.cpp compiled on 192.168.1.127 in 11.3617s, rate 512kB/s

distcc[4907] (dcc_unlock) release lock fd4

distcc[4907] compile dbus/remotecontrol.cpp on 192.168.1.127 completed ok

distcc[4907] elapsed compilation time 31.423067s

Still, the laptop is doing most of the work. It's cpu is pegged at nearly 100% while the quad core has a bump in cpu usage here and there.

The distcc monitor apps distccmon-text, distccmon-gui, distccmon-gnome do not work. None of them have displayed a single message about the distcc helping with the compiling.

The distccmon-* doesn't show anything when running emerge. When you compile a kernel you will see some output there. Also, not all packages use distcc. If the package is small, it will be compiled locally. Also the -j8 may be overkill for the laptop. If the package doesn't utilize distcc, the laptop will end up trying to compile with 8 threads, which will result in a huge load avg. I personally use (# of cores on the local host) + (# of distcc hosts) + 2. But I don't have a pc with more than 2 cores, so I can't be sure.
I think you can choose a different apcroach. You can compile the packages on the desktop and then export the distfiles to the laptop. But I don't have any experience on that field.

Quote:

Originally Posted by segmentation_fault (Post 4728739)

The distccmon-* doesn't show anything when running emerge. When you compile a kernel you will see some output there.

Really? http://i.imgur.com/zD6i1.jpg

Quote:

Originally Posted by segmentation_fault (Post 4728739)

Also the -j8 may be overkill for the laptop.

As you can see -j8 has the virtual machine that helping the compile overloaded. Just look at that cpu usage.. And from the previous picture, the laptop is over loaded with compiling processes in distcc. Actually, the laptop is not set to do ANY compiling when distcc is in use. Yes I know what distcc does and doesn't compile. It's not always enabled nor is the -j8. I comment it out when not using distcc.
http://img6.imagebanana.com/img/5so0...llRunningO.png

Quote:

Originally Posted by segmentation_fault (Post 4728739)

I personally use (# of cores on the local host) + (# of distcc hosts) + 2.

Quote:

(Modify MAKEOPTS to include -jN, where N is twice the number of CPUs +1 available) MAKEOPTS="-jN"

Quoted from http://www.gentoo.org/doc/en/distcc.xml It actually makes mention of using that formula twice.

Quote:

distcc is intended to be used with GNU Make's -j option, which runs several compiler processes concurrently. distcc spreads the jobs across both local and remote CPUs. Because distcc is able to distrib- ute most of the work across the network a higher concurrency level can be used than for local builds. The -j value should normally be set to about twice the total number of available CPUs, to allow for some tasks being blocked waiting for disk or network IO. distcc can also work with other build control tools such as SCons.

Quoted from the distcc man page. http://www.manpagez.com/man/1/distcc/
I just asked why my distcc was not working, not all that. I ended up solving it on my own (you can see in the screenshots, it is working), so I'll mark this thread as solved.

Well, I have never seen any output on distccmon, but I don't run it always. Maybe it's something I misconfigured.
Anyway, it's good you figured it out. :)

Here's how you get output on distccmon-gui or distccmon-text.

Create /etc/env.d/99local

Add these three lines to it.

DISTCC_VERBOSE=1
DISTCC_LOG="/var/log/distccd.log"
DISTCC_DIR="/var/tmp/portage/.distcc"

env-update && source /etc/profile

Btw, I read about creating this file under /etc/env.d/99local on the Gentoo Handbook, so it's supported, not a hack.