distcc fails to distribute

justwantin · 05-28-2017, 05:06 AM

I have two banana pros, one running current from around 08/01/17 and the other from early this May. I used distcc with x-toolchain on an x86 64 box to re-compile the kernels on both banana pros successfully post install.

Today without any changes to them that I can recall I tried to use distcc on both sbc's to compile with the same x86 box which likewise has not been changed in any way that I can recall.

For some reason I can no longer distribute from the arm sbc's to the x-toolchain on the x86. I just installed the latest x-toolchain and the result is the same. I have pasted in a snippet of stdout from when compiling the kernel below.

Perhaps someone on the list has had a similar problem and can give me a heads up.

Code:

distcc[4663] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CHK     include/generated/bounds.h
  CC      arch/arm/kernel/asm-offsets.s
distcc[4685] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  HOSTLD  scripts/dtc/dtc
  CHK     include/generated/asm-offsets.h
  HOSTLD  scripts/genksyms/genksyms

drmozes · 05-28-2017, 09:20 AM

Quote:

Originally Posted by justwantin

I have two banana pros, one running current from around 08/01/17 and the other from early this May. I used distcc with x-toolchain on an x86 64 box to re-compile the kernels on both banana pros successfully post install.

Today without any changes to them that I can recall I tried to use distcc on both sbc's to compile with the same x86 box which likewise has not been changed in any way that I can recall.

For some reason I can no longer distribute from the arm sbc's to the x-toolchain on the x86. I just installed the latest x-toolchain and the result is the same. I have pasted in a snippet of stdout from when compiling the kernel below.

Perhaps someone on the list has had a similar problem and can give me a heads up.

Code:

distcc[4663] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  CHK     include/generated/bounds.h
  CC      arch/arm/kernel/asm-offsets.s
distcc[4685] (dcc_build_somewhere) Warning: failed to distribute, running locally instead
  HOSTLD  scripts/dtc/dtc
  CHK     include/generated/asm-offsets.h
  HOSTLD  scripts/genksyms/genksyms

It could be that the compiler versions are mismatching.
Also, turn on distcc logging on x86 and see what it's doing

justwantin · 05-28-2017, 10:23 AM

gcc on banana pro is 5.4.0, on x86 gcc for most recent x-toolchain is 7.1.0, the older x-toolchain I had first tried was 5.3.0. So neither match gcc on client. Should I try building the x-tollchain with gcc-5.4.0?

This all that happens on the x86 box:

Code:

bash-4.3# tail /tmp/distcc.log 
distccd[7611] (dcc_create_kids) up to 3 children
distccd[7611] (dcc_create_kids) up to 4 children
distccd[7611] (dcc_create_kids) up to 5 children
distccd[7611] (dcc_create_kids) up to 6 children
distccd[7611] (dcc_create_kids) up to 7 children
distccd[7611] (dcc_create_kids) up to 8 children
distccd[7611] (dcc_create_kids) up to 9 children
distccd[7611] (dcc_create_kids) up to 10 children
distccd[7611] (dcc_create_kids) up to 11 children
distccd[7611] (dcc_create_kids) up to 12 children

drmozes · 05-28-2017, 01:45 PM

Quote:

Originally Posted by justwantin

gcc on banana pro is 5.4.0, on x86 gcc for most recent x-toolchain is 7.1.0, the older x-toolchain I had first tried was 5.3.0. So neither match gcc on client. Should I try building the x-tollchain with gcc-5.4.0?

I'm not sure of what adverse effects you'd have by having mismatched compilers -- it might not be that bad since the x-toolchain and really is used for gcc only, but I always keep them in sync.
Why not upgrade both current ARM machines to the latest 7.x gcc packages?

My build wrapper script for using distcc does this:

Code:

# Are the other machines powered up and running distccd?
timeout 1 bash -c 'cat < /dev/null > /dev/tcp/kitt/3632' > /dev/null 2>&1 && export DISTCC_HOSTS="$DISTCC_HOSTS kitt/9"
timeout 1 bash -c 'cat < /dev/null > /dev/tcp/turrican/3632' > /dev/null 2>&1 && export DISTCC_HOSTS="$DISTCC_HOSTS turrican/3"

This checks whether the build machine can reach the distccd on the x86 hosts before adding them to the list of available servers. Try adding that to your environment: did you check whether the ARM machines can reach the distccd on x86, and that there's enough disk space on the x86 machines where the disttcd will write its temporary files (/tmp, I think?)

justwantin · 05-29-2017, 10:36 PM

Thanks for the reply and I'll check out comms with that snippet of code the next try.

I just rsync'd current and am installing it on one of the b-pros. I'll leave the other one alone for now. It runs a weather station from the garden shed 24x7 and every 15 min takes a snap of the garden with mjpg-streamer, logs weather and market data and re-writes the webpage that is served on our lan. Theres I spike in cpu usage during all that. I'll leave that one alone for now.

Out of curiosity I tried a kernel compile via distcc on the two b-pros yesterday. Weather station as client and the other the server. Using distccmon-txt I could see that jobs were being sent to the 2nd b-pro for awhile but then now and again there would be a failure and fallback to localhost.

Code:

13682  Preprocess  exec_domain.c                              192.168.1.10[1]
13663  Preprocess  omap-secure.c                              192.168.1.10[2]
13792  Compile     page-writeback.c                              localhost[0]

This could be attributed to a couple things like poor wireless comms or not enough grunt for preprocessing during a 15 minute wakeup. At least I know that the client b-pro could distribute jobs on the lan with both b-pros using gcc 5.40.

My last successful compile with distcc and x-tool chain was with gcc 5.4 on a b-pro and 5.3 in the x-toolchain. Neither the new x-toolchain with gcc 7.1 or the old x-toolchain with 5.3 worked on Sunday. I'll have another go being a little more methodical after the install is completed and I have the time.

justwantin · 05-30-2017, 06:00 AM

OK, all good, gcc 7.1.0 on b-pro and in x-toolchain on X86. Here's a snippet of output from distccmon-test on the b-pro client:

Code:

 19619  Blocked                                                            [0]
 19602  Blocked                                                            [0]
 19632  Preprocess  trans.c                                     192.168.1.2[0]
 19535  Preprocess  extents.c                                   192.168.1.2[1]
 19555  Preprocess  scrub.c                                     192.168.1.2[2]
 19441  Preprocess  dns_resolve.c                               192.168.1.2[3]
 19591  Preprocess  quota.c                                     192.168.1.2[4]
 19572  Compile     rsa-pkcs1pad.c                              192.168.1.2[5]
 19624  Preprocess  catalog.c                                   192.168.1.2[6]
 19417  Compile     cifs_spnego.c                               192.168.1.2[6]
 19592  Preprocess  cifs_dfs_ref.c                              192.168.1.2[7]

Here's netstat on the X86 server:

Code:

bash-4.3# netstat -pant | grep distcc
tcp        0      0 0.0.0.0:3632            0.0.0.0:*               LISTEN      14811/distccd       
tcp        0      0 192.168.1.2:3632        192.168.1.10:45210      ESTABLISHED 23061/distccd       
tcp        0      0 192.168.1.2:3632        192.168.1.10:45186      ESTABLISHED 23199/distccd       
tcp        0      0 192.168.1.2:3632        192.168.1.10:45192      ESTABLISHED 23234/distccd       
tcp        0      0 192.168.1.2:3632        192.168.1.10:45196      ESTABLISHED 23465/distccd       
tcp        0      0 192.168.1.2:3632        192.168.1.10:45212      ESTABLISHED 23473/distccd       
tcp        0      0 192.168.1.2:3632        192.168.1.10:45190      ESTABLISHED 23212/distccd       
tcp        0      0 192.168.1.2:3632        192.168.1.10:45206      ESTABLISHED 23147/distccd       
tcp        0      0 192.168.1.2:3632        192.168.1.10:45208      ESTABLISHED 23140/distccd

I suppose that while this might have been possible between gcc 5.3.0 and gcc 5.4.0, it was not possible between gcc 5.4.0 and gcc 7.1.0

SCerovec · 06-06-2017, 03:18 AM

[SOLVED]? and recommend always keep gcc's in sync? eh B)

Didn't have such an suspense ggc read since the venerable gcc-2.96 form RedHat pollute my Mandrake :^] and then I was trying to make Mplayer...