Icecream distrobuted cross compiler

djsmiley2k · 12-23-2010, 10:21 AM

Howdy all...

I've got a HTPC I'm trying to setup with xbmc however under the newest ubuntu it doesn't seem to work correctly. X will randomly drop out and reset itself and other things are flaky too. When seeking help its pretty much dead end as none of the devs or anyone with real knowledge of the system hang around online, and the wiki is largely out of date.

Instead I've decided to install something I know and know well - Gentoo - and then stick on xbmc over the top, at least this way I'll be able to get the help of the gentoo community, who are amazing.

Onto my problem:

Its a HTPC..... Gentoo can involve lots of compiling, and while the system is a nice powerful system, it is going to have issues I think trying to compile everything I want and where possible I'd like to not heat it up too much

The answer I came up with is cross compiling on my two other systems. I'm trying to use icecream for the cross compliation as detailed in : http://dev.gentoo.org/~rane/icecream.xml however all I ever see is all the jobs going to the local machine.

Code:

[29365] 16:18:52: handle_local_job  137
[29365] 16:18:52: handle_local_job_done 137
[29365] 16:18:52: handle_local_job  138

and the slave never gets any:

Code:

tim@Musha ~ $ tail -f /var/log/iceccd 
[4855] 16:12:50: scheduler is on 192.168.1.252:8765 (net )
[4855] 16:12:50: scheduler not yet found.
[4855] 16:12:53: reconn Node Name: Musha
  Remote name: 
  Architecture: i686
  Current kids: 0 (max: 5)
  cpu: 937 idle, 0 nice
  load: 0, icecream_load: 0
  memory: 0 (free: 1233)

I don't know what else to try. I'm hoping someone here has used Icecream and has some ideas about what I've missed...

/etc/profile has been updated on both systems:

Code:

Slave:
export PATH="/usr/lib/ccache/bin:/usr/lib/icecc/bin:$PATH"

export ICECC_VERSION="x86_64:/var/icecc/host-i686-target-x86_64.tar.gz,i686:/var/icecc/host-i686-target-i686.tar.gz"

Code:

Master:
export PATH="/usr/lib/ccache/bin:/usr/lib/icecc/bin:$PATH"

export ICECC_VERSION="i686:/var/icecc/host-x86_64-target-i686.tar.gz,x86_64:/var/icecc/host-x86_64-target-x86-64.tar.gz"

Ideas?

ALSO:

If I set the number of jobs that the "Master (64bit system) can run to 0, the

Code:

scheduler reports:
[24802] 16:54:44: no job stats - looking at MushaV2 load: 273 can install: x86_64
[24802] 16:54:44: no job stats - looking at Musha load: 32 can install: 
[24802] 16:54:47: no job stats - looking at MushaV2 load: 405 can install: x86_64
[24802] 16:54:47: no job stats - looking at Musha load: 32 can install:

Showing it thinks that Musha (the slave) cannot run anything, and that the Master can ONLY do 64bit, it wont cope with i686....

hiptobecubic · 01-03-2011, 05:09 AM

I'm having the same problem. It's so bent on refusing to hand out jobs that if i start the daemon with "-m 0", it will just accept the number of jobs matching "make -j $NUMJOBS" and then hang.

Have you tried explicitly setting the network name "-n" ?

djsmiley2k · 01-03-2011, 07:07 AM

Hey!

At least I'm not going insane, I hadn't seen anyone else online with this issue and was convinced I'd setup something wrong, but it appears that icecream just has a very small userbase :/

I did try setting the network with -n - same issue.

derguteweka · 10-21-2018, 02:10 PM

Moin,

Just want to mention, that even 7 years later, i've the feeling, that nothing really changed in the behaviour of icecream:
Having here 4 PCs in the Network: 2x Intel i3, 1x Intel Celeron, 1x Intel P4 (=pretty old laptop), all connected via 100MBit Ethernet, which never showed any signs of poor performance during rsync, nfs, scp or surfing the www.
Trying to avoid slow compiling on the P4 and to transfer some jobs on the I3 machines or at least on the Celeron.
On all PCs: current icecream from github; one i3: LFS/BLFS approx. 6 years old; other 3 PCs: LFS/BLFS 8.3

I'm pretty sure, that i setup all things not to weired, because from time to time a few jobs indeed get distributed to a different machine. But overall it's just not at all working as expected:
Poor old P4 is compiling itself almost to death, while both i3 and the Celeron are pretty much ideling. Also it seems to be impossible to disable local compiling on the weak machine. Also never watched different compile jobs to be distributed to more than one PC during the same build process.
Pretty sad result.

Gruss
WK

derguteweka · 01-06-2021, 04:57 AM

Moin,

OK, 2 years later, pretty much still the same network and machines. But brandnew icecream from git, trying to compile some BLFS-10 packages for the old laptop.
Seems to work better in some cases than 2 years ago (also nice, that also clang is supported) , but seems to still have issues - currently i suspect, there are things (memory, time,... limits) hardcoded in icecream, which do not reflect proper settings in all cases.

E.g. it happened, that the compilejobs on the clients are running out of some memory limit (far below the machines limit). I changed that by crudly doubling it in the src (around line 203, in daemon/workit.cpp). Maybe gcc-10.2 needs more memory?
Also seems, that if compilation on the client takes longer, some timeout is tripped. This happened a lot during the compile of JS-68.11.0; much less often on packages like boost or linux(kernel). Result in the logs of clients then look e.g. like this:

Code:

[12033] 2021-01-06 00:59:58: Remote compilation completed with exit code 0
[11973] 2021-01-06 01:59:58: handle_end 3 192.168.178.55
[12033] 2021-01-06 00:59:58: </parent, waiting: 10094ms>
[12033] 2021-01-06 00:59:58: flush_writebuf() failed(Error: Broken pipe)
[12033] 2021-01-06 00:59:58: setting error state for channel 192.168.178.55: (B eof: 0)
[12033] 2021-01-06 00:59:58: write of obj chunk failed 100000
[11973] 2021-01-06 02:01:44: accepted 10 192.168.178.55 as 4


[15088] 2021-01-06 09:21:26: Remote compilation completed with exit code 0
[11973] 2021-01-06 10:21:26: handle_end 16 192.168.178.55
[15088] 2021-01-06 09:21:26: </parent, waiting: 3670ms>
[15088] 2021-01-06 09:21:26: flush_writebuf() failed(Error: Broken pipe)
[15088] 2021-01-06 09:21:26: setting error state for channel 192.168.178.55: (B eof: 0)
[15088] 2021-01-06 09:21:26: write of obj chunk failed 100000
[11973] 2021-01-06 10:22:14: accepted 10 192.168.178.55 as 17

Gruss
WK