LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 09-23-2009, 09:33 PM   #1
DreamAxe
LQ Newbie
 
Registered: Sep 2009
Distribution: Arch, Debian
Posts: 5

Rep: Reputation: 1
Question Use for spare PCs - network compile farm?


I recently got some old PC's from a friend who's moving to another city. The problem is I have no idea what to do with them. I thought I could make a compile farm to access over the LAN and the Internet to compile my projects faster and maybe offer some friends the ability to use the farm to compile their projects too. So I need to know a few things about compile farms:

1) Does distributed compiling offer an adequate increase in compile speed over local compilation?
2) My Internet connection is kinda slow (aDSL @ 8192/384). Is this adequate or will I need a faster connection? Generally speaking, do compile farms require fast network I/O or better hardware resources?
3) Any suggestions on which Linux distro to run on a compile farm?
4) Are there any specific hardware requirements for compile farms?
5) Any suggestions other than a compile farm that take advantage of the CPU and memory available?

Here is some quick hardware info about the two PCs
PC1: AMD 7750BE, 2GB RAM (works in DC), 1TB HDD, gigabit ethernet (works at 100mbps due to switch limitations)
PC2: AMD A64 3200+, 1.5GB RAM, 120GB HDD, gigabit ethernet (works at 100mbps due to switch limitations)

PS: I post this on the server section of LQ since it's more of a cluster question than a programming one. If a mod thinks it's not in the right forum, kindly move it. Thanks.
 
Old 09-24-2009, 01:25 PM   #2
tallship
Member
 
Registered: Jul 2003
Location: On the Beaches of Super Sunny Southern San Clemente, California USA
Distribution: Slackware - duh!
Posts: 534
Blog Entries: 3

Rep: Reputation: 118Reputation: 118
Thumbs up

Quote:
Originally Posted by DreamAxe View Post

1) Does distributed compiling offer an adequate increase in compile speed over local compilation?
2) My Internet connection is kinda slow (aDSL @ 8192/384). Is this adequate or will I need a faster connection? Generally speaking, do compile farms require fast network I/O or better hardware resources?
3) Any suggestions on which Linux distro to run on a compile farm?
4) Are there any specific hardware requirements for compile farms?
5) Any suggestions other than a compile farm that take advantage of the CPU and memory available?
1.) It can, it seems that you're hosting the farm elsewhere, so you need to take into account the d/l time following your compiles if you're going to be d/l'ing to your local network afterwards, although this will probably save time too since you are only d/l'ing the binaries following compilation of the huge source where your machines are located.

2.) It's totally adequate, since you're not d/l'ing source to your local network, only to the machinery co-located elsewhere, and then d/l'ing the binaries to your local network or distributing them elsewhere still over the higher speed backbone.

3.) Slackware 13.0 (64 bit)

4.) No. Just lots of memory and CPU power.

5.) Compile on the fastest machine with the greatest amount of memory.

Last edited by tallship; 03-06-2010 at 06:37 PM.
 
Old 09-24-2009, 02:40 PM   #3
johnsfine
LQ Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,286

Rep: Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197
When I think of a compile farm, I assume the files (both source and generated binaries) are on one system, while the compilers run on other system(s).

That has been on my todo list for a while, so I'd also like some insight into how to do it better.

Or if the DreamAxe has a different relationship in mind between the compiling and the files, please explain.

I assumed a 1Gb network and even there worry about the network speed. I've done a fair amount of compiling across the network one to one (one compiler system talking to one system with the files) and there are some flaws in caching:

My environment rereads many hpp files many times each in the course of a build. The systems involved have plenty of excess ram to allow the hpp files to sit in file cache from the time they are first read until past the end of the whole build, so each hpp is only read from disk once.

With files and compilation on the same computer, hpp file reads are very cheap. I would hope that the compilation computer could similarly cache hpp files even though they come from the network rather than a local hard drive. At best that has been imperfect. The same hpp files transfer from the file computer's file cache across the 1Gb network to the compilation system multiple times during a build.

With slower than 1Gb networking, I don't think this whole model of distributing files vs. compilation makes any sense.

When I think of a compile farm, I also think of running many compiles in parallel. I'm using an obsolete locally modified version of bjam (like make but smarter) to invoke the compile command and it is pretty good at keeping N compiles running in parallel for user specified N. But they're all on the same (multi core) computer (bjam runs where the compilation runs because it invokes compilers only on the local system).

I don't know what is available as a version of make or a replacement for make that could launch multiple builds on multiple systems. Without that, I don't think a "compile farm" has much point.

Last edited by johnsfine; 09-24-2009 at 02:44 PM.
 
Old 09-24-2009, 03:18 PM   #4
georg74
LQ Newbie
 
Registered: Mar 2007
Location: Austria
Distribution: Debian
Posts: 3

Rep: Reputation: 0
To reduce network traffic you can rsync the sources to the compile farm first (transferring only the changes), then compile and rsync the result back.
You could use cvs read-only checkout on the farm, but you would have to commit changes before each compilation...
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Get ip address of pcs on my network davholla Linux - Security 7 07-06-2007 04:15 AM
ssh to 2 PCs on my network remotely? ncsuapex Linux - Networking 5 05-16-2006 02:06 PM
Help all PCs on network system are down Trekker001 Linux - Networking 2 05-11-2005 01:49 PM
Network Mac And Pcs merlin371 Linux - Networking 0 02-18-2004 05:27 PM
How do you network 2 pcs with 1 ADSL connection? Rutnut Linux - Networking 5 05-13-2002 04:19 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 10:42 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration