Linux - ServerThis forum is for the discussion of Linux Software used in a server related context.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I recently got some old PC's from a friend who's moving to another city. The problem is I have no idea what to do with them. I thought I could make a compile farm to access over the LAN and the Internet to compile my projects faster and maybe offer some friends the ability to use the farm to compile their projects too. So I need to know a few things about compile farms:
1) Does distributed compiling offer an adequate increase in compile speed over local compilation?
2) My Internet connection is kinda slow (aDSL @ 8192/384). Is this adequate or will I need a faster connection? Generally speaking, do compile farms require fast network I/O or better hardware resources?
3) Any suggestions on which Linux distro to run on a compile farm?
4) Are there any specific hardware requirements for compile farms?
5) Any suggestions other than a compile farm that take advantage of the CPU and memory available?
Here is some quick hardware info about the two PCs
PC1: AMD 7750BE, 2GB RAM (works in DC), 1TB HDD, gigabit ethernet (works at 100mbps due to switch limitations)
PC2: AMD A64 3200+, 1.5GB RAM, 120GB HDD, gigabit ethernet (works at 100mbps due to switch limitations)
PS: I post this on the server section of LQ since it's more of a cluster question than a programming one. If a mod thinks it's not in the right forum, kindly move it. Thanks.
1) Does distributed compiling offer an adequate increase in compile speed over local compilation?
2) My Internet connection is kinda slow (aDSL @ 8192/384). Is this adequate or will I need a faster connection? Generally speaking, do compile farms require fast network I/O or better hardware resources?
3) Any suggestions on which Linux distro to run on a compile farm?
4) Are there any specific hardware requirements for compile farms?
5) Any suggestions other than a compile farm that take advantage of the CPU and memory available?
1.) It can, it seems that you're hosting the farm elsewhere, so you need to take into account the d/l time following your compiles if you're going to be d/l'ing to your local network afterwards, although this will probably save time too since you are only d/l'ing the binaries following compilation of the huge source where your machines are located.
2.) It's totally adequate, since you're not d/l'ing source to your local network, only to the machinery co-located elsewhere, and then d/l'ing the binaries to your local network or distributing them elsewhere still over the higher speed backbone.
3.) Slackware 13.0 (64 bit)
4.) No. Just lots of memory and CPU power.
5.) Compile on the fastest machine with the greatest amount of memory.
When I think of a compile farm, I assume the files (both source and generated binaries) are on one system, while the compilers run on other system(s).
That has been on my todo list for a while, so I'd also like some insight into how to do it better.
Or if the DreamAxe has a different relationship in mind between the compiling and the files, please explain.
I assumed a 1Gb network and even there worry about the network speed. I've done a fair amount of compiling across the network one to one (one compiler system talking to one system with the files) and there are some flaws in caching:
My environment rereads many hpp files many times each in the course of a build. The systems involved have plenty of excess ram to allow the hpp files to sit in file cache from the time they are first read until past the end of the whole build, so each hpp is only read from disk once.
With files and compilation on the same computer, hpp file reads are very cheap. I would hope that the compilation computer could similarly cache hpp files even though they come from the network rather than a local hard drive. At best that has been imperfect. The same hpp files transfer from the file computer's file cache across the 1Gb network to the compilation system multiple times during a build.
With slower than 1Gb networking, I don't think this whole model of distributing files vs. compilation makes any sense.
When I think of a compile farm, I also think of running many compiles in parallel. I'm using an obsolete locally modified version of bjam (like make but smarter) to invoke the compile command and it is pretty good at keeping N compiles running in parallel for user specified N. But they're all on the same (multi core) computer (bjam runs where the compilation runs because it invokes compilers only on the local system).
I don't know what is available as a version of make or a replacement for make that could launch multiple builds on multiple systems. Without that, I don't think a "compile farm" has much point.
To reduce network traffic you can rsync the sources to the compile farm first (transferring only the changes), then compile and rsync the result back.
You could use cvs read-only checkout on the farm, but you would have to commit changes before each compilation...
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.