LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices



Reply
 
Search this Thread
Old 05-06-2009, 03:55 PM   #1
johnsfine
Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,139

Rep: Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127Reputation: 1127
SGE (Sun Grid Engine) advice please


I want some advice on SGE, for use in distributing compile and test tasks across several Centos systems with 1Gb LAN connections.

I looked at the SGE site and read through the FAQ and looked for the "introduction" that was mentioned but with no obvious link. I couldn't find any place the basics were described (a level deeper than "it distributes work for you" but much less deep than most of what I saw there that assumed you already knew a lot about using SGE).

Is it the right tool at all? Or is there some other Linux tool to do the job I want?

What is the level of overhead? Would it make sense to hand it thousands of tasks most of which only take a second? Or do I need to group tasks up to a reasonable level to avoid adding lots of overhead?

Is it practical to have a lightweight task on the requesting computer waiting for the requested task to finish (usually on some other computer)? Is it practical to specify some notification task to be run after a requested task finishes?

Is it practical to specify a task not start until after a listed set of previous tasks has finished?

One major use I have is doing compile and link tasks in a project with thousands of .cpp files to be compiled and linked to create dozens of .so files, all multiplied by several different tool sets (32 bit vs. 64 bit target, debug build vs. release build, Intel compiler vs. GCC, different versions of each, etc.)

Currently it is all launched by an obsolete version of bjam, which has a j switch so it can be told how many compiles/links to run in parallel, which I base on either the number of cores or the amount of ram on the system (whichever is more limiting). A top level script runs several bjam commands (strictly sequentially) to do each of a few major subprojects times each of the currently enabled tool sets. That whole script is launched manually on one of a few Centos systems based on which isn't busy.

If something like SGE has very low overhead for tiny tasks, then I would want a very lightweight local task to be launched by bjam for every compile and every link, with bjam set to a much higher j value (number in parallel). Those lightweight local tasks would request the individual compile or link and wait for them (because bjam needs to think the task is local). So the top level scrip and bjam and all those lightweight tasks would run on one manually selected system and the real work would occur on several SGE selected systems. That would represent a minimum change from the current structure.

But I expect something like SGE has too much overhead for that idea. So another alternative would be handing whole bjam commands to SGE from the top level script. Most can run in parallel. But some of them cannot be started until specific earlier ones complete. So I would need extra logic to manage that.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Sun Grid Engine on linux me_spearhead Linux - General 5 03-12-2010 08:11 AM
Does anybody knows SUN GRID ENGINE?? meng_en Linux - General 4 02-25-2010 12:03 AM
sun grid(qmaster) installation srinivas2433 Linux - Software 3 07-10-2009 06:58 AM
Installation Of Sun Grid Engine parani86 Linux - Software 7 08-06-2008 08:56 AM
Grid engine SGE process hanging Red Hat haiders Linux - Software 0 09-18-2006 07:04 PM


All times are GMT -5. The time now is 09:57 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration