LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   execution many job in one node (https://www.linuxquestions.org/questions/linux-newbie-8/execution-many-job-in-one-node-4175505277/)

hoi 05-17-2014 12:48 AM

execution many job in one node
 
Lets, there are four nones and each node has 32 processors. If I submit 4 job with 8 processors per job then it distributes to each 4 node. So, the uses of node is 100% but the uses of processor is 50%. Is it possible to distribute 4 job in two nodes, that means uses of node is 50% and also the uses of processor is 50% ??

btmiller 05-17-2014 06:45 PM

What software are you using to distribute the jobs to nodes. There are many batch scheduler available such as TORQUE, Open Grid Engine, SLURM, LSF, etc. The way that job distribution is configured depends heavily on which one (if any) is in use. You need to provide more details about how you're launching and distributing jobs.

jpollard 05-19-2014 08:15 AM

In addition to the type batch system, it can also depend on the configuration of that batch system.

Most have ways to designate queues that define the number of CPUs available (say 32), and only allow one queue on a server - thus multiple jobs on one compute server are not possible. This is usually done to prevent overloading a particular compute resource. If the queue was configured for two concurrent jobs, then the CPUs would also tend to be restricted to 16 CPUs each... This would allow the fastest throughput.

Now on a low priority configuration NOT configured for high throughput (for testing jobs for example) the configuration COULD allow for two concurrent jobs, on a compute service that has only 32 CPUs available with jobs that use 32 CPUs - at the cost of potential thrashing.

The users desire may conflict with site policy, and available queue structures.

Where I worked, we had high priority queues and low priority queues - and a "whatever is left over" queue. In our case, the 16 CPUs (it was a Cray mainframe) had queues for 4 cpus (high priority, only one job possible), a queue for 4 more CPUS, but with two jobs possible (a medium priority), a production queue of 8 CPUs.... and for very high priority, a queue with 16 CPUs, but it was idled (jobs could be submitted, but they wouldn't run - and it required special permission to use it as operations had to stop all other queues before enabling this one to run).

The "whatever was left" was a low priority queue (it only ran if none of the higher priority queues had a job), and was permitted only 2 CPUs (but 4 concurrent jobs). Its purpose was just for testing.


All times are GMT -5. The time now is 08:33 AM.