Unable to utilize all CPUs on RedHat

ikmalf · 01-19-2012, 09:31 PM

Hi,

Currently, we have a system which shows 24 CPUs in 'top'. This system runs on Linux (RHEL 5.5 x86 64bit to be exact). This system/server is being used for data processing/analysis.

Previously, we were running individual processes. Each process will occupy 1 cpu 100%, this process is heavily CPU bound and will normally run for a few hours to few days (depending on data size analysed).

For some reason, when we run 24 of these processes concurrently, only 12 of the CPU is being utilized, or I should say, only the first 12 CPUs are serving these 24 processes.

Just to make sure that this is not an application error, we tried with OS-bundled program (e.g 'dd') and managed replicated the same symptoms. We run the following, 24 times :

$ dd if=/dev/urandom of=/dev/null &

..and again, only 12CPUs (from top output) is serving these 24 'dd' processes.

The question is, how can we enable the OS to utilize all 24 CPUs available on our system? E.g Is there any specific kernel parameter needed to be changed, etc ?

TIA.

sag47 · 01-19-2012, 10:26 PM

Perhaps by setting the processor affinity will resolve your issue.

I have it pre-installed on a RedHat 6.x server at work. It's part of the util-linux-ng package.

Code:

yum provides */taskset
yum install util-linux-ng
man taskset

You could test to see if taskset works as desired on your system with the following simple bash script.

Code:

#!/bin/bash
for i in {0..23};do
  dd if=/dev/urandom of=/dev/null &
  #$i is from the loop
  #$! is the pid of the last run background process
  taskset -c $i $!
done

When you're done killall dd.

SAM

ikmalf · 01-20-2012, 06:13 AM

sag47,

Thank you for your suggestion!
Indeed, 'taskset' does what it claims to do and I learn something new today, yay!

If I understand correctly, right now, if we were to use 'taskset', all the users have to manually check which CPUs are not 'busy' (so to say), say by using 'top' and run 'taskset' _manually_ to set the process to occupy idle CPUs.

I guess, we will use this manual way incase there is no other way.
But, for now, any other workarounds to disable the CPU count limit are welcome.