Very high system level processing

kenneho · 01-06-2009, 08:12 AM

Hello all.

One of my virtual RHEL 5.1 boxes is running slow, and I've found that system-level processing is very high:

Code:

[root@server ]# sar -u 2 50 
02:17:57 PM       CPU     %user     %nice   %system   %iowait    %steal     %idle 
02:17:59 PM       all     55.72      0.00     44.28      0.00      0.00      0.00 
02:18:01 PM       all     62.00      0.00     38.00      0.00      0.00      0.00 
02:18:03 PM       all     64.71      0.00     35.29      0.00      0.00      0.00 
02:18:05 PM       all     63.82      0.00     36.18      0.00      0.00      0.00

I'm not very experienced on these things, but I belive that such a high persentage of system level CPU usage is higher than normal. Can anyone confirm this? And does anyone know how to figure out exactly why the kernel is using so much CPU?

Regards,
kenneho

uks · 01-06-2009, 09:41 AM

Have you checked the sar logs(usually at /var/log/sa/) and see if that helps ?
You have the memory usage, swapping details etc in these files which will tell you if this high cpu usage is related to that.
And if it is then you drive your investigations on those lines.
Other than that, if its happening all the while, you need to check the normal things, like top and see if you can find the process thats saturating the cpu
And to answer your question, Yes, this is high usage of the cpu(though I do not know the configuration details.)

kenneho · 01-07-2009, 04:00 AM

Thanks for you reply.

The files found under /var/log/sa are very useful, as it displays a lot of information about the system. But do you know if there are any addons or something I can use to easier extrac relevant information for the files? As of now it's just a big text file, and I'm not quite sure what I'm looking for.

robertjinx · 01-07-2009, 04:24 AM

Why dont you try to use "top" or "htop" to see what is doing the high system load. I think is a bit more easier.

kenneho · 01-07-2009, 10:19 AM

Quote:

Originally Posted by robertjinx

Why dont you try to use "top" or "htop" to see what is doing the high system load. I think is a bit more easier.

I do use "top", but since this problem seem to come and go I'm often forced to search the historical data collected by for example sar.
I'm quite puzzled, since I don't (yet, at least) see any reason why the kernel (i.e. system-level) is using so much CPU power.

robertjinx · 01-07-2009, 10:29 AM

Try this small script, copy it and save it under the name "cpuload" for example and then add it to crontab like this:

*/10 * * * * ( su - root -c '/path/to/cpuload' ) >/dev/null 2>&1

PHP Code:



#!/bin/sh -e 
 
AWK=/usr/bin/awk 
SAR=/usr/bin/sar 
GREP=/bin/grep 
TR=/usr/bin/tr 
HEAD=/usr/bin/head 
PS=/bin/ps 
SORT=/usr/bin/sort 
HOSTNAME=$(hostname -f) 
SED=/bin/sed 
LOAD=49 
CAT=/bin/cat 
MAILFILE=/tmp/mailviews$$ 
MAILER=/usr/bin/mail 
mailto="somemail@example.org" 
 
for path in $PATH; 
        do 
        CPU_LOAD=$($SAR -P ALL 1 2 | $GREP 'Average.*all' | $AWK -F" " '{ print 100.0 -$NF}'|awk '{$1=sprintf("%.0f",$1)}1') 
    #CPU=$(echo $CPU_LOAD|awk '{$1=sprintf("%.0f",$1)}1') 
    echo $CPU_LOAD 
 if [ $CPU_LOAD -gt $LOAD ]; then 
                PROC="$($PS -eo user,pcpu,pid -o comm= --no-heading| $SORT -k2 -n -r|head -4)" 
                echo -e "$PROC\n" 
                echo "Please check your processes on ${HOSTNAME} the value of cpu load is $CPU_LOAD" > $MAILFILE 
        echo "" >> $MAILFILE 
        echo -e "USER\tCPU\tPID\tPROC" >> $MAILFILE 
        echo -e "$PROC\n" >> $MAILFILE 
                $CAT $MAILFILE | $MAILER -s "CPU Load is $CPU_LOAD % on ${HOSTNAME}" $mailto 
        rm -f $MAILFILE 
 fi 
 
done 
 
exit 0

Im using this on my servers and seems to work quite fine. Just try it and see in the mail whats making the high load.

robertjinx · 01-07-2009, 10:40 AM

I forgot to add there, that you might need to change some of the variables as some of the apps are in different location. Like "awk" or "mail" and so on. So good luck!

P.S. it doesn't work if the variables are not correct, also you can adjust the value of LOAD to lets say 55 or something similar.

rweaver · 01-07-2009, 11:47 AM

Quote:

Originally Posted by robertjinx

Try this small script, copy it and save it under the name "cpuload" for example and then add it to crontab like this:

*/10 * * * * ( su - root -c '/path/to/cpuload' ) >/dev/null 2>&1

Im using this on my servers and seems to work quite fine. Just try it and see in the mail whats making the high load.

Another neat trick is to use 'which' to set the locations after setting appropriate paths /bin:/sbin:/usr/bin:/usr/sbin etc... PROG=`which name` which makes things a bit more portable, although if you're using awk and sed and such you need to be careful that the output is the same, sometimes there are discrepancies.

kenneho · 01-08-2009, 01:39 AM

Thanks for the script - I'll give it a try and report back here.

uks · 01-12-2009, 04:55 AM

Yes the log files are too huge but I forgot to mention that there are ways to query them.
Let me give you an example:
sar -f /var/log/sa/sa08 -P ALL

will pull out just the CPU activity info from the accounting file for the 8th.
And if you combine this with the time interval like this:

sar -f /var/log/sa/sa08 -P ALL -s 15:00:00 -e 16:00:00

you can get it between the specified time frame, here between 3 and 4 pm.

Check sar -help and you can try the various combinations there.

--Ukesh Upendran

kenneho · 01-12-2009, 05:20 AM

Great, this was most helpful. Such features are not always easy to discover by oneself.