SAR Ouput Question
 

Each night my RH7.2 shows 0% idle at midnight. I can't find anything the logs that accounts for this.

Any ideas why the 110% usage?
Is this just sar starting up?
The daily sar logfiles are timestamed 11:50

11:40:00 PM CPU %user %nice %system %idle
11:50:00 PM all 0.01 0.00 0.09 99.90
12:00:00 AM all 109.71 109.74 109.73 0.00

thanks.

Probably a cron job scheduled to run at midnight. Something like updatedb can tie it up pretty tight. How long does it last? that does 12:10:00 say?

Try checking /etc/crontab and /etc/cron.d/*

12:10 is like 11:50, it's idle most of the night. 15% usage at 3:00Am when my backup kicks in.

I can't find anything in the cron logs at midnight. I thought maybe sar which runs every 10 min and creates a new log after the 11:50 run might be just spitting out garbage when it initializes the new file at midnight. I mean, how do you get 109% in 3 columns? They are all supposed to add upto 100%

-thanks

They're caused by nice tasks. The niced tasked get added to system time as well as user time, so sometimes the total ends up over 100%. So if sys = 0 and usr = 0, all will be equal to nice. It is kind of interesting to see the niced task take up 109% by itself. Do you remember prioritizing anything above normal? A snapshot of system activity at that time( something like top) will be useful to find out where the system is spending such a "nice time"!!

Whatever is happening at midnight was not my doing, maybe part of RH install which I made very minimal for a server setup.

My hardware router has "time initialize" in the logs each night at 11:55. I think that is a clock thing for the router only.

I'll keep looking.

I run a pretty vanilla RH 7.2 server and do not see the activity you see on yours. You might get a clue by viewing your processes (ps -ef or ps -aux) redirect it out to a file for comparison. Do one at 11:50 and every 60 seconds thereafter.
As mentioned earlier, Top or Gtop (gnome) might lead you right to the process, but if top shows a child process of the culprit as the top consumer of resources, then the ps output could show you all the processes which are active on your system at the time of all this activity. You could use the diff command to compare output (diff psoutput1 psoutput9)
Hope this helps;
Moon69

If you havent already found out what's eating up the system resources yet..

Try downloading the latest sysstat package from
http://perso.wanadoo.fr/sebastien.godard/

This one doesnt seem to have the niced task getting added to sys and user issue.

I'm going to run a script tonight to see if I can catch it. I'll post my results if I find it. I have a minimal install, no gui, so I'm a little baffled.

My redhat RPM is sysstat-4.0.1-2

Well I ran a script from 11:55 - 12:05 to ps -ef output and got nothing. I then looked at sar and the phantom deamon moved to 12:40 and went from 109% to 126%

12:10:00 PM all 0.03 0.00 0.01 99.97
12:40:00 PM all 126.41 126.49 126.47 0.00
12:50:00 PM all 0.01 0.00 0.11 99.88

Still nothing in the logs that makes sense. No evidence of a hack, no ftp, httpd activity. Here's the normal running processes.

UID PID PPID C STIME TTY TIME CMD
root 1 0 0 2002 ? 00:00:27 init [3]
root 2 1 0 2002 ? 00:00:00 [keventd]
root 3 1 0 2002 ? 00:00:00 [ksoftirqd_CPU0]
root 4 1 0 2002 ? 00:00:00 [ksoftirqd_CPU1]
root 5 1 0 2002 ? 00:00:17 [kswapd]
root 6 1 0 2002 ? 00:00:00 [bdflush]
root 7 1 0 2002 ? 00:00:12 [kupdated]
root 9 1 0 2002 ? 00:00:00 [khubd]
root 10 1 0 2002 ? 00:00:20 [kjournald]
root 136 1 0 2002 ? 00:00:00 [kjournald]
root 137 1 0 2002 ? 00:00:04 [kjournald]
root 138 1 0 2002 ? 00:00:02 [kjournald]
root 139 1 0 2002 ? 00:00:09 [kjournald]
root 496 1 0 2002 ? 00:00:06 syslogd -m 0
root 501 1 0 2002 ? 00:00:00 klogd -2
rpc 521 1 0 2002 ? 00:00:00 portmap
rpcuser 549 1 0 2002 ? 00:00:00 rpc.statd
root 609 1 0 2002 ? 00:00:15 /sbin/apcupsd -f /etc/apcupsd/ap
root 610 609 0 2002 ? 00:00:17 /sbin/apcupsd -f /etc/apcupsd/ap
root 611 610 0 2002 ? 00:00:00 /sbin/apcupsd -f /etc/apcupsd/ap
root 721 1 0 2002 ? 00:00:25 /usr/sbin/sshd
root 754 1 0 2002 ? 00:00:00 xinetd -stayalive -reuse -pidfil
root 764 1 0 2002 ? 00:00:00 /usr/knox/bin/nlservd start
lp 788 1 0 2002 ? 00:00:00 lpd Waiting
root 865 1 0 2002 ? 00:01:26 sendmail: accepting connections
root 907 1 0 2002 ? 00:00:07 crond
root 977 1 0 2002 ? 00:00:00 smbd -D
root 982 1 0 2002 ? 00:00:21 nmbd -D
root 1024 1 0 2002 tty2 00:00:00 /sbin/mingetty tty2
root 1025 1 0 2002 tty3 00:00:00 /sbin/mingetty tty3
root 1026 1 0 2002 tty4 00:00:00 /sbin/mingetty tty4
root 1027 1 0 2002 tty5 00:00:00 /sbin/mingetty tty5
root 1028 1 0 2002 tty6 00:00:00 /sbin/mingetty tty6
root 27168 1 0 Jan02 tty1 00:00:00 /sbin/mingetty tty1
root 10797 1 0 Jan15 ? 00:00:00 /bin/sh /usr/bin/safe_mysqld --d
mysql 10824 10797 0 Jan15 ? 00:00:00 /usr/libexec/mysqld --defaults-f
mysql 10826 10824 0 Jan15 ? 00:00:06 /usr/libexec/mysqld --defaults-f
mysql 10827 10826 0 Jan15 ? 00:00:00 /usr/libexec/mysqld --defaults-f
mysql 10828 10826 0 Jan15 ? 00:00:00 /usr/libexec/mysqld --defaults-f
root 17463 1 0 Jan17 ? 00:00:14 /usr/sbin/httpd -DHAVE_ACCESS -D
apache 16463 17463 0 04:02 ? 00:00:00 /usr/sbin/httpd -DHAVE_ACCESS -D
apache 16464 17463 0 04:02 ? 00:00:00 /usr/sbin/httpd -DHAVE_ACCESS -D
apache 16465 17463 0 04:02 ? 00:00:00 /usr/sbin/httpd -DHAVE_ACCESS -D
apache 16466 17463 0 04:02 ? 00:00:00 /usr/sbin/httpd -DHAVE_ACCESS -D
apache 16467 17463 0 04:02 ? 00:00:00 /usr/sbin/httpd -DHAVE_ACCESS -D
apache 16468 17463 0 04:02 ? 00:00:00 /usr/sbin/httpd -DHAVE_ACCESS -D
apache 16469 17463 0 04:02 ? 00:00:00 /usr/sbin/httpd -DHAVE_ACCESS -D
apache 16470 17463 0 04:02 ? 00:00:00 /usr/sbin/httpd -DHAVE_ACCESS -D

You could download "atop" from Freshmeat.net and run it with a small interval. I find it gives better output compared to running "top".
If you're bent on running "ps" then try I would suggest running it as "\ps axh -eo %C -eo priority,command | sort -nr". The backslash obsoletes any ps alias you have, h is w/o header, %C shows CPU usage, the rest should be clear.

I thought I had the latest sysstat rpm but found a newer one. I installed it yeaterday and I did not get the same results last night. I'm going to monitor it, I believe it was an issue with sysstat and smp when it rotates it's daily log, or something along those lines.