LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Server (http://www.linuxquestions.org/questions/linux-server-73/)
-   -   identifying cause of Consistent High IO Wait on RHEL 5.8 server (http://www.linuxquestions.org/questions/linux-server-73/identifying-cause-of-consistent-high-io-wait-on-rhel-5-8-server-4175451169/)

cxc718 02-21-2013 03:27 PM

identifying cause of Consistent High IO Wait on RHEL 5.8 server
 
I have an RHEL 5.8 server that has been experiencing high io wait times as queried via snmp, or displayed via top. The wait time is hovering in the 35-50% range.

I've tried using ps to look for processes that go into a 'D' state, by using a snippet like this:

for x in `seq 1 1 10`; do ps -eo state,pid,cmd | grep "^D"; echo "----"; sleep 5; done

which gives me the following processes as being in a D state -

syslogd
kjournald

I don't see anything unusual being written to the syslogd. /var/log/messages is about 4 mb, after 4 days of being written to.

I'm not sure where to go next with this. Can anyone suggest what to look for next?

Thanks

kbp 02-21-2013 05:25 PM

Are there any hardware issues reported? .. like a failed disk? .. or a raid controller issue?

chrism01 02-22-2013 02:14 AM

Tried iotop http://linux.die.net/man/1/iotop ?

cxc718 02-27-2013 04:48 PM

No disk or controller issues that I can find. I'm going to try IOTOP and see what that reveals.

abefroman 02-27-2013 08:17 PM

Get the load close to 0, then try to write a 2GB file to the harddrive, then read it. If its slow, then the drive is probably bad.

Code:

time dd of=testfile if=/dev/zero bs=1024 count=2048000;

time cat testfile >/dev/null


chrism01 02-27-2013 08:31 PM

BTW, if its only those 2, that's odd. This 'kjournald' is worrying (kernel journal 'process' for ext3, ext4 fs).
Maybe disk is going bad, so running out of good blocks?
Have a look at http://linux.die.net/man/8/smartctl


All times are GMT -5. The time now is 03:54 AM.