-   Linux - Server (
-   -   identifying cause of Consistent High IO Wait on RHEL 5.8 server (

cxc718 02-21-2013 02:27 PM

identifying cause of Consistent High IO Wait on RHEL 5.8 server
I have an RHEL 5.8 server that has been experiencing high io wait times as queried via snmp, or displayed via top. The wait time is hovering in the 35-50% range.

I've tried using ps to look for processes that go into a 'D' state, by using a snippet like this:

for x in `seq 1 1 10`; do ps -eo state,pid,cmd | grep "^D"; echo "----"; sleep 5; done

which gives me the following processes as being in a D state -


I don't see anything unusual being written to the syslogd. /var/log/messages is about 4 mb, after 4 days of being written to.

I'm not sure where to go next with this. Can anyone suggest what to look for next?


kbp 02-21-2013 04:25 PM

Are there any hardware issues reported? .. like a failed disk? .. or a raid controller issue?

chrism01 02-22-2013 01:14 AM

Tried iotop ?

cxc718 02-27-2013 03:48 PM

No disk or controller issues that I can find. I'm going to try IOTOP and see what that reveals.

abefroman 02-27-2013 07:17 PM

Get the load close to 0, then try to write a 2GB file to the harddrive, then read it. If its slow, then the drive is probably bad.


time dd of=testfile if=/dev/zero bs=1024 count=2048000;

time cat testfile >/dev/null

chrism01 02-27-2013 07:31 PM

BTW, if its only those 2, that's odd. This 'kjournald' is worrying (kernel journal 'process' for ext3, ext4 fs).
Maybe disk is going bad, so running out of good blocks?
Have a look at

All times are GMT -5. The time now is 02:13 PM.