I have an RHEL 5.8 server that has been experiencing high io wait times as queried via snmp, or displayed via top. The wait time is hovering in the 35-50% range.

I've tried using ps to look for processes that go into a 'D' state, by using a snippet like this:

for x in `seq 1 1 10`; do ps -eo state,pid,cmd | grep "^D"; echo "----"; sleep 5; done

which gives me the following processes as being in a D state -

syslogd
kjournald

I don't see anything unusual being written to the syslogd. /var/log/messages is about 4 mb, after 4 days of being written to.

I'm not sure where to go next with this. Can anyone suggest what to look for next?

Thanks

Are there any hardware issues reported? .. like a failed disk? .. or a raid controller issue?

Tried iotop http://linux.die.net/man/1/iotop ?

No disk or controller issues that I can find. I'm going to try IOTOP and see what that reveals.

Get the load close to 0, then try to write a 2GB file to the harddrive, then read it. If its slow, then the drive is probably bad.

BTW, if its only those 2, that's odd. This 'kjournald' is worrying (kernel journal 'process' for ext3, ext4 fs).
Maybe disk is going bad, so running out of good blocks?
Have a look at http://linux.die.net/man/8/smartctl