LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices



Reply
 
Search this Thread
Old 02-21-2013, 03:27 PM   #1
cxc718
LQ Newbie
 
Registered: Jul 2012
Posts: 4

Rep: Reputation: Disabled
identifying cause of Consistent High IO Wait on RHEL 5.8 server


I have an RHEL 5.8 server that has been experiencing high io wait times as queried via snmp, or displayed via top. The wait time is hovering in the 35-50% range.

I've tried using ps to look for processes that go into a 'D' state, by using a snippet like this:

for x in `seq 1 1 10`; do ps -eo state,pid,cmd | grep "^D"; echo "----"; sleep 5; done

which gives me the following processes as being in a D state -

syslogd
kjournald

I don't see anything unusual being written to the syslogd. /var/log/messages is about 4 mb, after 4 days of being written to.

I'm not sure where to go next with this. Can anyone suggest what to look for next?

Thanks
 
Old 02-21-2013, 05:25 PM   #2
kbp
Senior Member
 
Registered: Aug 2009
Posts: 3,758

Rep: Reputation: 644Reputation: 644Reputation: 644Reputation: 644Reputation: 644Reputation: 644
Are there any hardware issues reported? .. like a failed disk? .. or a raid controller issue?
 
Old 02-22-2013, 02:14 AM   #3
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.6, Centos 5.10
Posts: 16,324

Rep: Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041
Tried iotop http://linux.die.net/man/1/iotop ?
 
Old 02-27-2013, 04:48 PM   #4
cxc718
LQ Newbie
 
Registered: Jul 2012
Posts: 4

Original Poster
Rep: Reputation: Disabled
No disk or controller issues that I can find. I'm going to try IOTOP and see what that reveals.
 
Old 02-27-2013, 08:17 PM   #5
abefroman
Senior Member
 
Registered: Feb 2004
Location: lost+found
Distribution: CentOS
Posts: 1,278

Rep: Reputation: 53
Get the load close to 0, then try to write a 2GB file to the harddrive, then read it. If its slow, then the drive is probably bad.

Code:
time dd of=testfile if=/dev/zero bs=1024 count=2048000;

time cat testfile >/dev/null
 
Old 02-27-2013, 08:31 PM   #6
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.6, Centos 5.10
Posts: 16,324

Rep: Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041
BTW, if its only those 2, that's odd. This 'kjournald' is worrying (kernel journal 'process' for ext3, ext4 fs).
Maybe disk is going bad, so running out of good blocks?
Have a look at http://linux.die.net/man/8/smartctl
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Wait IO suddenly extremely high (exploit ?), crashing the server gcat Linux - Security 18 04-18-2012 01:14 PM
High I/O wait observed in Linux based Oracle Database Server rajaniyer123 Linux - General 3 01-16-2012 10:00 PM
Help troubleshooting Intermittent high IO Wait lbeachmike Linux - Server 6 02-05-2010 12:56 AM
Determine cause of high IO Wait on CPU? danraider Linux - Software 2 10-15-2007 10:10 AM
High I/O Wait times and TCP buffers? oswald21 Linux - General 0 08-26-2003 11:22 AM


All times are GMT -5. The time now is 06:04 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration