LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 02-02-2010, 06:32 PM   #1
lbeachmike
LQ Newbie
 
Registered: Feb 2010
Posts: 5

Rep: Reputation: 0
Help troubleshooting Intermittent high IO Wait


Hi there -

I am running a shared hosting server running Redhat Enterprise 5.4, and the latest stable cpanel/WHM build on Apache 2.2/PHP5.2.12/SQL5.

I am experiencing intermittent browser time-out issues when accessing the server during more busy periods, but sometimes also during less busy periods. The server is not particularly loaded, but what I do notice is that IO Wait spikes up intermittently to levels as high as 35-45% but for very short intervals.

Thus far, this is the only evidence I can find of any source of delay and I'm not sure how to dig into this further.

Can anybody provide some guidance?

I'll be happy to provide any further details if the above is not a sufficient starting point.

Thanks.

Mike
 
Old 02-02-2010, 11:59 PM   #2
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,269

Rep: Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028
Sounds very much like a DB issue. Short spikes can mean there's not enough RAM, so DB accesses have to go to the disk. This can be exacerbated by insufficient or inaccurate (ie wrong/unused) indexes on the DB tables.
Inside the DB you can use http://dev.mysql.com/doc/refman/5.0/...ocesslist.html to see what SQL cmds are being run & other useful info.
See also http://dev.mysql.com/doc/refman/5.0/en/show.html in general.

I'd also take a look at top, specifically swap when this happens.
 
Old 02-04-2010, 10:44 AM   #3
lbeachmike
LQ Newbie
 
Registered: Feb 2010
Posts: 5

Original Poster
Rep: Reputation: 0
Hi there -

I appreciate your insight and suggestions.

I've used the show processlist command in SQL and there is not much of anything going on during these very sluggish periods.

However, I do see this one suspicious entry that is waiting for a database insert for an extremely long period of time -

175441 | DELAYED | localhost | dcdeaf_wrdp1 | Delayed insert | 127 | Waiting for INSERT

In fact, it seems as if no matter when I do a show processlist, I see an entry with a delayed insert of some period of time with the same Waiting for INSERT status and the same process ID number.

What would I conclude from this, and what actions would you recommend?

Thanks.

Mike
 
Old 02-04-2010, 07:45 PM   #4
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,269

Rep: Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028
I believe there's a cmd (or check binlogs) to find out what user is doing that and what the full (SQL) cmd is.
Have a dig around the docs there, they're pretty good.
 
Old 02-04-2010, 09:09 PM   #5
lbeachmike
LQ Newbie
 
Registered: Feb 2010
Posts: 5

Original Poster
Rep: Reputation: 0
Hi there -

Apparently the command is "show full processlist" - however, I've simply killed off that process and it has not returned - so I suppose it was a query that was hung up for some reason.

I am still stuck with a server that has a problem of intermittent sluggishness and judging by the SQL activity, I do not think that SQL is the culprit. More specifically, when the server is being unresponsive, static pages are equally as unresponsive as dynamic pages.

I can't find anything screaming out at me from TOP or HTOP or iostat ...

Can anybody help point me in the right direction so that I can work systematically toward narrowing down the cause?

Thanks.

Mike
 
Old 02-04-2010, 10:09 PM   #6
Smartpatrol
Member
 
Registered: Sep 2009
Posts: 196

Rep: Reputation: 38
...

Last edited by Smartpatrol; 03-11-2010 at 10:03 PM.
 
Old 02-04-2010, 11:56 PM   #7
lbeachmike
LQ Newbie
 
Registered: Feb 2010
Posts: 5

Original Poster
Rep: Reputation: 0
Hi there -

Yes, it rather predictably happens between 11am and 2pm EST each day. Although apache requests increase during that time period, the server load is remaining relatively low and should have no problems handling it.

I believe systat is already installed on this server - I am simply not yet that familiar with the extent to which I can use it; although I have used SAR to a limited extent.

This is a cpanel server and it is setup to rotate the log files; however, perhaps there are log files that have grown out of control and need some pruning.

I appreciate the suggestion and I will definitely take a look at that.

If you have any advice for what to focus in on with systat, that would be appreciated.

Thanks.

Mike
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Troubleshooting high load - Possible IO / RAID issue? Keds Linux - Server 12 05-29-2010 04:02 PM
High load, high RAM usage and unresponsive VPS saeed22 Linux - Server 1 08-20-2009 11:58 AM
Need help troubleshooting high load averages jdw52 Linux - Server 6 12-31-2008 06:37 PM
Determine cause of high IO Wait on CPU? danraider Linux - Software 2 10-15-2007 09:10 AM
High I/O Wait times and TCP buffers? oswald21 Linux - General 0 08-26-2003 10:22 AM


All times are GMT -5. The time now is 09:06 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration