Help troubleshooting Intermittent high IO Wait
 

Hi there -

I am running a shared hosting server running Redhat Enterprise 5.4, and the latest stable cpanel/WHM build on Apache 2.2/PHP5.2.12/SQL5.

I am experiencing intermittent browser time-out issues when accessing the server during more busy periods, but sometimes also during less busy periods. The server is not particularly loaded, but what I do notice is that IO Wait spikes up intermittently to levels as high as 35-45% but for very short intervals.

Thus far, this is the only evidence I can find of any source of delay and I'm not sure how to dig into this further.

Can anybody provide some guidance?

I'll be happy to provide any further details if the above is not a sufficient starting point.

Thanks.

Mike

Sounds very much like a DB issue. Short spikes can mean there's not enough RAM, so DB accesses have to go to the disk. This can be exacerbated by insufficient or inaccurate (ie wrong/unused) indexes on the DB tables.
Inside the DB you can use http://dev.mysql.com/doc/refman/5.0/...ocesslist.html to see what SQL cmds are being run & other useful info.
See also http://dev.mysql.com/doc/refman/5.0/en/show.html in general.

I'd also take a look at top, specifically swap when this happens.

Hi there -

I appreciate your insight and suggestions.

I've used the show processlist command in SQL and there is not much of anything going on during these very sluggish periods.

However, I do see this one suspicious entry that is waiting for a database insert for an extremely long period of time -

175441 | DELAYED | localhost | dcdeaf_wrdp1 | Delayed insert | 127 | Waiting for INSERT

In fact, it seems as if no matter when I do a show processlist, I see an entry with a delayed insert of some period of time with the same Waiting for INSERT status and the same process ID number.

What would I conclude from this, and what actions would you recommend?

Thanks.

Mike

I believe there's a cmd (or check binlogs) to find out what user is doing that and what the full (SQL) cmd is.
Have a dig around the docs there, they're pretty good.

Hi there -

Apparently the command is "show full processlist" - however, I've simply killed off that process and it has not returned - so I suppose it was a query that was hung up for some reason.

I am still stuck with a server that has a problem of intermittent sluggishness and judging by the SQL activity, I do not think that SQL is the culprit. More specifically, when the server is being unresponsive, static pages are equally as unresponsive as dynamic pages.

I can't find anything screaming out at me from TOP or HTOP or iostat ...

Can anybody help point me in the right direction so that I can work systematically toward narrowing down the cause?

Thanks.

Mike

...

Hi there -

Yes, it rather predictably happens between 11am and 2pm EST each day. Although apache requests increase during that time period, the server load is remaining relatively low and should have no problems handling it.

I believe systat is already installed on this server - I am simply not yet that familiar with the extent to which I can use it; although I have used SAR to a limited extent.

This is a cpanel server and it is setup to rotate the log files; however, perhaps there are log files that have grown out of control and need some pruning.

I appreciate the suggestion and I will definitely take a look at that.

If you have any advice for what to focus in on with systat, that would be appreciated.

Thanks.

Mike