Hello, we have a few web servers that are monitored via nagios, the servers hardly ever go down, but from time to time, we get 10 second timeouts in nagios, but the pages load fine after a bit. The servers do get a high amount of traffic, but they don't run out of CPU or MEM.

We believe it may be file descriptors or hitting the TCP Connection limit, any idea how I can diagnose this? the logs don't seem to be of much help that I can see. Where can I verify its one of the two issues I stated? If so, I can just increase the limits

What distro/version?

And which service or services get high amounts of traffic? If you're hitting kernel limits, messages are generally barfed out to /var/log/messages. If you're hitting application limits, messages should be found in their specific logs.

Centos 5.5

Ill tail the logs for a while, but I don't expect to see much in there.

The service is just HTTPD

Apache's error logging will definitely make noise about hitting MPM limits. If you don't see it, then crank LogLevel up to info.

I changed the LogLevel to info, and tailed the error_log for a while.

When the server hit the mons, I saw the below log entry about 10 times for 35 different client IP's
Time to hit the googles

Interesting

That's the one. You know what to do, que no?