Hello Folks,
I'm getting so weird behavior of Apache Recently, which really driving me crazy.
Code:
Apache Server Status for localhost
Server Version: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/1.0.0-fips
FrontPage/5.0.2.2635 mod_bwlimited/1.4 mod_auth_passthrough/2.1
mod_fcgid/2.3.6
Server Built: Feb 3 2012 08:17:50
__________________________________________________________________
Current Time: Saturday, 17-Mar-2012 21:45:00 AST
Restart Time: Saturday, 17-Mar-2012 20:39:11 AST
Parent Server Generation: 67
Server uptime: 1 hour 5 minutes 49 seconds
Total accesses: 182195 - Total Traffic: 381.4 MB
CPU Usage: u126.89 s32.84 cu0 cs0 - 4.04% CPU load
101.1 requests/sec - 98.9 kB/second - 2194 B/request
1000 requests currently being processed, 0 idle workers
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
I don't really know, what's causing Apache to take that too much time to send a reply.
Also, once it reaches that amount of load, it throws tons of error:
Code:
[Sat Mar 17 21:53:14 2012] [warn] [client xx.xx.xx.xx] mod_fcgid: can't apply process slot for /usr/local/cpanel/cgi-sys/php5
[Sat Mar 17 21:53:14 2012] [warn] [client xx.xx.xx.xx] mod_fcgid: can't apply process slot for /usr/local/cpanel/cgi-sys/php5
[Sat Mar 17 21:53:16 2012] [warn] [client xx.xx.xx.xx] mod_fcgid: can't apply process slot for /usr/local/cpanel/cgi-sys/php5
I've made every possible effort, to find out what causing this issue, but I couldn't really get to the real cause.
Some of the causes I thought about
:
1- DDoS Attack [I've blocked most of the IPs with too much connections, didn't help].
2- MySQL server performing slowly [It doesn't seems so]
Code:
3869 mysql 20 0 2620m 103m 5128 S 34.1 1.3 4:47.48 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql
3- Few others causes.. but all led to nothing.
----
My setup:
- Cenots 6.2 - 2.6.32-220.7.1.el6.x86_64
- 2x Intel(R) Xeon(R) CPU E5320 @ 1.86GHz
- 8GB Ram
- 2x HDD 15k @ Hardware Raid-1
Would you please, be so kind to advise ?
Also, if you need any extra data.. I'd love to provide.
One thing to mention, that the issue happens all at a sudden.
NOTE:
I tried and restarted Apache, NginX, MySQL, CSF and LFD several times, but that didn't help.
BUT once I rebooted the whole server.. the issue seemed to be solved.
Is that maybe related to any system restrictions, variables or some kind of buffers settings that's being hit in peak hours?
My /etc/sysctl.conf:
Code:
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
Waiting for your help..!!
Regards..
- H
