Hi,
I've got a major site issue with my setup.
It was all working fine until I got hit with a DDOS & moved to a DDOS reverse proxy which is a nginx front-end and a apache2 backend.
There is a ping of ~170ms between the Frontend & Backend.
I have been getting these a lot
Code:
root@b020:~# cat /var/log/apache2/error.log
[Fri Feb 17 18:02:27 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/
[Fri Feb 17 18:02:51 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 18:03:05 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 18:04:19 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Feb 17 18:04:30 2012] [info] [client FrontendIP] Request body read timeout
[Fri Feb 17 18:04:30 2012] [info] [client FrontendIP] Request body read timeout
[Fri Feb 17 18:04:55 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 18:04:56 2012] [info] removed PID file /var/run/apache2.pid (pid=27081)
[Fri Feb 17 18:04:56 2012] [notice] caught SIGTERM, shutting down
[Fri Feb 17 18:05:02 2012] [notice] Apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze8 with Suhosin-Patch configured -- resuming normal operations
[Fri Feb 17 18:05:02 2012] [info] Server built: Feb 5 2012 21:35:42
[Fri Feb 17 18:05:02 2012] [debug] prefork.c(1013): AcceptMutex: sysvsem (default: sysvsem)
[Fri Feb 17 18:05:03 2012] [notice] Graceful restart requested, doing restart
[Fri Feb 17 18:05:03 2012] [notice] Apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze8 with Suhosin-Patch configured -- resuming normal operations
[Fri Feb 17 18:05:03 2012] [info] Server built: Feb 5 2012 21:35:42
[Fri Feb 17 18:05:03 2012] [debug] prefork.c(1013): AcceptMutex: sysvsem (default: sysvsem)
[Fri Feb 17 18:06:34 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 18:07:49 2012] [info] [client FrontendIP] Request body read timeout
[Fri Feb 17 18:07:49 2012] [info] [client FrontendIP] Request body read timeout
[Fri Feb 17 18:08:28 2012] [info] [client FrontendIP] Request body read timeout
[Fri Feb 17 18:08:28 2012] [info] [client FrontendIP] Request body read timeout
[Fri Feb 17 18:08:32 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/
[Fri Feb 17 18:08:42 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 18:08:42 2012] [info] [client FrontendIP] (104)Connection reset by peer: core_output_filter: writing data to the network
[Fri Feb 17 18:08:48 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Feb 17 18:08:49 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 18:09:10 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 18:09:30 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 18:09:54 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 18:10:03 2012] [info] removed PID file /var/run/apache2.pid (pid=27574)
[Fri Feb 17 18:10:03 2012] [notice] caught SIGTERM, shutting down
[Fri Feb 17 18:10:04 2012] [notice] Apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze8 with Suhosin-Patch configured -- resuming normal operations
[Fri Feb 17 18:10:04 2012] [info] Server built: Feb 5 2012 21:35:42
[Fri Feb 17 18:10:04 2012] [debug] prefork.c(1013): AcceptMutex: sysvsem (default: sysvsem)
[Fri Feb 17 18:10:40 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 18:10:59 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Feb 17 18:11:11 2012] [info] removed PID file /var/run/apache2.pid (pid=27903)
[Fri Feb 17 18:11:11 2012] [notice] caught SIGTERM, shutting down
[Fri Feb 17 18:11:12 2012] [notice] Apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze8 with Suhosin-Patch configured -- resuming normal operations
[Fri Feb 17 18:11:12 2012] [info] Server built: Feb 5 2012 21:35:42
[Fri Feb 17 18:11:12 2012] [debug] prefork.c(1013): AcceptMutex: sysvsem (default: sysvsem)
[Fri Feb 17 18:53:36 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/
[Fri Feb 17 18:53:43 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/
[Fri Feb 17 18:54:07 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 18:55:58 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 18:56:28 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 18:57:38 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 18:57:39 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 18:58:56 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Feb 17 18:59:40 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 18:59:46 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Feb 17 19:00:29 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:00:36 2012] [info] [client FrontendIP] Request body read timeout
[Fri Feb 17 19:00:37 2012] [info] [client FrontendIP] Request body read timeout
[Fri Feb 17 19:00:50 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:01:19 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:01:19 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:01:25 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:01:26 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:01:36 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:01:37 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:01:41 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:01:41 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:01:42 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:02:03 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:02:37 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Feb 17 19:02:44 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/
[Fri Feb 17 19:02:44 2012] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 14 idle, and 23 total children
[Fri Feb 17 19:02:51 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/
[Fri Feb 17 19:03:04 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:03:07 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:03:07 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:03:08 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:03:12 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:03:17 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Feb 17 19:03:48 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:04:27 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/
[Fri Feb 17 19:04:31 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:05:01 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/
[Fri Feb 17 19:05:18 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:05:24 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:05:35 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:05:38 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:06:28 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:06:31 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:06:37 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:06:37 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:07:32 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:08:34 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/
[Fri Feb 17 19:09:05 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/
[Fri Feb 17 19:09:22 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:09:39 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:09:45 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:11:30 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Feb 17 19:12:17 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:12:23 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:12:54 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Feb 17 19:13:25 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/
[Fri Feb 17 19:14:43 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:14:44 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:14:50 2012] [info] removed PID file /var/run/apache2.pid (pid=27971)
[Fri Feb 17 19:14:50 2012] [notice] caught SIGTERM, shutting down
[Fri Feb 17 19:14:51 2012] [notice] Apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze8 with Suhosin-Patch configured -- resuming normal operations
[Fri Feb 17 19:14:51 2012] [info] Server built: Feb 5 2012 21:35:42
[Fri Feb 17 19:14:51 2012] [debug] prefork.c(1013): AcceptMutex: sysvsem (default: sysvsem)
[Fri Feb 17 19:14:55 2012] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 13 idle, and 15 total children
[Fri Feb 17 19:15:21 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:15:22 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:15:24 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:15:48 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:15:55 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Feb 17 19:15:59 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:15:59 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:16:06 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:16:08 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:16:29 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:16:30 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:16:32 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/
[Fri Feb 17 19:16:36 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:16:41 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:16:54 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Feb 17 19:16:54 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:16:54 2012] [error] [client FrontendIP] request failed: error reading the headers, referer: http://mysite.com/
[Fri Feb 17 19:16:55 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:16:55 2012] [error] [client FrontendIP] request failed: error reading the headers, referer: http://mysite.com/
[Fri Feb 17 19:17:01 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:17:01 2012] [error] [client FrontendIP] request failed: error reading the headers, referer: http://mysite.com/
[Fri Feb 17 19:17:07 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/
[Fri Feb 17 19:17:13 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/
[Fri Feb 17 19:17:41 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:17:44 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:17:48 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:17:48 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:17:55 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:18:01 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:18:15 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Feb 17 19:18:25 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:18:42 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:19:00 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:19:14 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:19:25 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:19:41 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Feb 17 19:20:00 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:20:25 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/
[Fri Feb 17 19:20:59 2012] [info] [client FrontendIP] Request body read timeout
[Fri Feb 17 19:21:01 2012] [info] [client FrontendIP] Request body read timeout
[Fri Feb 17 19:21:01 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:21:04 2012] [info] [client FrontendIP] Request body read timeout
[Fri Feb 17 19:21:05 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/index.php
[Fri Feb 17 19:21:07 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:21:09 2012] [info] [client FrontendIP] Request body read timeout
[Fri Feb 17 19:21:30 2012] [error] [client FrontendIP] File does not exist: /var/www/mysite.com/images/editor/smilie.gif, referer: http://mysite.com/index.php
[Fri Feb 17 19:21:51 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:22:07 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:22:18 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:22:20 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:22:41 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:22:43 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:22:50 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:24:13 2012] [info] [client FrontendIP] Request body read timeout
[Fri Feb 17 19:24:13 2012] [info] [client FrontendIP] Request body read timeout
[Fri Feb 17 19:24:44 2012] [info] [client FrontendIP] Request body read timeout
[Fri Feb 17 19:24:45 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:24:45 2012] [info] [client FrontendIP] Request body read timeout
[Fri Feb 17 19:24:50 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:24:57 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:25:42 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:25:45 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:25:56 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:25:57 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:26:04 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:26:04 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:26:25 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:26:35 2012] [info] [client FrontendIP] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Feb 17 19:26:52 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:26:52 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:27:13 2012] [info] [client FrontendIP] Request body read timeout
[Fri Feb 17 19:27:14 2012] [info] [client FrontendIP] Request body read timeout
[Fri Feb 17 19:27:50 2012] [info] [client FrontendIP] Request header read timeout
[Fri Feb 17 19:27:57 2012] [info] [client FrontendIP] Request header read timeout
I have replaced actual frontend IP with >> FrontendIP
The above error log reflects very vividly in website's load times and operations.
Extremely slow! I've tried some different things, the website is a vBulletin forum.
I disabled suhsoin using the .htaccess method and it showed a little bit of improvement but these errors are wreaking havoc.
Nginx Config
Code:
limit_zone ZZZ $binary_remote_addr 20m;
upstream sarray {
ip_hash;
# server 1.2.3.4;
server 5.6.7.8;
}
server {
listen 10.10.20.30;
server_name mysite.com;
client_max_body_size 20M;
listen 80 default_server;
location /
{
limit_conn ZZZ 25;
proxy_pass http://sarray;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
Apache2 config file
http://paste2.org/p/1907478 (for better viewing)
Also at times 502 Badass gateway error shows up (apache being unavailable to nginx)
Can someone please assist me in debuggering off the errors
Any help would highly appreciated!
