Hello,

Our Apache web server is heavily used for downloads of large (100MB+) files.

Occasionally - every couple of months or so - we get a user who appears to have misconfigured download software. It attempts to download one of these large files and opens multiple simultaneous connections until all available Apache connections are used up. When this happens no one else can connect. In one case, our logs show that this denial of service lasted for 15 hours.

Restarting the web service does not help, since within two minutes all available connections are full again. MaxClients is currently set to 256. That is the most I can set this to without recompiling Apache but I don't think that increasing this will help because the client appears to use up all available connections no matter how many there are.

I have also tried installing mod_limitipconn.c - see http://dominia.org/djao/limitipconn.html for details. This module sends the user a 503 error message when exceeding a defined limit (currently set to 4). However this error message itself appears to use up a connection. So a client that keeps blindly making requests regardless of the error messages still uses up all available connections.

At this time the only way I have found to mitigate the problem is to manually block all connections from the user's IP address with iptables. This of course requires that I am aware of the problem when it happens and can log in to block the user before he brings down the server.

Do you have any suggestions to fix this problem?

Here are a few lines from the log file. This goes on for over 60,000 attempts!

222.66.100.122 222.66.100.122 - [28/May/2008:22:38:30 -0700] "GET /syn_dl/OKsGhzaOKMgXpczdaoKdZzESG/fpga922.exe HTTP/1.1" 503 328 "http://cct.me.ntut.edu.tw/chchting/aiahtm/display2007-2.html" "Mozilla/4.0 (compatible; MSIE 5.00; Windows 98)"
222.66.100.122 222.66.100.122 - [28/May/2008:22:38:30 -0700] "GET /syn_dl/OKsGhzaOKMgXpczdaoKdZzESG/fpga922.exe HTTP/1.1" 503 328 "http://cct.me.ntut.edu.tw/chchting/aiahtm/display2007-2.html" "Mozilla/4.0 (compatible; MSIE 5.00; Windows 98)"
222.66.100.122 222.66.100.122 - [28/May/2008:22:38:30 -0700] "GET /syn_dl/OKsGhzaOKMgXpczdaoKdZzESG/fpga922.exe HTTP/1.1" 503 328 "http://cct.me.ntut.edu.tw/chchting/aiahtm/display2007-2.html" "Mozilla/4.0 (compatible; MSIE 5.00; Windows 98)"
222.66.100.122 222.66.100.122 - [28/May/2008:22:38:32 -0700] "GET /syn_dl/OKsGhzaOKMgXpczdaoKdZzESG/fpga922.exe HTTP/1.1" 503 328 "http://cct.me.ntut.edu.tw/chchting/aiahtm/display2007-2.html" "Mozilla/4.0 (compatible; MSIE 5.00; Windows 98)"
222.66.100.122 222.66.100.122 - [28/May/2008:22:38:32 -0700] "GET /syn_dl/OKsGhzaOKMgXpczdaoKdZzESG/fpga922.exe HTTP/1.1" 503 328 "http://cct.me.ntut.edu.tw/chchting/aiahtm/display2007-2.html" "Mozilla/4.0 (compatible; MSIE 5.00; Windows 98)"
222.66.100.122 222.66.100.122 - [28/May/2008:22:38:32 -0700] "GET /syn_dl/OKsGhzaOKMgXpczdaoKdZzESG/fpga922.exe HTTP/1.1" 503 328 "http://cct.me.ntut.edu.tw/chchting/aiahtm/display2007-2.html" "Mozilla/4.0 (compatible; MSIE 5.00; Windows 98)"
222.66.100.122 222.66.100.122 - [28/May/2008:22:38:32 -0700] "GET /syn_dl/OKsGhzaOKMgXpczdaoKdZzESG/fpga922.exe HTTP/1.1" 503 328 "http://cct.me.ntut.edu.tw/chchting/aiahtm/display2007-2.html" "Mozilla/4.0 (compatible; MSIE 5.00; Windows 98)"


Thank you.

Sorry, I forgot to mention...

Server: Apache/1.3.33 (Unix) mod_ssl/2.8.22 OpenSSL/0.9.7g

The same IP each time? I'd just permanently firewall block the client and be done with it.

No, it usually comes from different locations, often from China. I don't know how it happens but there seem to be a lot of these resource hogs out there. One day recently, mod_limitipconn blocked over 364,000 requests! mod_limitipconn is helping some, but once in a while it still gets overwhelmed.

I see. There are plenty of DoS attacks out there. Sometimes the best you can do is just block the host, or IP range.

You can rate limit that IP or IP range, Consider also fail2ban: http://www.fail2ban.org/wiki/index.php/Main_Page

And a more complete overview of solutions can be found here: http://www.linuxquestions.org/questi...tempts-340366/

You should think about doing this on layer 3. Most firewalls and commercial load balancers easily do this for you with built in rate/connection limiting/DDOS functions. Some routers/switches can do this as well.

If you use iptables and only have one download server, fail2ban like Mr. C suggested works great.

Thank you for the suggestions everyone. I have installed fail2ban and it seems to be working well so far. I have set it to automatically block IP addresses that generate more than 100 excessive connection warnings in 10 minutes. 15 IP addresses have been blocked since yesterday. Interestingly they are all from China and all are using IE 5 or IE 6 according to the user agent string. I don't understand it. Is there something that would make IE open excessive connections when connecting from China? Anyway I am happy that I now have a tool to handle the problem. Thanks again.