Apache mod_security Setup Help?

helptonewbie · 12-19-2007, 09:01 AM

Hi all,
I've been on the mod_security website and can't say the doc's are great, i'm trying to work out if the following is possible:-
I can see that some parts are of the following like redirecting to different URL or blocking completely etc....
We have requests coming into our web servers and we get some annoying people on occasions screen scrapping us, which is basically requesting the same webpage or lots of different webpages in a short space of time, of which each request is taking cpu power and its quite obvious they are not interested in our products so we'd rather just stop them from being so annoying. We know its screen scrapping because we already have monitors that detect this behavior but currently manually just stop the ip address running the scrap we'd like to make this automatic. Therefore if a host (ipaddress) is constantly trying to access the same page over and over or various pages in a short space of time then we'd like to redirect to a holding page and then if it continues to drop their packets completely for a specified amount of time of which it would hopefully work like ip tables and have a timer in which will restart again everytime another request is sent and only after the period of the timer has waited without any requests from that host in question then to allow that host to send requests again. Now i dont know if mod_security is able to do that or perhaps if it might be better to use snort, in which case i'd like to know if there is any case study type things on the internet regarding this problem, i've been looking but not really found anything.

Thanks Regards

win32sux · 12-19-2007, 09:49 AM

Quote:

Originally Posted by helptonewbie

We have requests coming into our web servers and we get some annoying people on occasions screen scrapping us, which is basically requesting the same webpage or lots of different webpages in a short space of time, of which each request is taking cpu power and its quite obvious they are not interested in our products so we'd rather just stop them from being so annoying.

The way I understand it, screen scraping is the action of having a computer read output which was designed to be read by a human. Not sure how that applies here, as what you describe sounds like an attempt to carry-out a denial of service attack. That said, I don't know how to address this with mod_security, but while you wait for an answer from someone who does, I thought I might suggest a quick temporary workaround for you:

Use iptables to prevent individual IPs from starting more than a specified number of HTTP connections in a specified time-frame. If you calculate it right, you should be able to reduce the attacker's connection rate to something only a tad higher than that which you see from your friendly users. Like, for example, let's say that you determine the attacker is starting about eight (or whatever) HTTP connections every ten seconds, while none of your friendly users ever start more than three every ten seconds. In such a case, you could implement a set of rules like this (above the actual ACCEPT rule) on the server:

Code:

iptables -A INPUT -p TCP --dport 80 -m state --state NEW -j STOP-ABUSE

iptables -A STOP-ABUSE -m recent --set

iptables -A STOP-ABUSE -m recent --update --seconds 10 --hitcount 3 -j DROP

Once again, this is only intended as an anti-DoS workaround while you figure out your per-page technique.

helptonewbie · 12-19-2007, 10:10 AM

hello,
Thanks for the reply i should have perhaps said i've already looked at that, but thanks anyway it is useful for anyone that doesn't know it but looks at the thread.
Thanks for the info

Yes it is sort of DoS at the same time for different reasons, not going to explain to much because its never good to give to much away in these forums could shoot yourself in the foot.

win32sux · 12-19-2007, 10:16 AM

Quote:

Originally Posted by helptonewbie

hello,
Thanks for the reply i should have perhaps said i've already looked at that, but thanks anyway it is useful for anyone that doesn't know it but looks at the thread.
Thanks for the info

It's actually great that you have this going right now, as it drastically reduces the urgency of the mod_security solution - which was sort of the point of my post. And yes, you should have mentioned what temporary countermeasures you were currently using. It's all good. I'm sure someone familiar with mod_security will be posting soon. Hang in there.

helptonewbie · 12-19-2007, 10:26 AM

thanks for you help win32SUX, AS DOES 64

, i'll be hoping someone does...

Thanks or replying

helptonewbie · 12-19-2007, 12:20 PM

The other question i've always been curious about is i know its possible to re-write packets that are sent to servers just like NAT does basically but i know people also do it maliciously, so if the source host was re-written and then with the iptables example above, those packets sent to port 80 if reaching the limit, then i'm guessing the server would then block that source address for the specified time therefore creating a DoS on yourself, would that be the same if using snort and mod_security???? i'm guessing it would unless they could detect the packet has been tampered with which isn't possible unless using ip/sec i believe? Therefore this is a draw back to anything along these lines??

I believe that is right though isn't it, if you re-wrote the source host in the ip packet and then sent lots to the server and then it would block that source host but only for the time period even though its not from that actual source??

Cheers

b1v1r · 12-19-2007, 02:23 PM

I assume you are writing about someone spidering your site that is dynamically generated and that is causing you excessive CPU load?

You should be able to do this with ModSecurity using a SESSION or IP collection.

Basically, you want to keep trac of how many hits from a given SESSION/IP and then block (redirect, whatever) based on too many pages requested in a given timeframe.

Something like this will allow only 10 request in a minute from a given IP:

# Block > 10req/minute
SecAction "phase:1,pass,nolog,initcol:IP=%{REMOTE_ADDR},setvar:IP.pagecount=+1,expirevar:IP.pagecount=60"
SecRule IP:PAGECOUNT "@gt 10" "phase:1,t:none,deny,status:403,msg:'Blocking screen scraper'"

You will probably want to augment this with other checks by chaining them togeather:

# Block > 10req/minute for /cgi-bin and not in the IP whitelist
SecRule REQUEST_FILENAME "^/cgi-bin" "phase:1,log,auditlog,t:none,t:normalisePath,deny,status:403,msg:'Blocking screen scraper',chain"
SecRule REMOTE_ADDR "!^(?:127\.0\.0\.1|1\.2\.3\.4)$" "t:none,chain"
SecAction "nolog,initcol:IP=%{REMOTE_ADDR},setvar:IP.pagecount=+1,expirevar:IP.pagecount=60,chain"
SecRule IP:PAGECOUNT "@gt 10" "t:none"

This is pretty simplified and you could do a lot more, but hopefully you get the idea. A warning, though, that IP != user and you could end up blocking a proxy server's IP address with this (ie AOL, NetZero, etc), so probably best to stay away from IPs and use some sort of session or other more unique value (SESSION collection vs IP).

To get into a more detailed discussion on implementation, send a note to the ModSecurity user list (see contact page on website)

later,
-B

b1v1r · 12-19-2007, 02:25 PM

See also #modsecurity on freenode IRC for more help.

later,
-B

win32sux · 12-19-2007, 02:45 PM

Quote:

Originally Posted by helptonewbie

The other question i've always been curious about is i know its possible to re-write packets that are sent to servers just like NAT does basically but i know people also do it maliciously, so if the source host was re-written and then with the iptables example above, those packets sent to port 80 if reaching the limit, then i'm guessing the server would then block that source address for the specified time therefore creating a DoS on yourself, would that be the same if using snort and mod_security???? i'm guessing it would unless they could detect the packet has been tampered with which isn't possible unless using ip/sec i believe? Therefore this is a draw back to anything along these lines??

I believe that is right though isn't it, if you re-wrote the source host in the ip packet and then sent lots to the server and then it would block that source host but only for the time period even though its not from that actual source??

One thing you can do to stop spoofed packets from entering the STOP-ABUSE chain (and possibly causing a denial of service for friendly users) is to activate TCP SYN cookies in order to discard any spoofed SYN packets. Then you follow this up with an iptables rule to filter any packets of state NEW which aren't proper SYN packets. This should in effect make sure that any packets getting sent to STOP-ABUSE have been checked for spoofing. Example:

Code:

echo "1" > /proc/sys/net/ipv4/tcp_syncookies

Code:

iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

iptables -A INPUT -p TCP ! --syn -m state --state NEW -j DROP

iptables -A INPUT -p TCP --dport 80 -m state --state NEW -j STOP-ABUSE

iptables -A INPUT -p TCP --dport 80 -m state --state NEW -j ACCEPT

iptables -A STOP-ABUSE -m recent --set

iptables -A STOP-ABUSE -m recent --update --seconds 10 --hitcount 3 -j DROP

helptonewbie · 12-20-2007, 09:46 AM

I like that, some great ideas there. Still if anyone has more information in any particular websites or books even that might be particularly helpful with things along these lines then please please post away!!

Regards