Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Hi all,
I've been on the mod_security website and can't say the doc's are great, i'm trying to work out if the following is possible:-
I can see that some parts are of the following like redirecting to different URL or blocking completely etc....
We have requests coming into our web servers and we get some annoying people on occasions screen scrapping us, which is basically requesting the same webpage or lots of different webpages in a short space of time, of which each request is taking cpu power and its quite obvious they are not interested in our products so we'd rather just stop them from being so annoying. We know its screen scrapping because we already have monitors that detect this behavior but currently manually just stop the ip address running the scrap we'd like to make this automatic. Therefore if a host (ipaddress) is constantly trying to access the same page over and over or various pages in a short space of time then we'd like to redirect to a holding page and then if it continues to drop their packets completely for a specified amount of time of which it would hopefully work like ip tables and have a timer in which will restart again everytime another request is sent and only after the period of the timer has waited without any requests from that host in question then to allow that host to send requests again. Now i dont know if mod_security is able to do that or perhaps if it might be better to use snort, in which case i'd like to know if there is any case study type things on the internet regarding this problem, i've been looking but not really found anything.
We have requests coming into our web servers and we get some annoying people on occasions screen scrapping us, which is basically requesting the same webpage or lots of different webpages in a short space of time, of which each request is taking cpu power and its quite obvious they are not interested in our products so we'd rather just stop them from being so annoying.
The way I understand it, screen scraping is the action of having a computer read output which was designed to be read by a human. Not sure how that applies here, as what you describe sounds like an attempt to carry-out a denial of service attack. That said, I don't know how to address this with mod_security, but while you wait for an answer from someone who does, I thought I might suggest a quick temporary workaround for you:
Use iptables to prevent individual IPs from starting more than a specified number of HTTP connections in a specified time-frame. If you calculate it right, you should be able to reduce the attacker's connection rate to something only a tad higher than that which you see from your friendly users. Like, for example, let's say that you determine the attacker is starting about eight (or whatever) HTTP connections every ten seconds, while none of your friendly users ever start more than three every ten seconds. In such a case, you could implement a set of rules like this (above the actual ACCEPT rule) on the server:
Code:
iptables -A INPUT -p TCP --dport 80 -m state --state NEW -j STOP-ABUSE
iptables -A STOP-ABUSE -m recent --set
iptables -A STOP-ABUSE -m recent --update --seconds 10 --hitcount 3 -j DROP
Once again, this is only intended as an anti-DoS workaround while you figure out your per-page technique.
hello,
Thanks for the reply i should have perhaps said i've already looked at that, but thanks anyway it is useful for anyone that doesn't know it but looks at the thread.
Thanks for the info
Yes it is sort of DoS at the same time for different reasons, not going to explain to much because its never good to give to much away in these forums could shoot yourself in the foot.
Last edited by helptonewbie; 12-19-2007 at 10:24 AM.
hello,
Thanks for the reply i should have perhaps said i've already looked at that, but thanks anyway it is useful for anyone that doesn't know it but looks at the thread.
Thanks for the info
It's actually great that you have this going right now, as it drastically reduces the urgency of the mod_security solution - which was sort of the point of my post. And yes, you should have mentioned what temporary countermeasures you were currently using. It's all good. I'm sure someone familiar with mod_security will be posting soon. Hang in there.
The other question i've always been curious about is i know its possible to re-write packets that are sent to servers just like NAT does basically but i know people also do it maliciously, so if the source host was re-written and then with the iptables example above, those packets sent to port 80 if reaching the limit, then i'm guessing the server would then block that source address for the specified time therefore creating a DoS on yourself, would that be the same if using snort and mod_security???? i'm guessing it would unless they could detect the packet has been tampered with which isn't possible unless using ip/sec i believe? Therefore this is a draw back to anything along these lines??
I believe that is right though isn't it, if you re-wrote the source host in the ip packet and then sent lots to the server and then it would block that source host but only for the time period even though its not from that actual source??
I assume you are writing about someone spidering your site that is dynamically generated and that is causing you excessive CPU load?
You should be able to do this with ModSecurity using a SESSION or IP collection.
Basically, you want to keep trac of how many hits from a given SESSION/IP and then block (redirect, whatever) based on too many pages requested in a given timeframe.
Something like this will allow only 10 request in a minute from a given IP:
You will probably want to augment this with other checks by chaining them togeather:
# Block > 10req/minute for /cgi-bin and not in the IP whitelist
SecRule REQUEST_FILENAME "^/cgi-bin" "phase:1,log,auditlog,t:none,t:normalisePath,deny,status:403,msg:'Blocking screen scraper',chain"
SecRule REMOTE_ADDR "!^(?:127\.0\.0\.1|1\.2\.3\.4)$" "t:none,chain"
SecAction "nolog,initcol:IP=%{REMOTE_ADDR},setvar:IP.pagecount=+1,expirevar:IP.pagecount=60,chain"
SecRule IP:PAGECOUNT "@gt 10" "t:none"
This is pretty simplified and you could do a lot more, but hopefully you get the idea. A warning, though, that IP != user and you could end up blocking a proxy server's IP address with this (ie AOL, NetZero, etc), so probably best to stay away from IPs and use some sort of session or other more unique value (SESSION collection vs IP).
To get into a more detailed discussion on implementation, send a note to the ModSecurity user list (see contact page on website)
The other question i've always been curious about is i know its possible to re-write packets that are sent to servers just like NAT does basically but i know people also do it maliciously, so if the source host was re-written and then with the iptables example above, those packets sent to port 80 if reaching the limit, then i'm guessing the server would then block that source address for the specified time therefore creating a DoS on yourself, would that be the same if using snort and mod_security???? i'm guessing it would unless they could detect the packet has been tampered with which isn't possible unless using ip/sec i believe? Therefore this is a draw back to anything along these lines??
I believe that is right though isn't it, if you re-wrote the source host in the ip packet and then sent lots to the server and then it would block that source host but only for the time period even though its not from that actual source??
One thing you can do to stop spoofed packets from entering the STOP-ABUSE chain (and possibly causing a denial of service for friendly users) is to activate TCP SYN cookies in order to discard any spoofed SYN packets. Then you follow this up with an iptables rule to filter any packets of state NEW which aren't proper SYN packets. This should in effect make sure that any packets getting sent to STOP-ABUSE have been checked for spoofing. Example:
Code:
echo "1" > /proc/sys/net/ipv4/tcp_syncookies
Code:
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -p TCP ! --syn -m state --state NEW -j DROP
iptables -A INPUT -p TCP --dport 80 -m state --state NEW -j STOP-ABUSE
iptables -A INPUT -p TCP --dport 80 -m state --state NEW -j ACCEPT
iptables -A STOP-ABUSE -m recent --set
iptables -A STOP-ABUSE -m recent --update --seconds 10 --hitcount 3 -j DROP
I like that, some great ideas there. Still if anyone has more information in any particular websites or books even that might be particularly helpful with things along these lines then please please post away!!
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.