LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Security (https://www.linuxquestions.org/questions/linux-security-4/)
-   -   Restrict access to x pages by IP address to prevent abuse and harvesting (https://www.linuxquestions.org/questions/linux-security-4/restrict-access-to-x-pages-by-ip-address-to-prevent-abuse-and-harvesting-4175489586/)

andrew777 12-30-2013 09:58 AM

Restrict access to x pages by IP address to prevent abuse and harvesting
 
We have a Linux (GNU/Linux 3.6.6) web server with hundreds of sites and some of these sites have hundreds of millions of html pages.

As you can imagine we have occasional abuse problems and harvesting problems. So we are looking for a way to restrict access to some of these domains (web sites) by limiting the access to about 500 pages per 24 hour period per IP address. This will ensure that the people consulting the sites are consulting them for valid reasons and are not harvesting the data.

Can someone please direct me to a document or html page that explains how to restrict access to a web site or domain name on a Linux server in the manner explained above.

Thank you

unSpawn 12-31-2013 03:07 AM

Quote:

Originally Posted by andrew777 (Post 5089067)
(..) This will ensure that the people consulting the sites are consulting them for valid reasons and are not harvesting the data.

No it will not. What it does is limit access. You're reading too much into it. Besides web sites have different characteristics so what would be "good" for one web site would be absolutely bad for another, not only from the point of view of client usage but also with respect to how one should combat symptoms. If you're only responsible for hosting those web sites then you should work with the web site owners to have them implement measures on the application level (UA filtering, captcha, anti-leech, caching, maybe disabling some features, etc, etc) and separately from what you should implement as a hosting company on the layers below (mod_security, mod_evasive, mod_bandwidth, reverse proxy, caching proxy, firewall, bottleneck analysis, resource migration / separation, etc, etc).

andrew777 01-01-2014 12:28 AM

unSpawn...

With all due respect, we have made the decision to implement such a restriction based on our clientèle. Visitors to the sites in question do not need to access more than a few hundred pages in a single 24 hour period to get what they need. This has been established and we are now seeking a solution to our problem.

We are not a hosting company. We host only our own sites and we have several which are huge (over a hundred million static html pages) and are regularly targeted by hackers and harvesters. As a result we have taken the decision to implement some sort of IP access restriction unless we can find a better solution.

This having been said, could you kindly recommend something that would help solve our harvesting issue.

Thanks

unSpawn 01-01-2014 08:15 AM

See the iptables limit, recent, hashlimit and connlimit extensions. Note implementing only rate limiting access may suppress "your clientèle" but it won't help with "occasional abuse problems and harvesting problems" or "targeted by hackers" as each requires a different approach but I already hinted at that in my reply.


All times are GMT -5. The time now is 03:25 AM.