LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices

Reply
 
Search this Thread
Old 12-30-2013, 09:58 AM   #1
andrew777
LQ Newbie
 
Registered: Feb 2009
Posts: 14

Rep: Reputation: 0
Restrict access to x pages by IP address to prevent abuse and harvesting


We have a Linux (GNU/Linux 3.6.6) web server with hundreds of sites and some of these sites have hundreds of millions of html pages.

As you can imagine we have occasional abuse problems and harvesting problems. So we are looking for a way to restrict access to some of these domains (web sites) by limiting the access to about 500 pages per 24 hour period per IP address. This will ensure that the people consulting the sites are consulting them for valid reasons and are not harvesting the data.

Can someone please direct me to a document or html page that explains how to restrict access to a web site or domain name on a Linux server in the manner explained above.

Thank you
 
Old 12-31-2013, 03:07 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,003
Blog Entries: 54

Rep: Reputation: 2756Reputation: 2756Reputation: 2756Reputation: 2756Reputation: 2756Reputation: 2756Reputation: 2756Reputation: 2756Reputation: 2756Reputation: 2756Reputation: 2756
Quote:
Originally Posted by andrew777 View Post
(..) This will ensure that the people consulting the sites are consulting them for valid reasons and are not harvesting the data.
No it will not. What it does is limit access. You're reading too much into it. Besides web sites have different characteristics so what would be "good" for one web site would be absolutely bad for another, not only from the point of view of client usage but also with respect to how one should combat symptoms. If you're only responsible for hosting those web sites then you should work with the web site owners to have them implement measures on the application level (UA filtering, captcha, anti-leech, caching, maybe disabling some features, etc, etc) and separately from what you should implement as a hosting company on the layers below (mod_security, mod_evasive, mod_bandwidth, reverse proxy, caching proxy, firewall, bottleneck analysis, resource migration / separation, etc, etc).
 
Old 01-01-2014, 12:28 AM   #3
andrew777
LQ Newbie
 
Registered: Feb 2009
Posts: 14

Original Poster
Rep: Reputation: 0
unSpawn...

With all due respect, we have made the decision to implement such a restriction based on our clientèle. Visitors to the sites in question do not need to access more than a few hundred pages in a single 24 hour period to get what they need. This has been established and we are now seeking a solution to our problem.

We are not a hosting company. We host only our own sites and we have several which are huge (over a hundred million static html pages) and are regularly targeted by hackers and harvesters. As a result we have taken the decision to implement some sort of IP access restriction unless we can find a better solution.

This having been said, could you kindly recommend something that would help solve our harvesting issue.

Thanks
 
Old 01-01-2014, 08:15 AM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,003
Blog Entries: 54

Rep: Reputation: 2756Reputation: 2756Reputation: 2756Reputation: 2756Reputation: 2756Reputation: 2756Reputation: 2756Reputation: 2756Reputation: 2756Reputation: 2756Reputation: 2756
See the iptables limit, recent, hashlimit and connlimit extensions. Note implementing only rate limiting access may suppress "your clientèle" but it won't help with "occasional abuse problems and harvesting problems" or "targeted by hackers" as each requires a different approach but I already hinted at that in my reply.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] IPTABLES: Restrict Internet access based on time of day and MAC address ScottSmith Linux - Security 7 02-09-2010 02:25 AM
Limiting access to web-pages by IP-address with PHP alienDog Programming 3 12-16-2008 08:10 AM
LXer: How to prevent Linux man pages from clearing after you quit reading LXer Syndicated Linux News 0 05-09-2008 11:40 PM
Samba: How to restrict access to server via MAC-address? johnny1959 Linux - Networking 5 03-27-2008 04:43 AM
restrict server access by mac address? stinkpot Linux - Software 4 11-22-2005 07:05 AM


All times are GMT -5. The time now is 06:40 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration