Referrer Spam is killing my site. over 1million hits in the last 5 days
Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Referrer Spam is killing my site. over 1million hits in the last 5 days
Since Friday I get inundated with hits on my site coming from sites that do not have a link to my site. I did some research and found out how to stop some of them. I am still getting thousands of hits that show like this on my logs.
<Limit GET HEAD POST>
order allow,deny
deny from 210.214.47.44
deny from "I just keep adding ip here"
allow from all
</LIMIT>
Also, I use this for spam referrers
Code:
SetEnvIfNoCase Referer "^http://(www.)?fillbest.com" spam_ref=1
SetEnvIfNoCase Referer "^http://(www.)?pharma-cy.info" spam_ref=1
SetEnvIfNoCase Referer "^http://(www.)?wikili.com" spam_ref=1
SetEnvIfNoCase Referer "^http://(www.)?forex-here.com" spam_ref=1
SetEnvIfNoCase Referer "^http://(www.)?I KEEP ADDING REFERESR HERE" spam_ref=1
# block all referrers that have spam_ref set
<FilesMatch "(.*)">
Order Allow,Deny
Allow from all
Deny from env=spam_ref
</FilesMatch>
In the last 5 days my logs recorded over one million hits. It is slowing my site down to a crwal and performance wise it's hurting.
I tried tired denying the ip addresses in my htaccess but they keep showing up. Not only to these ip addresses keep shoing up, but the sites I was able to block changed ip addresses and now I have to add the ip again.
Any ideas no how to block this traffic.
Last edited by win32sux; 05-14-2008 at 12:35 PM.
Reason: Removed QUOTE tags, added CODE tags.
Here is the portion of my .htaccess that I use to block referer spam, and I get practically no referer spam. Note that the first couple of lines of each section are there to determine if the referer is from my site; if it is, then there is no need to process the rest of the list. This speeds things up a lot.
This list was developed and tuned over a period of a couple of years; it seems to catch what I want to catch while not throwing out valid connects.
This is part 1 of the relevant portion of my .htaccess file; this site won't let me post the whole thing as one file. Part 2 is in the next post; just append it to part 1 to have the whole thing.
RewriteCond %{REQUEST_URI} !(/myunattendeddownloadscriptdirectory/index.php)
RewriteCond %{HTTPS} off
RewriteRule ^cart/(.*)$ https://mysecureshoppingcarturl/cart/$1 [R,L]
I don't have a ecommerce or cart on my site and don't have a downloads section. So, I don't need this. If I did, I just replace the Myunattendeddowloads...etc with my own path?
Also, My .htaccess file is about 350kb to big since I've been adding IP addresses to block. Will it be a good idea to just use your .htaccess? Or is it better to just include your spam rules into my file?
One more thing, I've been reading about a way to check and see if the referer has an acutal link on their site before allowing access to my site but I can't find how to set that up. Do you now where can I get information about that?
I don't know much about htaccess and I am learning the as I go. This past week has been avery humbling experience.
Thank for your help.
Last edited by win32sux; 05-14-2008 at 12:37 PM.
Reason: Removed QUOTE tags, added CODE tags.
xchido, in your future posts please refrain from using QUOTE tags when posting this sort of content. It messes-up the page layout and it makes it difficult to read. Use CODE tags instead - they are designed precisely for this. I've edited your previous posts.
RewriteCond %{REQUEST_URI} !(/myunattendeddownloadscriptdirectory/index.php)
RewriteCond %{HTTPS} off
RewriteRule ^cart/(.*)$ https://mysecureshoppingcarturl/cart/$1 [R,L]
I don't have a ecommerce or cart on my site and don't have a downloads section. So, I don't need this. If I did, I just replace the Myunattendeddowloads...etc with my own path?
Also, My .htaccess file is about 350kb to big since I've been adding IP addresses to block. Will it be a good idea to just use your .htaccess? Or is it better to just include your spam rules into my file?
One more thing, I've been reading about a way to check and see if the referer has an acutal link on their site before allowing access to my site but I can't find how to set that up. Do you now where can I get information about that?
I don't know much about htaccess and I am learning the as I go. This past week has been avery humbling experience.
Thank for your help.
My site permits unattended downloads after purchase. The secure server doesn't permit downloads, so I have to redirect purchasers to an http: rather than an https: for the actual download. I do this by giving them a download link after they purchase.
The code that actually handles the download is in a subdirectory under my shopping cart. So this specific rule is intended to force all visitors to only access the shopping cart using https: UNLESS they are in the unattended download subdirectory, which they can only access using http:
So you don't need that rule at all.
I only block specific IP addresses or IP ranges when I identify a cracker, or someone who tries to use my forms (which are hardened) for open relay or SQL injection attacks.
I don't think Apache can check a referer for validity through .htaccess but I wouldn't do it anyway; slows everything down. Just nuke the ones that don't belong and forget about it.
My robots.txt file tells Google (actually all search engines) to stay away from my shockwave flash files. But googlebot refuses to obey that rule. So I enforce it.
xchido, in your future posts please refrain from using QUOTE tags when posting this sort of content. It messes-up the page layout and it makes it difficult to read. Use CODE tags instead - they are designed precisely for this. I've edited your previous posts.
Well, oops, I didn’t realize I was using the wrong tag. I just wanted it to emphasize the text. I know better now to stick with just regular tags.
I went ahead and replaced my htaccess file with yours and added my php flags that I need. It all seems to be working perfectly. Site traffic for the last few hours has returned to normal. Still have to check the access logs and see what it says there.
I don’t have flash files so that rewrite rule does not affect me. I left it in anyways.
Can you direct me to a link where I can read about how to block traffic from entire countries. I was doing that with my old htaccess but the file got over 300KB. I tried different sites that provide the ip ranges but most of them are not the same. Is there an official list that shows the countries ip ranges? Aldo, do they stay set or they change with time? I had problems with specific countries hackers trying to and actually hacking my site that I would like to ban.
How big is too big for an htaccess file?
I cannot begin to tell you how much I appreciate your help. I really appreciate it.
I use the database IP2Nation (google for it) and I execute some code on a per-page basis to block entire nations that I do not want accessing particular parts of my site (especially the message boards, since most of the spammers are coming from outside the US. I use phpbb for my message board and I have altered it to enhance security. I won't detail the enhancements because if known they could be compromised.
However here is a PHP code fragment I use to query the IP2Nation database and block those I don't want to have access:
Code:
$sql = 'SELECT
country
FROM
ip2nation
WHERE
ip < INET_ATON("'.$_SERVER['REMOTE_ADDR'].'")
ORDER BY
ip DESC
LIMIT 0,1';
if( !($result = $dbnation->sql_query($sql)) )
{
message_die(CRITICAL_ERROR, "Could not query nations database", "", __LINE__, __FILE__, $sql);
} else {
if ($row = $dbnation->sql_fetchrow($result)) {
$warn1="Due to major spam problems, ";
$warn2=" are banned from the message boards. If you are a legimate visitor, contact Just So Software via our contact form and we can arrange to whitelist you.";
switch ($row[0]) {
case 'ru':
// Block russians
die($warn1."Russian visitors".$warn2);
case 'cn':
// block chinese
die($warn1."Chinese visitors".$warn2);
case 'nl':
// block netherlands
die($warn1."visitors from the Netherlands".$warn2);
case 'br':
// block brazil
die($warn1."visitors from Brazil".$warn2);
case 'be':
// block belgium
die($warn1."visitors from Belgium".$warn2);
case 'kr':
// Block koreans
die($warn1."Korean visitors".$warn2);
case 'jp':
// Block japanese
die($warn1."visitors from Japan".$warn2);
case 'ua':
// Block Ukraine
die($warn1."visitors from Ukraine".$warn2);
case 'de':
// Block Germany
die($warn1."visitors from Germany".$warn2);
case 'cz':
// Block Czech Republic
die($warn1."visitors from the Czech Republic".$warn2);
case 'es':
// Block Spain
die($warn1."visitors from Spain".$warn2);
case 'ar':
// Block Argentina
die($warn1."visitors from Argentina".$warn2);
case 'in':
// Block India
die($warn1."visitors from India".$warn2);
case 'gr':
// Block Greece
die($warn1."visitors from Greece".$warn2);
case 'bg':
// Block Bulgaria
die($warn1."visitors from Bulgaria".$warn2);
case 'ro':
// Block Romania
die($warn1."visitors from Romania".$warn2);
case 'co':
// Block Columbia
die($warn1."visitors from Columbia".$warn2);
case 'tw':
// Block Taiwan
die($warn1."visitors from Taiwan".$warn2);
case 'pl':
// Block Poland
die($warn1."visitors from poland".$warn2);
case 'se':
// Block Sweden
die($warn1."visitors from Sweden".$warn2);
default:
}
} else {
message_die(CRITICAL_ERROR, "IP range not found", "", __LINE__, __FILE__, $sql);
}
Actually, I think there are only a handful of people out there who are responsible for all of the referer spam. I no longer even have the attempts appearing on my sites; I think they gave up and removed my domains from their spam lists. After all, it does take time, and they only get 403 error back so if they can't get through why bother.
LOL. I just checked. So far this month, my site has had a total of 28 (that is twenty eight) 403 (forbidden) errors.
When I was fighting this battle to get the referer spam blocked, I was getting several thousand 403s a day. They finally gave up on me.
Actually, the spam abruptly stopped one day last july. I had thought that someone had been put out of business, but I still show zero referer spam attempts and since you say you are getting hammered, it must mean my domains were just removed from someone's list.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
Referrer Spam is killing my site. over 1million hits in the last 5 days
