LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > LinuxQuestions.org > LQ Suggestions & Feedback
User Name
Password
LQ Suggestions & Feedback Do you have a suggestion for this site or an idea that will make the site better? This forum is for you.
PLEASE READ THIS FORUM - Information and status updates will also be posted here.

Notices


Reply
  Search this Thread
Old 02-18-2012, 01:25 PM   #1
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: distro hopper
Posts: 9,982

Rep: Reputation: 4569Reputation: 4569Reputation: 4569Reputation: 4569Reputation: 4569Reputation: 4569Reputation: 4569Reputation: 4569Reputation: 4569Reputation: 4569Reputation: 4569
What spambots are and how to deal with them


How Spammers Think
Quote:
My bots don't have feelings and could give a rats a$$ about your forum anyway. No reg, no prob, he'll just move on to the 999999999 other ...
-- a spammer

The forum that the quote is from is thankfully down. I recovered that quote from Google's search results page.

How Spammers Post

Spammers typically post via tools that everyone else call "spambots".

Before I continue, I want to make sure that everyone who reads this knows what a spambot is. Therefore, watch this video I previously posted, showing one in action, and the come back.

Video: a forum spambot in action

Now that you've seen a spambot, you know what it is. It's an automated program that opens accounts on multiple forums and posts to them.

Why do people run them? The main reasons go under the umbrella term of "SEO." "[Back]link building" to drive traffic to a particular website is very typical reason.

Who runs them? Well, it varies, but at least one spammer here left enough information in his posts for me to find his linkedin profile:

http://www.linkedin.com/in/ashutoshtiwari86

How Spammers Succeed

The usual purpose of running a spambot is to boost a website's visibility. This implies that the links being spammed need to be there for visitors to read. While the stereotypical spammer strategy is to simply post lots of links, the reality is that spammers also try to ensure that their links will not get taken down.

On the blackhat communities, you will find recommendations to limit your "link velocity", which is the rate at which you add links to a particular site. Blackhat urban legend has it that if Google detects that a site's link velocity exceeds a certain threshold, they will recognize that the site is being spammed and penalize it. It is, therefore, not uncommon for spammers to post on a schedule known by the self explanatory name of a "drip feed". I would make an analogy to r/K Selection Theory in biology.

One strategy popular spam-hiding strategy is automated text generation. Taking text somewhere on the web (the same forum thread, in some particularly stupid cases I've seen here) and then substituting words and and phrases with others is a common one. Here's another strategy that one bot used:

http://www.linuxquestions.org/questi...6/#post4560158

I have personally met one computer science master's students whose specialization was programming computers to generate original text. We can only expect this to become even more difficult to detect in the future. Furthermore, remember that all forums broadcast their topics via META tags (which are meant to be consumed by software). Therefore, we should not be surprised to see spambots posting with computer-generated text tailored to the topic of the forum.

That is not to say, of course, that a spambot would not be programmed to post the exact same text to as many forums as possible. That still happens, and will continue to happen forever.

How To Recognize Spam

When I report a spambot, my report typically includes several of the following:

The Payload

A spam payload, which is usually going to be in the sig (but is sometimes in the post body) is a very good indicator. When I see a sig containing a link to an off-topic business, I google the username, payload, and (sometimes) parts of the post. If it's been posted to multiple other forums at around the same time, it's a pretty good indicator.

If the payload link is to a doorway page or site that appears to have been written with article spinning software, then draw the obvious conclusion.

The Spambot Databases

Check if the username has an entry in a spambot database, such as:

The Off-Topic-ness

Posts by new users that are off-topic for the sub forum are an indicator. Note: I said "an indicator." Not "proof".

The Writing Style

I can tell the difference between a poster having trouble with English, and a writing style that's off for other reasons. I look for other posts by the same username (elsewhere) and try to analyze how the bot was programmed to write.

The Boasts

Several spam accounts here have openly identified themselves as SEO firms in their profile. Stupid, but it happens.

The Necroposts

I've regularly seen spambots programmed to search up the post with the most desired combination of keywords and then reply to that, regardless of how old it is. Therefore, necroposts are another indicator.

How To Stop Them

Subscribe to a filtering service

First, the number of spambots that get through make me wonder which spam filters LQ use. They're obviously not sufficient.

Akismet tends to be highly regarded, and it does offer a service for forums.

Remove Those That Get Through

You found a spambot account on your forum? And you're going to leave the account available for the spammer to use during the next iteration of his drip feed blast? Really? Ban the account and remove all of its posts now!

What Not To Do

Warn The Spammer

E.g. "You are advertising. You agreed not to do that when you registered."

The spammer won't be there to read the warning.

Removing the payload from the signature or post, and then leaving the account unbanned, falls into the "warning" category.

In the cases where you're not sure whether the account is a spam account, then I would recommend issuing a warning (possibly privately) that requires a response, removing the any commercial links from the signature and any of its posts, and then banning the account if no response is received within one week. The last step is important. It has also never been done.

Implement Spam-Friendly Policies

Quote:
Remember that our policies are more relaxed towards advertising in sigs. They don't show up in search engines, and members can disable them.
I would recommend narrowing this down so that it only applies to noncommercial advertising.

First, spammers aren't aware that their payloads here don't show up in search engines and can be disabled by logged-in members. Second, visibility to humans is still visibility.

These measures do not deter spammers (who won't program their bots to make a special case for LQ) and they do not reduce the amount of spam they send you. The policy that is based on these measures, however, has made spammers much more difficult to stop.

As it is, this policy allowed one spammer to spam for two months while multiple people (including me) spent the entire time trying to get the problem taken seriously.

Reference:
How A Spammer Slipped Through

Punish People For Reporting Spam

People who point out that certain accounts were almost certainly opened by spambots are not the evil ones. They should not be "warned" or otherwise admonished for doing so. They should, instead, be treated as the valuable, perceptive and responsible citizens that the are.

The problem of spambots being here is real. People who complain about it should be taken seriously.

Last edited by dugan; 03-04-2012 at 10:21 PM.
 
Old 02-18-2012, 01:47 PM   #2
jeremy
root
 
Registered: Jun 2000
Distribution: Debian, Red Hat, Slackware, Fedora, Ubuntu
Posts: 13,431

Rep: Reputation: 3988Reputation: 3988Reputation: 3988Reputation: 3988Reputation: 3988Reputation: 3988Reputation: 3988Reputation: 3988Reputation: 3988Reputation: 3988Reputation: 3988
FWIW, we use a combination of Akismet, Mollom, StopForumSpam (although username alone has an astonishingly high false positive rate; in fact both jeremy and dugan come back with positives), Bad Behavior and a couple others. When you're as large of a site as we are, spammers are *remarkably* dedicated, persistent and crafty. The amount of spam we catch in an automated way is astounding and it's always a matter of maintaining a balance between stopping the bad guys and not catching or inconveniencing the good guys (which is really a difficult balance to strike). We have made a conscious decision that we'd prefer to err on the side of letting a small amount of spam though which can be addressed manually as opposed to erring on the side that blocks legitimate members as collateral damage. That being said we're always looking for new and better ways to combat what we acknowledge is a serious issue. As always we appreciate the feedback and I'll address a couple points above in subsequent posts.

--jeremy
 
1 members found this post helpful.
Old 02-18-2012, 02:04 PM   #3
jeremy
root
 
Registered: Jun 2000
Distribution: Debian, Red Hat, Slackware, Fedora, Ubuntu
Posts: 13,431

Rep: Reputation: 3988Reputation: 3988Reputation: 3988Reputation: 3988Reputation: 3988Reputation: 3988Reputation: 3988Reputation: 3988Reputation: 3988Reputation: 3988Reputation: 3988
A couple comments on...

Us "Implementing Spam-Friendly Policies": I do not think we have. In fact, you admit that spammers don't read the rules so changing our .sig policy would not stop the amount of spam we get. It would however punish legitimate members whom we allow to link to commercial (and non-commercial) sites as part of participating at LQ. I think we can improve on some of the obvious cases (such as "SEO" links), however. Also note, we have now seen multiple cases of custom written bots specifically aimed at LQ.

Us "Punishing People For Reporting Spam": This should never happen. If it does, let me know. We're extremely grateful when members take the time to report and track spam and abuse. Your continued dedication is much appreciated, even if we don't always take the exact recommendations you've given.

Thanks for the continued feedback.

--jeremy
 
Old 02-18-2012, 02:36 PM   #4
H_TeXMeX_H
LQ Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292
I think spam is mostly under control here. There are bursts here and there from efficient spam bots, but they are handled quite quickly. Either way, I report ones that I see.
 
Old 02-18-2012, 03:10 PM   #5
k3lt01
Senior Member
 
Registered: Feb 2011
Location: Australia
Distribution: Debian Wheezy, Jessie, Sid/Experimental, playing with LFS.
Posts: 2,900

Rep: Reputation: 637Reputation: 637Reputation: 637Reputation: 637Reputation: 637Reputation: 637
We had a flurry of new members with single posts about a month ago and the typical theme of the posts was "I love Ubuntu (or Linux) its the best" and they all come from machines running Windows 7. There were a few of us who picked up on it while others thought, and some were adamant, they were legitimate.

I know LQ is huge and that it takes a varied approach to deal with this issue because of the unique nature of LQ itself. I personally think the overall job being done is excellent and that the occasional spurt of activity that does get through is not typical of what is seen in the background.
 
Old 03-04-2012, 09:28 PM   #6
cin_
Member
 
Registered: Dec 2010
Posts: 281

Rep: Reputation: 24
Spam As (Null) Issue

Damn, H_TeXMeX_H's sig is crazy applicable here...
Quote:
Those Who Sacrifice Liberty For Security Deserve Neither.
~ Benjamin Franklin
As for dugan's OP I found it an interesting and well documented tutorial. With little need to make it into anything more.
Remove everything from and below the header `How To Stop Them', change the title and, I'd say make it sticky under General or Newbie.


In response to k3lt01's post, might I make a shameless plug.


I did not know this sort of activity upset anyone. I know my post count is lower than the others who have weighed in, but more than that I think my unawareness means you guys at LQ are doing a great job.

Seriously, I use LQ because I like how open it is to members and non members alike, people who use it similarly and differently than me.
The clean layout, and the great collection of dedicated users.

Last edited by cin_; 03-04-2012 at 10:11 PM. Reason: gramm`err
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
sendmail and spambots pbaumgar Linux - Server 1 10-04-2009 06:26 AM
What's the deal with xv? rkelsen Slackware 9 12-18-2007 03:56 AM
LXer: The Fourth ‘Patent Deal’ was with Europe… and the Sixth Deal That Won’t be LXer Syndicated Linux News 0 10-24-2007 03:40 PM
Tweaking to prevent spambots at my site Whitestar General 5 07-19-2007 03:32 AM
Okay, here's the deal. . . musicnut01 Linux - Networking 5 06-29-2001 06:01 PM

LinuxQuestions.org > Forums > LinuxQuestions.org > LQ Suggestions & Feedback

All times are GMT -5. The time now is 03:46 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration