A couple weeks ago, a spammer used my server to send out 1000s of emails.

I found this out by receiving an email from one of my clients alerting me that AOL wasn't accepting emails from his address. As it turns out, this was because AOL added one of our IP addresses to its spam list.

At first I thought one of our clients was breaking the contract and using our server for spam but the Apache logs told me a different story. The error logs showed a ridiculous amount of 404 errors as the spammers had some robot search every possible directory/filename combo for for script that could be used to send email. They found an old Perl script that I completely forgot about and sent 100s of thousands of emails from it. I have the actual email that they sent but I need more information to track these guys down.

I've never sued anyone in my life but I would love to nail these guys. I'm sure that they are clever enough to cover their tracks but I want to throughly examine every possible trace. These spammers are the scum of the internet. I can't even count the hours I've spend deleting spam, filtering it, etc. Now they broke into my server.

Anyone here good at examining log files that can offer me some advice?

Thanks.

Save all the logs immediately to some form of read-only backup media, such as CD-R. The ideal thing is to actually make a disk image that saves all the original file and directory attributes.

Now, your Apache logs should have the IP address of the attacker who was trying to gain access to your scripts, right? That's where you start. Also there were probably several different messages that got sent (i.e. different campaigns), so if you could get a copy of each unique message, that would be great. Often the source code from those messages is very useful (showing obfuscated HTML links, hidden web bugs, etc).

If you're willing to share your logs, that would be fantastic. I could do so under an NDA if you wish. I happen to work for an e-mail security company and I run the e-mail security site www.smtps.net/email-sec/ . Being able to determine what kind of tools they were using to exploit your site would be very helpful, since these same people will definitely be targeting other sites as well.

Thanks for the reply.

It's been a couple weeks since this happened but we save all the logs so I should be able to find the relevant ones. It sounds like we may be able to help each other out with this so I'm going to gather the data and email it to you. Is there anything beside the Apache logs that may be helpful? It's RH9 with cpanel doing most of the administration.


I browsed your post history and it seems like you're one of the good guys so no worries about the NDA.

The Apache logs certainly, and also your maillog. Your PERL script probably invoked Sendmail, right? The maillog should have information about all the messages sent through Sendmail. Obviously this only needs to be from the relevant time period. Depending how your syslog was/is setup, there may or may not be useful information in the messages (/var/log/messages) log.

On your own you should probably review your other logs, such as the auth log to make sure the spammers didn't actually compromise your machine.

Chort, would you be willing to keep us informed wrt your approach to solving this? Simple chronological bullet list could do. You know I'm interested.

Certainly, with permission. If anything of interest is found, and if it's OK with kidestranged, I'll publish the finished research on my website. Of course this could turn out to be straight-forward and relatively uninteresting, or a lot of dead-ends because of the elapsed time. In any case it should serve as a warning that spammers are extremely resourceful and they can damage the repuation of organizations that aren't supremely vigilant.

I don't understand the point of the spam if you cannot contact them from the email they sent. How can they make money from it ? Or are they just vandals ?
I have never replied to spam nor tried to buy anything from a spam email but I thought it was a dishonest way of making money.

Basically they sell the use of a list of names. They make money from the people who are offering to sell you Viagra or something by claiming to be able to send X millions of emails. They short and dirty is people with a product pay a "mail" service (i.e.spammer) a set figure based on number of emails sent. The product owners in turn hope for a 1% return for purchase.

If you can contact the product owners to buy products sure would it not be possible for the police to find out who they paid to do the spamming ?

Not really. Usually the people advertising the product claim they had no idea that the people they hired to "market" their product were using such underhanded methods, and surprise those nasty spammer disappeared, no idea how to contact them! Of course it's all a big lie, but it usually works.

Chort,

I've sent you an email with a zip of the logs. Thanks for your help.

Just out of curiosity, was there anything of interest found? I have to admit that I'd be interested in finding out what the conclusion was for this thread. -- J.W.

Alright guys - I'll keep you in the loop

After looking over my logs, Chort replied to me via email with the following response and said that he'll try to make a post explaining how he came up with the following conclusions:


Sorry for taking so long with the response. When I got your message I had landed smack in the middle of two other projects and I was backlogged. There isn't a huge amount of info in the logs provided, obviously just the originating source IP (which is a school in Korea).
Most likely an automated exploit (perhaps an e-mail worm) compromised some of the computers at the school and the spammer logged in remotely to use that machine (machines?) to run scripts looking for other open machines to send spam through.

The casino being advertised is registered to a company in Norway and from the name of it, they probably run at least several on-line casinos (although their registrar's database was not helpful in finding any others).

The random link at the end of the URL could be significant for several
reasons:
a) It identifies the "advertiser" ID, so if they're paying by referrals they'll know who to pay
b) It could just identify which advertising "campaign" generated the hit
c) It might just be a redirect back to their main page that has a long random string to throw off anti-spam software that is based on matching URLs

Sorry I couldn't be more helpful than that, but it's both simple (in
concept) and stale (trail gone cold), so there isn't a whole lot to deal with.

The moral of the story is: Make sure you don't have any web scripts that could send mail to an arbitrary recipient.



***********************

I'll keep you posted if the on-line casinos help me track the spammers down but I'm sure they know what the hired "advertisers" are doing and don't really care as long as their ad is delivered.