Help needed with trying to identify spammer

countrydj · 11-09-2011, 04:38 PM

Hi Guys...

I have been notified that a spammer is sending mail through my server.
I need help in finding out how this is happening.

As an example, I found this email in the root mailbox, returned mail:

Quote:

Return-Path: <MAILER-DAEMON@blackvelvet.gvl99.co.uk>
Date: Wed, 9 Nov 2011 12:19:40 GMT
To: <root@blackvelvet.gvl99.co.uk>
Subject: Returned mail: see transcript for details
Auto-Submitted: auto-generated (failure)

This is a MIME-encapsulated message

--pA9CJe9d024580.1320841180/blackvelvet.gvl99.co.uk

The original message was received at Wed, 9 Nov 2011 12:14:33 GMT
from redvelvet.gvl99.co.uk [127.0.0.1]

----- The following addresses had permanent fatal errors -----
<tasobello@hotmail.it>
(reason: 550 Requested action not taken: mailbox unavailable)

----- Transcript of session follows -----
451 4.4.1 reply: read error from mx1.hotmail.com.
... while talking to mx3.hotmail.com.:
>>> DATA
<<< 550 Requested action not taken: mailbox unavailable
550 5.1.1 <tasobello@hotmail.it>... User unknown
<<< 503 Need Rcpt command.

--pA9CJe9d024580.1320841180/blackvelvet.gvl99.co.uk
Content-Type: message/delivery-status

Reporting-MTA: dns; blackvelvet.gvl99.co.uk
Received-From-MTA: DNS; redvelvet.gvl99.co.uk
Arrival-Date: Wed, 9 Nov 2011 12:14:33 GMT

Final-Recipient: RFC822; tasobello@hotmail.it
Action: failed
Status: 5.1.1
Remote-MTA: DNS; mx3.hotmail.com
Diagnostic-Code: SMTP; 550 Requested action not taken: mailbox unavailable
Last-Attempt-Date: Wed, 9 Nov 2011 12:19:39 GMT

--pA9CJe9d024580.1320841180/blackvelvet.gvl99.co.uk
Content-Type: message/rfc822

Return-Path: <root@blackvelvet.gvl99.co.uk>
Received: from blackvelvet.gvl99.co.uk (redvelvet.gvl99.co.uk [127.0.0.1])
by blackvelvet.gvl99.co.uk (8.13.8/8.13.8) with ESMTP id pA9CEU9d024577
for <tasobello@hotmail.it>; Wed, 9 Nov 2011 12:14:33 GMT
Received: (from root@localhost)
by blackvelvet.gvl99.co.uk (8.13.8/8.13.8/Submit) id pA9CET3k024573;
Wed, 9 Nov 2011 12:14:29 GMT
Date: Wed, 9 Nov 2011 12:14:29 GMT
Message-Id: <201111091214.pA9CET3k024573@blackvelvet.gvl99.co.uk>
From: Promotions Department <promotions@places-cazino.info>
To: tasobello@hotmail.it
Subject: Get Free 1000 EURO to Play!
MIME-Version: 1.0
Content-Type: multipart/related;
boundary="=_a9713fc79164888ab50b927c8b0c2650"

--=_a9713fc79164888ab50b927c8b0c2650
Content-Type: multipart/alternative;
boundary="=_d8df151bd01e463b26b81cbf2741e6bb"

--=_d8df151bd01e463b26b81cbf2741e6bb
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 7bit

I looked up the corresponding entry in the maillog and found this:

Quote:

Nov 9 12:19:40 blackvelvet sendmail[25823]: pA9CJd8i025823: from=root, size=140075, class=0, nrcpts=1, msgid=<201111091219.pA9CJd8i025823@blackvelvet.gvl99.co.uk>, relay=root@localhost
Nov 9 12:19:40 blackvelvet sendmail[25823]: pA9CJd8i025823: to=jonny_be_good30@hotmail.com, ctladdr=root (0/0), delay=00:00:01, xdelay=00:00:00, mailer=relay, pri=170075, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection refused by [127.0.0.1]
Nov 9 12:19:40 blackvelvet sendmail[24580]: pA9CEU9d024577: pA9CJe9d024580: DSN: User unknown

I tend to think that this abuse may be via a mail script hosted on my server. There must be about 20 - 30 of them.

Is there anybody that can help me ???

Regards,

John C

frankbell · 11-09-2011, 08:10 PM

Is it possible you have an open relay?

http://www.cyberciti.biz/tips/test-m...pen-relay.html

countrydj · 11-10-2011, 03:59 AM

frankbell said:

Quote:

Is it possible you have an open relay?

Hi frankbell..

Thanks for taking the time to help.
NO. The server is NOT open relay.
Checked by: http://www.dnsgoodies.com/
That is why I tend to think that it may be originating from a mail() script.

Regards,

Noway2 · 11-10-2011, 09:47 AM

I agree that based upon what you have posted so far, there is a good possibility that this is originating on your system. You need to determine where this is happening from. One of the first questions I would ask is, what services are you running on this server? Some services, such as Apache, have history of this type of exploit. Also, what content manager, server manager and monitoring software are you running (e.g. plesk, webmin, etc). Also, what distribution and rev level are you running?

If you suspect that you have been compromised in some fashion, the first thing to do is isolate the machine either by disconnecting the network cable or putting up the firewall to allow SSH connections only from a trusted source. This will avoid disturbing the crime scene and buy you time to investigate. Here is a link to the CERT intruder detection checklist. It outlines the steps that you need to take: http://www.scribd.com/doc/6398/Cert-...tion-Checklist. Basically, you will need to scour your system for hidden and strange files and scripts as well as determine what, if anything, has been modified. You will also want to be careful to capture a copy of your log files as normally there is lots of noise created by an intrusion attempt.

One of the things that you will need to do is to identify a suspected time frame of the compromise. Using a tool such as logwatch will help you in analyzing your logs.

I would recommend the following set of commands to begin gathering information. You might want to consider capturing them into a big log file for analysis:

Code:

/bin/ps acxfwwwe 2>&1;
/usr/sbin/lsof -Pwln 2>&1;
/bin/ls -al /var/spool/cron 2>&1;
/usr/bin/lastlog 2>&1;
/usr/bin/last 2>&1;
/usr/bin/who -a 2>&1

countrydj · 11-12-2011, 05:32 AM

Hi Noway2...
Thank you very much indeed for taking the time and trouble to try and help me.

I don't pretend to understand all the test commands that you have suggested, but this one:

Code:

/usr/bin/last 2>&1;

threw up a strange entry:

Quote:

altys-of pts/0 193.61.254.65 Thu Oct 13 08:56 - 08:56 (00:00)

Although this domain is hosted on my server, he does not have access other that ftp access.
I have a file /bin/ftponly and has no shell access.
All the other entries I can acount for.

However, having said that, I have only just (2 or 3 days ago) stopped 'root' login. I now login via my User acount and then su into root.

As for all the other commands, I don't know what to do with them.
It would appear that:

Code:

/usr/sbin/lsof -Pwln 2>&1;

Gives me a list of all the log files for all the domains on the server.
However, I don't know what to look for and what to do.

My server is running CentOS release 5.5 (Final). The installation was done from Centos 5.5 x86_64
CDs.

These are some of the services that are running, taken from the command:

Code:

/bin/ps acxfwwwe 2>&1;

and I don't know what they are:

Code:

    2 ?        S<     0:00 migration/0
    3 ?        SN     0:00 ksoftirqd/0
    4 ?        S<     0:00 watchdog/0
    5 ?        S<     0:00 migration/1
    6 ?        SN     0:00 ksoftirqd/1
    7 ?        S<     0:00 watchdog/1
    8 ?        S<     0:00 events/0
    9 ?        S<     0:00 events/1
   10 ?        S<     0:00 khelper
   79 ?        S<     0:00 kthread
  549 ?        S<s    0:00 udevd
 2080 ?        S<sl   0:03 auditd
 2112 ?        Ss     0:14 syslogd
 2115 ?        Ss     0:00 klogd
 2238 ?        Ss     0:00 portmap
 2277 ?        Ss     0:00 rpc.statd
 2309 ?        Ss     0:00 rpc.idmapd
 2332 ?        Ss     0:00 dbus-daemon
 2345 ?        Ss     0:00 hcid
 2349 ?        Ss     0:00 sdpd
 2374 ?        S<     0:00 krfcommd
 2418 ?        Ssl    0:00 pcscd
 2432 ?        Ss     0:00 acpid
 2445 ?        Ss     0:00 hald
 2486 ?        Ss     0:00 hidd

Any advice on these would be appreciated.

I really do appreciate any help and advice that is given...

countrydj · 11-13-2011, 08:28 AM

I believe that my server has been compromised.
I an getting a lot of email activity shown in /var/log/maillog.
This is a typical example:

Code:

Nov 13 14:24:00 blackvelvet sm-msp-queue[21871]: pADDkOl9020649: to="David Greaves" <snapz1@dgreaves.orangehome.co.uk>, ctladdr=root (0/0), delay=00:37:36, xdelay=00:00:00, mailer=relay, pri=252926, relay=[127.0.0.1], dsn=4.0.0, stat=Deferred: Connection refused by [127.0.0.1]

Does this mean that this item is NOT sent, but just queued for retrying ???

leslie_jones · 11-13-2011, 08:39 AM

Quote:

Originally Posted by countrydj

I believe that my server has been compromised.
I an getting a lot of email activity shown in /var/log/maillog.
This is a typical example:

Code:

Nov 13 14:24:00 blackvelvet sm-msp-queue[21871]: pADDkOl9020649: to="David Greaves" <snapz1@dgreaves.orangehome.co.uk>, ctladdr=root (0/0), delay=00:37:36, xdelay=00:00:00, mailer=relay, pri=252926, relay=[127.0.0.1], dsn=4.0.0, stat=Deferred: Connection refused by [127.0.0.1]

Does this mean that this item is NOT sent, but just queued for retrying ???

If my memory serves me correct it means just that, the connection is refused, but I'd want to find what is trying to fire that because it won't stop. Is this a server hosting web pages for clients and what is the MTA? (Exim/Qmail/Sendmail) ?

countrydj · 11-13-2011, 08:59 AM

Hi leslie_jones..

Thanks for your response.

Quote:

Is this a server hosting web pages for clients and what is the MTA?

Yes it is.
This has been going on for a few days now and I can't figure out where it is comming from.
I am now starting to believe that my server has been compromised and should be trashed.
I use sendmail as my MTA.
It is the only MTA that I have ever used although people tell me to change to Exim. What do you think ???

I noticed this morning that my /etc/passwd file had some strange entries:

Code:

zarafa:x:101:104:Zarafa Service Account:/var/lib/zarafa:/sbin/nologin
xfs:x:43:43:X Font Server:/etc/X11/fs:/sbin/nologin         
nx:x:102:105::/var/lib/nxserver/home:/usr/bin/nxserver   
snort:x:551:535:Snort:/var/log/snort:/bin/false

btw:

Code:

:/var/lib/zarafa
::/var/lib/nxserver/home
:/var/log/snort

Do NOT exist !!! Very strange ???

I also found a login that sould not have happened:

Code:

altys-of-preston pts/0    193.61.254.65    Thu Oct 13 08:56:12 +0100 2011

This user does not have shell login permissions. Ftponly permission.
Unfortunately Now that I've found this intrusion, I don't know how to find out what he did or where he went.

Are you aware of any way of telling ???

leslie_jones · 11-13-2011, 09:48 AM

Hi John,

I personally don't speak 'sendmail' so-to-speak. I hear people always banging on about how insecure it is supposed to be - but in reality I would take an educated guess that the compromise has come by way of a vulnerable web script. I must stress that is just a guess.

While you could trash it and start again, I'd personally not do that because if there is a hole in something, particularly a web script, it will only come back the minute user files are restored.

I'd start by looking at some other very similar posts here to get yourself going:

http://www.linuxquestions.org/questi...-point-905282/

And this resource is a common answer/starting point at LQ for such problems: CERT CHECK LIST

I think the first goal will be to find out *what* is trying to dump mail into the sendmail queue before getting hung up on anything else. Usually when this kind of post is made at LQ a number of senior people come along and make great suggestions on how to deal with it and I'm sure they will post better answers than I can give shortly John.

If you had an FTP user added BTW, do your FTP server logs not show you what he uploaded/downloaded/deleted?

leslie_jones · 11-13-2011, 09:54 AM

Just to add, this was suggested to a user how had similar problems:

http://www.rfxn.com/projects/linux-malware-detect/

but I've not tried it myself.

countrydj · 11-13-2011, 10:37 AM

Hi leslie_jones

Quote:

While you could trash it and start again, I'd personally not do that because if there is a hole in something, particularly a web script, it will only come back the minute user files are restored.

This is my big concern, but I may have to take the risk.
The bombardment is happening very often.

Quote:

I think the first goal will be to find out *what* is trying to dump mail into the sendmail queue before getting hung up on anything else.

This is what I am trying to do.

Quote:

http://www.linuxquestions.org/questi...-point-905282/

I'll have a look at this.

Quote:

And this resource is a common answer/starting point at LQ for such problems: http://web.archive.org/web/200801092...checklist.html

This link doesn't seem to work.

Thanks for your help and advise.
If I don't get this sorted pretty quick my isp has threatened to cut me off.

Kind Regards,

leslie_jones · 11-13-2011, 11:01 AM

If your ISP is threatening disconnection that would suggest that the attack is successful and you are spitting out shed loads of spam - ummm - that's not good.

This problem comes up often - a compromised server spitting out spam. It's a pity that there is no definitive guide on LQ to dealing with this.

It's stating the obvious to say that if this is a web script or exploited application it can be a royal pain in the backside to find it. You know how it is, Apache is running away, a PHP or Perl script is being used/manipulated and finding clues as to which file and which user is like looking for a needle in as many haystacks as you have users - and the size of the needle will depend on how busy your server is in general.

Something does stand out about your sendmail line - and this may be a pure coincidence - the timestamp: Nov 13 14:24:00
It's exactly 00 seconds. Either that's a fluke or I'd look in the crontabs to make sure something in there is not calling an emailer script. If it is a fluke it will still be helpful - try and match occurrences of the refusals in your maillog, with the names of scripts being called in the apache access log with timestamps a few moments either side. POST requests *may* suggest uploads of recipient lists. Of course, all this assumes it is a web based exploit - and not a core package or system exploit.

If the attack is fairly constant I'd consider shutting all websites down, then bringing them back up one at a time a few minutes apart whilst tailing the mailog. Naturally if you've 100's of site this is not a practical proposition.

It may also be worth TCPDumping traffic coming in on port 80 for a bit, then open the capture in Wireshark and do a text search for any of the recipient addresses in maillog that have occurred during the capture. Wireshark will then allow a simple right click, follow TCP stream, and you should be able to see the HTTP request calling it.

Sincerely hope that is helpful in some way.

leslie_jones · 11-13-2011, 11:06 AM

Just adding in this - TCPDUMP to capture port 80 on eth0 to file:

Quote:

tcpdump -vv -x -X -i eth0 port 80 -s 1500 -w /tmp/port80.pcap

Note you'll get the requests/uri's OK but the content may be garbled due to gzip/inflate encoding in the HTML response.

And that link should be:
http://web.archive.org/web/200801092...checklist.html

countrydj · 11-13-2011, 12:54 PM

Hi Leslie...

Code:

tcpdump -vv -x -X -i eth0 port 80 -s 1500 -w /tmp/port80.pcap

I've set this running and like you say, the output is garbled.
Now I have installed wireshare to read it.
At the moment I can't see how to use wireshark.
I run from the command line.
I did stop apache (httpd) and the attack still carried on which leads meto doubt if the problem is within a web site.

Quote:

Something does stand out about your sendmail line - and this may be a pure coincidence - the timestamp: Nov 13 14:24:00

I think that this is just a fluke.

leslie_jones · 11-13-2011, 01:04 PM

Wireshark is a gui job needing some x system. It makes trawling capture files bearable and easier. You can trawl the naked tcpdump by hand, but it's awful having to work that way.

If you only have a server environment don't install it there, put it on something else - even a windows box if you have to. You won't need the capture side of it, just the opening file and read side of it.

I also found this with a google:

http://www.halecomm.net/blog/2010/04...is-sending-it/

It looks like it may be a great tool in finding this kind of thing, but I've not tested it myself.