[SOLVED] My Postfix server used to send SPAM, please help identify entry point!

AcorpComputers · 09-29-2011, 05:52 PM

Quote:

Originally Posted by snooly

Do you know which executable is being called to send mail?

I'm not sure which executable...I don't even know what they all are! I know of "mail" and "sendmail" are there others?

Quote:

Originally Posted by snooly

Can you take the mail server out of service for a few hours or days until you sort the problems out?

No, can't really do that. I do have multiple backup servers, but at different DC's, so I'd have to muck with changing DNS, etc. for all my client domains. Best to leave it up. However, since the problem appears to be using a local executable, and none of my clients need to send mail via a local executable, I have turned off the ability to send mail locally. That gives me, "breathing room" without impacting client services for smtp/imap/pop mail.

Quote:

Originally Posted by snooly

If so, you might be able to replace the mail executable with a shell script which logs all the information you can think of, rather than sending mail. You might even make the script still send mail through the original executable, renamed to something else, if the mail is legitimate mail you want to send.

Oh, I like that idea! I think I'll start with replacing sendmail and mail, and have it log, accept, but defer delivery of any messages. Then maybe I'll have a better idea of what is going on. I'm pretty decent with bash scripting, but rather than re-invent the wheel, do you happen to have or know of a base/model I could start from?

Thanks, that is a great idea!

- Paul

kbp · 09-29-2011, 06:40 PM

You might be able to start with fakemail and modify it a little, its available in perl and python.

http://sourceforge.net/projects/fakemail/

LBM · 09-29-2011, 07:03 PM

Quote:

Originally Posted by AcorpComputers

Can you clarify what you mean by, "sending through a relay?" I usually think of that as connecting, externally, to the smtp or submission port of my server, to send a message. I'm not seeing that in the logs, or am I missing something? I would expect an external connection to be prefaced with "postfix/smtpd" in the logs, rather than "postfix/pickup" as here.

You have this in your log, which looks to me that you are sending through a relay.

Code:

relay=mxs.mail.ru[94.100.176.20]:25

It is completely normal doing so, I am just woundering, if it is a script that is sending through this relay, or if you main.cf file have been edited, which where the reason I asked.

kbp · 09-29-2011, 07:52 PM

I had a look at the python version of fakemail and the changes below will probably work:

Code:

    def process_message(self, peer, mailfrom, rcpttos, data):
        message("Incoming mail")
        print("[*] connection info matching inbound client:")
        print(os.system('lsof -i:' + peer[1]))
        print("[*] process commandline for matching inbound client")
        print(os.system('lsof -i:' + peer[1] + ' | awk \'{if ($2 ~ /0[-9]/) print $2}\' | xargs -I pid cat /proc/pid/cmdline'))
        for recipient in rcpttos:
        ...

This should output enough info to track the offending process hopefully

AcorpComputers · 09-29-2011, 08:48 PM

Quote:

Originally Posted by LBM

You have this in your log, which looks to me that you are sending through a relay.

Code:

relay=mxs.mail.ru[94.100.176.20]:25

That did look strange, so I looked through the rest of my logs, and I guess it is a postfix thing. When I send a message to a gmail account, the log says:

Code:

relay=gmail-smtp-in.l.google.com

And to one of our local ISP's:

Code:

relay=smtp.mchsi.com

So, I think that is just Postfix logging that it handed the message off to the receiving MX server, and who that server was. In that context, I think the "relay=mxs.mail.ru" is just saying the message was handed off to the mail.ru smtp server (where all of the spam went).

If I've understood that wrong, please feel free to correct me.

Thanks,

- Paul

leslie_jones · 09-30-2011, 12:36 AM

IMHO It's most likely going to be a web/php script somebody has '''helpfully''' uploaded. Tracking it down will probably be an utter nightmare if you have multiple hosted sites and users. How many of those scripts are going to make calls to sendmail one way or another?

I guess a logical extension/supplementary question to ask would be this;

If there were a public facing PHP/Perl script on my server that was able to dump mail into the queue - how would you approach uncovering which script it was out of potentially dozens/hundreds/thousands? It's probably helpful that whatever is doing it is, at least, dumping it so it appears in maildrop - rather than opening sockets and sending it direct to MX.

It crosses my mind that you may be able to partially correlate the Apache logs (what page/script was called) within the timeout window (180 seconds typically) with the mail logs. It's a needle in a haystack, but it may drill things down if the sever was not busy at the time.

Users running old/vulnerable versions of Wordpress (and other web applications for that matter) may not be inclined to update anything where they don't see a direct benefit themselves. They don't tend to care about vulnerabilities. It's a awkward and painful position to find yourself in as an admin and I don't envy you.

AcorpComputers · 09-30-2011, 01:53 PM

Quote:

Originally Posted by leslie_jones

IMHO It's most likely going to be a web/php script somebody has '''helpfully''' uploaded. Tracking it down will probably be an utter nightmare if you have multiple hosted sites and users. How many of those scripts are going to make calls to sendmail one way or another?
[SNIP]
Users running old/vulnerable versions of Wordpress (and other web applications for that matter) may not be inclined to update anything where they don't see a direct benefit themselves. They don't tend to care about vulnerabilities. It's a awkward and painful position to find yourself in as an admin and I don't envy you.

You are exactly right - it was a Wordpress exploit from a non-updated WP installation, which allowed the "Web Shell by oRb" malware to be installed on the server. I don't know if this is typical or variable, but in my case it had placed itself in almost every WP installation (even updated ones) at /wp-contents/plugins/index.php

I was pointed to a nifty tool via a thread on Web Hosting Talk: Linux Malware Detect. While I had found many of the malware programs on the server, it was a time-consuming and error-prone process. LMD searched and found many more, quickly and efficiently. I HIGHLY recommend its use for posterity, if anyone else hits on this thread while searching for a similar problem. Linux Malware Detect is currently found here.

So, I have now changed all passwords, removed all malware, maintained the block on local email sending for the current time, and have advised all clients that they are required to upgrade their WP installs and change their WP admin passwords by the end of the weekend.

I am thinking about modifying the "Web Shell by oRb" and putting a sanitized version up to at least obtain the IP's of anyone attempting to connect to my server...don't know if I have the time, but sure would feel nice to, "fight back!"

I'll hang around the thread for a while yet in case anyone has any questions or anything else to add.

Thanks for all the help!

- Paul

leslie_jones · 10-01-2011, 02:32 AM

Quote:

Originally Posted by AcorpComputers

You are exactly right - it was a Wordpress exploit from a non-updated WP installation, which allowed the "Web Shell by oRb" malware to be installed on the server. I don't know if this is typical or variable, but in my case it had placed itself in almost every WP installation (even updated ones) at /wp-contents/plugins/index.php

Magic :-) I'm really glad you found it. These can be a royal PITA.

Quote:

Originally Posted by AcorpComputers

I was pointed to a nifty tool via a thread on Web Hosting Talk: Linux Malware Detect. While I had found many of the malware programs on the server, it was a time-consuming and error-prone process. LMD searched and found many more, quickly and efficiently. I HIGHLY recommend its use for posterity, if anyone else hits on this thread while searching for a similar problem. Linux Malware Detect is currently found here.

That's a really useful link and piece of software - thanks :-)

Quote:

Originally Posted by AcorpComputers

I am thinking about modifying the "Web Shell by oRb" and putting a sanitized version up to at least obtain the IP's of anyone attempting to connect to my server...don't know if I have the time, but sure would feel nice to, "fight back!"

On fighting back: It depends where the offenders are TBH. It's OK if they are in a civilised country where larting the ISP will kill their service - but many are in places like China, South America, Eastern Europe where it's almost considered rude not to attack other peoples servers. Mix this in with the proxy/tor users and things get more messy. I operate the policy of blocking TOR exit nodes, known proxies, and IPs from blocks that I see routinely running scanning tools with IPTables - but I'm at hundreds of lines and it's not ideal. I also rate control port 80 connections with IPTables so scanning tools tend to cough after about 20 GET's. Daily (sometimes hourly) checks in the logs will always reveal a stack of attacks/door knocking. The 403's, 400's, Morpheus Scanners, you name it - it's a grind.

But as for modifying that script - it's not a bad idea if you have the time. Whilst the IP address may not be useful, spammers will always leave a path back to them somewhere in the actual message they send - be it an email address of a freemailer, or a website. In the past I've set up honeypots with Exim that log the entire message without ever sending it, and then let spammers pump their messages in just to obtain this kind of data. It's probably possible to set something up with Postfix so it pipes a copy of messages submitted by the script to a mail spool without actually sending them. Depends how much time you want to give to it - or even if you want to give it any.

Quote:

Originally Posted by AcorpComputers

Thanks for all the help!
- Paul

Thanks for a great read and really useful link.

LBM · 10-01-2011, 07:39 AM

Good work, and nice you got it working.