[SOLVED] My Postfix server used to send SPAM, please help identify entry point!
Linux - ServerThis forum is for the discussion of Linux Software used in a server related context.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
My Postfix server used to send SPAM, please help identify entry point!
Hoping someone can help me out here:
This morning, I received over a hundred bounce messages from mail.ru. My first impression was back scatter SPAM, and I started composing an email about incorrectly configured mail servers...then I thought I should check my server just in case, and sure enough, it looks like I DID send SPAM.
I am running a Postfix 2.3.3 on a CentOS server. The server is not an open relay - have tested in the past, and again (verify.abuse.net) after the incident this morning. All tests return "Relay Access Denied" and after Relay Test 9 my server, as I have configured it, breaks the connection with "Too many errors."
I've been managing my linux servers for about five years now, so not a long time, but not a short time either, however at this point I am stumped...I can't even tell for sure how my mail server accepted the SPAM.
Things I have checked:
1) Not an open relay
2) mynetworks is as restrictive as it should be
3) No "strange" ssh users logged in to the system, in fact all logins are from recognized IP's
4) No "strange" user/mail logins
5) No strange cron jobs
What I "think" I know:
1) My server did send SPAM
2) It looks like it might have accepted it from the postfix sendmail command line program, and/or
3) It looks like my "Acorp" account is the one that sent the messages by putting them in the maildrop directory, where they got picked up and sent by pickup...am I seeing that correctly?
I don't want to post wholesale configurations here that might open me up to any future hacking, but if you can tell me what might be relevant I'll certainly post it as-is or sanitized if it looks dangerous.
Excerpt from the first relevant entry in the maillog:
Code:
Sep 27 07:02:08 Calevra postfix/sendmail[21701]: fatal: No recipient addresses found in message header
Sep 27 07:03:05 Calevra postfix/pickup[17060]: 8949521A0FB: uid=500 from=<acorp>
Sep 27 07:03:05 Calevra postfix/cleanup[21716]: 8949521A0FB: message-id=<20110927120305.8949521A0FB@calevra.acorp.net>
Sep 27 07:03:05 Calevra opendkim[1106]: 8949521A0FB: no signing table match for 'maxdertyh@gmail.com'
Sep 27 07:03:05 Calevra opendkim[1106]: 8949521A0FB: no signature data
Sep 27 07:03:05 Calevra postfix/qmgr[4761]: 8949521A0FB: from=<acorp@calevra.acorp.net>, size=390, nrcpt=2 (queue active)
Sep 27 07:03:08 Calevra postfix/smtp[21722]: 8949521A0FB: to=<dfsgdfhdgfgdf@mail.ru>, relay=mxs.mail.ru[94.100.176.20]:2$
Sep 27 07:03:08 Calevra postfix/smtp[21722]: 8949521A0FB: to=<niturew@mail.ru>, relay=mxs.mail.ru[94.100.176.20]:25, del$
Sep 27 07:03:08 Calevra postfix/qmgr[4761]: 8949521A0FB: removed
Sep 27 07:03:09 Calevra postfix/smtpd[21724]: connect from mx18.mail.ru[94.100.176.143]
Sep 27 07:03:10 Calevra postfix/smtpd[21724]: setting up TLS connection from mx18.mail.ru[94.100.176.143]
Sep 27 07:03:10 Calevra postfix/smtpd[21724]: TLS connection established from mx18.mail.ru[94.100.176.143]: TLSv1 with c$
Sep 27 07:03:11 Calevra milter-greylist: (unknown id): addr 94.100.176.143 flushed, removed 0 grey and autowhite (ACL 29$
Sep 27 07:03:11 Calevra postfix/smtpd[21724]: 33DC821A0F7: client=mx18.mail.ru[94.100.176.143]
Sep 27 07:03:11 Calevra postfix/cleanup[21716]: 33DC821A0F7: message-id=<E1R8WNR-0002zc-00@mx18.mail.ru>
Sep 27 07:03:11 Calevra milter-greylist: (unknown id): addr mx18.mail.ru[94.100.176.143] from <> to <acorp@calevra.acorp$
Sep 27 07:03:11 Calevra postfix/cleanup[21716]: 33DC821A0F7: milter-reject: END-OF-MESSAGE from mx18.mail.ru[94.100.176.$
I have increased verbosity of internal/local SMTP connections by adding -vvvvv to SMTP on Unix in master.cf, but I can't find out how to get more in depth logging of the pickup program itself.
I have also removed the ability for ALL users except root to send messages using the "authorized_submit_users" directive in main.cf. I think, if I did that correctly!
Am I jumping to conclusions, missing anything, or otherwise off-track? Any thing else I can do? Short of preventing local submission of emails (a short term solution at best, though in reality no one should be sending messages locally - unless webmail uses that method?) I haven't really STOPPED anything, so I feel I am still at risk here.
Your messages are truncated and just appears to be a random sample, please grep for a specific pid so we can see the whole transaction or post a longer sample.
You'll need to determine where acorp is submitting from, maybe see if you have any logs for the webmail app. You could also try changing acorp's password as a quick test.
Thanks for the suggestion. Any ideas how I go about determining where acorp submitted from? That does seem to be the key to the whole thing.
I have changed the password for that account...and actually disabled log in as well, though I don't see, from the logs, that the account actually logged in.
As to webmail, I checked my application (Roundcube), and it submits mail (for all users) as the user apache, so I'm pretty sure it didn't come through webmail, as the maillog pretty clearly identifies acorp as the culprit.
So (I think) this is what I know, and I just can't find the missing piece:
1) Acorp didn't log in to the server via SSH
2) Acorp didn't "log in" via smtp
3) Acorp didn't submit these messages via webmail (else the logs would have shown them as coming from apache)
4) The messages were submitted locally, directly to the pickup queue by Acorp
So, the missing piece is, "how did these messages get submitted to the pickup queue?" Anyone know any logs I can check, or anything I can set up at this point to monitor that for the future?
Is acorp.net the real/your domain ? Have you tested with telnet to see if you can send emails with non-authorized accounts, if they seem to come from your domain? http://www.tech-recipes.com/rx/381/s...or_open_relay/
Secondly, are you using a relayhost? (mxs.mail.ru)
Have you tested with telnet to see if you can send emails with non-authorized accounts, if they seem to come from your domain?
Yes - I get "relay access denied"
Quote:
Secondly, are you using a relayhost? (mxs.mail.ru)
No, since my server is directly connected to the internet, I just let it do the normal MX lookup and delivery.
Quote:
Any cron jobs, scripts or other listening services that could be causing it?
The "Acorp" user does not have any cronjobs (at least crontab -e) when logged in as Acorp shows nothing.
I've searched for other listening services (netstat, I think) and haven't seen anything unusual, but I'm not super-experienced in that, so if somebody has an idea of what to look for and exactly what command to use, I'd be open to checking again.
I guess I am beginning to think it was some type of script or maybe a php program that was installed...I going to search all publicly exposed websites for strange php and or pl programs, but since many of my client sites run Wordpress, that could be fun. Anybody have any ideas on how to do that efficiently?
Thanks for the ideas, hope we can keep them coming until I get this figured out!
According to the log, it is sending through an relay.
A wild guess, is that the server is comprimited and a script is running, if you are sure you are not an openrelay?
I'm not familiar with wordpress, but as you mention, there could be a security issue, where a script/code have been injected.
According to the log, it is sending through an relay.
Can you clarify what you mean by, "sending through a relay?" I usually think of that as connecting, externally, to the smtp or submission port of my server, to send a message. I'm not seeing that in the logs, or am I missing something? I would expect an external connection to be prefaced with "postfix/smtpd" in the logs, rather than "postfix/pickup" as here.
Quote:
Originally Posted by LBM
A wild guess, is that the server is comprimited and a script is running, if you are sure you are not an openrelay?
About as positive as I can be, based on multiple automated online and direct connection tests, that server is not an open relay.
Quote:
Originally Posted by LBM
I'm not familiar with wordpress, but as you mention, there could be a security issue, where a script/code have been injected.
Beginning to think this is the only possibility. Am looking into NMAP as snooly suggested, too though.
You may be able to lock the acorp account - 'passwd -l acorp', then check /var/log/secure to see where the login attempts are coming from.
Would /var/log/secure show me anything that "last" doesn't? last shows no logins by Acorp since April, and the login at that time was legitimate, and from a recognized IP address. I have removed login access to the Acorp account as a precaution, regardless, though the -l option might be a better choice since root would still have access to that account if I wanted. I'll test now and make sure that a [failed] login attempt shows up in /var/log/secure.
Quote:
Originally Posted by snooly
You might like to try "nmap" to scan your ports to see what's open.
nmap shows no unexpected ports open, thanks!
Looking more and more like it has to be a script or program of some type that dropped email messages in the postfix pickup directory...think I have prevented a repeat by blocking access to the postfix pickup program from all users except root, but I'd really like to get to the bottom of the issue, instead of just working around it.
I did a text search for "sendmail" on all outward facing files (websites) and didn't see anything unusual, but with so many files to sift through thanks to Wordpress maybe I missed something.
Well, it does look like "some" program (?) is trying to send email via the postfix sendmail program. The latest from my logs:
Sep 28 12:36:05 Calevra postfix/sendmail[19025]: fatal: User acorp(500) is not allowed to submit mail
Anybody have any ideas how I can track down WHAT is trying to use sendmail to send a message? There is NO more information in the log then that single line - repeated irregularly since I removed sendmail rights from all users earlier today.
Do you know which executable is being called to send mail? Can you take the mail server out of service for a few hours or days until you sort the problems out?
If so, you might be able to replace the mail executable with a shell script which logs all the information you can think of, rather than sending mail. You might even make the script still send mail through the original executable, renamed to something else, if the mail is legitimate mail you want to send.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
