Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I'm really in trouble, somehow my server got compromised and is sending out spam non stop. My guess is that the intruder somehow was able to upload a php mailer script that lets him send out spam by parameters to his liking. The server hosts about 30 websites, all well visited, so the logs are extremely hard to look at.
Here's an excerpt from /var/logs/maillog
Is there ANY way that I can find out which script launches these emails?
Code:
Oct 25 18:27:09 server1 courieresmtp: id=00B105AD.4CC57AD3.000044B7,from=<paul-bryant777@live.com>,addr=<nigelprice@lineone.net>,status: deferred
Oct 25 18:27:09 server1 courierd: Waiting. shutdown time=none, wakeup time=Mon Oct 25 18:30:22 2010, queuedelivering=398, inprogress=40
Oct 25 18:27:09 server1 courierd: started,id=00B105B0.4CC57AE0.000045C0,from=<paul-bryant777@live.com>,module=esmtp,host=lineone.net,addr=<nikhart@lineone.net>
Oct 25 18:27:09 server1 courierd: started,id=00A70403.4CC231BD.0000551D,from=<paul-bryant777@live.com>,module=esmtp,host=lineone.net,addr=<martinthefish@lineone.net>
Oct 25 18:27:09 server1 courierd: started,id=00A4819E.4CBF1AFE.00000799,from=<paul-bryant777@live.com>,module=esmtp,host=boxerzwinger.de,addr=<aywdgfhv1@boxerzwinger.de>
Oct 25 18:27:09 server1 courierd: Waiting. shutdown time=none, wakeup time=Mon Oct 25 18:30:22 2010, queuedelivering=398, inprogress=40
Oct 25 18:27:09 server1 courieresmtp: id=00B105B0.4CC57AE0.000045C0,from=<paul-bryant777@live.com>,addr=<nikhart@lineone.net>: Connection reset by peer
Oct 25 18:27:09 server1 courieresmtp: id=00B105B0.4CC57AE0.000045C0,from=<paul-bryant777@live.com>,addr=<nikhart@lineone.net>,status: deferred
Oct 25 18:27:10 server1 courierd: started,id=00A481AF.4CBF1B57.00000F5D,from=<paul-bryant777@live.com>,module=esmtp,host=boxerzwinger.de,addr=<aywz1@boxerzwinger.de>
Oct 25 18:27:10 server1 courierd: Waiting. shutdown time=none, wakeup time=Mon Oct 25 18:30:22 2010, queuedelivering=398, inprogress=40
Oct 25 18:27:10 server1 courieresmtp: id=00A70403.4CC231BD.0000551D,from=<paul-bryant777@live.com>,addr=<martinthefish@lineone.net>: Connection reset by peer
Oct 25 18:27:10 server1 courieresmtp: id=00A70403.4CC231BD.0000551D,from=<paul-bryant777@live.com>,addr=<martinthefish@lineone.net>,status: deferred
Oct 25 18:27:10 server1 courierd: started,id=00A70B88.4CC25BEC.0000349A,from=<paul-bryant777@live.com>,module=esmtp,host=jhmi.edu,addr=<vclark5@jhmi.edu>
Oct 25 18:27:10 server1 courierd: Waiting. shutdown time=none, wakeup time=Mon Oct 25 18:30:22 2010, queuedelivering=398, inprogress=40
Oct 25 18:27:10 server1 courieresmtp: id=00A70B88.4CC25BEC.0000349A,from=<paul-bryant777@live.com>,addr=<vclark5@jhmi.edu>: Connection refused
Oct 25 18:27:10 server1 courieresmtp: id=00A70B88.4CC25BEC.0000349A,from=<paul-bryant777@live.com>,addr=<vclark5@jhmi.edu>,status: deferred
Oct 25 18:27:10 server1 courierd: started,id=00A70B88.4CC25BEC.0000349A,from=<paul-bryant777@live.com>,module=esmtp,host=sic.shell.nl,addr=<veer3@sic.shell.nl>
Oct 25 18:27:10 server1 courierd: Waiting. shutdown time=none, wakeup time=Mon Oct 25 18:30:22 2010, queuedelivering=398, inprogress=40
Oct 25 18:27:12 server1 courieresmtp: id=00A810F0.4CC3B40C.0000177E,from=<paul-bryant777@live.com>,addr=<aburke2@gmu.edu>: Connection reset by peer
Oct 25 18:27:12 server1 courieresmtp: id=00A810F0.4CC3B40C.0000177E,from=<paul-bryant777@live.com>,addr=<aburke2@gmu.edu>,status: deferred
Oct 25 18:27:12 server1 courierd: started,id=00A48EEB.4CBF671B.00001E5A,from=<paul-bryant777@live.com>,module=esmtp,host=werwer.com,addr=<wefwef@werwer.com>
Oct 25 18:27:12 server1 courierd: completed,id=00A810F0.4CC3B40C.0000177E
Oct 25 18:27:12 server1 courierd: Waiting. shutdown time=none, wakeup time=Mon Oct 25 18:30:22 2010, queuedelivering=398, inprogress=40
Oct 25 18:27:12 server1 courieresmtp: id=00A70B88.4CC25BEC.0000349A,from=<paul-bryant777@live.com>,addr=<veer3@sic.shell.nl>: DNS lookup failed.
Oct 25 18:27:12 server1 courieresmtp: id=00A70B88.4CC25BEC.0000349A,from=<paul-bryant777@live.com>,addr=<veer3@sic.shell.nl>,status: deferred
Oct 25 18:27:12 server1 courierd: started,id=00A480E1.4CBF1706.000015BA,from=<paul-bryant777@live.com>,module=esmtp,host=dest.gov.au,addr=<aytoagoncillo.org@dest.gov.au>
Oct 25 18:27:12 server1 courierd: completed,id=00A70B88.4CC25BEC.0000349A
Oct 25 18:27:12 server1 courierd: Waiting. shutdown time=none, wakeup time=Mon Oct 25 18:30:22 2010, queuedelivering=398, inprogress=40
Oct 25 18:27:18 server1 courieresmtp: id=00A6033B.4CC1C9D3.00005548,from=<paul-bryant777@live.com>,addr=<azennoeepwveo@berlin.3d-game.com>: Connection timed out
Oct 25 18:27:18 server1 courieresmtp: id=00A6033B.4CC1C9D3.00005548,from=<paul-bryant777@live.com>,addr=<azennoeepwveo@berlin.3d-game.com>,status: deferred
Oct 25 18:27:18 server1 courierd: started,id=00A480E1.4CBF1706.000015BA,from=<paul-bryant777@live.com>,module=esmtp,host=tmusic.com,addr=<aytoantillon1@tmusic.com>
Oct 25 18:27:18 server1 courierd: started,id=00A480E1.4CBF1706.000015BA,from=<paul-bryant777@live.com>,module=esmtp,host=tmusic.com,addr=<aytoazuaga1@tmusic.com>
Oct 25 18:27:18 server1 courierd: started,id=00A480E1.4CBF1706.000015BA,from=<paul-bryant777@live.com>,module=esmtp,host=tmusic.com,addr=<aytoba_equal1@tmusic.com>
Oct 25 18:27:18 server1 courierd: started,id=00A480E1.4CBF1706.000015BA,from=<paul-bryant777@live.com>,module=esmtp,host=tmusic.com,addr=<aytobinefar1@tmusic.com>
Oct 25 18:27:18 server1 courierd: started,id=00A480E1.4CBF1706.000015BA,from=<paul-bryant777@live.com>,module=esmtp,host=tmusic.com,addr=<aytobody1@tmusic.com>
Oct 25 18:27:18 server1 courierd: completed,id=00A6033B.4CC1C9D3.00005548
Oct 25 18:27:18 server1 courierd: Waiting. shutdown time=none, wakeup time=Mon Oct 25 18:30:22 2010, queuedelivering=398, inprogress=40
Oct 25 18:27:18 server1 courieresmtp: id=00A68144.4CC1DAEE.00004F7E,from=<paul-bryant777@live.com>,addr=<azoun@axiowave.com>: DNS lookup failed.
Oct 25 18:27:18 server1 courieresmtp: id=00A68144.4CC1DAEE.00004F7E,from=<paul-bryant777@live.com>,addr=<azoun@axiowave.com>,status: deferred
Oct 25 18:27:18 server1 courieresmtp: id=00A48192.4CBF1ACD.00000589,from=<paul-bryant777@live.com>,addr=<ayvon@axiowave.com>: DNS lookup failed.
Oct 25 18:27:18 server1 courieresmtp: id=00A4819E.4CBF1AFE.00000799,from=<paul-bryant777@live.com>,addr=<aywe@axiowave.com>: DNS lookup failed.
Oct 25 18:27:18 server1 courieresmtp: id=00A48192.4CBF1ACD.00000589,from=<paul-bryant777@live.com>,addr=<ayvon@axiowave.com>,status: deferred
I forgot to mention that the root has NOT been compromised (whew). There are around 20 other user accounts on the host though, each with their websites.
My guess is that the intruder somehow was able to upload a php mailer script that lets him send out spam by parameters to his liking.
The way this forum handles security breaches is to eliminate guessing and focus on gathering and analyzing facts about the situation. We're not trying to be obnoxious, it is just that when it comes to breaches, people tend to assume a lot of things that aren't necessarily true. So in that spirit, why do you suspect a php mailer?
Quote:
I forgot to mention that the root has NOT been compromised (whew). There are around 20 other user accounts on the host though, each with their websites.
What is the evidence for root not being compromised? Again, I'm not trying to be a jackass, but we need to deal in facts. Usually in these situations it is also a good idea to lock down access to the machine if not take it off the net altogether. Can you do that here? Can you limit access to SSH from trusted IP addresses only? Is there some backup plan you can put into play so this server can be investigated?
Quote:
Is there ANY way that I can find out which script launches these emails?
Do you know when the spam started? That would at least give you and idea of when to look for new/changed files. Do you have anything like a HIDS/NIDS running (Aide, Osiris, Samhain, Tripwire)? You also might look at the output of lsof -Pwn. That should help identify what files might be responsible. The output of ps -axfwwwe also might be helpful. In both of these, you're looking for processes that shouldn't be there.
Other bits that would be helpful
- Distro and status of patching
- Software known to be served (are we talking lots of PHP applications here?) and their update status
- How are the users segregated from each other? VMs? Chroot?
- Have a look at the CERT Checklist for some other ideas on how to evaluate the machine.
One last thing, the spam here is the symptom, not the problem. Some weakness allowed this to happen and you need to focus on finding and fixing that weakness, not solely on stopping the spam.
I'm buying the services from a managed hosting company and it costs me an arm and a leg. So they're pros and have looked into the matter and said that 99.9% likely it's a bad PHP script that's being used and suggested that I would go through all the site's logs and look for the scripts. The sites I host are all PHP/Mysql sites but the users don't even have access to the server.
Some passwords to the server were published on the internet, which is about the dumbest thing that ever happened. I took care of that, but that's about when the breach happened. We got the first blacklisting notifications on Oct 23.
Server's distro is CentOS release 4.3
PHP 5.2.5
My biggest question is how I could trace where the sendmail orders originate. In my mind, I should be able to find the PHP script behind the spam that way, no?
Here's the part from the ps output that has to do with the mailer
Here's the part from the ps output that has to do with the mailer
...and there's part of your problem. You're focusing on MTA processes while Hangdog42 gave you a checklist and specific commands to run. At this point the more information you provide (post in BB code tags, host, attach as plain text files, pastebin, docs.google or email) the better you help us help you.
So they're pros and have looked into the matter and said that 99.9% likely it's a bad PHP script that's being used and suggested that I would go through all the site's logs and look for the scripts.
If you're paying them through the nose, I would hope they would be cooperative and give you some idea (and hopefully some documentation) as to why they think that. Blaming PHP isn't a bad assumption, but it is an assumption and unless they have some evidence to support their conclusion, they may be giving you the brush-off. I would think that your patronage should entitle you to some indication of what kind of investigation they did and what they found.
You certainly could look for files that have been created since right before the spamming started. Presumably that would be a day or so before your blacklisting notice. At any rate that would be a time to start looking for files that shouldn't be there. Similarly that would be a good date to look in the log files.
Quote:
The sites I host are all PHP/Mysql sites but the users don't even have access to the server.
The users having access or not may not be hugely important at this point. Poorly written PHP can be abused and can grant access at the web-server level. Are these "commercial" PHP programs like Joomla or phpBB or are they custom jobs?
Quote:
Server's distro is CentOS release 4.3
PHP 5.2.5
Um, ouch. Is CentOS 4.3 still receiving security patches? I'm thinking that since PHP is still on 5.2.5, it isn't or they aren't being applied.
Quote:
My biggest question is how I could trace where the sendmail orders originate. In my mind, I should be able to find the PHP script behind the spam that way, no?
I think that is why you need to look at the output I suggested for stuff that shouldn't be there. As unSpawn is suggesting, focusing on processes that should be there anyways probably isn't going to get us very far. I know the files and output we're requesting are large so do feel free to post them in GoogleDocs or email them if you wish.
Wow, I really appreciate your willingness to help me out - it's good to have people in this world that help without wanting get paid in return!
For me this whole thing is a learning process. I think I know my way around in server administration good enough for day to day things but when it comes to crashes like this I'm helpless. The staff of the managed hosting company did finally go in after I told them after 3 hours that there was no PHP script I could find in the logs that was obviously being abused. I thought, for now, I would share their comments. BTW, the spamming has ceased for now.
Code:
Tue, 26 Oct 2010 21:03:12 -0400
Those are not emails being sent now, that's your mail queue backed up with the spam that bounced on the first delivery attempt. In other words the spammer could of sent the spam days ago, but its in the queue stuck because you been blacklisted/blocked from delivery to the final destination(s) so your mail queue just fills up.
That mail log is showing just the mail queue
to view the queue type "mailq" while ssh'd in as root.
to purge the que i put a script in /root/purgemailq.sh for you and ran it to clean it out.
ok that said, that email of paul-bryant777@live.com goes back to oct 19th, /var/log/maillog.1 to find this entry
Oct 19 03:29:50 server1 courierd: newmsg,id=00A0015B.4CBD48EB.00000D94:
dns; User (themaximizedliving.com [::ffff:67.228.87.98])
Oct 19 03:29:50 server1 courierd: started,id=00A0015B.4CBD48EB.00000D94,from=<paul-bryant777@live.com>,
module=esmtp,host=gmail.com,addr=<personalosas@gmail.com>
the source ip 67.228.87.98 is in seatle, coming from another web server (probably one that's been hijacked, now using your server to relay mails).
Since the ip address is shown in maillog.1 for oct 19th, that means they were NOT using the rinetd system (rinetd will mask the ip with 127.0.0.1 due to how it proxy forwards the ports), so we can likely rule that out as the cause/reason.
The question is how does a user at 67.228.87.98 get permission to send an email. It looks like this all started at that point, becasue the emails (spams) were going through and delivering to gmail and others then in the logs (but shortly later got blocked due to spam complaints/filtering). So that looks to be ground zero on Oct 19th for you there.
The issue now looks like this person at that ip has permission to send emails through your server, filling up the mail queue. How could they do it, well the answer is that all someone has to do is authenticate with any valid pop mail box, to have their ip opened up for 15 minutes as a permitted email relayer. So it's possible this person has a pop user of one of your users (likely someone clever enough to use 'default' passwords for an account like 'support' email box and 'support' as the password for the mail box). This script in seatle must be pop authenticating that pop user, then shooting out a batch of spam via smtp since the relay is open at that point for 15 minutes, then they just check in again with pop, and it resets the relay permissions again for that ip.
I'm looking through the logs for that ip more thoroughly as i should be able to find the pop user that's doing this but the logs are a bit overwhelming to process while im deleting out 14,000 emails from your mail queue right now. I will update you as soon as im done.
Code:
Wed, 27 Oct 2010 09:12:13 -0400
unfortunately, there is no record of ip 67.228.87.98 connecting to your server via POP to authenticate. Which unfortunately means we haven't figured this one out yet.
The log on server (its a bit too long to include here, is located on the server at /root/spammer.txt
You will see about 1700 lines of log entries of this user connecting to the smtp daemon to send emails but not one authentication request via pop to open up the relay.
It almost appears like your server is setup as an open mail relay (letting anyone send mail through the server).
Your mailque was purged last night, and is down to 38 messages (just normal emails since last night), so thats good, no new spam. Looks like all the mail relay was done previously and what you saw in the logs was just mail queue left overs for lack of a better term on this.
We are unsure how they are or did send those emails - for now i've blocked 67.228.87.98 at the server firewall entirely.
Mail relaying is blocked by default, and opens only when someone's ip is permitted during authentication, OR someome manually updates the file /etc/courier/smtpaccess/default and restarts the mail daemon. When someone 'pops' in for access, the file /etc/courier/smtpaccess/pbs gets updated with their ip for 15 to 60 minutes to permit email relaying (pbs stands for Pop Before Smtp). So there should of been a pop entry around this time for that ip, to allow him to send emails, but the logs did not indicate it like it should of, so the theory of this person has one of your clients email's and simply is pop-before-smtp'ing doesn't seem to fit here.
Due to how rinetd port forwards, via port 1025, there is potential for tricks/abuse to do such a mail relay by which it could bypass some restrictions, i don't think that is the case here because the mail logs clearly indicated the traffic was from the originating ip (and not showing a localhost proxy forward), none the less at this time we don't have an answer to how this person sent those emails. We still can't rule out anything at this point.
We almost have to catch it 'in the act' to determine with certainty at this point, and right now since we blocked the ip and turned off RINETD, and you closed up some un-used accounts that may do the trick.
The only thing left I can suggest to your team - go through each mail box account and remove any un-used ones, and if possible update individual email box accounts. It's possible this person is using a valid username/password for smtp relaying but its not indicated in the logs (catching this in the act to watch sql querry transactions to the auth database is almost the only way I could catch this to prove 100% though)
Interesting. What is very clear from what you posted is that your hosting service doesn't really know how the spammer got access, and that is disturbing. To be honest, I'm really not that impressed with their solution. Blocking a single IP address isn't going to do much if the spammer wants to start up again. I would suspect that the lack of spam currently could have as much to do with the operations of the spammer as the effectiveness of the remedy.
Spammers almost certainly know that the lifetime of a spam spewing server is relatively limited, and it seems to me likely that they would use a server to spew spam for a bit and then stop since it is likely that the server gets blacklisted. The question is whether they permanently discard the server or simply wait for the owner to do a little bit of clean-up, get un-blacklisted, and then spam some more.
I guess that is the long way of saying that my opinion is that you're not out of the woods. It is a good thing that the spam stopped, but if I were in your shoes, I wouldn't have the warm fuzzies just yet. There are a lot of issues from the facts you've presented so far:
- Why is it assumed root wasn't compromised?
- Why is CentOS 4.3 still being used? If it isn't being actively maintained, you've got a real problem
- Are any of the sites being hosted here vulnerable to attack?
I can go on for a bit, but I think you get the idea from this, and previous, posts. Unless you want to get a lot more first-hand experience in computer forensics, you need to work on your operations a bit.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
