more than one uid 0 - centOS/Plesk 10.x
I thought our VPS server was compromised this morning for these reasons:
- Got an email from cron. (btw: I can't find the cron script in etc or in roots crontab): Code:
To: root@jam - maillog/messages are empty - wtmp begins Sat Jul 2 11:29:26 2011. Shorty after the cronjob. - a virtual terminal was open via ssh (sshd: sendmail@pts/0) - ps outputs, "Internal error!" for some processes - also user sendmail is at the end of the pass file (plesk was installed weeks ago): Code:
[root@jam ~]# cat /etc/passwd | tail -1 Code:
[root@jam ~]# awk -F: '($3 == "0") {print}' /etc/passwd Steps I took to lock down: - I disabled the user sendmail by adding a asterisk in the beginning pass field in /etc/shadow - And killed virtual terminal process: Code:
[root@jam ~]# ps waux | grep sendmail I was under the impression there should never be another user with UID 0 right? |
I'm sure there are exotic situations where multiple UID 0's are acceptable but
- no sane Linux distro runs their MTA as UID 0 by default. - Adding a user using a cronjob is suspect as it's not an action one would willingly want to repeat. * If one process still has a deleted .bash_history (or cronjob!) on a fd ('lsof -Pwln|grep dele') you can copy it out. - When did maillog / messages become empty and do you have rotated logs slash backups (also see lsof)? - wtmp may be rotated but the cronjob points to malicious activity. As only root can add users that's disconcerting. - Apart from not listing which distro+release this is about user "sendmail" should not be allowed use of interactive shell let alone SSH access (hardening?). - "ps outputs, "Internal error!" for some processes" nice, but I like details ('/bin/ps axfwwwe'). - "And killed virtual terminal process" by doing that w/o listing details (ps, netstat, lsof) first you destroy potential evidence. Have a go at the http://web.archive.org/web/200801092...checklist.html ? |
Quote:
Quote:
Code:
[root@jam ~]# lsof -Pwln|grep dele Quote:
Quote:
Quote:
Code:
[root@jay ~]# /bin/ps axfwwwe Quote:
I'm going through the link you provided (thanks) so far no suspicious setuid files so far: Code:
/sbin/unix_chkpwd |
Quote:
Quote:
Quote:
Quote:
'( rpm -Vv SysVinit glibc binutils procps strace findutils lsof coreutils net-tools shadow-utils rpm grep 2>&1 | grep -v '^\.\{8\}'; )' '( /usr/bin/strings -an1 /bin/ps 2>&1; /lib/ld-linux.so.2 --verify --list /bin/ps 2>&1; /usr/bin/strace -v -s 1024000 -f -ff /bin/ps - -o /dev/shm/ps.strace 2>&1; /usr/sbin/lsof -Pwln 2>&1; /bin/netstat -anpe 2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last 2>&1; /usr/bin/who -a 2>&1; )' '/usr/bin/find / -printf "%T@ %A@ %C@ \"%p\"\n" 2>&1' '/bin/rpm -Vva 2>&1|/bin/grep -v "\.\{8\}" 2>&1' * the latter two might take some time so I would appreciate it if you make the first two available as soon as possible. Please email me the D/L URI or use my designated drop-off (check your email). Also a couple of question while you're gathering nfo: - When did this start? Reviewing your past threads doesn't show any previous occurrences. Were there no previous anomalies or breaches of security? - which services does the machine provide? Please include web-based management panels, statistics, web log, forum, shopping cart, plugins and other software running in the web stack if any and post exact versions, - was the software was kept up to date? - are SELinux and auditd active? - Are there any setuid-root or oddly named files in writable (temp) dirs? - did you check user shell history files? - which access restrictions are in place and what hardening was performed? Quote:
Quote:
|
Quote:
Quote:
Quote:
Quote:
Quote:
Plesk provides DNS, HTTP, HTTPS, FTP, POP/IMAP, POPS/IMAPS, MYSQL, statistics all the goodies. I'm selling hosting to my friends and family so anyone of them could be installing forums and shopping carts. There are 79 domains I was migrating (downgrading) to this VPS server in question. I've moved about 37 of the sites when this happened. bummer' Quote:
Quote:
Quote:
Quote:
Quote:
|
Just to be clear:
- I purchased a VSP server - Had it hardened by third party - Tried to migrate my sites/mail etc with Plesk software that failed - Because the failed migration messed up the Plesk panel/database my provider said the only way would be to re image the VPS and reinstall plesk. - The server was not hardened yet after reimaging when this incident occured. |
Thanks for the ordered output, well done.
From 1rpm.out: Code:
SM5..... /bin/ps From 2strings.out, running '/usr/bin/strings -an1 /bin/ps 2>&1': Code:
176 libproc.so.2.0.6 Code:
libproc.so.2.0.6 => /lib/libproc.so.2.0.6 (0xb7f63000) Code:
1 execve("/bin/ps", ["/bin/ps", "-", "-o", "/dev/shm/ps.strace"], # etc |
Quote:
|
Lets start with the obvious, who had access:
Quote:
Unfortunately, the log files were wiped, yet, messages, secure, and maillog were moved to /tmp too, and then deleted? Could this be part of an attempt to hide some specific activity? Does the output of netstat show any open connections? It also seems interesting that they would attempt to put a (second) mail user in place with root level access. It is almost as if their intentions were to setup something to feed information back to them at a later date. Quote:
I would also recommend having a SERIOUS conversation with both your hosting provider and your "security" company. It is beyond absurd that the hosting provider pretty much told you to "piss off" when you presented information that your system had been compromised. Also as a reminded, which I didn't see mentioned in the thread, if you haven't already, lock this system down. Forget about trying to migrate to your new setup, etc. You mentioned that your friends, family, and customers could be installing shopping carts and things. These need to be secured and verified, straight away. |
Quote:
Quote:
Quote:
Quote:
Quote:
|
Sorry for the silence, R/L kicking in. Your provider, together with a few other well-known ones, probably is on everybody's scan list for the easy targets such clusters of servers often provide but still it amazes me your server got compromised that soon. I'm not happy there's no evidence left wrt the point of entry because that leaves us without a clue as to how this happened. This means that any hardening advice will have to be as broad as possible. As you've already started configuring your new server what are your hardening plans? If you could start a new thread we'd be glad to offer advice.
|
Quote:
Quote:
Speaking of my provider during the re-image ticket there where several "Linux Admin" posting to the ticket. Every other post would be a different guy helping me. So I asked how many people are over there and they said they employ 600+ couldn't tell me how many where Linux admins. With that being said it could easily be internal leak, maybe the guy that said I wasn't compromised. I still need to talk to a manager about what happened. Quote:
- internal leak from my provider or the security company - intercepted the passwords in email transmission - exploited software to gain priv Quote:
- Re-imaged the server completely, reinstalled the Operating System (obviously after a rootkit you can't trust anything) - Implemented a secure firewall + additional security software - Hardened the web services & permissions - Requiring stronger passwords for email - Changed all user passwords - Policy to sms passwords without usernames or ips (for people that have login shells) - Engaged an security firm to conduct a investigation if my provider will let me access the infected image. I may start another thread for additional setup tips. For now I feel pretty good about the server now I've just changed glue for my domains. Logs are slowing to a stop on the old box. |
I have been looking over the data and I have a few things to add. I will also forward this information to Unspawn and ask him to double check, but here are some things I would like to ask you to look into and / or comment on too.
1) In the file with the strings output, I am curious as to the nature of the file. I am operating under the impression that it is the output of the strings command on some of the compromised binaries and libraries. If this is the case, please look on or about line 8710, where there are references to user names and IP addresses. If, like I said, this is from the compromised file(s) the attack would seem to be awfully targeted, suggesting or implicating an information leak. 2) in the FD3 file, around line 10606, there seems to be a change in the attack style. The brute force attempts on your SSH server are happening at a level that I find astounding. The fact that your security firm didn't provide a defense against this and the fact that the hosting provider is not reacting to this kind of traffic surprises me. What is interesting is that shows what appears to be a lockout on the plesk database along with an inability to connect to the database. I am wondering if this was a potential crack in the foundation that let something in? 3) How long has this server been online? You are getting WAY too many DNS queries for 3rd party (recursive) look ups for a new system. They are almost all coming from the same Italian domain too. It makes me think that you have were sold a bad IP or bad server that had a known history and were ill prepared for it. 4) Looking in the same file, fd3, Line 17231, it Looks like proftp may have gotten hit with access from user 0 (0 = root?) Was this a potential crack? 5) File FD5, I notice the spam filter running through a socket connection in /tmp. /tmp wouldn't be my first choice for this sort of thing since it is typically 777 permission. My concern would be possible injection point. 6) Same file, look on or about 3340, it looks like the mailer system may have been compromised as it looks to be sending spam based upon the email address. If I am reading this correctly, it looks like these are outgoing messages as the SENT=OK was set. I didn't follow the SMTP ID's closely enough to confirm, but it caught my attention. Again, as with the DNS, for a server that (I think) is rather new, you are getting hit awfully hard and the defense both locally and at the hosting provider level are woefully inadequate for dealing with it. |
Micxz, I was looking over some of the information and I came across something that was potentially interesting.
Does the user yelapan mean anything to you as well as the IP 187.173.218.171, which is dsl-187-173-218-171-dyn.prod-infinitum.com.mx. which appears to be a residential, or at least a DSL with DHCP provider in Mexico? |
Quote:
The only users with login shells at the time of entry was me, admin (used for Plesk), root (of course you could obtain a shell from some exploit) I did try on the new box (well re-imaged same IPs) to replicate the adding a user via cron from the Plesk admin interface. I was able to escalate myself from admin to user micxzsendmail (uid 0) within 5 mins by adding a cron job for root. But when I tired to logging in failed as root is now only allowed key login or use su of course. Maybe Plesk should be notified? Changing all the passwords sure does make everyone catch up on the bills. I got paid for over four hosting accounts for people wanting to login but didn't want to ask for support until they're paid up. I'm rich! he he. Also I would like to add though maybe unrelated. There was a users pending in my billing system which is older software. I got the client signup on 06/26/2011 and even though the account was not setup (no payment or verificati0n). If that user somehow got access to that DB there are usernames and passwords for FTP accounts like above (no shells/chrooted ftp). Then right around the time of re-imaging the tainted box I get another signup 07/03/2011 and another 07/05/2011 this time he actually paid me via paypal and emailed asking when his account will be setup. All three from different public emails and IPs in South Africa. They are suspect and I believe they wanted to get "back on". Just for the record I do not setup new accounts right off. I have people verify email first, I call the phone #, and have them present a valid referral (e.g. Your mom told me to host my site here.) |
All times are GMT -5. The time now is 07:45 PM. |