Possible server intrusion
 

Hi fellow forum members, I suspect one of my servers has been compromised by a local user. Here's some evidence I've gathered.

A setuid file was brought to my attention, it however appears to be the same as bash and since bash drops privileges, it can't be used to gain root:
ls -la /sbin/.sash
---s--s--x 1 root root 2197406 Sep 19 04:35 .sash
$ ls -la /bin/bash
-rwxr-xr-x 1 root root 2197406 Jul 20 22:34 /bin/bash
$ sha256sum /bin/bash
add83decc3326c666cada75e1c189151853f025dd08b82cca06ab2a02a478dcf /bin/bash
$ sha256sum /sbin/.sash
add83decc3326c666cada75e1c189151853f025dd08b82cca06ab2a02a478dcf /sbin/.sash

All users have .bash_history chattr +a and owned like so:
-rw--w---- 1 root john 1317 Jul 16 04:58 .bash_history

This means they cannot get rid of their bash_history. However, one user's history is missing.
$ ls -la /home/freeshell
total 4280
drwx-----x 6 freeshell freeshell 4096 Sep 19 04:51 .
d--x--x--x 345 root root 12288 Sep 21 13:51 ..
-rw-r--r-- 1 freeshell freeshell 33 Jul 10 22:36 .bash_logout
-rw-r--r-- 1 freeshell freeshell 176 Jul 10 22:36 .bash_profile
-rw-r--r-- 1 freeshell freeshell 654 Sep 2 16:43 .bashrc
drwxr-xr-x 2 freeshell freeshell 4096 Jul 14 06:15 .mc
-rw-r--r-- 1 freeshell freeshell 658 Jul 10 22:36 .zshrc
drwxr-xr-x 10 freeshell freeshell 4096 Jul 14 06:02 eggdrop
-rw-r--r-- 1 freeshell freeshell 3111941 Jul 14 06:03 eggdrop-compiled.tgz
-rw-r--r-- 1 freeshell freeshell 1208409 Jul 14 06:16 eggdrop-installed.tgz
drwxr-xr-x 10 freeshell freeshell 4096 Jul 14 06:01 eggdrop1.6.19
drwxr-xr-x 2 freeshell freeshell 4096 Aug 9 21:39 public_html

Login details:
$ last -a freeshell
freeshell pts/30 Sat Sep 19 04:39 - 04:50 (00:11) ip-67-189.interbild.net
freeshell pts/41 Sat Sep 19 04:16 - 04:36 (00:20) ip-67-189.interbild.net

wtmp begins Sat Sep 19 01:03:45 2009

This IP leads to the following member account on a forum: http://www.cyberarmy.net/~r3v3r
Evidence of this: http://www.kocccka.hu/normal-opers.log
2009-06-21 13:28:09 <&L> 2009-06-19 13:23:03 !irc.phoenixchat.hu *** Notice -- Client exiting: rotacioskapa (r3v3r@ip-67-189.interbild.net) [SIGTERM: leaving]

Any guidance as what to do from here onwards is appreciated. I considered leaving the server online and the user account in tack, just so I could watch the attacker. Is this a good idea?

Yeah I don't know what to make of that. I tried copying bash over and making it suid, then tested the euid, etc with PS, and everything looked 'secure.'

I'd be worried though, the fact that a file was placed in /sbin means that someone has root access (if you know nothing you did put it there).

Yeh, bash drops root privileges, maybe the attacker forgot about this?

ShellPwn

Egg-drop is a well known program and almost certainly is a hack. I would not personally leave it there. Your time would be better spent reimaging the server and finding out how this program normally gains access and putting preventative measures in place. It can be used as part of a botnet.

Eggdrop is not a hack or malicious program, many of my users use it. I operate a shell server that provides IRC bouncing and hosting people's IRC bots (not necessarily malicious, but I do find the odd one).

Also, this user doesn't have any processes running.

Sounds like you definitely got rooted, especially if he deleted a file that was chattr +a and put a file in /sbin, which is only root writable.

Also block the IP associated with that user.

Indeed, but the fact that 0) it was dropped in (root-owned) /sbin (recently), 1) has a filename starting with a dot, 2) the users shell history is missing and 3) login details not matching home directory MAC times is bad news.


No, it is a bad idea (in fact it is as bad as telling you to reinstall the OS w/o telling you to investigate) because as Internet user you are responsible for your machines not harming others: leaving the machine in an exploitable state should not be acceptable for you and is not for us.

Here's in short what you should do to help us help you determine what goes on.
* Bear in mind that since you indicated being responsible for multiple servers to check the others for possible compromises as well.

0. Start reading: it helps you focus on what is important and what is not. This makes things less error-prone and more efficient for you and us. Please read the CERT Steps for Recovering from a UNIX or NT System Compromise: http://www.cert.org/tech_tips/win-UN...ompromise.html for an overview of the major steps we will go through (namely: "Regain control" and "Analyze the intrusion"). Please read the CERT Intruder Detection Checklist: http://web.archive.org/web/200801092...checklist.html. (Note this is an archived copy as CERT in all their wisdom decided to take it down w/o providing proper replacement). Please consider the checklist as leading if you have no Incident Response procedure or work document to work with.

1. Notify users and (upstream) provider if necessary.

2. As root account user list open files (\lsof -P -w -n), process (\ps ax -o ppid,pid,uid,cmd --sort=uid) and network data (\netstat -anpe) listings to a location where you do not overwrite data or pipe data through ssh.
* It would be good to verify the integrity of your binaries before executing commands.
* You may need to prefix paths instead of using backslashes but be careful not to mix in common args used in aliases and such.
* Saving listings is obviously hampered by reboots. Let us know if that is the case.

3. Mitigate the situation. Regain control by shutting down all non-essential services and raising the firewall to only allow SSH traffic to and from your management IP (range). If the machine is local to you and you have any doubts power down the box and only boot with a Live CD unless you tell us the machine is remote.
- Tell us the servers OS and precise release version,
- Tell us if the server was properly kept up to date,
- Tell us when you first noticed this situation,
- Tell us if the server exhibited any odd behaviour in the past,
- Tell us (attach logs?) if you have recent logs from running Chkrootkit / Rootkit Hunter or Aide / Samhain (or Integrit, Osiris or even tripwire) (don't install anything),
- Tell us what services the server provides (exact versions please) and if it is a LAMP machine also what runs on top of Perl/PHP/Ruby (forum, web log, stats) and their exact versions,
- Tell us (attach output?) what the /tmp, /var/tmp and webserver docroot directories hold (find /dirname -ls),

4. After you've answered those questions (do not install or delete anything) we'll move on to preparing backups for reference (not reuse) and investigate further using system authentication data (logrotated wtmp), IDS logs, filesystem integrity checkers, package manager (if good enough), all system, daemon and firewall logs, temp files, unusual (setuid root) files, user shell histories. When you report back include any information, hints, hunches or gut feelings you think would help. Please attach logs if possible, else please use BB code tags to preserve formatting and efficient reading. And please ask before doing things if you have any doubts.


* I have little time to spend but I also realize LQ is low on members with proper background knowledge and Incident Response capability (considering only those members who have demonstrated it and as I want to see it, namely: helpful, structured and of a certain quality). If you would like a second opinion you are invited to contact me by email to facilitate bulk exchange of logs et cetera.

It'd be interesting to see a checksum of your sha256sum (when you've taken care of everything else).

I know that, I was just notifying you of that information as it seems whoever it was, forgot about bash dropping privileges. Perhaps the attacker only created a setuid file and then exited? This means patching the system in time/fixing the hole, would be excellent, and
re-installation of the OS would not be required.

I do not plan to notify users currently.

Nothing seemed out of the ordinary here.

The server is running Cent OS 5.3, it was fully updated (still is) and I first noticed the SUID, whilst checking logs at the same time that I posted this thread. The server has never experienced anything odd. The server provides a LAMP with Apache and mod_python (this isn't even used). It runs a forum, radio, wiki and several other web applications. I have to run off now, so thanks for your help.

I forgot to mention, it also runs Sendmail v8.13.8

If you think you can manage things yourself, fine, else your posted information sorely lacks detail.

I wonder what people think. Is it worth paying for services offered by a host that takes this kind of stance towards integrity verification?.. Even after admitting there might be something wrong?

His service is free.

Free hosting for shell bots.....

OK. Regardless of services being free or not, what would services be worth anyway if offered by a host who has more important things to do then make sure integrity is maintained or restored? Would you feel secure using them?