I got rootkitted, recommendations for recovery?

haertig · 05-28-2009, 09:43 PM

Looks like I got rootkitted!!! Best I can tell from my analysis below it happened on Mar 27, 2009 at about 15:19 CDT.

This is actually my parents computer, and I am ssh'ing into it remotely (I live a two day drive away). I will be there in about two weeks to reinstall the OS from scratch, but in the meantime I'd like to do what cleanup I can remotely. Their ISP will be shutting off service within 24 hours if I don't clear up the outgoing ssh connection abuse. I'd like to have my parents computer running safely, hobbling along, until I can get there to to do a bare-metal install. No other way to 100% for sure recover from a rootkit.

I have no idea how this happened. Ubuntu 8.10 (whatever the "Ibis" version is) standard install. sshd listening, but only allows three specific userids in, and mandates pubkey authentication - no passwords. The only other server running that I'm suspicious of is vino, which I use in combination with FreeNX to get into their GUI when that need arises.

My initial guess at quick-pseudo-cleanup is to kill that new crontab that was created by the rootkit and rm -rf "/var/tmp/ /" and tell them NOT to reboot until I get there. Maybe add an iptables entry denying all outgoing ssh.

Any other suggestions?

thanks!!!

Code:

root> netstat -tanp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      4750/sshd
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      4797/cupsd
tcp        0    300 70.124.xxx.xxx:22        76.120.xxx.xxx:1812      ESTABLISHED 5998/sshd: doris [p  (this is ME, incoming ssh)
tcp        1      0 70.124.xxx.xxx:36232     205.128.92.126:80       CLOSE_WAIT  5814/gweather-apple
tcp6       0      0 ::1:5900                :::*                    LISTEN      5889/vino-server
tcp6       0      0 :::22                   :::*                    LISTEN      4750/sshd

From chkrootkit:

Searching for suspicious files and dirs, it may take a while...
/usr/lib/xulrunner-1.9.0.5/.autoreg /usr/lib/firefox-3.0.5/.autoreg /lib/modules/2.6.27-7-generic/volatile/.mounted /lib/init/rw/.ramfs

root> ls -l /usr/lib/xulrunner-1.9.0.5/.autoreg /usr/lib/firefox-3.0.5/.autoreg /lib/modules/2.6.27-7-generic/volatile/.mounted /lib/init/rw/.ramfs
-rw-r--r-- 1 root root 0 2009-05-28 15:36 /lib/init/rw/.ramfs
-rw-r--r-- 1 root root 0 2009-05-28 15:36 /lib/modules/2.6.27-7-generic/volatile/.mounted
-rw-r--r-- 1 root root 0 2008-12-21 10:16 /usr/lib/firefox-3.0.5/.autoreg
-rw-r--r-- 1 root root 0 2008-12-21 10:16 /usr/lib/xulrunner-1.9.0.5/.autoreg

From chkrootkit:

Searching for common ssh-scanners default files...
/var/tmp/ /.ssh/.S008/pscan2
/var/tmp/ /.ssh/.S008/vuln.txt
/var/tmp/ /.ssh/.S008/ssh-scan

From chkrootkit:

Checking `sniffer'... lo: not promisc and no packet sniffer sockets
eth0: PACKET SNIFFER(/sbin/dhclient3[5804])


root> ls -l /var/spool/cron/crontabs
total 4
-rw------- 1 doris crontab 235 2009-05-27 15:21 doris


root> cat /var/spool/cron/crontabs/doris
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (cron.d installed on Wed May 27 15:21:00 2009)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
* * * * * /var/tmp/ /.ssh/.bash/update >/dev/null 2>&1


root> find "/var/tmp/ /" -ls
410441    4 drwxr-xr-x   3 doris    doris        4096 May 27 15:19 /var/tmp/\ /
410490    4 drwxr-xr-x   4 doris    doris        4096 May 27 15:20 /var/tmp/\ /.ssh
410492   12 -rwxr-xr-x   1 doris    doris        9780 Jul 25  2006 /var/tmp/\ /.ssh/clean
410493    4 drwxr-xr-x   2 doris    doris        4096 May 28 07:39 /var/tmp/\ /.ssh/.S008
410502    4 -rwxr-xr-x   1 doris    doris         265 Nov 24  2004 /var/tmp/\ /.ssh/.S008/gen-pass.sh
410507   16 -rw-r--r--   1 doris    doris       13021 May 28 07:39 /var/tmp/\ /.ssh/.S008/209.9.pscan.22
410614   28 -rwxr-xr-x   1 doris    doris       25503 May 28 07:39 /var/tmp/\ /.ssh/.S008/pscan2
410501    4 -rwxr-xr-x   1 doris    doris        4019 May 15  2007 /var/tmp/\ /.ssh/.S008/start
410612    0 -rw-r--r--   1 doris    doris           0 May 28 05:08 /var/tmp/\ /.ssh/.S008/vuln.txt
410509    4 -rwxr-xr-x   1 doris    doris          50 Apr  7  2007 /var/tmp/\ /.ssh/.S008/a6
410510   12 -rw-------   1 doris    doris       12288 Jan 18  2007 /var/tmp/\ /.ssh/.S008/.vuln.txt.swp
410613    4 -rwxr-xr-x   1 doris    doris         159 Jan 10  2006 /var/tmp/\ /.ssh/.S008/test.sh
410644   16 -rw-r--r--   1 doris    doris       13021 May 28 07:39 /var/tmp/\ /.ssh/.S008/mfu.txt
410609    4 -rwxr-xr-x   1 doris    doris         208 Jan 10  2006 /var/tmp/\ /.ssh/.S008/a5
410505  192 -rw-------   1 doris    doris      190654 May 28 07:39 /var/tmp/\ /.ssh/.S008/nohup.out
410511    8 -rwxr-xr-x   1 doris    doris        7213 Jan 10  2006 /var/tmp/\ /.ssh/.S008/a4
410504    4 -rwxr-xr-x   1 doris    doris         527 May 15  2007 /var/tmp/\ /.ssh/.S008/a1
410506    4 -rwxr-xr-x   1 doris    doris         208 Jan 10  2006 /var/tmp/\ /.ssh/.S008/a
410499 1068 -rwxr-xr-x   1 doris    doris     1089450 Aug  6  2008 /var/tmp/\ /.ssh/.S008/pass_file
410643   12 -rwxr-xr-x   1 doris    doris        9940 May 27 15:35 /var/tmp/\ /.ssh/.S008/189.1.pscan.22
410494  832 -rwxr-xr-x   1 doris    doris      846832 May 27 16:36 /var/tmp/\ /.ssh/.S008/ssh-scan
410503   24 -rwxr-xr-x   1 doris    doris       22354 Dec  1  2004 /var/tmp/\ /.ssh/.S008/common
410500    4 -rwxr-xr-x   1 doris    doris         215 Jan 10  2006 /var/tmp/\ /.ssh/.S008/a2
410611  176 -rwxr-xr-x   1 doris    doris      172060 May 28 07:39 /var/tmp/\ /.ssh/.S008/pico
410615    4 drwxr-xr-x   3 doris    doris        4096 May 28 09:50 /var/tmp/\ /.ssh/.bash
410641    4 -rw-------   1 doris    doris           5 May 27 15:21 /var/tmp/\ /.ssh/.bash/m.pid
410631   24 -rwxr-xr-x   1 doris    doris       22882 Oct 30  2006 /var/tmp/\ /.ssh/.bash/m.help
410629    4 -rwxr-xr-x   1 doris    doris         713 Jan  7 15:25 /var/tmp/\ /.ssh/.bash/start
410652    4 -rw-r--r--   1 doris    doris        1026 May 28 09:50 /var/tmp/\ /.ssh/.bash/m.ses
410630  488 -rwxr-xr-x   1 doris    doris      492135 Oct 30  2006 /var/tmp/\ /.ssh/.bash/bash
410646    4 -rw-r--r--   1 doris    doris         970 May 28 09:50 /var/tmp/\ /.ssh/.bash/justin.seen
410635    4 -rw-r--r--   1 doris    doris         225 May 28 09:50 /var/tmp/\ /.ssh/.bash/70.124.xxx.xxx.user
410637    4 -rw-r--r--   1 doris    doris         152 May 28 09:50 /var/tmp/\ /.ssh/.bash/70.124.xxx.xxx.user3
410491    4 -rw-r--r--   1 doris    doris          14 May 27 15:20 /var/tmp/\ /.ssh/.bash/vhosts
410638    4 -rw-r--r--   1 doris    doris          22 May 27 15:21 /var/tmp/\ /.ssh/.bash/mech.dir
410639    4 -rw-r--r--   1 doris    doris          55 May 27 15:21 /var/tmp/\ /.ssh/.bash/cron.d
410640    4 -rwxr--r--   1 doris    doris         193 May 27 15:21 /var/tmp/\ /.ssh/.bash/update
410634    4 -rw-r--r--   1 doris    doris        1663 May 27 15:21 /var/tmp/\ /.ssh/.bash/m.set
410636    4 -rw-r--r--   1 doris    doris         152 May 28 09:50 /var/tmp/\ /.ssh/.bash/70.124.xxx.xxx.user2
410653    4 -rw-r--r--   1 doris    doris        1043 May 28 09:50 /var/tmp/\ /.ssh/.bash/m.lev
410371    0 -rw-r--r--   1 doris    doris           0 May 28 09:50 /var/tmp/\ /.ssh/.bash/miguel.seen
410632    4 -rwxr-xr-x   1 doris    doris          29 Oct 30  2006 /var/tmp/\ /.ssh/.bash/run
410616    8 -rwxr-xr-x   1 doris    doris        5115 Jan  7 15:17 /var/tmp/\ /.ssh/.bash/inst
410628    4 -rwxr-xr-x   1 doris    doris         317 Oct 30  2006 /var/tmp/\ /.ssh/.bash/autorun
410645    4 -rw-r--r--   1 doris    doris         970 May 28 09:50 /var/tmp/\ /.ssh/.bash/gustavo.seen
410642    8 -rw-r--r--   1 doris    doris        6874 May 28 09:50 /var/tmp/\ /.ssh/.bash/LinkEvents
410617    4 drwxr-xr-x   2 doris    doris        4096 Jan  7 15:28 /var/tmp/\ /.ssh/.bash/r
410624    4 -rw-r--r--   1 doris    doris         830 Oct 30  2006 /var/tmp/\ /.ssh/.bash/r/rkicks.e
410626    4 -rw-r--r--   1 doris    doris         519 Oct 30  2006 /var/tmp/\ /.ssh/.bash/r/rnicks.e
410625    4 -rw-r--r--   1 doris    doris        1465 Oct 30  2006 /var/tmp/\ /.ssh/.bash/r/rversions.e
410623    4 -rw-r--r--   1 doris    doris        3982 Oct 30  2006 /var/tmp/\ /.ssh/.bash/r/rinsult.e
410618    4 -rw-r--r--   1 doris    doris        3651 Oct 30  2006 /var/tmp/\ /.ssh/.bash/r/rsignoff.e
410622    8 -rw-r--r--   1 doris    doris        5195 Oct 30  2006 /var/tmp/\ /.ssh/.bash/r/raway.e
410620    4 -rw-r--r--   1 doris    doris        2495 Oct 30  2006 /var/tmp/\ /.ssh/.bash/r/rpickup.e
410619   60 -rw-r--r--   1 doris    doris       55316 Oct 30  2006 /var/tmp/\ /.ssh/.bash/r/rsay.e
410621   60 -rw-r--r--   1 doris    doris       55316 Oct 30  2006 /var/tmp/\ /.ssh/.bash/r/rtsay.e
410627  172 -rwxr-xr-x   1 doris    doris      167964 Jun 27  2007 /var/tmp/\ /.ssh/.bash/pico
410633   28 -rwxr-xr-x   1 doris    doris       28489 Oct 30  2006 /var/tmp/\ /.ssh/.bash/xh

haertig · 05-28-2009, 10:45 PM

Here it is, straight from /var/log/auth.log:

Code:

May 27 15:16:09 doris sshd[9202]: reverse mapping checking getaddrinfo for 189-109-33-18.customer.tdatabrasil.net.br [189.109.33.18] failed - POSSIBLE BREAK-IN ATTEMPT!
May 27 15:16:11 doris sshd[9202]: Accepted password for doris from 189.109.33.18 port 41638 ssh2
May 27 15:16:35 doris passwd[9224]: pam_unix(passwd:chauthtok): password changed for doris

My question is WHY did sshd allow password authentication? My config should have ruled out this authentication method. I believed I had things configured for sshd to ONLY allow pubkey authentication, no passwords under any circumstance. What did I do wrong in the config below?

[edit]
p.s. - Another interesting thing... the IP address that succesfully hacked in as userid 'doris' apparently guessed this userid on it's first attempt, since the above auth.log lists the ONLY entries for the hackers IP address in the logs. The hacker guessed the correct userid on the first try?! You can see in my config below that only three userids are allowed to ssh into the server, 'doris' being one of those. 'doris' is not exactly a common userid that I would expect to be guessed on the first try, but that's what my auth.log appears to show.
[/edit]

Code:

Port 22
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
UsePrivilegeSeparation yes
KeyRegenerationInterval 3600
ServerKeyBits 768
SyslogFacility AUTH
LogLevel INFO
LoginGraceTime 120
PermitRootLogin no
StrictModes yes
AllowUsers david doris nx
RSAAuthentication no
PubkeyAuthentication yes
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server
UsePAM no

win32sux · 05-28-2009, 11:25 PM

I've merged your other post into this thread, in order to keep the discussion in one place.

haertig · 05-28-2009, 11:58 PM

Quote:

Originally Posted by win32sux

I've merged your other post into this thread, in order to keep the discussion in one place.

OK. That's fine with me! :-)

I initially made two different threads because I thought of these as two separate questions: (1) How do I recover from a rootkitting, and (2) How do I configure sshd to not allow password authentication. True, in my case, (2) is what led to (1) and I should not have made the two thread titles so similar. I initially thought I'd get in more trouble for combining two topics in one thread, so I decided to go for two. Man I just can't win! ;-)

billymayday · 05-29-2009, 12:05 AM

I believe PasswordAurthentication defaults to yes, although I'm not sure what it authenticates against once UsePAM is set to no.

haertig · 05-29-2009, 12:47 AM

Quote:

Originally Posted by billymayday

I believe PasswordAurthentication defaults to yes...

Oh man, what happened to my PasswordAuthentication line in sshd_config?! I must have accidently deleted that during some editing session. Yeah - you spotted it. I was so busy looking at what WAS in the config file, that I missed that important part that WAS NOT in there! I hate stupid mistakes on my part. Thanks!

unSpawn · 05-29-2009, 02:00 AM

With regards to the title "recommendations for recovery" and your current findings, I'd say no. The machines security appears to be compromised but no evidence of a rootkit is shown (file listing shows "XH", a process hider, and the regular layout of IRC mech/SSH scan tools you would find in most situations where a machine got compromised by crackers looking for easy preys) and therefore there is no valid, compelling reason to keep it running. So the end goal must be a complete version of the three R's: reformat, repartition and reinstall from scratch plus proper host hardening. Investigation is optional and can be done later on if you made a bit-by-bit disk backup. Since you're not able to deal with it right now and if the machine must be used what you could do is (have them) save detailed process, network and open files listings, is prep an USB stick and let them boot a Live CD/USB for the time being, reading configuration files from and saving their data on the stick.

* If you find time to investigate you might want to start by reading the Intruder Detection Checklist (CERT): http://web.archive.org/web/200801092...checklist.html.