[SOLVED] suspected hacking on RHEL 6.2

cortexa · 02-01-2013, 09:13 AM

Hi there,

I have reason to believe my system has been hacked and would like some advice about what to do next. I should preface this post by pointing out that I am rather inexperienced with Linux systems; I have been using RedHat for some years with a very limited set of applications, but up until now have had full IT support. My current institution does not support Linux so I'm trying to resolve these issues alone, and am not really equipped to do so.

I'm running RHEL 6.2 on a single workstation connected to a university network. In recent days I began noticing some strange errors and warning messages when performing ordinary operations, including the following:

1) ls returns the warning:

Quote:

ls: unrecognized prefix: rs
ls: unparsable value for LS_COLORS environment variable

2) top returns the error:

Quote:

top: error while loading shared libraries: libncurses.so.4: cannot open shared object file: No such file or directory

3) opening a new terminal returns the warning:

Quote:

while Unknown HZ value! (20) Assume 100.
Warning: /boot/System.map has an incorrect kernel version.

I had changed nothing on the system, and several online posts on these errors led me to believe that my system may have been compromised, so I ran chkrootkit and rkhunter which detected several rootkits and infected files.

Summary output from rkhunter log file:

Quote:

[14:36:37] System checks summary
[14:36:37] =====================
[14:36:37]
[14:36:37] File properties checks...
[14:36:37] Files checked: 140
[14:36:38] Suspect files: 14
[14:36:38]
[14:36:38] Rootkit checks...
[14:36:38] Rootkits checked : 317
[14:36:38] Possible rootkits: 3
[14:36:38] Rootkit names : cb Rootkit, SHV4 Rootkit, SHV5 Rootkit
[14:36:38]
[14:36:38] Applications checks...
[14:36:38] Applications checked: 4
[14:36:38] Suspect applications: 2

I've read many posts on various linux forums suggesting methods of removing/reinstalling the infected files, but many of them are contradictory, or just over my head. I'm very wary of following any of these suggestions as my inexperience might result in my making a catastrophic error. Would my best bet be to reinstall RedHat? I have the original installation discs.

I'm also not sure whether these issues have the ptential to infect external media attached to my computer. Is this a reasonable fear? Is there a way of checking this?

Finally, and possibly most importantly, what security steps - other than regularly changing passwords - should I take to prevent this happening again? I have read a number of posts on this topic but wasn't able to decipher most of them.

I'd really appreciate any assistance you can offer in this matter. Please let me know if you would like me to provide any further information.

Thanks!

unSpawn · 02-01-2013, 10:12 AM

Quote:

Originally Posted by cortexa

I have reason to believe my system has been hacked and would like some advice about what to do next.

Welcome to LQ, even if on such a sad occasion, hope you like it here.

Quote:

Originally Posted by cortexa

I'm also not sure whether these issues have the ptential to infect external media attached to my computer. Is this a reasonable fear? Is there a way of checking this?

No. Even though it may contain "infected" tools a rootkit (if it is) is not a virus.
We'll address your questions wrt reinstalling and hardening later on.
But first things first.

Quote:

Originally Posted by cortexa

I had changed nothing on the system

Installing software counts as a change too but otherwise: good.
0. First of all please don't reboot the machine (but please indicate if you already did).
1. Please do not use this machine (or let it be used) anymore: use another one. If it's on the same subnet please ensure it's clean before using it.
2. If you have a router then (if possible: log and) block traffic to and from the machine. If you haven't or can't then if the machine has a wired connection then disconnect the cable. If it's wireless then bring the interface down.
3. Please attach the full rkhunter log file to your reply (do obfuscate your host name and IP address). If you do not feel like doing that then attach it to an email to me RSN. (As you can read from the Rootkit Hunter documentation) I'm at unspawn at hushmail dot com ;-p

cortexa · 02-01-2013, 10:47 AM

Quote:

Welcome to LQ, even if on such a sad occasion, hope you like it here.

Thank you! And thank you for replying.

Quote:

Installing software counts as a change too but otherwise: good.

I haven't installed anything in several weeks.

Quote:

0. First of all please don't reboot the machine (but please indicate if you already did).

I'm afraid I already did, before I realised what was going on.
OK, I've disconnected the network cable and attached the rkhunter log file. Thanks for any help you can offer!

unSpawn · 02-01-2013, 11:48 AM

Quote:

Originally Posted by cortexa

I haven't installed anything in several weeks.

Not exactly good, kernel 2.6.32 alone shows 13 CVEs but OK, what's done is done.

Quote:

Originally Posted by cortexa

I'm afraid I already did, before I realised what was going on.

It's just that rebooting means volatile process, open files, utmp and network connection data is gone.

Quote:

Originally Posted by cortexa

OK, I've disconnected the network cable and attached the rkhunter log file.

Thanks. The log file is rather clear about things:

Code:

[14:34:40] Warning: File '/sbin/ifconfig' has the immutable-bit set.
[14:34:47] Warning: File '/bin/ls' has the immutable-bit set.
[14:34:47] Warning: File '/bin/netstat' has the immutable-bit set.
[14:34:48] Warning: File '/bin/ps' has the immutable-bit set.
[14:34:52] Warning: File '/usr/sbin/lsof' has the immutable-bit set.
[14:34:55] Warning: File '/usr/bin/find' has the immutable-bit set.
[14:34:57] Warning: File '/usr/bin/md5sum' has the immutable-bit set.
[14:34:58] Warning: File '/usr/bin/pstree' has the immutable-bit set.
[14:35:01] Warning: File '/usr/bin/top' has the immutable-bit set.

This is the first tell-tale sign. Sure you could set the immutable bit yourself but then you would know you did, right?

Code:

[14:35:17] Warning: cb Rootkit                               [ Warning ]
[14:35:17]          File '/lib/libproc.so.2.0.6' found

[14:35:35] Warning: SHV4 Rootkit                             [ Warning ]
[14:35:35]          File '/lib/lidps1.so' found
[14:35:35]          File '/lib/libproc.a' found
[14:35:35]          File '/lib/libproc.so.2.0.6' found
[14:35:35]          File '/usr/include/file.h' found
[14:35:35]          File '/usr/include/hosts.h' found
[14:35:35]          File '/usr/include/log.h' found
[14:35:35]          File '/usr/include/proc.h' found

[14:35:36] Warning: SHV5 Rootkit                             [ Warning ]
[14:35:36]          File '/etc/sh.conf' found
[14:35:36]          File '/lib/libproc.a' found
[14:35:36]          File '/lib/libproc.so.2.0.6' found
[14:35:36]          File '/lib/lidps1.so' found
[14:35:36]          File '/lib/libsh.so/bash' found
[14:35:36]          File '/usr/include/file.h' found
[14:35:36]          File '/usr/include/hosts.h' found
[14:35:36]          File '/usr/include/log.h' found
[14:35:36]          File '/usr/include/proc.h' found
[14:35:37]          File '/lib/libsh.so/shhk' found
[14:35:37]          File '/lib/libsh.so/shhk.pub' found
[14:35:37]          File '/lib/libsh.so/shrs' found
[14:35:37]          File '/usr/lib/libsh/.bashrc' found
[14:35:37]          File '/usr/lib/libsh/shsb' found
[14:35:37]          File '/usr/lib/libsh/hide' found
[14:35:37]          File '/usr/lib/libsh/.sniff/shsniff' found
[14:35:37]          File '/usr/lib/libsh/.sniff/shp' found
[14:35:37]          Directory '/lib/libsh.so' found
[14:35:37]          Directory '/usr/lib/libsh' found
[14:35:37]          Directory '/usr/lib/libsh/utilz' found
[14:35:37]          Directory '/usr/lib/libsh/.backup' found

Unsurprisingly somebody used a default kit layout ;-p

Code:

[14:36:15] Warning: Checking for possible rootkit strings    [ Warning ]
[14:36:15]          Found string 'fucknut' in file '/sbin/ttymon'. Possible rootkit: SHV5 Rootkit
[14:36:16]          Found string 'lamersucks' in file '/sbin/ttymon'. Possible rootkit: SHV5 Rootkit
[14:36:16]          Found string 'skillz' in file '/sbin/ttymon'. Possible rootkit: SHV5 Rootkit
[14:36:16]          Found string 'propert of SH' in file '/sbin/ttyload'. Possible rootkit: SHV5 Rootkit
[14:36:16]          Found string 'ttyload' in file '/etc/inittab'. Possible rootkit: SHV5 Rootkit

Forensic basics teaches you to not trust anything you (think you) see but to confirm it. So while extended attributes and process or file names are nice as first indication it's good to have 'strings' confirm it.

As an encore you seem to have company:

Code:

[14:36:25] Info: Found password file: /etc/passwd
[14:36:25]   Checking for root equivalent (UID 0) accounts   [ Warning ]
[14:36:25] Warning: Account 'oracle' is root equivalent (UID = 0)
[14:36:25] Warning: Account 'plesk' is root equivalent (UID = 0)

OK. So that's the log. Now you have a choice:

If you want to find out how / how long this has been going on you should save output and copy files to another machine or external storage for processing:
- /var/log/wtmp*, /var/log/btmp* and /var/log/lastlog,
- all system and daemon logs,
- a list of all files with ownership and MAC time stamps (as in 'find / -xdev printf "\"%p\":%U:%G:%m:%T%:%A@:%C:%Z\n" 2>&1|tee /mnt/mountpoint/find.txt;'),
- all users shell history files,
- router logs (if any).
Elif you want to move on then just say so.

cortexa · 02-01-2013, 02:06 PM

Quote:

Not exactly good, kernel 2.6.32 alone shows 13 CVEs but OK, what's done is done.

OK, I guess that's part one of the answer to my 'what should I do differently in future' question...

Quote:

OK. So that's the log. Now you have a choice:

If you want to find out how / how long this has been going on you should save output and copy files to another machine or external storage for processing:
- /var/log/wtmp*, /var/log/btmp* and /var/log/lastlog,
- all system and daemon logs,
- a list of all files with ownership and MAC time stamps (as in 'find / -xdev printf "\"%p\":%U:%G:%m:%T%:%A@:%C:%Z\n" 2>&1|tee /mnt/mountpoint/find.txt;'),
- all users shell history files,
- router logs (if any).
Elif you want to move on then just say so.

To be honest, I just want to move on. This is the only linux box I have, so I'm falling further behind on work every hour it's out of commission.
What's my next move?

Thanks for all your help so far!

unSpawn · 02-01-2013, 06:34 PM

OK, moving on. In short: Inform, backup, install, harden.

I. Inform Rootkit installations are quite exceptional these days. Compromised systems can be used as springboard. Without further analysis there is no time line or point of origin. If this machine is owned by the institution or is part of an institutional network then the IT dept should be informed of the breach so they can decide if an investigation is needed or not. If your machine was used by other people (local account or as authenticated user of a service) then inform them of the breach as well.

II. Backup If you kept regular backups fine, else backup 0) only human readable configuration files from /etc (and other common locations like /usr/local/etc) 1) your /home/ directory contents and 2) the whole of /var to external storage. The /etc configuration files (no passwd, group or shadow!) serve as reference, meaning you copy relevant lines but not complete files and not without inspection. Your /home/ directory should contain configuration files and documents but not any binaries. Backing up /var is for reference too (wrt log analysis). If you have a router then change the admin password and ensure any port forwarding is disabled for now.

III. Installation Before you install an (any) OS you should be aware of the basics. CentOS documentation resides at http://www.centos.org/docs/ and TUV (aka RHEL) documentation resides at https://access.redhat.com/knowledge/.../?locale=en-US. Review at least the Installation and Deployment guides. If your IT department has installation then follow those.
Use the CentOS 6.3 DVD (or netboot ISO) and let the installer reformat the partitions. Select workstation mode, meaning an OS without network accessible services. When creating accounts ensure you change the password for root and for your personal account. Do not install software you do not need right now.

IV. Hardening If your IT department has hardening guidelines follow those. Check the firewall. For a quick can run 'yum install -y system-config-firewall-tui && system-config-firewall-tui' and ensure no unwanted ports are open[1]. Check SELinux is activated (system-config-securitylevel). Check unwanted and unnecessary services are deactivated (chkconfig or system-config-services). Check basic authentication (the "options" section of system-config-authentication). Check user accounts (passwd and chage or system-config-users) for password aging, shell usage[2].
Install or check availability of: iptables, audit service, Sudo, AIDE (or tripwire), GNU Tiger (www.nongnu.org/tiger/), Logwatch (set detail to "High" and configure a non-root email address), fail2ban (the latter requires the EPEL repo. Run Tiger and review its report. Review how you're going to be alerted of crucial software updates. Check and test your backup process. Confirm the warnings and alerts you're sending goes to an email address where somebody actually regularly reads email. Otherwise alerting makes no sense.
Review the RHEL 6 Security Guide, the NSA Hardening Tips For Default Installation of RHEL 5 and the NSA Guide to the Secure Configuration of Red Hat Enterprise Linux 5. Finish off by doing another Tiger run and comparing the reports, address any leftover issues and if you want to check against the Cisecurity RHEL 5 benchmark.

V. The Rest ;-p Now you have a basic workstation you can add network services to. Before you do that this would be a good moment to make a baseline backup.

That's about it. Any questions just ask.

[1] A minimalistic workstation firewall (/etc/sysconfig/iptables) could look like:

Code:

*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -m state --state INVALID -j DROP 
-A INPUT -p icmp -m icmp --icmp-type any -j ACCEPT 
-A INPUT -j REJECT --reject-with icmp-host-prohibited 
-A OUTPUT -o lo -j ACCEPT 
-A OUTPUT -j ACCEPT 
COMMIT

[2] Change the passwords for any on-line services you use while you're at it.

cortexa · 02-05-2013, 09:53 AM

Hi, thanks for all that. I've reinstalled RedHat and am working my way through the hardening measures, but I'm having trouble running tiger. I keep getting the error

Quote:

./config: line 398: tempfile: command not found
--ERROR-- [init006e] `log/' does not exist (file LOGDIR).

The config script seems to be calling a command (tempfile) that doesn't exist, and the README and USING files were no help. Have you come across this before?

Thanks.

unSpawn · 02-05-2013, 10:35 AM

Quote:

Originally Posted by cortexa

Hi, thanks for all that. I've reinstalled RedHat and am working my way through the hardening measures, but I'm having trouble running tiger. I keep getting the error

Yeah, unfortunately I didn't log how I fixed it though but basically what they reference to as 'tempfile' in TigerInstallDir/config and TigerInstallDir/util/genmsgidx we would call 'mktemp'. Could try and substitute it.

Quote:

Originally Posted by cortexa

The config script seems to be calling a command (tempfile) that doesn't exist, and the README and USING files were no help. Have you come across this before?

For any warning messages consult TigerInstallDir/doc/config.txt instead (that's the directory TigExp gets its explanations from if you run 'tiger -e'):

Code:

%init006e
An input file required for performing a test is not available.  This
indicates that there is a configuration error.

Check your tigerrc for the TigerLogDir directive or run tiger with "-l /some/path".