Hello I believe I got a problem with someone tryn to use a rootkit on my box. Can someone tell me if this is true, how they are doing it and how I sould go about fixing this. I started to notice an error message a couple days ago.

[root@x1-6-00-01-03-23-8f-53 root]# service httpd start
sh: line 1: /usr/bin/(swapd): No such file or directory
Starting httpd2: [ OK ]
[root@x1-6-00-01-03-23-8f-53 root]#


It does that on every service as I start, shutdown, and check status.

so I looked looked at my memory and swap space.

[root@x1-6-00-01-03-23-8f-53 root]# cat /proc/meminfo
MemTotal: 515856 kB
MemFree: 109892 kB
Buffers: 147508 kB
Cached: 121972 kB
SwapCached: 0 kB
Active: 250036 kB
Inactive: 60528 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 515856 kB
LowFree: 109892 kB
SwapTotal: 1020024 kB
SwapFree: 1020024 kB
Dirty: 96 kB
Writeback: 0 kB
Mapped: 57804 kB
Slab: 90408 kB
Committed_AS: 188376 kB
PageTables: 1236 kB
VmallocTotal: 507896 kB
VmallocUsed: 2220 kB
VmallocChunk: 505400 kB


This is my Top

11:58pm up 2 days, 10:49, 1 user, load average: 0.18, 0.05, 0.01
105 processes: 103 sleeping, 2 running, 0 zombie, 0 stopped
CPU states: 0.0% user, 0.3% system, 0.2% nice, 99.8% idle
Mem: 503K av, 0K used, 0K free, 0K shrd, 0K buff
Swap: 109K av, 0K used, 0K free 0K cached

I think my box gets to a point and services start to get turned off (or someone else is doing it)

I also noticed something went wrong with ifconfig as well it stopped listing how much data was being transmitted and recieved. As well I have tons of TX overruns all over the place!

eth0 Link encap:10Mbps Ethernet HWaddr 00:01:03:23:8F:53
inet addr:###.###.###.### Bcast:255.255.255.255 Mask:255.255.255.128
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:181784328 errors:781329 dropped:0 overruns:0
TX packets:0 errors:0 dropped:0 overruns:55037868
Interrupt:10 Base address:0xdc00

Please someone help! rootkit or no rootkit something is off!

What do the first few lines of /etc/init.d/httpd say? That is a lot of errors being reported by your NIC. Perhaps try a new CAT5 cable (or a new NIC).

If you're concerned about rootkits, install rkhunter, update it, and run it.

The first line of every script is the shebang. its just not httpd its every service.

I will try that, my connection seems fine. Its reporting that theres not TX packets. not a single one. But I have been moving a couple hundred meg a day, so clearly its works.

As for that rootkit checker, I have tried to install 2 now... I get this.

[root@x1-6-00-01-03-23-8f-53 root]# urpmi rkhunter

http://mirrors.usc.edu/pub/linux/dis...mdk.noarch.rpm
installing /var/cache/urpmi/rpms/rkhunter-1.1.6-2mdk.noarch.rpm
Preparing... ##################################################
1:rkhunter ##################################################
error: unpacking of archive failed on file /usr/sbin/rkhunter;44bb1372: cpio: open failed - Permission denied

Yes I am in as root, yes I have disk space.

[root@x1-6-00-01-03-23-8f-53 root]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md2 17G 3.9G 13G 24% /
/dev/md0 981M 19M 912M 3% /boot


This is getting annoying

Thanks for that lesson, but I'm asking for the first few lines of your script. That may give some insight into why the Bourne shell interpreter is complaining about it.

As for being unable to install rkhunter, you may want to take that up with the package maintainer for whatever distro you're using. An alternative is chkrootkit.

Here is the first 10 lines my httpd

[root@x1-6-00-01-03-23-8f-53 new]# head -10 /etc/init.d/httpd
#!/bin/sh
#
# Startup script for the Apache Web Server
#
# chkconfig: 345 92 8
# description: Apache is a World Wide Web server. It is used to serve \
# HTML files and CGI.
# processname: httpd
# pidfile: /var/run/httpd.pid


I managed to get chkrootkit to work. I got some issues lol



Checking `ifconfig'... INFECTED
Checking `ls'... INFECTED
Checking `netstat'... INFECTED
Checking `ps'... INFECTED
Checking `top'... INFECTED
Checking `aliens'... /tmp/982235016-gtkrc-429249277
Searching for Madalin rootkit default files... Possible Madalin rootkit installed

So all the wierd stuff Is becuase of a rootkit. Any suggestions on how to remove?

I do not know how to remove a rootkit, but you could disconnect the computer from the network and move the infected files to some place make sure your setup is not using them and they are not running as daemons. Next reinstall the files. Before putting the computer on the network, figure out how rootkits can be installed by gathering hacking and security books. After you figure out a way to protect your system from rootkits and update apache, you can put the system back on the network. Hopefully from doing all this that you do not get any more rootkits installed.

Very good information about rootkits http://www.sans.org/reading_room/whi.../linux/901.php

Thanks for the help. I have been spending the last 15 minutes chasing down process ID's and trying to kill them. But they dont seem to exist, or constantly chaning? anyhow I think I know what got them in my box, it wasn't apache, it was mysql I have 11 process hidden back there.

Please read these docs first:
Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html
Steps for Recovering from a UNIX or NT System Compromise: http://www.cert.org/tech_tips/root_compromise.html
Formatting and Reinstalling after a Security Incident (SF): http://www.securityfocus.com/infocus/1692


I do not know how to remove a rootkit, but you could disconnect the computer from the network
Once a box is compromised you must cease using it and cease allowing it to be used. There is no way to remove a rootkit. Yes, you can remove the files but the damage is already done. For newbies there is no "easy" way to figure out what the damage is. You must curb risks and stabilise the situation. If the box is local to you then power the box down and reboot it using a Live CD with security tools like KNOPPIX(-STD), Helix, Auditor or alike. If the box is remote: notify colo people and disconnect the box from the network by stopping unnecessary services and raising the firewall to only allow traffic to and from your management IP (range).

Before you do, if you still have access to the box, make and save off-site: a process listing (ps axwwwe), a connection listing (netstat -an), list open files (lsof -n) and list users (last; w -f; who -a). Also save your password/shadow files and your system and daemon logs.


service httpd start
sh: line 1: /usr/bin/(swapd): No such file or directory
/usr/bin/(swapd) (note the brackets) isn't a regular binary but placed on the system by the intruder. This single sign should be enough to power the box down after getting some info.


This is my Top
No it isn't, because you're focussing on the wrong details (understandable though). A full process listing would have been more helpful: the more info the better.


I also noticed something went wrong with ifconfig as well it stopped listing how much data was being transmitted and recieved.
Most likely fall-out from the intruders ability to hide traffic. Tools used may be a wee bit crude.


error: unpacking of archive failed on file /usr/sbin/rkhunter;44bb1372: cpio: open failed - Permission denied
Probably extended attributes where set (chattr). In any case installing stuff on a compromised box doesn't work and hampers any investigation later on.


Tasks you must do *now* are, in this order:
- make a backup with "dd", mark and store separately for later perusal,
- get the listings and bring the box down,
- prepare to repartition, reformat and reinstall from scratch.

When you're done don't bring the box online but make sure to partially harden the box first. Use different passwords on all accounts, setup an unprivileged user and sudo to avoid network root logins, stop unnecessary services from running (any r* services, mysql, httpd. About the only thing you need is ssh), remove software you don't need *now* and raise your firewall to only accept traffic from your management IP (range) and distribution software repo's. Unpdate your software and keep services off until you properly hardened the box.

Thank you for the process of dealing with this situation. Now the main top for me is Prevention. Is the best process to reduce the chances of this happenening again listed in this order.

1. Firewall policies
2. Update Kernel as soon as available
3. locked up services (I believe they exploited my mysql)
4. Locate and find patches for services (do they have mailing lists?)
5. Install chrookit, and other tools in cron to run daily and email me
6. long password phrases, do this change anything?

thanks

Thank you for the process of dealing with this situation.
I hope with "you" you mean all of us. LQ would be nothing without us *all* chipping in. That said I think people know by now I'm one of the few people here that positively enjoy dealing with situations like these.


Now the main top for me is Prevention. Is the best process to reduce the chances of this happenening again listed in this order.
I think not. I'll try and explain.
* IMHO the first thing is to follow the three R's: repartition, reformat and reinstall from scratch. Only this way you can make certain nothing can come back to haunt you. Fresh slate thing.

* The second thing is to minimise risk. Think:
Availability: less software = less worries = less chances. The less SW you install the less you have to configure, audit and update. This means
pre-installtime choosing and downloading packages that can help you maintain a certain security posture if they are not in your distro's repo's like
- file integrity checkers like Aide (passive, needs libmhash) or Samhain (active) or even tripwire,
- generic auditing tools like Tiger, Chkrootkit, Rootkit Hunter and if you fancy it something like Bastille-Linux, and
- logfile watchers like Logcheck, Tenshi or Logwatch or Swatch.
at installtime opting to choosing packages that can help you maintain a certain security posture and forego on services that need the box hardened first (esp anything non-critical, publicly accessable and PHP related), and
post-installtime keeping network down, disabling services you don't need *now*, liposucking the fat out of what's installed and running initial file integrity check (baseline database) and initial Tiger audit.

Configuration: taking away unnecessary and setting necessary options for operation. For instance setting up sudo + unprivileged user account for remote access, checking root can't login except for console and is disallowed access to any networked service, setting password policies (using PAM_passwdqc), setting up PAM listfile.so's, checking inert shells on system accounts (UID's between 1 and 500), cron and at allow/deny files, mount flags, etc, etc.

Access controls: explicitly allowing something. For instance SELinux, GRsecurity, LIDS. Also daemon-specific ways like Xinetd.d allowed ranges (also backup-ssh), any sshd_config settings, tcp wrappers, firewall.

Auditing: the more you know, the easier it is to make adjustments. For instance making sure syslog dumps nearly all info (above debug level), making sure any daemon logs, have files rotated and watched or inspected. Also mod_security for Apache, an IDS like Snort or Prelude, if you need it a process monitor like for instance Monit, maybe remote syslogging and remote checking (say Nagios).

* Backups. Can't say it enough. If you invest time setting it up, why stop there? Set a good backup scheme!

* * This is by no means a complete list of what to do, but it should prove enough to be able to connect the box to the network in a more controlled and more secure way.

I hope it's somewhat clear, any questions (or corrections anyone?): just post.


long password phrases, do this change anything?
With the right use of punctuation, caps and numbers they're a wee bit more difficult to crack (at the word level) and a wee bit easier to remember. But that's my personal opinion. I do use phrases for everything critical.

at the risk of sounding like a book-sallesman, i have to recommend a good and easy to understand linux security book:

title: Hardening Linux
publisher: McGraw-Hill / Osborne
ISBN: 0-07-225497-1

has everything to help prevent incursions on you server, hardening firewall policies, the kernel and user access rights.

I wish to thank everyone that posted. I have been backing up all my files, configs. I tried to get a quick database save off too (got month old backup) but I have been locked out of it. I chowned the folder to root and copied the files.

In a couple hours I am going to bring it down and format. I have one concern, I know some windows virus/torgans can write code into the MBR has that ever been the case in linux?

I have been making a list of all the things I am gonna do differently this time around, your advice hasn't fallen on deaf ears. I haven't been monitoring my box like a hawk, I am gonna get tripwire and run security scripts in cron. I have had my box running for 2 years and it just got hurt now, I seem to be a little better off than the guy that got hacked in 10 hours.

Thanks all! It's better to know whats going on instead of having a massive question mark over my head.

It is entirely possible that a rootkit could write to the MBR, though I'm not aware of any that do. If you are concerned, however, there is a way to eliminate this:

1. Make sure all data you need is backed up.
2. Boot from a livecd, like knoppix or a distro install cd that lets you get to a console.
3. As root, type:
   Code:

   dd if=/dev/zero of=/dev/hda bs=512 count=1

   where /dev/hda is your disk. This will overwrite the MBR and partition table with zeros.

Wish me luck, I am bring it down now. I got my backups, I got newest version of services. Should I keep my logs to send to CERt or are they worthless?

If you got backups thats cool. I doubt you need to send them to CERT but you might learn from them.

Good luck.