I was curious as to what steps can be taken to, although obviously not with 100% certainty, determine whether or not a machine has been compromised.
I'd like to preface this by talking about something else first. We all know a Linux box is powerful because it is versatile, reliable, performs well and has excellent networking capabilities. But do we
realise the data it contains, the services it provides, and the time invested in configuring, running and maintaining a Linux box represent a value? Things of value should be protected (hardened and regularly audited) and be treated with care. What I mean is that taking into account the value it represents, the automatic reaction to any sign (clear or vague) of compromise should be to
treat a box as infected (threatening and urgent condition) which means it should be dealt with defensively, immediately.
Moving on, although there are basic steps for checking any box after a (perceived) breach of security, it helps to first know (read "make certain", not "guess") the purpose of the box and where it's placed in the network: knowing its purpose should lead to a list of services it provides. Knowing the place in the network shows its exposure (who has access), any devices to check in front and (where necessary) a list of (adjacent/"trusted") machines to check (it communicates with). In a professional environment you'll also have to know about network and security policies which should include a list of people to contact (like helpdesk, network admin, S.O.). Somewhere in between this all we've got to wedge in knowledge of the level of hardening of the box and network. Having remote access to network device logs, (remote) syslogs and (N)IDS logs can provide pieces of the puzzle. What I mean is that thinking should come before acting. More on reasons later on.
Now, if there are clear signs an intrusion is in progress or the box was compromised the first steps should be to isolate the box from the network (fire up a "known good" spare server to take over services if necessary) and responsable people and users should be warned. If there are no clear signs please realise that, whatever you do, you will need to
make certain, not "guess" things are really OK.
I've done the obvious of running chkrootkit
That's a good thing though you'll have to take into account that output of tools like Chkrootkit, Rootkit Hunter or OSSEC Rootcheck (ossec.net) can be circumvented not only at the kernel level but also due to use of hardcoded stuff[0] like searchpaths. That's why having a file integrity checker[1] is important and not relying on the report of one single tool.
and ocassionally nmaping and running nessus against my own machines.
Portscanning a host (from another box) can show the status of a port, however it doesn't show ports that are not open (proto/port/bit-knock-alike mechanism), it uses a static port-to-service mapping (use version scanning) but most importantly if a cracker is watching a scan will tip her off. Nice.
Just wondering what everyone else is doing.
Leaving out things already mentioned and checks performed by Chkrootkit, Rootkit Hunter, Rootcheck here's some things in no particular order:
- logs. Always check the logs,
- (config) files that should not be touched that have their MAC times changed,
- any files that have their checksums changed,
- processes with unusual characteristics (name[2], deleted file usage, work directory),
- network connections (for instance to an IRCd),
- (any libpcap) processes using device promiscuous mode,
- new or changed setuid root binaries (in publicly accessable directories).
Output of listing modules, processes and network connections can be changed, so I prefer to use additional tools for example "unhide" (security-projects.com) for both process and network checks, "ip" for promiscuous mode checks, "kern_check" (la-samhna.de), an IDT checker (see Phrack, can be subverted too). If I'm unsure and I need more info there's nothing left but to drop to runlevel 1 or reboot with KNOPPIX-STD, PSK, FIRE or any other CDROM-based toolkit in case there's enough leads something weird is going on.
Are there any clear signs and indications of a hacked machine?
I'd say anything that is not "normal" behaviour is worth looking at like hight of or spikes in load or network usage, fill rate or full disks or logs. Basically anything that doesn't work "as expected".
Just happen to be pondering upon this item tonight.
Which means you're on a road that leads to hightened awareness, which is a Good Thing. The next thing to do would be to recheck the box (and network) for any holes wrt hardening and auditing capabilities (see the
LQ FAQ: Security references). After that comes practicing. Practice builds familiarity and knowledge wrt tools to use and confidence and efficiency wrt execution.
[0] It's rather easy to for instance make Adore-ng invisible to Chkrootkit's "chkproc".
[1] Chkrootkit also has the nasty habit of relying on the "ifconfig" binary instead of "ip" which can, under certain circumstances, skew results of promiscuous mode checks. Choosing which file integrity checker to deploy should depend on the expected threat level (exposure) of the host and if it's using MAC (SELinux) or RBAC (GRSecurity). Samhain is an active one (includes an LKM) which should be a good choice for hosts that have less restrictions due to the services they provide. Aide (a "better" tripwire replacement) is a passive one meaning it needs to be run periodically. (Also see "Saint Jude LKM" where no MAC or RBAC can be used).
[2] Do
not assume just because the process name reads "httpd" it *is* the webserver:
Code:
#!perl
$cmd = "httpd";
$system = '/bin/sh';
$0 = $cmd;
sleep
