i runned chkrootkit at my server. i got the message 'possible zk rootkit detectet'.

how can i find out, wheater i really have it or not??

First of all, Chkrootkit is known to show what is known as "false positives", which can make using it a liability for your blood pressure levels :-] This however does not mean this will be one, be prepared to see odds switching both ways...


I'm running (a modified) Chkrootkit 0.42b(+u0.0.2, with way better interface promiscuous mode detection on Linux kernels 2.4.x but that's besides the point), if you "grep chkrootkit -e ZK" you should see that as a part of detection (and next to looking for suspicious files like sbin/init.zk), around lines 540 (expert mode) and 960 it looks for existence of any of two files: /usr/bin/run and /etc/sysconfig/console/load.zk. Rerunning Chkrootkit in debug and expert mode with only that test (chkrootkit -d -x aliens) should show which files show up to investigate.
I. Next you should verify these files are part of your distribution, and if they are, if their contents check out (run "file" or "strings" on them, visually inspect text files) and md5/sha1sums check out comparing it with "trusted" copies from say cdrom or one of your distro's ftp mirrors.

II. Else if the files are not part of your distribution, and since this is a rootkit, assert someone has got root. This means the box is no longer under your sole control and is not to be trusted for whatever it's task is, and you should be prepared to turn off the box for investigation.
From now on, regard this system as UNTRUSTED. DO NOT USE IT, AND DO NOT ALLOW OTHERS TO USE IT.
...and forget about getting that one last minute thing done or making changes to the filesystem like making backups etc etc.

OK. What to do in case of suspected compromise?
Precaution 0: read Steps for Recovering from a UNIX or NT System Compromise before doing anything else.

Precaution 1: For any operations on the disks, boot your distro's rescue cd, any cd-based distro (Trinux, Knoppix, Finnix, FIRE, PSK) or a one-floppy distro like tomsrtbt, LOAF. DO NOT BOOT the kernel from the harddisk or let it automatically mount the disks. If you need to mount the disks, mount them readonly.

2. Stay calm.

3. While this also means extracting memory, process and network (connection and interface) details of the running system may be tampered with on the fly, you should still do so. For details on running ps, netstat and lsof I'll hook you up with this post a little down. The find command is wrong, you should run "find /proc/[0-9]* -name exe -printf "%p\n" | xargs -iX cp -aL --parents 'X' /tmp2>&1|tee /tmp/procs.log" to save copies of running binaries. Try running "ifconfig -a" or better "ip link show".
Make sure you don't choose VFS partitions a lot happens on like /, /tmp, /var for output of files. If unsure, don't. If you took process and network snapshots, bring it down to runlevel 1 first, then take another snapshot. Now power down the box. (If you've got magic sysreq keys (see printscreen key) the ALT+SYSRQ+{S then U then B} would sync disks, remount readonly and reboot at which you can power off). This will render the system useless to the cracker and protect you from doing stuff on the box.

4. Boot your systems rescue floppy/cdr, Knoppix, FIRE or PSK. Mount partitions read-only. Check "chkrootkit" again and have your package manager check the files again. Inspect auth files (passwd, shadow, group) and system logs. Please report back any anomalies.

5. In the situation you think the box could be compromised, and if you want to know what the intruder did, proceed by making a full backup of the paritions using "dd". We'll use it to try and track down what the intruder did. If you have to, take out the disks and add to another machine. Make sure you have enough free diskspace. For all partitions do "dd if=/dev/<partition> of=/mnt/disk/file_name_disk_partition_number" and don't forget the swap. Make sure these "dd" images never get mounted in read-write mode. To preserve data.



Good luck.

For the above, some notes:

- If this is a remote box, you obviously can't bring it down and proceed. Ask your colo people to do it for you. If it costs too much money, then make 'em save /etc and /var/(log) in a tarball and use the 3 R's: reformat, repartition and reinstall your box. Note restoration is not feasable, unless you have external means to verify the backup was sane.

- If you have a huge amount of logs and system logs you want me to check out, or can't make head or tail out of it, make a bzipped tarball of it and upload it, mail me the URI and I'll help you check it. Use "split" for archives over 10MB. (This rootkit ain't in my personal collection, so yeah, I'm interested.)

- Make sure to be verbose when you post a follow up, it will speed up the process. Also, I invest time writing these posts, and the last thing I want to see is a follow up which sez "I did X but I found nothing": if unsure, posting as much as you can will be valuable. It will enable us to help you.

thx, unspawn.

first, unfortunately the grep chkrootkik -e ZK didn't work on my version of chkrootkit... (v 0.42b)

afterwards, i was looking for .zk files with locate....none showed up...just the /usr/bin/run is there, thats all...even a netstat -an shows no strange connection...

what do you think? should the init.zk be visible?

first, unfortunately the grep chkrootkik -e ZK didn't work on my version of
chkrootkit... (v 0.42b) That was only part of my example, to make clear,
how Chkrootkit tries to find evidence of the ZK rootkit. I told you to run
Chkrootkit in expert mode. And what to check afterwards.


afterwards, i was looking for .zk files with locate....none showed up...just
the /usr/bin/run is there, thats all...even a netstat -an shows no strange
connection... what do you think? should the init.zk be visible?
For some reason however, you managed to read just the first few lines of my
reply, neglect the rest, didn't perform any test I suggested and started doing
your own thing. Besides that, you answered terse, something I explicitly asked
you not to do. If something I said in my first or second reply wasn't clear:
ask. If it is: answer it. If you won't: then I've got nothing more to say.

hi unspawn,

as i am some kind of rookie, first of all my excuses not to follow your instructions. so i try to follow step-by-step.

Have to say, that first of all, i immediatly unmounted my data-disks and reinstalled again fedora core 1. The first application i installed afterwards was chkrootkit. so i did the following:
[root@morpheus etc]# chkrootkit -x | grep ZK
[root@morpheus etc]# chkrootkit -x | grep zk
### Output of: /bin/ls /usr/lib/pt07 /usr/bin/atm /tmp/.cheese /dev/ptyzx /dev/ptyzg /usr/bin/sourcemask /dev/ida /dev/xdf* /usr/lib/libx?otps /sbin/init.zk
/bin/ls: /sbin/init.zk: Datei oder Verzeichnis nicht gefunden (means init.zk not found...)
...
and then 'chkrootkit -x > chkrootkit.txt' and then 'more chkrootkit.txt | grep zk' and more chkrootkit.txt | grep ZK with the following results:

###
### Output of: /bin/ls /usr/lib/pt07 /usr/bin/atm /tmp/.cheese /dev/ptyzx /dev/ptyzg /usr/bin/sourcemask /dev/ida /dev/xdf* /usr/lib/libx?otps /sbin/init.z
k
###
/bin/ls: /usr/lib/pt07: Datei oder Verzeichnis nicht gefunden
/bin/ls: /usr/bin/atm: Datei oder Verzeichnis nicht gefunden
/bin/ls: /tmp/.cheese: Datei oder Verzeichnis nicht gefunden
/bin/ls: /dev/ptyzx: Datei oder Verzeichnis nicht gefunden
/bin/ls: /dev/ptyzg: Datei oder Verzeichnis nicht gefunden
/bin/ls: /usr/bin/sourcemask: Datei oder Verzeichnis nicht gefunden
/bin/ls: /dev/xdf*: Datei oder Verzeichnis nicht gefunden
/bin/ls: /usr/lib/libx?otps: Datei oder Verzeichnis nicht gefunden
/bin/ls: /sbin/init.zk: Datei oder Verzeichnis nicht gefunden
/dev/ida:
c0d0
c0d0p1
c0d0p10
c0d0p11
c0d0p12
c0d0p13
c0d0p14
c0d0p15
c0d0p2
c0d0p3
c0d0p4
c0d0p5
c0d0p6
c0d0p7
c0d0p8
c0d0p9
c0d1
c0d10
c0d10p1
c0d10p10
c0d10p11
c0d10p12
c0d10p13
c0d10p14
c0d10p15
c0d10p2
c0d10p3
c0d10p4
c0d10p5
c0d10p6
c0d10p7
c0d10p8
c0d10p9
c0d11
c0d11p1
c0d11p10
c0d11p11
c0d11p12
c0d11p13
c0d11p14
c0d11p15
c0d11p2
....
and so on....

so, i got a list with about 2'000 lines.... with chkconfig -d -x aliens and i more'd with 'zk' getting the same lines as above. do i assume right, that i have to verify, if those files should exist now in my distro?

thx

as i am some kind of rookie, first of all my excuses not to follow your instructions. so i try to follow step-by-step.
Well, it's not that it's mandatory to follow instructions if you know what you're doing, it's only that it saves time when you don't, which I think is critical in such situations.


so, i got a list with about 2'000 lines.... with chkconfig -d -x aliens and i more'd with 'zk' getting the same lines as above.
I'm sorry. I keep forgetting most people don't troubleshoot Bash scripts like I do.
If you run it and redirect output like "chkconfig -d -x aliens 2> /tmp/aliens.err 1>/tmp/aliens.log", the /tmp/aliens.log will hold the files you need to check, and you get the whole "conversation" in /tmp/aliens.err, which means you have a way to verify chkrootkit is running right.


do i assume right, that i have to verify, if those files should exist now in my distro?
No, not "if", it's the other way around: files chkrootkit [u]finds[/i] should be inspected.
About false positives, I know of two packages that include a file called "/usr/bin/run".
If your distro has a package manager that allows you to query files for the package they're in, and provided you have a "sane" copy of your package managers database of installed apps, it should be easy to find the name of that package. For that package, verify the contents of what is on your filesystem with the package from your install cdroms or distro mirror, or remote site where it is made available.

yes everybody how he wants....i tried this and hot huge files in /tmp. so i seems the most interesting is aliens.log. as this was huge, i was filtering all with zk. there he found no files:
### Output of: /bin/ls /usr/lib/pt07 /usr/bin/atm /tmp/.cheese /dev/ptyzx /dev/ptyzg /usr/bin/sourcemask /dev/ida /dev/xdf* /usr/lib/libx?otps /sbin/init.z
k
###
/bin/ls: /usr/lib/pt07: Datei oder Verzeichnis nicht gefunden
/bin/ls: /usr/bin/atm: Datei oder Verzeichnis nicht gefunden
/bin/ls: /tmp/.cheese: Datei oder Verzeichnis nicht gefunden
/bin/ls: /dev/ptyzx: Datei oder Verzeichnis nicht gefunden
/bin/ls: /dev/ptyzg: Datei oder Verzeichnis nicht gefunden
/bin/ls: /usr/bin/sourcemask: Datei oder Verzeichnis nicht gefunden
/bin/ls: /dev/xdf*: Datei oder Verzeichnis nicht gefunden
/bin/ls: /usr/lib/libx?otps: Datei oder Verzeichnis nicht gefunden
/bin/ls: /sbin/init.zk: Datei oder Verzeichnis nicht gefunden

but a huge list of devices...(i do not paste full if it isn't needed..
/dev/ida:
c0d0
c0d0p1
c0d0p10
c0d0p11
c0d0p12
c0d0p13
c0d0p14
c0d0p15
c0d0p2
c0d0p3
c0d0p4
c0d0p5
c0d0p6
c0d0p7
c0d0p8
c0d0p9
c0d1
c0d10
c0d10p1
c0d10p10

he even doesn't check for /usr/bin/run....or did i miscomprehend? for me, it looks like making specified tests by rootkit, so all for the zk-rootkit should be together....

i was remarking not only in this a little bit strange behaviour of the Fedora Core 1, even if i find it excellent...the antivir-kit from antivir.de even seems to dislike the prelink-crontab (no ELF-support), so it dies...but this is another story...

thanks for your patience...

ah...btw 'Datei oder Verzeichnis nicht gefunden' means 'File or Directory not found'

ah...btw 'Datei oder Verzeichnis nicht gefunden' means 'File or Directory not found'
Ja ist das, was translate.google.com sagte.
//Of course translate.google doesnt care eight bits for syntactically correct sentences :-]


he even doesn't check for /usr/bin/run....or did i miscomprehend?
Yes, it does. See the lines at:

for me, it looks like making specified tests by rootkit, so all for the zk-rootkit should be together....
Hmm. I didn't get that ligne.


i was remarking not only in this a little bit strange behaviour of the Fedora Core 1, even if i find it excellent...
Wow. OK. Stupid me. I should have asked you your distro/release right away.
Since I installed FC1 as well to do some dev on, I ran Chkrootkit-0.42b (new patched version should be out in December) and I got this
Well well. Since it doesn't detect the "/etc/sysconfig/console/load*", the *.zk files, nor a trojaned /sbin/init, let's see (note I'm using my own aliases, but the output matters):
At this point we haven't established anything except that:
1. Chkrootkit didn't detect all files that are part of the ZK rootkit (which means they are either hidden or not there),
2. there is a file "usr/bin/run" in the "run" package, the filetype, access permissions and ownership match, but the size and MD5sum dont.
This is the point where you would:
1. Use your filesystem integrity checker (Aide, Samhain, tripwire, md5sum database) to verify the files' attributes in the database match (database on read-only media)
2. Download a copy of the rpm from FTP to compare contents with.



the antivir-kit from antivir.de even seems to dislike the prelink-crontab (no ELF-support), so it dies...but this is another story...
Yes, set up another thread for that, or better, email Antivir.de and let them fix it.


thanks for your patience...
You're welcome.


Concluding notes.
It would be easy to say the system is safe. However this is a way of deceiving yourself. Having a copy of the rpm database on read-only media and checking the filesystem with that is one way, but running a filesystem integrity scanner should be mandatory when you install your linux OS.

Chkrootkit has been known to cause FP's (false positives) under a variety of circumstances ranging from finding dotfiles to relying on names or port numbers and from probs with threading apps to detecting short-lived processes or network connections or the state of network interfaces (see http://www.rootshell.be/~unspawn/pac...hkrootkit.html or the Chkrootkit mailinglist archives). For more than one reason it should be clear no system should rely on using just one application to provide you with filesystem "health" signs.
Chkrootkit also is an OSS effort to help people, so this FP should be reported and a patch should be submitted, to help Chkrootkit evolve.

Filesystem integrity checking is something a lot of people don't know about, but (when configured and used properly) can provide invaluable info on the state of the system. Please spread the word and check out/link to the LQ FAQ: Security references, post #3 "Intrusion detection, integrity checks: IDS, NIDS, HIDS, Antivirus, software.".

In the same post you'll find info about Linux and viruses. When newbies ask about viruses, be sure to warn them about the things that are really important instead of just posting a link to someone's rant on AV or AV software.


HF

have the same results, hard to check...is this because of ELF?? I am not a dev at all, but as i read in manpages, there runs one a day a prog called prelink, that optimizes apps for startup...

Yes, it's because of the prelink stuff.
The error message itself is no cause for alarm.

thanks a lot...makes me more convenient again...