First I am sure this is a FAQ. If there is one please point me to it.
Second sorry for such a basic question.

Okay, I ran rkhunter on my debian system. This is the summary.
-----------------------------------------------------
System checks summary
=====================

File properties checks...
Files checked: 137
Suspect files: 1

Rootkit checks...
Rootkits checked : 248
Possible rootkits: 2
Rootkit names : Xzibit Rootkit, Xzibit Rootkit

Applications checks...
All checks skipped

The system checks took: 1 minute and 23 seconds
--------------------------------------------------

What do I do now? I suppose "possible rootkit" should be confirmed?
But what would I do to remove a rootkit if it is there? Rootkits do not seem to be removed by "virus" scanning and quarantining applications.
How would I trace back to find the origin of the rootkit?

These are all pretty basic questions. If anyone could just point me to the best documentation that is not written exclusively for security experts I would appreciate it.

Thanks

Rootkits are pretty rare and there is a high probability that it is a false positive. The first thing I would do is to use Google. For example, using the information you provided, debian system and Xzibit Rootkit, I looked up these terms. Here are the first two results:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=561308, http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=576680 both of which are known false positives with Debian for this rootkit. You didn't specify which file(s) have these errors and you may want to throw this term into your search.

The second thing I would do is read the Debian readme files as well as the rkhunter documentation, which will have some further help for your investigation.

well, find the file(s) that it thinks may be part of a rootkit and investigate those files closely (study their hex code, compare to known rootkit code, etc). roots usually have a c2 component, so does the box act oddly or not like it used to? roots can hide stuff from users (root or anyone), but they cannot hide their activity on the network, so take a look at network traffic looking for odd traffic, etc.

if your one tool is non-conclusive then you need other tools, etc.

conclusive positive rootkit then i suggest you rebuild that OS (box) from scratch (hopefully you have a build image or a vm copy, etc). "cleaning" is no longer a option.

While in the case of a real root kit infection this may be the ultimate solution, at LQ we strongly advocate performing a thorough investigation into the cause and infection vector. Performing an otherwise blind re-install accomplishes next to nothing in regards to preventing future infections and is highly discouraged here at LQ. I don't mean to be preachy and I mention this because a number of LQ-Security members have worked very hard at developing this culture and take real offense at recommendations for wipe and reinstall without prior justification.

Your recommendation to use other tools is spot on. This is getting a few steps ahead of ourselves, but since you raised the issue, normally we recommend following the CERT intruder detection check list. Here is a link. We often times recommend looking at the output of netstat -pane and lsof -pwn, which will show you the open file and socket connections.

i didnt suggest wipe/reinstall w/o justification. what i said was "conclusive positive rootkit" then the machine can no longer be used or cleaned or fixed or patched or whatever, its toast. the ability to do forensics depends on the resources available to the responsible owner of the box, etc. sure, a live mem dump along with images of all the filesystems would be good preservation, but if the host mucks around in 50TB of disk then where exactly does one image 50TB to? the live host can be somewhat preserved by taking it off the network, but at that point it cant provide services, etc.

in red, hmmmm, not sure i would trust any output from anything that executed into memory/cpu of a possible-rooted box.

btw, the link you provided doesnt work for me, and the url on that page leads to a 404.

OP - can you post the log for rkhunter. from searching around the net it seems that rkhunter flags anything that has string "hdparm" in it (or anything it itself thinks should not have that string, etc).

RKhunter should be able to remove it.

0 members found this post helpful.

Like Noway2 already said rootkits are pretty rare these days and often Zxibit is a false positive. The first to do however is check the rkhunter-users mailing list archives (as most questions have been asked already) or search LQ (http://www.linuxquestions.org/questi...3/#post3815419) or your rkhunter.conf for clues. For some odd reason people seem to think that installing and running stuff equals gaining insights one would get from actually reading documentation...


Due to false positives best start the other way around: if available use file integrity verification through package management or other applications or methods (Samhain, Aide, package content verification from trusted repo).


Next time maybe read up on things before you embarrass yourself posting stuff like that? RKH is a passive, post-incident auditing tool, not some antivirus tool with "cleaning" capabilities.

any file integrity checker or rk hunter that runs in/on the suspect host cannot be trusted. at minimum, root hunters have to be run from a different OS (liveCD, flash, etc). perhaps i am somehow suggesting systems be taken offline for short period of time to run rk hunters / file checkers via liveCD/flash on a routine basis....? with the sophistication of roots and other crap advancing day-by-day we might need to resort to such a practice.

In case of a suspected "true" rootkit installation that is correct (and it's something one shouldn't forget pointing that out in a first reply). However RKH doesn't come with a standard configuration that is tuned for one OS, distribution or release so after installation FPs will occur. In these cases (asserting proper attention was given to proper OS installation, hardening, logging, auditing and alerting prior to exposing it to hostile networks) it is more practical to first look for FPs than to start a full scale investigation if the system doesn't exhibit anomalous behaviour and no other alerts were raised.


In short rootkit installations have dwindled substantially if one looks back over the past decade and emphasis has shifted to malware riding the web stack. Ergo, while that may be nice to do at home situation, I wouldn't mark that as practical nor necessary. It certainly is not a best practice for those situations in which Linux excels: providing networked services in a stable, secure and continuous way.


Some of us have practiced Incident Response and Forensics for over a decade here and elsewhere so their opinion may differ from yours. You're invited to watch what comes our way the coming year and let IR cases we handle here provide you with the insights to change your opinion...

thats's awesome. i've been doing forensics/IR and computer/network security for 15yrs. the bottom line is, if a machine is suspect (which can have its own meaning) then it is not acceptable to run anything on/in the suspect machine, it must be checked offline, including checking BIOS. practical or not is a operations issue hence why deployments need to account for these security related scenarios.

oh btw,not sure if you keep up with things, but some recent reports say that many places had infected systems, yet they didnt know (didnt detect it) and the systems didnt exhibit anything oddly.

cheers.

Interesting. Any of that available for reading?

I think the point of my original post may not have been clear, or has become obscured.
Let me start with a basic premise. Anyone with the root password on a linux or unix box is a System Administrator. Well at least fledgling SA.
We may remember, "With great power comes great responsiblity". Part of that responsiblity relates to security. This is a function one should be performing for ourselves and the larger community.

There is a lot to know and I don't and can't know everything. Noone can. But I can have a set of basic procedures for regular security auditing that I follow. It is really asking a lot to expect our fledgling SA to compare hex code.
But there probably are some basic steps that could be followed before even posting here. Personally, I have fooled around with some forensic tools. But I really don't know what I am doing.

In the same way if our abecendrian SA wanted to learn the basics of crash analysis, I could at least show her the basics steps of running strings on a core file, and how to investigate error messages found in the logs. I would be nice if they could use a debugger. But you got to start somewhere.

So I guess the more basic questions for you security experts are these.

1. What kinds of regular security audits should anyone responsible for a linux or unix box be running?
2. When some issue or problem is reported by some security analyisis tool, what basic steps should be taken.
3. Basically, what would you like our junior SA to have done before posting. Is there a step by step basic procedure?

Thanks and Cheers

I think that the number one thing you should do is regularly look at your logs. Doing this will help you to gain insight into what is normal for your system. Without understanding what is normal, it is nearly impossible to know if something is abnormal. There are tools, such as logwatch, which can help with this but until you are comfortable with the logs themselves you shouldn't rely on log watch. The logs will provide indications of threats as well as tell you if programs are running properly. The second thing I would recommend is using a HIDS tool, which will alert you when something in your system has changed and makes an excellent complement to (but not a replacement for) watching the logs . It is best to install a HIDS during the initial setup on a known clean system.

Another important aspect of system security is to regularly check to make sure your system is up to date. In some systems the package manager will notify you of updates automatically while in others it is up to you to check. Keeping you applications up to date with the latest security patches if far more beneficial in modern Linux systems than anti-virus and rootkit scanners.
First, don't panic. Second, identify the message and research it's meaning. Third, try to identify a potential cause. Make not of things like time of occurrence and what other things may have happened at or around that time. Look for data such as is it a known or suspected bug, and are there potential solutions. Follow this process up with a review of your logs. Are there indications of intruder activity? Examine what applications are running, who is logged in, what network connections are open.
Back in post #4 of this thread, I posted a link to the CERT intruder detection check list. Here is the link again. It provides a step by step procedure for things to investigate.

Here is a recent example that I am currently facing: I receive a notification this morning from the OSSEC monitor that "there is a problem in the system and the following message was detected in my Apache error log - "zend_mm_heap corrupted". According to OSSEC, this message was received at 04:41. Looking a the error logs, shows just this error message without any other time date stamping (this is where OSSEC provided an important clue - the time). Looking through the log I see that there is something going on about once a week or so at 04:41 and typically the message reads "SIGHUP Received. Attempting to restart". This particular case, the heap corrupted message was followed by one that said "child process still did not exit, sending a SIGTERM.". The fact that the timing is always the same, 04:41 suggests a process related to CRON or some form of system network activity at that time - perhaps a security program or scan. As I am currently investigating, I don't yet have a lot of details to offer, however, my approach will be to Google the error messages, look at the other logs for activity at around 04:41, look at my SNORT logs to see if there is strange network activity at this time, and so forth. Initial indication from the known bug reports indicate that this is a problem that has existed for a few years and is commonly caused by unset() being called in a script on a non-existent memory object. The bug also commonly manifests as a segmentation fault, which I have seen on occasion. Putting all of the above together, I will try to see what activity occurs at this time of day, what was being accessed and then look for memory operations in these scripts.

Edit - Followup(1): I discovered the cron.daily is run at 04:40. The /etc/cron.daily folder contains a couple of scripts, such as ones to check for expiration of any SSL certificates, logwatch, and logrotate. One possible avenue of investigation is to change the time associated with cron.daily to see if the error moves with the time. If it does, it indicates that it is being caused by one of these scripts. My initial guess is logrotate. I suspect this because I noticed that a new log file was created today, along with the error message. Looking at the date time stamps associated with these messages shows a direct correspondence with logrotate. Consequently, I can conclude that logrotate is causing the shutdown and restart of apache, which is expected or at least understood in order to update the log files. Whether or not this is the root cause, I do not yet know.

Followup(2) - there was an error message in the apache logs at time of restart, complaining about an API mismatch in the PDO module. Investigation shows that the extensions=pdo.so line is no longer required in PHP.ini as of version 5.3 and later. I recently upgraded to PHP 5.3.8 and I am reasonably certain that the segfault and now heap corruption messages appeared after the upgrade. I removed this configuration line and restarted Apache (without the error). Time will now indicate if this has addressed the problem.

imho, what you're asking for is not "basic". i'll suggest you start here (http://csrc.nist.gov/publications/ni...800-61rev1.pdf) and then tailor your IR/Security program accordingly. you didnt say how mature your IR/Security program is so its hard (at least for me) to try and suggest things you should be doing and how you should be doing them. granted the suggestions already mentioned are all good but they only touch the surface, etc. besides investigating the host there is a wealth of info to be had from network flow data, so if you have that to look at it can provide some very useful info (something like a niksun per segement or on the outside fw iface, etc).

i think what Noway2 has suggested is an investigative tactic (in reference to the heap log entry being investigated). i am in trained in RCA/RCCA so my thinking on how to find the problem is likely different than others. i might have concluded sooner that the issue was related to apache settings, or maybe not (i would have went after all the logs all at once for the suspect timeframe, grep the syslog or filter in a tool, etc).

as for the strategic approach Noway2 suggested (logs, hid, integrity checkers, etc etc) i think that is a great idea, but use commercial products when possible (like now defunct MARS, ArcSight, Tripwire, yada yada yada). another approach might be numerous layers of freeware, but that is up to you.

having the skillset to dig into it when the tools fail (or seem to fail, or dont provide enough info) is a golden technical quality to have in-house, but it seems to be rare as many people rely heavily on cots and freeware tools to provide assurance/analysis (investing in tools is easier than educating the people, which is definitely a wrong strategic approach to security, you need a matched mix, etc).