What the best way to detect a rootkit that clam/avg/rkhntr/lysis etc aren't detecting

TIA

What are the symptoms?

Yeah, there's not enough info here. Care to provide some more details ?

I mean, how do you know there is a rootkit ? because you seem convinced. Hardware rootkits are nearly impossible to detect without specifically designed software and maybe not even then.

You can try asking the question a different way as well in case you don't have more info.

There is something that replaces all the IPs in /var/log/secure every second or so. And it also clears the history. Plus I found some suspicios PHP files.

I don't usually deal with these, but you should try to investigate further. Some useful commands:
When you find out more about the names of the suspicious running programs, search the internet for the exact names of the files. If the malware is known it should give a hit and possible solutions.

You could probably also use a packet sniffer to find out if it is phoning home and who it is contacting.

Replacing data in root owned files is not a good sign: I would urge you to isolate the machine while you investigate. (There are no valid reasons why you shouldn't but if you want me to argue them: BMG.) "the history" doesn't tell us much: please be precise. Who's history? And what "suspicious PHP files" exactly? (Send a copy to my Gmail address?) In what location did you find them? What's their ownership? And MAC times? What other files are associated with the owner? Did any change as well? Are those other files associated with an application? Was that application kept up to date?..


0) Check the Intruder Detection Checklist (CERT): http://web.archive.org/web/200801092...checklist.html (formerly: http://www.cert.org/tech_tips/intrud...checklist.html) if you don't have structured steps to take,
- notify users of the system(s) to take preventive measures,
- isolate the machine or at least firewall it to only allow access from your management IP (range),

1) Spill some more nfo:
- location, the distribution and release version and purpose of the machine,
- which services the machine or machines provide (including web-based management panels, statistics, web log, forum, shopping cart, plugins, themes, addons and other software if any),
- which logging, access restrictions is in place and hardening was performed,
- what audit and auth data, system and daemon logs, user shell histories and cron spools you've checked,
- which exact software versions and if the software was kept up to date,
- if there have been earlier breaches or anomalies,
- found any setuid root files?
- complete listings of running (piping through SSH or saving in /dev/shm may be a substitute for "/path/to/"): '( /bin/ps axfwwwe -opid,ppid,gid,uid,cmd 2>&1; /usr/sbin/lsof -Pwln 2>&1; /bin/ls -al /var/spool/cron 2>&1; /bin/netstat -anTpe 2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last -wai 2>&1; /usr/bin/who -a 2>&1 ) > /path/to/data.txt', '/bin/rpm --nodeps --noscripts --notriggers -Vva 2>&1|/bin/grep -v "\.\{8\}" 2>&1> /path/to/rpmvfy.log'. Run all system and daemon logs through Logwatch with the "--detail High --service All --range All --archives --numeric --save /path/to/logwatch.log" args. (With perl-Date-Manip installed a range can also be expressed like "--range 'between 2012/11/26 and 2012/12/01'": see --range Help). MAC times: 'find / -type f -printf "%T@ %A@ %C@ \"%p\"\n" 2>&1;'.(or find /tmp /var /tmp /usr/tmp -printf "%T@ %A@ %C@ %u %g %m %y \"%p\"\n" 2>&1).
- results from the actions performed as per the CERT Intruder Detection Checklist.

2) Please compress and attach (rename to .txt extension) or if file size prohibits attaching it please contact Noway2 or me to discuss dropping logs off.

*Please ask specific questions before performing if necessary and please reply verbosely.
**Please stay with the thread (subscribe?) until completion and reply as soon as possible when replies are posted...

Replacing data in root owned files is not a good sign: I would urge you to isolate the machine while you investigate. (There are no valid reasons why you shouldn't but if you want me to argue them: BMG.)
>>Its isolated

"the history" doesn't tell us much: please be precise. Who's history?
>>Root's history
And what "suspicious PHP files" exactly? (Send a copy to my Gmail address?)
>>It was just a php shell

In what location did you find them? What's their ownership? And MAC times?
>>/home/user, owned by the user, 7/20

What other files are associated with the owner?
>>Nothing

Did any change as well? Are those other files associated with an application? Was that application kept up to date?..
>>It appears the shell was uploaded by the application, yet, it was created by a 3rd party and I suspect it to be vulnerable, it was suppose to run as the user though, so they had to also escalated their priv another way.

Still there's data in root owned files being changed so answering the other questions could help.

I noticed a user had no password.

Is there any way to get a user to have no password without root access?

Ex, in the shadow file it had something like this:
user::16293:0:99999:7:::

That user in particular was the only user with sudo access.

Upon further review, there does not appear to be a root kit, only commands added /root/.bashrc to unset the histfile each time you log in, and to run a sed on the logs to replace whatever IP you log in with with another.

have you tried rkhunter?
you could also try installing open source tripwire, though the machine needs to be in a known clean state when tripwire is installed.

RK hunter didn't find anything.

It depends on the file permissions. Check then and you will get your answer. In general only root or a configured sudo can edit system config files in /etc.

If that is not a common practice for you (and your fellow admins if any) then that again isn't a good sign. Rootkit Hunter, any file system integrity checker, should alert you to changes in the user database. But only if you run it daily. And that only makes sense if one actually responds immediately to correct mistakes or worse. So what are the MAC times of passd, group and shadow? What files and processes is this user associated with? Are there any logins with that account?


Depends on what method was used, like tricking a privileged process into executing commands. If it, like metaschima suggests, boils down to file permissions then you've got Layer 8 problems.


The reasons I post a set of suggestions and instructions is to have it serve as guideline for you and others who aren't that well-versed in analysing such situations and for me to gauge, based on the information supplied, if the investigation adheres to at least the most rudimentary rules of incident handling and subsequently stands a chance solving the problem the way it should be done. (There's only about five LQ members I trust to do incident handling the way I want to see it done BTW.)

I don't know if this is your MO nowadays but I do notice you've mostly talked about the incident and you've carefully managed to avoid sharing any relevant data that could help us help you. That means that we should take your word for it that "Upon further review" somehow equals a proper investigation as suggested above. What's worse is that finding "only commands added /root/.bashrc" could lead to the conclusion that this is an isolated incident that warrants "correcting and cleaning up"...