reviewing today's logs/notices I found the following

I also see the following

Nothing else seems out of the ordinary, I don't see anything alarming (or don't realize it should be alarming) with a netstat -a, nmap localhost shows only ports I've opened, and my smtp and ftp banner messages are still identical. My root password hasn't changed, my ssh service disallows root, and only allows 2 users specified in AllowUsers to log into ssh.

Doing a who never shows anyone but me logged in, and the last users never shows anyone but myself (I understand that could have been modified).

I do allow passwords for SSH, but only two users are allowed access via the AllowedUsers statement (myself and a good friend). I do use iptables for a firewall, have a default policy of drop, allow established connections, and only allow new incoming connections for services I need to allow. I've verified that my mail server (postfix) isn't an open relay, my mysql server only listens on localhost and vsftpd only allows local users, and disallows root. I am runnign Debian Etc, and keep up to date via apt at least twice a week, or sooner if I get notification via email that a package I have installed has a security risk.

Is there any way I can find out if this box has been truely compromised? I'll reinstall the OS if need be, but this is a production box, running a mail and www server and it would be a real hassle to reinstall without a good reason.

If the box has been compromised, how can I find out how they got in, so I can be sure it's secured with a fresh OS install?

Check the logs for your http server. If you really were compromised, that's the most likely avenue (bad php scripts, etc).

I did a grep -are for 'bash' 'history' and 'bash_history' in all the access.log* files I have and came up with nothing. Any ideas on other keywords to grep for?

I was also under the impression that in Debian, while Apache starts via the root users, that session then dies, and all child processes run under the non privilidged www-data users (my primary reason for not goign through the hassle of chrooting Apache). Is this not the case?

My primary IT admin background is with Win 2000 and 2003 servers, and linux has been an ongoing project to broaden my horizons. I've been running Debian for well over a year and this is the first incident I've had where I wonder if I've been compromised. Being from a Windows background, I don't know all the ins & outs to linux type forensic work, so any examples you can provide of how to best track this down would be much appreciated.

For clarity:
0. Notice it is better to post complete output, unless you're onehundred percent sure this is the only error,
1. A shell history file being zeroed out in itself does not mean the box is compromised, we need more evidence,
2. Notice the timestamp being exactly 8 PM. While a coincidence you could check for some form of legitimate "logrotating".


Nothing else seems out of the ordinary
And that is as it should be. I mean, this swings both ways. If the box is fine, it's fine, if the box was compromised in a "qualitatively good way" evidence would be hidden and you have to take that into account while auditing. This means the best way to unhide malarky (unless the box is in colo) is to reboot the box using a Live CD like Helix, KNOPPIX or your distro's rescue cd if any.


If you have no idea and if you're not accustomed to checking things the first to read is the Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html. Apps doing multiple checks like Chkrootkit, USAT, Rootkit Hunter and Tiger, a file integrity checker like Aide, Samhian or even tripwire (if you installed it) and single-purpose tools like unhide, skdet can supply part of the necessary audit caps but using a checklist provides you with the necessary overview and an ordered list of steps to follow so you know why you're doing it.


My POV is we need to build an understanding of the situation. The more we know the better we can tailor our advice to your situation. Here's some points in addition to the checklist:
- location and purpose of the box (any other apart from SMTP, FTP and HTTP?),
- audit and auth data returned from your checks,
- any logged anomalies from an IDS (if you run one),
- any logged anomalies from checking system, daemon and firewall logs,
- running services,
- process and open files listing,
- setuid root files (any anomalous files in temp dirs).
Add anything else you think may be important.


If a box is (percieved) compromised it should not be part of business ops for the duration of the investigation (contamination). It should not see any applications introduced on the filesystem during the investigation (disturbing "evidence"), if the box isn't in colo run your stuff from a CDR. Make an entry in your local log and add entries while you step through the checklist. Notify fellow users from your local box and start your investigation.
When done please post as much information as you can, but guard for posting repetitive lines.


Chances are that you find the box was not compromised. Then at least you get some experience from auditing the box and maybe a thing or two to change that will make future auditing more complete, easier, so this will not have been wasted time. Good luck.

It happens to me, like once a month my root history on debian is also cleared and that's very annoying. I haven't taken the time to search the reason, but I'm 100% sure the box is clean apart from this (running integrity checkers). So I would say that this only problem its not a proof of compromised.

Here is the complete logging notification from chkrootkit, the packet sniffer is intentional.

I do have logrotate going, but unfortunately I never noticed that the default config has zero system logging in it.

That's what I'm worried about. [i]if[/if] I was compromised it was done in a 'good way' and not just some hackbot script kiddie. Any evidence that it was done is pretty well hidden as far as I can tell on the live OS. I have Knoppix, and can go to the datacenter this box is located in, but I'm not really sure what to look for.

Thanks, I'll follow up on that link.

I haven't found any anomalies in the logs, but that doesn't mean they couldn't have been altered. I run both chkrootkit and rkhunter as well as snort. Chkrootkit has never warned about anything other than finding snort, and Rkhunter only ever complains about
From the reading I've done on the net, this seems like a false warning. I *think* I emailed the author about it once, but I'm not 100% positive anymore.

[QUOTE=unSpawn]
- running services,
- process and open files listing,
[quote]

What's the best way to get this for you? a ps aux > filebalh is a
12k file. Is there a cleaner way to get that information?

my /tmp dir has noexec and nosetuid and has nothing listed but lost+found and clamav files

Here are all files with a setuid of root.

There are no running processes that match the pids 11454 or 11456.

This box runs SSH, Bind, Vsftpd, Postfix, Courier, Amavisd, Spamassassin, Apache 1.33 (Apache2 was installed by apt when I installed php5, but doesn't run), PHP5, Mysql 5.0.27, Roundcube Webmail, Chkrootkit, Rkhunter, Snort, and has been running breakinguard, but I've removed it as it's too buggy.

The ftp service only allows local users, chroots everyone but me, and doesn't allow root, or anonymous users to log in. SSH disallows root explicitly, and only 2 users are allowed via the AllowedUsers command.

Mail is local delivery only, for a few of my domains. Users are stored in a mysql db with a md5 hash for passwords, and mail is stored in Maildir format. Postfix is chrooted.

Bind is run chrooted.

Apache hosts three of my personal domains and a single domain for a friend. They do not have shell access to the server. Apache master process starts as root, which immediately dies, the actual child processes are owned by www-data. According to my reading this should prevent Apache from gaining root access, which is why I never bothered to go through the hassle of chrooting it. If this is incorrect, I'd love to know.


Here is a list of a nmap from another server on a completely seperate class C

I haven't done anything to the box to distrube any 'evidence', and haven't even rebooted it. The box however is located in a colo facility, about an hour away. I do have access, but as I stated above, I'm not sure exactly what to look for if I go there and boot to a live cd.

Hopefully I've given you enough information to paint a complete picture. I really appreciate all the advice so far.

//Reply sort order prioritised.

Thanks, I'll follow up on that link.
That actually would be the first thing to do after making a log entry and notifying people.


Any evidence that it was done is pretty well hidden as far as I can tell on the live OS.
It still would be good to take a chance, one of the tools to use would be unhide. Rootkit Hunter will recognise it as "plugin" and use it (currently only CVS version) if it's in your $PATH. That does not solve the "problem" of introducing stuff on the FS, but as with all things it's a trade-off. At the end of this post I'll post some output from a live system with a PID hidden with Adore-NG as reference.


I do have logrotate going, but unfortunately I never noticed that the default config has zero system logging in it.
No, I phrased it as being "some form of legitimate logrotating". With nx5000' reply above you now have a clue what was the cause, so
I have Knoppix, and can go to the datacenter this box is located in, but I'm not really sure what to look for.
that won't be necessary if there are no more doubts. If the box runs has a rootkit installed no tool can provide onehundred percent trustable output. Rebooting the system with a Live CD turns your live system into a "corpse" which, as goes for (nearly) all corpses, remains inert and doesn't run the perceived subverted kernel or init so that kind of trickery has no effect. Files remain unhidden (at the expense of loosing process info at reboot). Make sure to mount disks readonly against tampering. Stuff to look for basically means a complete verification of the system. Manual inspection of auth and config files and logs, filesystem integrity check using Aide, Samhain or similar or your distro's package manager (caveat: scope) and what single-purpose tools and Chkrootkit and Rootkit Hunter can offer.


What's the best way to get this for you? a ps aux > filebalh is a 12k file. Is there a cleaner way to get that information?
Can't be helped. If the checksums would check out OK, you would just run "ssh user@host 'ps axfwwwe; lsof -w -n' 2>&1| bzip2 > /some/log.bz2" and post a D/L link.


I did a grep -are for 'bash' 'history' and 'bash_history' in all the access.log* files I have and came up with nothing. Any ideas on other keywords to grep for?
I'd rather check all logs for any anomalies.


From the reading I've done on the net, this seems like a false warning. I *think* I emailed the author about it once, but I'm not 100% positive anymore.
I haven't received any email from you, I'm sure ;-p


I was also under the impression that in Debian, while Apache starts via the root user, (..). Is this not the case?
That separation of bind to port vs request handling tasks is not Debian-specific but default Apache behaviour.


This box runs (..)
If 1) the versions are up to date, 2) all checksums are OK and 3) auth data and logs show no anomalies then there is little more to check. If you did install Aide or Samhain (with a copy of the binary, config and databases off-site) that would be a good start to check. You could also verify against a backup, but you would have to have independant and unambiguous means to verify the integrity of the *backup* is OK in the first place. Circular dependency and such.


Concluding this post with output from a live system with a PID hidden with Adore-NG as reference.
HTH