LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 12-05-2006, 11:21 PM   #1
Jukas
Member
 
Registered: Mar 2005
Posts: 141

Rep: Reputation: 15
Compromised?


reviewing today's logs/notices I found the following

Quote:
/etc/cron.daily/chkrootkit:
Warning: `//root/.bash_history' file size is zero
I also see the following

Quote:
nix:/etc/ssh# ls -lah //root/.bash_history
-rw------- 1 root root 314 2006-12-05 20:00 //root/.bash_history
nix:/etc/ssh# ls -lah /root/.bash_history
-rw------- 1 root root 314 2006-12-05 20:00 /root/.bash_history
Nothing else seems out of the ordinary, I don't see anything alarming (or don't realize it should be alarming) with a netstat -a, nmap localhost shows only ports I've opened, and my smtp and ftp banner messages are still identical. My root password hasn't changed, my ssh service disallows root, and only allows 2 users specified in AllowUsers to log into ssh.

Doing a who never shows anyone but me logged in, and the last users never shows anyone but myself (I understand that could have been modified).

I do allow passwords for SSH, but only two users are allowed access via the AllowedUsers statement (myself and a good friend). I do use iptables for a firewall, have a default policy of drop, allow established connections, and only allow new incoming connections for services I need to allow. I've verified that my mail server (postfix) isn't an open relay, my mysql server only listens on localhost and vsftpd only allows local users, and disallows root. I am runnign Debian Etc, and keep up to date via apt at least twice a week, or sooner if I get notification via email that a package I have installed has a security risk.

Is there any way I can find out if this box has been truely compromised? I'll reinstall the OS if need be, but this is a production box, running a mail and www server and it would be a real hassle to reinstall without a good reason.

If the box has been compromised, how can I find out how they got in, so I can be sure it's secured with a fresh OS install?
 
Old 12-05-2006, 11:47 PM   #2
chort
Senior Member
 
Registered: Jul 2003
Location: Silicon Valley, USA
Distribution: OpenBSD 4.6, OS X 10.6.2, CentOS 4 & 5
Posts: 3,660

Rep: Reputation: 69
Check the logs for your http server. If you really were compromised, that's the most likely avenue (bad php scripts, etc).
 
Old 12-05-2006, 11:58 PM   #3
Jukas
Member
 
Registered: Mar 2005
Posts: 141

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by chort
Check the logs for your http server. If you really were compromised, that's the most likely avenue (bad php scripts, etc).
I did a grep -are for 'bash' 'history' and 'bash_history' in all the access.log* files I have and came up with nothing. Any ideas on other keywords to grep for?

I was also under the impression that in Debian, while Apache starts via the root users, that session then dies, and all child processes run under the non privilidged www-data users (my primary reason for not goign through the hassle of chrooting Apache). Is this not the case?

My primary IT admin background is with Win 2000 and 2003 servers, and linux has been an ongoing project to broaden my horizons. I've been running Debian for well over a year and this is the first incident I've had where I wonder if I've been compromised. Being from a Windows background, I don't know all the ins & outs to linux type forensic work, so any examples you can provide of how to best track this down would be much appreciated.
 
Old 12-06-2006, 05:52 AM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,409
Blog Entries: 55

Rep: Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582
For clarity:
Code:
/etc/cron.daily/chkrootkit:
Warning: `//root/.bash_history' file size is zero
nix:/etc/ssh# ls -lah //root/.bash_history
-rw------- 1 root root 314 2006-12-05 20:00 //root/.bash_history
0. Notice it is better to post complete output, unless you're onehundred percent sure this is the only error,
1. A shell history file being zeroed out in itself does not mean the box is compromised, we need more evidence,
2. Notice the timestamp being exactly 8 PM. While a coincidence you could check for some form of legitimate "logrotating".


Nothing else seems out of the ordinary
And that is as it should be. I mean, this swings both ways. If the box is fine, it's fine, if the box was compromised in a "qualitatively good way" evidence would be hidden and you have to take that into account while auditing. This means the best way to unhide malarky (unless the box is in colo) is to reboot the box using a Live CD like Helix, KNOPPIX or your distro's rescue cd if any.


If you have no idea and if you're not accustomed to checking things the first to read is the Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html. Apps doing multiple checks like Chkrootkit, USAT, Rootkit Hunter and Tiger, a file integrity checker like Aide, Samhian or even tripwire (if you installed it) and single-purpose tools like unhide, skdet can supply part of the necessary audit caps but using a checklist provides you with the necessary overview and an ordered list of steps to follow so you know why you're doing it.


My POV is we need to build an understanding of the situation. The more we know the better we can tailor our advice to your situation. Here's some points in addition to the checklist:
- location and purpose of the box (any other apart from SMTP, FTP and HTTP?),
- audit and auth data returned from your checks,
- any logged anomalies from an IDS (if you run one),
- any logged anomalies from checking system, daemon and firewall logs,
- running services,
- process and open files listing,
- setuid root files (any anomalous files in temp dirs).
Add anything else you think may be important.


If a box is (percieved) compromised it should not be part of business ops for the duration of the investigation (contamination). It should not see any applications introduced on the filesystem during the investigation (disturbing "evidence"), if the box isn't in colo run your stuff from a CDR. Make an entry in your local log and add entries while you step through the checklist. Notify fellow users from your local box and start your investigation.
When done please post as much information as you can, but guard for posting repetitive lines.


Chances are that you find the box was not compromised. Then at least you get some experience from auditing the box and maybe a thing or two to change that will make future auditing more complete, easier, so this will not have been wasted time. Good luck.
 
Old 12-06-2006, 06:12 AM   #5
nx5000
Senior Member
 
Registered: Sep 2005
Location: Out
Posts: 3,307

Rep: Reputation: 57
It happens to me, like once a month my root history on debian is also cleared and that's very annoying. I haven't taken the time to search the reason, but I'm 100% sure the box is clean apart from this (running integrity checkers). So I would say that this only problem its not a proof of compromised.
 
Old 12-06-2006, 02:19 PM   #6
Jukas
Member
 
Registered: Mar 2005
Posts: 141

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by unSpawn
For clarity:

0. Notice it is better to post complete output, unless you're onehundred percent sure this is the only error,
Here is the complete logging notification from chkrootkit, the packet sniffer is intentional.

Code:
/etc/cron.daily/chkrootkit:
Warning: `//root/.bash_history' file size is zero
eth0: PACKET SNIFFER(/usr/sbin/snort[8978])
eth0:1: PACKET SNIFFER(/usr/sbin/snort[8978])
eth0:2: PACKET SNIFFER(/usr/sbin/snort[8978])
eth0:3: PACKET SNIFFER(/usr/sbin/snort[8978])
eth0:4: PACKET SNIFFER(/usr/sbin/snort[8978])
eth0:5: PACKET SNIFFER(/usr/sbin/snort[8978])
Quote:
Originally Posted by unSpawn
1. A shell history file being zeroed out in itself does not mean the box is compromised, we need more evidence,

2. Notice the timestamp being exactly 8 PM. While a coincidence you could check for some form of legitimate "logrotating".
I do have logrotate going, but unfortunately I never noticed that the default config has zero system logging in it.

Quote:
Originally Posted by unSpawn
Nothing else seems out of the ordinary
And that is as it should be. I mean, this swings both ways. If the box is fine, it's fine, if the box was compromised in a "qualitatively good way" evidence would be hidden and you have to take that into account while auditing. This means the best way to unhide malarky (unless the box is in colo) is to reboot the box using a Live CD like Helix, KNOPPIX or your distro's rescue cd if any.
That's what I'm worried about. [i]if[/if] I was compromised it was done in a 'good way' and not just some hackbot script kiddie. Any evidence that it was done is pretty well hidden as far as I can tell on the live OS. I have Knoppix, and can go to the datacenter this box is located in, but I'm not really sure what to look for.

Quote:
Originally Posted by unSpawn
If you have no idea and if you're not accustomed to checking things the first to read is the Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html.
Thanks, I'll follow up on that link.

Quote:
Originally Posted by unSpawn
- audit and auth data returned from your checks,
- any logged anomalies from an IDS (if you run one),
- any logged anomalies from checking system, daemon and firewall logs,
I haven't found any anomalies in the logs, but that doesn't mean they couldn't have been altered. I run both chkrootkit and rkhunter as well as snort. Chkrootkit has never warned about anything other than finding snort, and Rkhunter only ever complains about
Code:
Found warnings:
[06:25:34] WARNING, found:  /dev/.static (directory)  /dev/.udev (directory) 
/dev/.initramfs (directory)
From the reading I've done on the net, this seems like a false warning. I *think* I emailed the author about it once, but I'm not 100% positive anymore.

[QUOTE=unSpawn]
- running services,
- process and open files listing,
[quote]

What's the best way to get this for you? a ps aux > filebalh is a
12k file. Is there a cleaner way to get that information?

Quote:
Originally Posted by unSpawn
- setuid root files (any anomalous files in temp dirs).
my /tmp dir has noexec and nosetuid and has nothing listed but lost+found and clamav files

Here are all files with a setuid of root.

Code:
nix:/# find / -user root -perm -4000 -print
/usr/bin/newgrp
/usr/bin/chfn
/usr/bin/chsh
/usr/bin/gpasswd
/usr/bin/passwd
/usr/bin/gpg
/usr/bin/traceroute.lbl
/usr/bin/mtr
/usr/bin/procmail
/usr/bin/sudo
/usr/bin/sudoedit
/usr/lib/pt_chown
/usr/lib/apache/suexec.disabled
/usr/lib/openssh/ssh-keysign
/usr/lib/apache2/suexec
/usr/lib/libfakeroot-tcp.so
/usr/lib/libfakeroot-sysv.so
/bin/su
/bin/mount
/bin/umount
/bin/ping
/bin/ping6
/lib/uncompress.so
find: /proc/11454/task/11454/fd/4: No such file or directory
find: /proc/11454/fd/4: No such file or directory
/sbin/unix_chkpwd

nix:/# find / -group kmem -perm -2000 -print
find: /proc/11456/task/11456/fd/4: No such file or directory
find: /proc/11456/fd/4: No such file or directory
There are no running processes that match the pids 11454 or 11456.

Quote:
Originally Posted by unSpawn
Add anything else you think may be important.
This box runs SSH, Bind, Vsftpd, Postfix, Courier, Amavisd, Spamassassin, Apache 1.33 (Apache2 was installed by apt when I installed php5, but doesn't run), PHP5, Mysql 5.0.27, Roundcube Webmail, Chkrootkit, Rkhunter, Snort, and has been running breakinguard, but I've removed it as it's too buggy.

The ftp service only allows local users, chroots everyone but me, and doesn't allow root, or anonymous users to log in. SSH disallows root explicitly, and only 2 users are allowed via the AllowedUsers command.

Mail is local delivery only, for a few of my domains. Users are stored in a mysql db with a md5 hash for passwords, and mail is stored in Maildir format. Postfix is chrooted.

Bind is run chrooted.

Apache hosts three of my personal domains and a single domain for a friend. They do not have shell access to the server. Apache master process starts as root, which immediately dies, the actual child processes are owned by www-data. According to my reading this should prevent Apache from gaining root access, which is why I never bothered to go through the hassle of chrooting it. If this is incorrect, I'd love to know.


Here is a list of a nmap from another server on a completely seperate class C

Code:
Not shown: 1667 filtered ports
PORT    STATE  SERVICE
20/tcp  closed ftp-data
21/tcp  open   ftp
22/tcp  open   ssh
25/tcp  open   smtp
53/tcp  open   domain
80/tcp  open   http
106/tcp closed pop3pw
110/tcp open   pop3
143/tcp open   imap
443/tcp open   https
587/tcp open   submission
783/tcp closed spamassassin
993/tcp open   imaps
Quote:
Originally Posted by unSpawn
If a box is (percieved) compromised it should not be part of business ops for the duration of the investigation (contamination). It should not see any applications introduced on the filesystem during the investigation (disturbing "evidence"), if the box isn't in colo run your stuff from a CDR. Make an entry in your local log and add entries while you step through the checklist. Notify fellow users from your local box and start your investigation.
I haven't done anything to the box to distrube any 'evidence', and haven't even rebooted it. The box however is located in a colo facility, about an hour away. I do have access, but as I stated above, I'm not sure exactly what to look for if I go there and boot to a live cd.

Hopefully I've given you enough information to paint a complete picture. I really appreciate all the advice so far.
 
Old 12-06-2006, 08:16 PM   #7
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,409
Blog Entries: 55

Rep: Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582
//Reply sort order prioritised.

Thanks, I'll follow up on that link.
That actually would be the first thing to do after making a log entry and notifying people.


Any evidence that it was done is pretty well hidden as far as I can tell on the live OS.
It still would be good to take a chance, one of the tools to use would be unhide. Rootkit Hunter will recognise it as "plugin" and use it (currently only CVS version) if it's in your $PATH. That does not solve the "problem" of introducing stuff on the FS, but as with all things it's a trade-off. At the end of this post I'll post some output from a live system with a PID hidden with Adore-NG as reference.


I do have logrotate going, but unfortunately I never noticed that the default config has zero system logging in it.
No, I phrased it as being "some form of legitimate logrotating". With nx5000' reply above you now have a clue what was the cause, so
I have Knoppix, and can go to the datacenter this box is located in, but I'm not really sure what to look for.
that won't be necessary if there are no more doubts. If the box runs has a rootkit installed no tool can provide onehundred percent trustable output. Rebooting the system with a Live CD turns your live system into a "corpse" which, as goes for (nearly) all corpses, remains inert and doesn't run the perceived subverted kernel or init so that kind of trickery has no effect. Files remain unhidden (at the expense of loosing process info at reboot). Make sure to mount disks readonly against tampering. Stuff to look for basically means a complete verification of the system. Manual inspection of auth and config files and logs, filesystem integrity check using Aide, Samhain or similar or your distro's package manager (caveat: scope) and what single-purpose tools and Chkrootkit and Rootkit Hunter can offer.


What's the best way to get this for you? a ps aux > filebalh is a 12k file. Is there a cleaner way to get that information?
Can't be helped. If the checksums would check out OK, you would just run "ssh user@host 'ps axfwwwe; lsof -w -n' 2>&1| bzip2 > /some/log.bz2" and post a D/L link.


I did a grep -are for 'bash' 'history' and 'bash_history' in all the access.log* files I have and came up with nothing. Any ideas on other keywords to grep for?
I'd rather check all logs for any anomalies.


From the reading I've done on the net, this seems like a false warning. I *think* I emailed the author about it once, but I'm not 100% positive anymore.
I haven't received any email from you, I'm sure ;-p


I was also under the impression that in Debian, while Apache starts via the root user, (..). Is this not the case?
That separation of bind to port vs request handling tasks is not Debian-specific but default Apache behaviour.


This box runs (..)
If 1) the versions are up to date, 2) all checksums are OK and 3) auth data and logs show no anomalies then there is little more to check. If you did install Aide or Samhain (with a copy of the binary, config and databases off-site) that would be a good start to check. You could also verify against a backup, but you would have to have independant and unambiguous means to verify the integrity of the *backup* is OK in the first place. Circular dependency and such.


Concluding this post with output from a live system with a PID hidden with Adore-NG as reference.
Code:
]# gfind /var/log/ksymoops/ adore | head -2
.ksyms:dcd0a000 __insmod_adore_O/some/dir/adore.o_M45587176_V132129    [adore]
.ksyms:dcd0a060 __insmod_adore_S.text_L5081     [adore]

]# gfind /var/log/ksymoops/ cleaner | head -2
.ksyms:dcd0e060 __insmod_cleaner_S.text_L31     [cleaner]
.ksyms:dcd0e000 __insmod_cleaner_O/some/dir/cleaner.o_M45587177_V132129        [cleaner]

]$ rkscan | grep -i detect
  ADORE rootkit NOT DETECTED on this system.
  KNARK rootkit NOT DETECTED on this system.
]# no result

]# /tmp/skdet -a | grep invis
13174   [invisible]

]# unhide sys
Found HIDDEN PID: 13174

]# unhide-tcp 
Found HIDDEN PORT: 22
Found HIDDEN PORT: 33850
HTH
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Compromised ? ./2[1].6.12 DaveQB Linux - Security 4 10-10-2006 07:47 PM
Compromised??? redice Linux - Security 5 02-25-2006 02:14 PM
Compromised? I can't tell. Chuck23 Linux - Security 11 02-15-2005 08:33 AM
Help! My system's been compromised.... DaVenom Linux - Security 1 11-12-2004 03:49 PM
Am I compromised? dripter Linux - Security 5 01-27-2004 01:31 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 09:14 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration