Looking for any suggestions, comments, etc...

model: Every two hours I collect some system statistics and they get compared, along with program and file system info, versus the previous collection of information, emailing me with any discrepancies....


cron:
0 3 * * * /usr/bin/rkhunter --update
0 */2 * * * /usr/local/sbin/collector.pl
20 * * * * /usr/local/sbin/ids.sh
30 3 * * * /usr/local/sbin/backup.sh

-----------------------------------------------
[collector.pl]

#!/usr/bin/perl -w

use strict;

my %Cmds;
my $host = qw(XXXXX);
my $user = "root";
my $externalip = "X.X.X.X";

chdir "/data";

my @md5files = qw(/bin/login
/usr/bin/passwd
/bin/ps);

my ($Second, $Minute, $Hour, $Day, $Month, $Year, $WeekDay, $DayOfYear, $IsDST) = localtime(time);

if ($Hour == 8) {
$Cmds{'disk.usage'} = "df -lk";
$Cmds{'packages'} = "yum check-update"; }

$Cmds{'md5sigs'} = "md5sum @md5files";
$Cmds{'suidfiles'} = "find / ! -wholename '/proc*' -type f -perm +6000 |xargs ls -l";
$Cmds{'cron.root'} = "crontab -l -u root";
$Cmds{'nmap'} = "nmap -sS $externalip | egrep -v '^(Nmap|Starting)'";
#$Cmds{'chkroot'} = "/usr/bin/chkrootkit";
$Cmds{'/dev/null'} = "updatedb";
#$Cmds{'/dev/null'} = "/usr/bin/rkhunter --update";
$Cmds{'rootkithunt'} = "/usr/bin/rkhunter -c --no-mail-on-warning --rwo --noappend-log --sk --nocolors";
#$Cmds{'iptables'} = "/sbin/iptables --list";
$Cmds{'listening'} = "netstat -utan | grep -i listen";
#$Cmds{'rootkithunt'} = "cat /var/log/rkhunter/rkhunter.log";

### main loop ###
for my $file (keys %Cmds) {
my $cmd = $Cmds{$file};

### run each command on $host and print the
### output to $file
&run_command($cmd, $file, $host);
}
exit 0;

sub run_command() {
my ($cmd, $file, $host) = @_;

my ($stdout, $stderr, $exit) = system($cmd." > $file");
return;
}


-----------------------------------------------
[ids.sh]

#!/bin/bash

## look for discrepanices

/usr/bin/perl /usr/local/sbin/mail-output.pl --subject "XXXXX.domain.net ETC Change" --recip admin@domain.net "diff -b -B -p -r -I \"Updated\" -X /home/backup/backup-excludes -u /home/backup/etc /etc"

/usr/bin/perl /usr/local/sbin/mail-output.pl --subject "XXXXX.domain.net Config Change" --recip admin@domain.net "diff -a -b -B -p -r -u -I \"The system checks took\" -I \"Host is up\" /home/backup/files /data/"

## copy files

echo "Starting IDS data sync at `date`." > /home/backup/backup-ids.log

echo "" >> /home/backup/backup-ids.log

echo "Backing up /etc/..." >> /home/backup/backup-ids.log rsync -a --delete /etc/ /home/backup/etc/ >> /home/backup/backup-ids.log 2>> /home/backup/backup-ids.err

echo "" >> /home/backup/backup-ids.log

echo "Backing up config files..." >> /home/backup/backup-ids.log rsync -a --delete /data/* /home/backup/files >> /home/backup/backup-ids.log 2>> /home/backup/backup-ids.err echo "" >> /home/backup/backup-ids.log

echo "Backing up system files..." >> /home/backup/backup-ids.log rsync -a --delete /usr/local/sbin/* /home/backup/usr/local/sbin/ >> /home/backup/backup-ids.log 2>> /home/backup/backup-ids.err echo "" >> /home/backup/backup-ids.log

echo "Backup finished at `date`." >> /home/backup/backup-ids.log


-----------------------------------------------
[backup.sh]

#!/bin/bash

## zip & send

tar -czvf /home/XXXXX.tar.gz /home/backup/*

ls -alR /home/backup > /home/dirlist.txt

mail -s "XXXXX Backup Configs" -r backups@domain.net -a /home/XXXXX.tar.gz admin@domain.net < /home/dirlist.txt

Welcome to LQ, hope you like it here.

...then comments is what you'll get.


- That's good to know, but what's missing is your decision for choosing a "poor man's IDS" over existing Off The Shelf solutions.
- What's also missing is a list of OSes or Linux distributions these scripts run on and the role of these machines. Knowing their role (what services and software gets exposed, how / if they're hardened properly and who accesses these machines) gives us a better idea of the attack surface and what would be missing from your "poor man's IDS".
* In the logical order of things IDS is complementary (running a hardened system w/o IDS is possible but running a system Out of the Box w/o hardening with an IDS is a waste of time and effort) and should therefore IMHO be focused on prevention (early warning and if possible proactive measures). And wrt early warning also have a look at LogWatch as it reports anomalies and errors for common system and daemon log files.


You don't need to check for RKH updates that often. One reason is that it strains the (remote mirror) excessively but more importantly because the developers just don't update RKH that often (yeah, I know, we should, but there's only two of us and it's not like we haven't got anything else to do).


To start with if you want to watch system watch directories and files for changes (early warning) there's different options:
- SELinux / setroubleshootd: targeted policy when configured OK will fire AVC messages if any discrepancies are seen (doesn't care about hashes though),
- the audit service: stock on Red Hat and equivalent, loads rules into memory so you just collect from /var/log/audit/audit.log,
- Samhain (not in any repo IIRC but compiles OK on CentOS w/o much dependencies) a continuously running process which has gotten a whole lot more efficient since using Inotify (can remotely load databases, do a gazillion other checks and email alerts as well),
and then there's a host of commands like 'rpm -V [packagename]|grep -v "^\.\{8\}";', 'yum verify', 'md5deep -r /sbin -c /path/to/checksumfile', AIDE, RKH's hash check. The downside of these commands is that they have to be executed manually or via a cron job so unless you set an insane timing scheme they'll not be reporting changes as fast as you would like.


While it's not the same (then again you don't white list allowed setuid / setgid root files either ;-p) RKH already incorporates a check looking for changed file permissions.
BTW the 'xargs ls -l' may be superfluous as 'find' has a "-ls" switch itself?


We've had a case a year or so ago where the Apache user loaded a crontab. "crontab -l" will use the 'crontab' binary while you could just 'find /var/spool/cron'. If you use /etc/cron.{allow,deny} (why not? ;-p) maybe parse that first and then decide which users cron spools to list?


...should yield better results when scanned from a remote machine.


Chkrootkit hasn't been updated in ages so you did well commenting it out.
While it's not the same (then again you don't white list allowed listening ports either ;-p) RKH already incorporates a check looking for listening processes.
And what would happen if you did find signs of a compromise? I wouldn't want to run 'updatedb' then. Or what if there's no change on the system? Would running 'updatedb' make sense then?



Efficiency-wise, should this really take two emails? And should you email when you're still halfway through the script?


Why don't you pipe through 'usr/bin/logger' instead? It can send to file, or better: syslog (time stamps) and you don't need these empty echo lines for readability.
And 'rsync' itself can log any output you want with "--log-file=" which is configurable with "--log-file-format="?
And why not 'rsync' over the 'net? Is having a local copy really that safe (if the machine gets compromised) or convenient (for example when a disk breaks down)?
Also you use static file names (I'd use 'mktemp') don't set an umask.


IMHO that's overkill as using 'tar -czv' already provides a list so why list twice? (Or else 'tar -tf /home/XXXXX.tar.gz' could but what would you need a separate list for?) Also you're not catching stderr which means you miss out on any logged errors.


To recap:
- proper hardening first,
- role / service-centric checks,
- use OTS SW and don't duplicate.


HTH

If you can get another box on the network, and can either put a tap on your existing network or switch that network card to promiscuous mode, I'd suggest checking out Security Onion. It uses a lot of good IDS tools to analyze anomalous traffic.

I am no security master, but I have been extremely impressed with Samhain. It is highly configurable, allows for file-watching of different types (read-only, log files which grow, etc.).

If your plan is to do it yourself, your interval (2 hours?) seems pretty long to me.