LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices

Reply
 
Search this Thread
Old 08-04-2011, 05:07 PM   #1
sneakyimp
Member
 
Registered: Dec 2004
Posts: 791

Rep: Reputation: 49
Intrusion Detection and File Integrity Monitoring on Amazon EC2 using samhain?


Thanks to the amazing unSpawn's herculean efforts at educating me, I have managed to instantiate, lock down, and harden (at a basic level anyway) a compute instance hosted by Amazon EC2. This has been tricky because the EC2 compute instances are on the Spamhaus policy block list which means they can't send outgoing mail. Amazon offers SES as a mail solution. If anyone is interested in configuring postfix to send mail through SES, please check this post on the AWS forums for sorting out SES integration problems and this other post here at LQ detailing how to prevent local mail delivery by postfix and to canonicalize recipient addresses which are both useful steps when dealing with EC2 and SES.

I am now setting up samhain (apt-cache search samhain for debian/ubuntu users) which should act as a watchdog for my server and inform me if certain changes are made, if the server reboots, etc. I'm hoping to get some answers from the pros about how best to set this up and what all I should do to get things really secure. While I don't expect to use the insane renamed-binary-and-steganography approach, I would like to implement an append-only log setup (MySQL or otherwise).

Per unSpawn's directions, I am posting a comment-filtered output of my samhain config with incriminating details redacted:
Code:
$ sudo grep -v ^# /etc/samhain/samhainrc|grep .;
[Misc]
[Attributes]
file=/etc/mtab
file=/etc/ssh_random_seed
file=/etc/asound.conf
file=/etc/resolv.conf
file=/etc/localtime
file=/etc/ioctl.save
file=/etc/passwd.backup
file=/etc/shadow.backup
file=/etc/postfix/prng_exch
file=/etc/adjtime
file=/etc/network/run/ifstate
file=/etc/lvm/.cache
file=/etc/ld.so.cache
file=/etc
[LogFiles]
file=/var/run/utmp
file=/etc/motd
[GrowingLogFiles]
file=/var/log/warn
file=/var/log/messages
file=/var/log/wtmp
file=/var/log/faillog
file=/var/log/auth.log
file=/var/log/daemon.log
file=/var/log/user.log
file=/var/log/kern.log
file=/var/log/syslog
[IgnoreAll]
file=/etc/resolv.conf.pcmcia.save
file=/etc/nologin
file=/etc/network/run
[IgnoreNone]
[Prelink]
[ReadOnly]
dir=/usr/bin
dir=/bin
dir=/boot
dir=3/sbin
dir=/usr/sbin
dir=/lib
dir=3/etc
file=/usr/lib/pt_chown
[User0]
[User1]
[EventSeverity]
SeverityReadOnly=crit
SeverityLogFiles=crit
SeverityGrowingLogs=warn
SeverityIgnoreNone=crit
SeverityAttributes=crit
SeverityIgnoreAll=info
SeverityFiles=crit
SeverityDirs=crit
SeverityNames=warn
[Log]
MailSeverity=crit
PrintSeverity=none
LogSeverity=info
SyslogSeverity=alert
ExportSeverity=none
 
LoginCheckActive = True
SeverityLogin=info
SeverityLoginMulti=crit
SeverityLogout=info
LoginCheckInterval = 300
[Misc]
Daemon = yes
ChecksumTest=check
SetLoopTime = 600
SetFileCheckTime = 7200
SetReverseLookup = False
SetMailTime = 86400
SetMailNum = 10
SetMailAddress=admin@mydomain.com
SetMailAddress=dev@mydomain.com
SetMailSender=noreply@mydomain.com
SetMailRelay = localhost
MailSubject = [Samhain at %H] %T: %S
SyslogFacility=LOG_LOCAL2
[EOF]
Question 1: How to set up append-only log?
I see in the samhainrc file that one can set MySQL credentials:
Code:
  SetDBName=db_name 
  SetDBTable=db_table
  SetDBHost=db_host 
  SetDBUser=db_user 
  SetDBPassword=db_password
  UsePersistent=yes/no
Unfortunately, the documentation does not indicate what permissions are needed by this database user. I suspect only INSERT is required and possibly SELECT. If it's just INSERT and SELECT then I believe this would be an effective way to implement an append-only log. If not, I would like to learn more about SetLogServer usage as the documentation neglects to mention what sort of protocol is required or how this function might work at all.

Question 2: Tweaks to samhainrc?
It looks like I'll need to tweak samhainrc to remove references to files which don't exist on my machine and add references for files that correspond to the Apache/MySQL/PHP stack I expect to install. Some things already learned:
* Ubuntu doesn't support the kernel check.
* this command yields a list of files mentioned in samhainrc which do not exist on my machine:
Code:
sudo awk -F'=' '/file=/ {print $NF}' samhainrc|xargs -iX stat -c '%N' 'X' >/dev/null
The result:
Code:
stat: cannot stat `/etc/ssh_random_seed': No such file or directory
stat: cannot stat `/etc/asound.conf': No such file or directory
stat: cannot stat `/etc/ioctl.save': No such file or directory
stat: cannot stat `/etc/passwd.backup': No such file or directory
stat: cannot stat `/etc/shadow.backup': No such file or directory
stat: cannot stat `/etc/postfix/prng_exch': No such file or directory
stat: cannot stat `/etc/adjtime': No such file or directory
stat: cannot stat `/etc/network/run/ifstate': No such file or directory
stat: cannot stat `/etc/lvm/.cache': No such file or directory
stat: cannot stat `/var/log/warn': No such file or directory
stat: cannot stat `/etc/resolv.conf.pcmcia.save': No such file or directory
stat: cannot stat `/etc/nologin': No such file or directory
stat: cannot stat `/etc/network/run': No such file or directory
stat: cannot stat `/usr/lib/apache/suexec': No such file or directory
stat: cannot stat `/usr/lib/apache/suexec.disabled': No such file or directory
Do I need to remove these or change them to the Debian/Ubuntu equivalent? Or just leave them?
* This command should yield a list of the open files in /var/log, excluding those results containing one of these words: cache, run, db, lib, and spool.
Code:
$ sudo lsof -Pwln +D/var/|awk '/\// {print $NF}'|egrep -v "(cache|run|db|lib|spool)/"|sort -u | xargs ls -l;
ls: cannot access NAME: No such file or directory
-rw-r----- 1 syslog adm  117749 2011-08-04 21:40 /var/log/auth.log
-rw-r--r-- 1 root   root  12164 2011-08-04 21:30 /var/log/ConsoleKit/history
-rw-r----- 1 syslog adm    3304 2011-08-04 19:09 /var/log/daemon.log
-rw-r----- 1 syslog adm    4342 2011-08-04 19:10 /var/log/debug
-rw-r----- 1 syslog adm   40220 2011-08-04 19:10 /var/log/kern.log
-rw-r----- 1 syslog adm       0 2011-07-20 18:56 /var/log/lpr.log
-rw-r----- 1 syslog adm   94410 2011-08-04 20:12 /var/log/mail.err
-rw-r----- 1 syslog adm  495054 2011-08-04 20:12 /var/log/mail.info
-rw-r----- 1 syslog adm  495054 2011-08-04 20:12 /var/log/mail.log
-rw-r----- 1 syslog adm  285893 2011-08-04 20:12 /var/log/mail.warn
-rw-r----- 1 syslog adm   37454 2011-08-04 19:09 /var/log/messages
-rw-r----- 1 syslog adm       0 2011-07-20 18:56 /var/log/news/news.crit
-rw-r----- 1 syslog adm       0 2011-07-20 18:56 /var/log/news/news.err
-rw-r----- 1 syslog adm       0 2011-07-20 18:56 /var/log/news/news.notice
-rw-r----- 1 syslog adm   32771 2011-08-04 21:17 /var/log/syslog
-rw-r----- 1 syslog adm       0 2011-07-20 18:56 /var/log/ufw.log
-rw-r----- 1 syslog adm       0 2011-07-24 06:42 /var/log/user.log
I am not sure whether to put these files in the LogFiles section of samhainrc or the GrowingLogFiles section or whether to leave them out entirely. lpr, news, user, ufw do not appear to be used at all. The rest are likely to grow until they get rotated by the log rotation function. Advice would be much appreciated.
* Any advice/thoughts/heuristic guidelines regarding the other settings?

I've also got questions about the notifications I'm (gratefully) receiving. Here's one:
Code:
-----BEGIN MESSAGE-----
[2011-08-01T21:01:42+0000] ip-WWW-XXX-YYY-ZZZ.ec2.internal
ALERT  :  [2011-08-01T21:01:42+0000] msg=<START>, program=<Samhain>, userid=<0>, path=</etc/samhain/samhainrc>, hash=<AB929C7CA3F2E147DDDF18FA37568940B6F75F82912816E6>, path=</var/state/samhain/samhain_file>, hash=<D02B86008E9343562721282B8997F5083D26176A0C3F6B94>
-----BEGIN SIGNATURE-----
DEE70E65B40FBB3234F52ABDD65FF61D7B3F56A6C4C2B479
000000 1312232562::ip-10-100-237-252.ec2.internal
-----END MESSAGE-----
This appears to be the type of mesg sent whenever samhain starts -- either at bootup or after I manually reboot it. I see that it's signed. I've no idea how to validate this signature. I would need a public key, wouldn't I? Wondering where this key might be.

Here's another notification I received. I don't know what it's telling me:
Code:
-----BEGIN MESSAGE-----
[2011-08-01T21:02:43+0000] ip-WWW-XXX-YYY-ZZZ.ec2.internal
CRIT   :  [2011-08-01T21:02:42+0000] msg=<POLICY [ReadOnly] C-------TS>, path=</etc/samhain/samhainrc>, size_old=<15470>, size_new=<15559>, ctime_old=<[2011-08-01T02:54:40]>, ctime_new=<[2011-08-01T21:01:32]>, mtime_old=<[2008-11-11T21:12:42]>, mtime_new=<[2011-08-01T21:01:32]>, chksum_old=<D2A96989673435CF2BC499D37DBD76EF3E743E6D1F9E4F0B>, chksum_new=<AB929C7CA3F2E147DDDF18FA37568940B6F75F82912816E6>,
CRIT   :  [2011-08-01T21:02:42+0000] msg=<POLICY [ReadOnly] C--I----TS>, path=</etc/aliases>, inode_old=<181511>, inode_new=<181563>, dev_old=<8,1>, dev_new=<8,1>, size_old=<51>, size_new=<72>, ctime_old=<[2011-07-28T00:09:41]>, ctime_new=<[2011-08-01T20:43:28]>, mtime_old=<[2011-07-28T00:09:41]>, mtime_new=<[2011-08-01T20:43:28]>, chksum_old=<820C14F10AFCE63DD039005F3BB5C0795CF724D39EA50573>, chksum_new=<A6B503E31CD9DDBAE4C670690D325ED1FE11D4694E3DD951>,
CRIT   :  [2011-08-01T21:02:43+0000] msg=<POLICY [ReadOnly] --------T->, path=</etc/rc1.d>, ctime_old=<[2011-07-28T00:09:40]>, ctime_new=<[2011-08-01T03:25:52]>, mtime_old=<[2011-07-28T00:09:40]>, mtime_new=<[2011-08-01T03:25:52]>,
CRIT   :  [2011-08-01T21:02:43+0000] msg=<POLICY ADDED>, path=</etc/rc1.d/K19samhain>, mode_new=<lrwxrwxrwx>, attr_new=<------------>, imode_new=<41471>, iattr_new=<0>, hardlinks_new=<1>, idevice_new=<0>, inode_new=<181556>, dev_new=<8,1>, owner_new=<root>, iowner_new=<0>, group_new=<root>, igroup_new=<0>, size_old=<0>, size_new=<17>, ctime_new=<[2011-08-01T03:25:52]>, atime_new=<[2011-08-01T03:31:15]>, mtime_new=<[2011-08-01T03:25:52]>, chksum_new=<000000000000000000000000000000000000000000000000>, link_new=</etc/rc1.d/../init.d/samhain>
CRIT   :  [2011-08-01T21:02:43+0000] msg=<POLICY [ReadOnly] --------T->, path=</etc/rc6.d>, ctime_old=<[2011-07-28T00:09:40]>, ctime_new=<[2011-08-01T03:25:52]>, mtime_old=<[2011-07-28T00:09:40]>, mtime_new=<[2011-08-01T03:25:52]>,
CRIT   :  [2011-08-01T21:02:43+0000] msg=<POLICY ADDED>, path=</etc/rc6.d/K19samhain>, mode_new=<lrwxrwxrwx>, attr_new=<------------>, imode_new=<41471>, iattr_new=<0>, hardlinks_new=<1>, idevice_new=<0>, inode_new=<181557>, dev_new=<8,1>, owner_new=<root>, iowner_new=<0>, group_new=<root>, igroup_new=<0>, size_old=<0>, size_new=<17>, ctime_new=<[2011-08-01T03:25:52]>, atime_new=<[2011-08-01T03:31:16]>, mtime_new=<[2011-08-01T03:25:52]>, chksum_new=<000000000000000000000000000000000000000000000000>, link_new=</etc/rc6.d/../init.d/samhain>
CRIT   :  [2011-08-01T21:02:43+0000] msg=<POLICY ADDED>, path=</etc/aliases.db>, mode_new=<-rw-r--r-->, attr_new=<------------>, imode_new=<33188>, iattr_new=<0>, hardlinks_new=<1>, idevice_new=<0>, inode_new=<181521>, dev_new=<8,1>, owner_new=<root>, iowner_new=<0>, group_new=<root>, igroup_new=<0>, size_old=<0>, size_new=<12288>, ctime_new=<[2011-08-01T20:43:33]>, atime_new=<[2011-08-01T21:00:11]>, mtime_new=<[2011-08-01T20:43:33]>, chksum_new=<E89C32A2F6007AFE4845DA7A9966CFB30492314DD330ECBE>
CRIT   :  [2011-08-01T21:02:43+0000] msg=<POLICY [ReadOnly] --------T->, path=</etc/rc5.d>, ctime_old=<[2011-07-28T00:09:40]>, ctime_new=<[2011-08-01T03:25:52]>, mtime_old=<[2011-07-28T00:09:40]>, mtime_new=<[2011-08-01T03:25:52]>,
CRIT   :  [2011-08-01T21:02:43+0000] msg=<POLICY ADDED>, path=</etc/rc5.d/S19samhain>, mode_new=<lrwxrwxrwx>, attr_new=<------------>, imode_new=<41471>, iattr_new=<0>, hardlinks_new=<1>, idevice_new=<0>, inode_new=<181561>, dev_new=<8,1>, owner_new=<root>, iowner_new=<0>, group_new=<root>, igroup_new=<0>, size_old=<0>, size_new=<17>, ctime_new=<[2011-08-01T03:25:52]>, atime_new=<[2011-08-01T03:31:16]>, mtime_new=<[2011-08-01T03:25:52]>, chksum_new=<000000000000000000000000000000000000000000000000>, link_new=</etc/rc5.d/../init.d/samhain>
CRIT   :  [2011-08-01T21:02:43+0000] msg=<POLICY [ReadOnly] --------T->, path=</etc/rc2.d>, ctime_old=<[2011-07-28T00:09:40]>, ctime_new=<[2011-08-01T03:25:52]>, mtime_old=<[2011-07-28T00:09:40]>, mtime_new=<[2011-08-01T03:25:52]>,
-----BEGIN SIGNATURE-----
C201A07222694F86DDFA88F3C4518C7E6E471E5388754030
000001 1312232562::ip-WWW-XXX-YYY-ZZZ.ec2.internal
-----END MESSAGE-----
And one last question: How can I be sure that samhain is always running? I know that I've received EXIT notifications from it, but what happens when someone gives it a kill -9 ?

I'd appreciate any input you kind folks might have to help me get a leg up on this.

Last edited by sneakyimp; 08-08-2011 at 11:22 AM. Reason: Updated settings.
 
Old 08-05-2011, 07:58 AM   #2
idlehands
Member
 
Registered: Mar 2010
Distribution: zLinux, RHEL, Ubuntu, SUSE
Posts: 50

Rep: Reputation: 16
What is the benefit of using samhain for this vs auditd and auditsp plugin to remotely send audit events?
 
Old 08-05-2011, 02:20 PM   #3
sneakyimp
Member
 
Registered: Dec 2004
Posts: 791

Original Poster
Rep: Reputation: 49
I'm not familiar with auditd/auditsp. I know that samhain has a variety of neat options: you can specify a database server for append-only logs, you can specify which mail server to use for alerts, it's highly configurable, etc.
 
Old 08-06-2011, 03:00 AM   #4
idlehands
Member
 
Registered: Mar 2010
Distribution: zLinux, RHEL, Ubuntu, SUSE
Posts: 50

Rep: Reputation: 16
auditd is the linux audit subsystem. You can configure what system calls to audit, and also files, directories etc. i.e. if anyone reads or updates anything in /etc/* a record can be cut

auditsp is a plugin that lets you reroute audit records to a remote auditd

you can also add things on top, say TSIEM (enterprise$$) , splunk and do event notification, correlation(5 failed logins across 2 accounts send me an email) kind of stuff based on your audit records/syslog

again, I'm not familiar with samhain but it sounds like it is outside of the kernel hooks that auditd can monitor. though it has some overlap, so if you were really paranoid both might be good to have.
 
Old 08-06-2011, 06:39 AM   #5
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,944
Blog Entries: 54

Rep: Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731
Quote:
Originally Posted by idlehands View Post
I'm not familiar with samhain
That is something you can fix yourself by reading up on things.


Quote:
Originally Posted by idlehands View Post
it sounds like it is outside of the kernel hooks that auditd can monitor.
As stated in the other thread Samhains kernel watch doesn't work with kernels that have been patched to deny /dev/kmem access.


Quote:
Originally Posted by idlehands View Post
if you were really paranoid both might be good to have.
As you don't explain the difference between "not paranoid" and "paranoid" your comment, like the regularly seen "I don't think that" and "don't worry" ones, is subjective, has no value and is therefore best left out.


Quote:
Originally Posted by idlehands View Post
you can also add things on top, say TSIEM (enterprise$$) , splunk and do event notification, correlation(5 failed logins across 2 accounts send me an email) kind of stuff based on your audit records/syslog
Indeed a good thing. In short security is a continuous process and it requires several layers.
 
1 members found this post helpful.
Old 08-06-2011, 06:45 AM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,944
Blog Entries: 54

Rep: Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731
Quote:
Originally Posted by idlehands View Post
What is the benefit of using samhain for this vs auditd and auditsp plugin to remotely send audit events?
Samhain is a complete intrusion detection application. Auditd logs near real-time while Samhain doesn't. You wouldn't be able to express 'HTTPDUID=$(getent passwd apache | awk -F ':' '{print $3}'); rpm -ql perl-libwww-perl wget curl elinks lynx|grep bin/|awk '{ print "-a always,exit -F path="$1" -F perm=x -F auid=%HTTPDUID -k HTTPD_problem"}'|sed -e "s|%HTTPDUID|$HTTPDUID|g"' in Samhain or tag network traffic. In short they complement each other in more than one way.
 
Old 08-06-2011, 08:58 PM   #7
sneakyimp
Member
 
Registered: Dec 2004
Posts: 791

Original Poster
Rep: Reputation: 49
I hope that we might keep this thread focused on my configuration efforts.

I am struggling to get the append-only log set up. I believe I have altered the samhainrc file properly because samhain no longer complains, but it does not appear to write anything to my database (which is an Amazon RDS instance). I cannot tell if this is because I have not granted permissions properly on the remote machine (which I tend to doubt) or whether it's something else. Is there anywhere I can look to see error messages generated by samhain in relation to a database? I've been watching /var/log/samhain/samhain.log and it doesn't seem to say anything about mysql or a database and yet nothing is written to the db. I'm also unsure how to check any logs on the RDS instance because I don't have ssh access to check any logs there.

My settings:
Code:
# IMPORTANT:  these settings apparently must appear in this order for some weird reason
DatabaseSeverity=warn
I kept getting an error about an unrecognized option until I put this directive after ExportSeverity. Apparently samhain requires these options to be in order
Code:
[Database]
SetDBName = mydomain_samhain
SetDBTable = log
SetDBUser = samhain
SetDBPassword = myPassword
SetDBHost = mydomain-samhain.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com
UsePersistent = False
I also realize today that samhain had not been running since about August 4th because of a missing /var/run/samhain folder. I'm not sure how this folder might have gotten deleted, but it resulted in samhain quitting each time I restarted it. Because I'm not really versed in interpreting samhain notifications, I did not realize this. The first log entry from Aug 4th:
Code:
[SOF]
ERROR  :  [2011-08-04T19:09:52+0000] msg=<Could not write PID file>, userid=<0>, path=</var/run/samhain/samhain.pid>
35098D4976AB68C316F6F71FF715D3D57C245199B09BBA45[2011-08-04T19:09:59+0000]
ALERT  :  [2011-08-04T19:09:59+0000] msg=<EXIT>, program=<Samhain>, status=<None>
87FBAF3D77C5F907DE0E711404792FD833EF7C99E6499B9D
I was definitely fiddling around with various configuration files at that point and may have deleted the file inadvertently, but I don't see anything in my bash_history that looks like it would have done this.
 
Old 08-07-2011, 02:26 AM   #8
sneakyimp
Member
 
Registered: Dec 2004
Posts: 791

Original Poster
Rep: Reputation: 49
After trying to enable some additional checks, samhain now complains about unrecognized section headers and configuration values. These were in the original samhainrc and are described in the documentation. Why would samhain claim they are unrecognized?

For example, it complains about line 386 which contains "[Utmp]" here:
Code:
[Utmp]
##
## --- Logging of login/logout events
##

## Switch on/off
LoginCheckActive = True

## Severity for logins, multiple logins, logouts
#
SeverityLogin=info
SeverityLoginMulti=crit
SeverityLogout=info

## Interval for login/logout checks
#
LoginCheckInterval = 300
And also about line 406 which is "[Database]".

It also complains about line 305:
Code:
ExportSeverity=none
and 308:
Code:
DatabaseSeverity=warn
What the heck? The errors from samhain.log:
Code:
WARN   :  [2011-08-07T06:48:18+0000] msg=<Invalid line 305 in configuration file: incorrect format, unrecognized option, or missing section header>
WARN   :  [2011-08-07T06:48:18+0000] msg=<Invalid line 308 in configuration file: incorrect format, unrecognized option, or missing section header>
WARN   :  [2011-08-07T06:48:18+0000] msg=<Unrecognized section heading in line 386 of configuration file>
WARN   :  [2011-08-07T06:48:18+0000] msg=<Unrecognized section heading in line 406 of configuration file>
 
Old 08-07-2011, 05:20 PM   #9
sneakyimp
Member
 
Registered: Dec 2004
Posts: 791

Original Poster
Rep: Reputation: 49
*sigh*. It has become apparent to me that the ubuntu (and possibly debian) package version of samhain does not support the [Utmp] and [Database] features because the binary was compiled without them. If I want these features (which sound necessary if I want a read-only log) then it sounds like I'll need to compile samhain from source myself. This sounds like a fairly daunting task as the package installed numerous files. I find myself wondering if the configure/make/make install will result in these features:
* samhain starts as daemon at startup or any time server reboots
* samahin can be controlled by command /etc/init.d/samhain restart/stop/start/reload/force-reload
* samhain configuration file is located at /etc/samhain/samhainrc
* samhain performs file integrity checks and intrusion detection for ubuntu/debian specific file locations and os features
* samhain logs to a MySQL database
* samhain signs all messages
* samhain writes a lot to /var/log/samhain/samhain.log
 
Old 08-07-2011, 06:52 PM   #10
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,944
Blog Entries: 54

Rep: Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731Reputation: 2731
Quote:
Originally Posted by sneakyimp View Post
If I want these features (which sound necessary if I want a read-only log) then it sounds like I'll need to compile samhain from source myself.
Alternatively you could contend yourself right now by keeping a copy of the config, binary and database at a remote location, and forward Samhain logging to remote syslog. It's not what you want but keeping stuff remote offers at least one more hurdle (as long as the syslog server is secured properly) and allows you to move on and revisit it later on. Generic rsyslog recipe, do adjust:
Code:
$InputFileName /var/log/samhain.log
$InputFileTag samhain
$InputFileStateFile samhain.state
$InputFileSeverity info
$InputFileFacility local10
$InputRunFileMonitor

Quote:
Originally Posted by sneakyimp View Post
This sounds like a fairly daunting task as the package installed numerous files.
Compiling a package usually needs a './configure; make; make install' triplet. If you replace the last with 'make -n install 2>&1 | tee logfile' it will --dry-run the installation and dump its log in "logfile" for your perusal. Should show you where everything goes.


Quote:
Originally Posted by sneakyimp View Post
I find myself wondering
Source tarball doc/ directory, running './configure --help'?


Quote:
Originally Posted by sneakyimp View Post
starts as daemon at startup or any time server reboots / can be controlled by command /etc/init.d/samhain restart/stop/start/reload/force-reload
If you look at the tarball source init/samhain.startLinux.in shows it supports Debians /etc/default/rcS.


Quote:
Originally Posted by sneakyimp View Post
configuration file is located at /etc/samhain/samhainrc
Configure option "--with-config-file=/path/to/file".


Quote:
Originally Posted by sneakyimp View Post
performs file integrity checks and intrusion detection for ubuntu/debian specific file locations and os features
OS yes, Debian / Ubuntu-specific no: you have to configure what you want to audit yourself.


Quote:
Originally Posted by sneakyimp View Post
logs to a MySQL database
Configure option "--enable-xml-log --with-database=mysql".


Quote:
Originally Posted by sneakyimp View Post
signs all messages
You mean signed configuration file (docs/MANUAL-2_3.tar/signed-files.html)? (Configure option "--with-gpg=/path/to/gpg".)


Quote:
Originally Posted by sneakyimp View Post
writes (..) to /var/log/samhain/samhain.log
Configure option "--with-log-file=/path/to/file".
 
Old 08-07-2011, 08:45 PM   #11
sneakyimp
Member
 
Registered: Dec 2004
Posts: 791

Original Poster
Rep: Reputation: 49
Very helpful info. THANK YOU. Especially the configuration options.

Quote:
Originally Posted by unSpawn View Post
Alternatively you could contend yourself right now by keeping a copy of the config, binary and database at a remote location, and forward Samhain logging to remote syslog. It's not what you want but keeping stuff remote offers at least one more hurdle (as long as the syslog server is secured properly) and allows you to move on and revisit it later on. Generic rsyslog recipe, do adjust:
Code:
$InputFileName /var/log/samhain.log
$InputFileTag samhain
$InputFileStateFile samhain.state
$InputFileSeverity info
$InputFileFacility local10
$InputRunFileMonitor
What is rsyslog? I've tried man rsyslog with nothing returned. I see there are a few debian packages for rsyslog but, given my problems with the samhain package, I'm not feeling a lot of love or trust for .deb packages. Additionally, it is one more learning curve I must tackle. This is taking weeks now and I'm feeling quite frustrated. I believe I'll take my chances with recompiling samhain -- although the missing /var/run/samhain folder problem that started on 8/4 really bothers me. I could have fat-fingered something and deleted this, but I don't think that's likely. Another possibility is that a hard reboot forced by the EC2 system when I take a snapshot may have caused a problem (although I doubt this too).

Quote:
Originally Posted by unSpawn View Post
If you look at the tarball source init/samhain.startLinux.in shows it supports Debians /etc/default/rcS.
We've briefly discussed /etc/default/rcS, but the man page for this just says it is "variables that affect the behavior of boot scripts" and says that one may set TMPTIME, SULOGIN, DELAYLOGIN, UTC, VERBOSE, and FSCKFIX. I don't understand what it means for samhain's tarball to support Debian's /etc/default/rcs.

Quote:
Originally Posted by unSpawn View Post
OS yes, Debian / Ubuntu-specific no: you have to configure what you want to audit yourself.
I figure this might be feasible by keeping a copy of the samhainrc installed by the debian package. However, I had questions about that in the original post above which are still unanswered.

Quote:
Originally Posted by unSpawn View Post
You mean signed configuration file (docs/MANUAL-2_3.tar/signed-files.html)? (Configure option "--with-gpg=/path/to/gpg".)
To be honest, my understanding of the use of keys in the context of samhain is still quite muddled. The samhainrc file that came via apt-get install samhain did not have anything resembling a signature. On the other hand, every single email that I received from samhain had a signature on it. The documentation you so kindly linked seems to have lots of information about the preferred setup, but really don't provide much context for newcomer to understand what this means. I've read it numerous times but I still don't understand how signing the configuration file makes anything more secure if the private key used to sign it exists on the machine where samhain is running. Being quite unfamiliar with gpg, I'm also unsure how to verify the signature of an email received from samhain. I understand that I should be able to use gpg, feed it the message, and that I would need a public key from somewhere. Or should I be using these LOGKEYs that samhain was mailing to me:
Code:
-----BEGIN MESSAGE-----
[2011-08-07T01:37:29+0000] ip-WWW-XXX-YYY-ZZZ.ec2.internal
ALERT  :  [2011-08-07T01:37:29+0000] msg=<LOGKEY>, program=<Samhain>, hash=<7101E0B654A1608844BA757E1CC17B11F96A283D3A4BEAEE>
-----BEGIN LOGKEY-----
7101E0B654A1608844BA757E1CC17B11F96A283D3A4BEAEE[2011-08-07T01:37:29+0000]
-----BEGIN SIGNATURE-----
5DC697C943CFB6F7C0FAE1E374DE21D8FE32FF341815A46B
000001 1312681049::ip-WWW-XXX-YYY-ZZZ.ec2.internal
-----END MESSAGE-----
And then there's the question of what the signature in the logkey email means.

I'm wondering a few things:
1) Should I use my own private key should be used to sign a samhainrc file? Shouldn't this key be kept OFF the server one is trying to secure?
2) I see that one can specify --with-keyid=0x<hex KeyID> when configuring samhain for compilation. Should this be the complimentary public key of the private key used in step 1? Does this key need to be imported into the gpg keyring of the samhain user? at the point i run ./configure? at the point I run make? at the point i run make install ?
3) Does this public key need to remain in the gpg ring as long as I'm running samhain or does samhain somehow retain a copy of it?
4) Does samhain need a private key to sign the database file? If so, doesn't this present a security risk because the private key lives on the server which may in fact be compromised?
5) What private key is used to sign the samhain email messages? Where does this private key live? What public key is used to check the signature on these messages?

As always, help is *much* appreciated.

Last edited by sneakyimp; 08-07-2011 at 08:52 PM.
 
Old 08-07-2011, 10:42 PM   #12
sneakyimp
Member
 
Registered: Dec 2004
Posts: 791

Original Poster
Rep: Reputation: 49
OK I'm leaning toward this configuration command:
Code:
./configure --enable-login-watch --enable-xml-log --with-database=mysql --with-recipient=dev@mydomain.com --with-sender=samhain@mydomain.com --with-config-file=/etc/samhain/samhainrc --with-log-file=/var/log/samhain/samhain.log --with-pid-file=/var/run/samhain/samhain.pid --with-state-dir=/var/state/samhain --with-data-file=/var/state/samhain/samhain_file
This skips the gpg step -- I don't want to exclude it but do not yet understand its functioning or what keys are required and where they are used -- and results in the following configuration:
Code:
 samhain has been configured as follows:
     System binaries: /usr/local/sbin
  Configuration file: /etc/samhain/samhainrc
        Manual pages: /usr/local/man
                Data: /var/state/samhain
            PID file: /var/run/samhain/samhain.pid
            Log file: /var/log/samhain/samhain.log
            Base key: 1881432157,1883906182

    Selected rc file: samhainrc.linux
These appear to mimic those specified by the current debian samhain package but I'll be compiling it myself.

Last edited by sneakyimp; 08-08-2011 at 11:21 AM.
 
Old 08-07-2011, 10:47 PM   #13
sneakyimp
Member
 
Registered: Dec 2004
Posts: 791

Original Poster
Rep: Reputation: 49
I just did the configure, the make, the make install on my desktop machine. Some interesting results:
Code:
mkdir /etc/samhain
mkdir /var/run/samhain
mkdir /var/log/samhain
mkdir /var/state/samhain
./samhain-install.sh --destdir= --express --verbose install-data
  cp samhainrc.linux samhainrc
  cp samhainrc samhainrc.pre
  mv -f samhainrc.pre samhainrc.install
  ./samhain-install.sh --install-sh  -m 600 samhainrc.install /etc/samhain/samhainrc
  checking whether paths are trustworthy
  configuration file /etc/samhain/samhainrc ... OK
  state directory /var/run/samhain ... OK
  state directory /var/log/samhain ... OK
  data directory /var/state/samhain ... OK

  You can use 'samhain-install.sh uninstall' for uninstalling
  i.e. you might consider saving that script for future use

  Use 'make install-boot' if you want samhain to start on system boot
 
Old 08-08-2011, 11:12 AM   #14
sneakyimp
Member
 
Registered: Dec 2004
Posts: 791

Original Poster
Rep: Reputation: 49
Ok this morning is looking very promising but with a couple of difficulties. I compiled samhain from source as described above and replaced the default samhainrc with mine from before. I tried to start it with sudo /etc/init.d/samhain start and it complained about self resolving, about not having a database, and about the samhain.pid file permissions being world-writable:

So I changed the /var/run/samhain folder to be root:root, 770. This solved the last problem.

To solve the 2nd problem I initialized the samhain db as described in the docs:
Code:
sudo samhain -t init -p info > samhain_start.txt
This resulted in the creation of a database and an enormous amount of output text.
Difficulty 1: For some reason the text did not go into the file samhain_start.txt but instead was echoed to the terminal window.

Difficulty 2: Running a samhain check results in a persistent complaint about self-resolving:
Code:
$ sudo samhain -t check

---------   sh_unix.c  ---   1683 ---------

According to uname, your nodename is foo-bar-64, but your resolver
library cannot resolve this nodename to a FQDN.
Rather, it resolves this to foo-bar-64.
For more information, see the entry about self-resolving under
'Most frequently' in the FAQ that you will find in the docs/ subdirectory.

----------------------------------------------
I checked the docs it mentioned and the referenced section mentions the /etc/hosts file but it would seem that this matters mostly in a client/server arrangement which I don't have. Do I really need to fix this?

Difficulty: 3Log rotation
The docs have a page that says log rotation may be necessary -- this makes me think there is no built-in log rotation. Being new to logrotate, I'm a little nervous about breaking something. The Debian samhain package includes this config in /etc/logrotate.d/samhain:
Code:
/var/log/samhain/*.log {
	weekly
	missingok
	rotate 52
	compress
	delaycompress
	notifempty
	create 640 root adm
	sharedscripts
	postrotate
	   if [ -f /var/run/samhain/samhain.pid ]; then \
		/etc/init.d/samhain reload > /dev/null; fi
	endscript
}
This script doesn't look like it's very careful about stopping/starting samhain or acquiring locks. I also don't understand what the reload is for. Unfortunately, the recommended samhain logrotate script is not tested:
Code:
/usr/local/var/log/samhain_log {
    size 100k
    nocreate
    compress
    mail root@localhost
    maillast

prerotate
        if test -f /usr/local/var/run/samhain.pid; then \
          PIN=`cat /usr/local/var/run/samhain.pid`; \
          /bin/kill -TTI $PIN; \
          sleep 1; \
          AA=0; \
          while test "x$AA" != "x120"; do \
           let "AA = $AA + 1"; \
           if test -f /usr/local/var/log/samhain_log.lock; then \
             sleep 1; \
           else \
             break; \
           fi \
          done; \
        fi
    endscript
}
Any advice you have about constructing a good logrotate config would be much appreciated.

Good news:
* I have records in my database now, which is tremendous and makes me exceedingly happy. Append-only log! W00T.
* I think I've almost got all the big questions answered.

Last edited by sneakyimp; 08-08-2011 at 11:23 AM.
 
Old 08-08-2011, 05:01 PM   #15
sneakyimp
Member
 
Registered: Dec 2004
Posts: 791

Original Poster
Rep: Reputation: 49
Alrighty I believe I have managed to get samhain compiled from source and configured for log rotation et. al. I'm now turning to the still-unanswered questions in my original post.

I've resolved Question 1: I have samhain logging with Select and insert permissions to a remote MySQL instance.

Question 2:
These files do not exist on my system, so I am disabling checks for them in /etc/samhain/samhainrc:
Code:
stat: cannot stat `/etc/ssh_random_seed': No such file or directory
stat: cannot stat `/etc/asound.conf': No such file or directory
stat: cannot stat `/etc/ioctl.save': No such file or directory
stat: cannot stat `/etc/passwd.backup': No such file or directory
stat: cannot stat `/etc/shadow.backup': No such file or directory
stat: cannot stat `/etc/postfix/prng_exch': No such file or directory
stat: cannot stat `/etc/adjtime': No such file or directory
stat: cannot stat `/etc/network/run/ifstate': No such file or directory
stat: cannot stat `/etc/lvm/.cache': No such file or directory
stat: cannot stat `/var/log/warn': No such file or directory
stat: cannot stat `/etc/resolv.conf.pcmcia.save': No such file or directory
stat: cannot stat `/etc/nologin': No such file or directory
stat: cannot stat `/etc/network/run': No such file or directory
stat: cannot stat `/usr/lib/apache/suexec': No such file or directory
stat: cannot stat `/usr/lib/apache/suexec.disabled': No such file or directory
I have added these to the GrowingLogFiles section:
Code:
/var/log/ConsoleKit/history
/var/log/debug
/var/log/fail2ban.log
/var/log/lpr.log
/var/log/mail.err
/var/log/mail.info
/var/log/mail.info
/var/log/mail.warn
/var/log/news/news.crit
/var/log/news/news.err
/var/log/news/news.notice
/var/log/ufw.log

Still wondering
* How to use signatures in email notifications to validate the messages therein?
* What is the significance of the logkey?
* Which keys are needed and how are they used in order to run samhain using gpg?
* Is there some way to check up on samhain to make sure it's running? I'm worried a kill -9 might take it down without any sort of notification.
 
  


Reply

Tags
samhain, ubuntu


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
SSH to Amazon EC2 Arlc Linux - Security 3 07-22-2011 07:41 PM
LXer: Host Based Intrusion Detection - Samhain LXer Syndicated Linux News 0 01-19-2011 03:20 PM
File Integrity Through Samhain (windows) s3cur3ity General 1 10-04-2010 10:47 AM
What are the Pros versus Cons with File Integrity Detection Systems 2backitup Linux - Security 7 03-02-2006 06:47 PM
Suggestions for file integrity monitoring? Phaethar Linux - Software 1 06-11-2005 02:07 AM


All times are GMT -5. The time now is 10:22 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration