file integrity monitoring: grokking samhain

sneakyimp · 10-26-2016, 05:11 PM

I first heard of samhain here and have installed it to monitor the integrity of some of my more important servers. This has resulted in a flood of email lately, most of which looks fairly benign, but I need to start cleaning things up. I'm hoping to get some advice about how to go about this. My understanding of how the samhain configuration works is reasonably good, but I would like to hone it -- and some of the things it reports do not look familiar to me, so I could use some help.

The system is Ubuntu 14.x (patched today), running as an Amazon EC2 instance.

The latest notification (with some details changed to protect my server's identity:

Code:

-----BEGIN MESSAGE-----
2016-10-26T21:09:59+0000 my.host.name
<log sev="CRIT" tstamp="2016-10-26T21:09:46+0000" msg="POLICY ADDED" path="/var/log/upstart/systemd-logind.log" mode_new="-rw-r-----" attr_new="------------" imode_new="33184" iattr_new="0" hardlinks_new="1" idevice_new="0" inode_new="56" owner_new="root" iowner_new="0" group_new="root" igroup_new="0" size_old="0" size_new="108" ctime_new="2016-10-26T21:01:47" atime_new="2016-10-26T20:55:39" mtime_new="2016-10-26T21:01:47" chksum_new="C06C34687BB0893357A27BD6BECB3BC091C5B59A8210E9ED"  />
<log sev="CRIT" tstamp="2016-10-26T21:09:45+0000" msg="POLICY [ReadOnly] --------T-" path="/var/lib/update-notifier/fsck-at-reboot" ctime_old="2016-10-25T00:49:27" ctime_new="2016-10-26T20:55:45" mtime_old="2016-10-25T00:49:27" mtime_new="2016-10-26T20:55:45"  />
<log sev="CRIT" tstamp="2016-10-26T21:09:45+0000" msg="POLICY [ReadOnly] ---I----T-" path="/var/lib/update-notifier/hwe-eol" inode_old="6228" inode_new="4831" ctime_old="2016-10-25T00:49:27" ctime_new="2016-10-26T20:55:44" mtime_old="2016-10-25T00:49:26" mtime_new="2016-10-26T20:55:44"  />
<log sev="CRIT" tstamp="2016-10-26T21:09:45+0000" msg="POLICY [ReadOnly] C-------T-" path="/var/lib/update-notifier/updates-available" ctime_old="2016-10-25T00:49:26" ctime_new="2016-10-26T20:55:44" mtime_old="2016-10-25T00:49:26" mtime_new="2016-10-26T20:55:44" chksum_old="F3957B51C066E96725902F13293816955C26EFEED37F232A" chksum_new="AFDCAE6165007B240E2140532E3F58D4B810B6806EE8D589"  />
<log sev="CRIT" tstamp="2016-10-26T21:09:45+0000" msg="POLICY [ReadOnly] --------T-" path="/var/lib/update-notifier" ctime_old="2016-10-25T01:46:13" ctime_new="2016-10-26T21:01:48" mtime_old="2016-10-25T01:46:13" mtime_new="2016-10-26T21:01:48"  />
<log sev="CRIT" tstamp="2016-10-26T21:09:24+0000" msg="POLICY [ReadOnly] --------T-" path="/var/lib/ubuntu-release-upgrader/release-upgrade-available" ctime_old="2016-10-25T00:49:28" ctime_new="2016-10-26T20:55:45" mtime_old="2016-10-25T00:49:28" mtime_new="2016-10-26T20:55:45"  />
<log sev="CRIT" tstamp="2016-10-26T21:09:21+0000" msg="POLICY [ReadOnly] C--I----T-" path="/var/git/myproject-v3/refs/heads/master" inode_old="5022" inode_new="15805" ctime_old="2016-10-10T19:47:20" ctime_new="2016-10-26T20:55:45" mtime_old="2016-10-10T19:47:20" mtime_new="2016-10-26T20:55:45" chksum_old="3062131FCA23F52D2D305A96A5D8015F131D90B53043114B" chksum_new="D3967385F7E6F90A691D5230B97C20CE2C2F2656EEBE9F97"  />
-----BEGIN SIGNATURE-----
2589396F5B0C7641D2027F59117EF2DA101382DF0C38FE26
000082 1476128504::my.host.name
-----END MESSAGE-----

Most of these notifications appear to be related to Ubuntu's default behavior that checks for available updates/upgrades. Does this automatic check present any security risk? Should I disable the security check or should I ignore changes to the files listed here?

The last notification is due to some code being checked into a repository. I'm not sure what to do about this. If I make the changes myself, I don't want samhain to tell me about it. If someone else makes them, I'd want to know if it was some stranger, but not if it was a trusted companion. Anyone have suggestions?

BlackRider · 10-27-2016, 12:36 PM

I don't use shamain, but I use aide and my OpenBSD boxes have security scripts running limited integrity checks.

Any tool that gives you reports with massive amounts of false positives in them is not helping. You end up looking them over quickly and ignoring the lines out of pure boredoom. My advice is to monitor only things that are not supposed to change. Some tools are more flexible than others and allow you to decide what to look for in order to make a report.

An option for things like repositories is to have an integrity database just for them, make checks just before you start working with them, and update the database once you are over. Probably a lot of hassle anyway.

sneakyimp · 10-27-2016, 12:48 PM

EDIT: thanks for your response!

Quote:

Originally Posted by BlackRider

Any tool that gives you reports with massive amounts of false positives in them is not helping. You end up looking them over quickly and ignoring the lines out of pure boredoom. My advice is to monitor only things that are not supposed to change. Some tools are more flexivle than others and allow you to decide what to look for in order to make a report.

Samhain is highly configurable, so I'm not really asking about samhain. I'm asking about the security implications really. I could easily suppress this output in samhain. I'm actually interested in knowing that it happens. I don't consider it especially risky for these files, but was looking for other opinions.

Quote:

Originally Posted by BlackRider

An option for things like repositories is to have an integrity database just for them, make checks just before you start working with them, and update the database once you are over. Probably a lot of hassle anyway.

This does sound like a hassle. My thinking (which may be flawed from a security perspective) is that a source code repo is going to change a lot and should therefore be excluded from integrity checking altogether. On the other hand, perhaps there are certain types of file integrity that I should be checking to make sure only authorized people change these files?

BlackRider · 10-27-2016, 01:35 PM

I would exclude the repository out.

I have heard some people out there is using git with signed commits, so you can audit who commited what change. Maybe you can investigate such an option.

unSpawn · 10-31-2016, 05:22 PM

Quote:

Originally Posted by sneakyimp

Most of these notifications appear to be related to Ubuntu's default behavior that checks for available updates/upgrades. Does this automatic check present any security risk? Should I disable the security check or should I ignore changes to the files listed here?

The last notification is due to some code being checked into a repository. I'm not sure what to do about this. If I make the changes myself, I don't want samhain to tell me about it. If someone else makes them, I'd want to know if it was some stranger, but not if it was a trusted companion. Anyone have suggestions?

First of all these /var/lib/update-notifier/ and /var/lib/ubuntu-release-upgrader/ files are clearly traceable to the application that uses them, meaning you can gauge from the application in which way local or networked users (ab)use them (or not). So if for example this application would have a port open, like say MySQL listening on TCP/3306, you can mitigate that by reconfiguring the service. In your case I'd guess both applications are short-lived, command line or cron driven and only provide information. Next most changes are done on purpose (whatever that purpose is): so unwanted changes in /sbin/init, /etc/ssh/sshd_config, /usr/bin/traceroute or "/usr/local/sbin/httpd -DSSL" and such, sure, but ask yourself what Indicators Of Compromise a change in these specific files would actually yield? They're most likely log files and not binaries, libraries, configuration files or such the system relies on for any reason. Security and auditing remains a trade-off and while you would like full coverage that's not always practical or necessary. IIRC Samhain has a "growing only" section for log files. Maybe these files fit into that category. And I agree you should find a GIT-centric way to validate its integrity.