Unix administrator auditing - suggestions?
 

Hey guys,

I work with application management for an enterprise class application, and I am having a really hard time dealing with the people working on our server infrastructure (Application server admins who have privileged access to production servers). The exact problem is that, the crontab schedules of our application's unix scripts keeps gets messed up lately:
1. One was removed from crontab
2. Two were disabled from crontab
3. One script was scheduled to run on another server (backup live server), resulting to simultaneous execution of the script causing chaos.

The thing is, they don't have any mechanism to determine who were making such changes in the crontab schedule of the servers. Although they have their process which regularly creates a backup of crontab, they could no longer determine who were making any changes to the crontab of the servers.

Additonal details:
1. Around 15 administrators could access the servers (4 Unix boxes)
2. Challenge: each of them connect to the Unix servers using ssh, under a single username@server

Do you guys have any suggestions on how we could possible perform audit on the commands that each admin executes on the server? I was thinking of regularly checking the history file against the ssh signon times, but I might get more productive/feasible from you guys.

Cheers!

Situation clear, constraints not. Basic requisites should be remote syslog and NTP enabled everywhere, and what you add then from workstation logging, Inotify watches, FUSE LoggedFS, audit service, rootsh to obviously each user logging in from their own unprivileged user account and enough rights only using /etc/sudoers depends on what you are allowed / willing to modify / install.

Wow, that's so much to take in, thanks I'll look into those.

Sorry, I was kind of strapped for time and I needed to wedge in a reply pretty quick. Just ask me to expand on any aspects if needed though it would help to know what you can and can not do or suggest.

Attempting to expand upon unSpawn's suggestions, requiring sudo coupled with time stamping on the crontabs might be a an easy way to obtain the information you're after. By requiring sudo, you should get a log entry for the action performed as well as a time stamp for the action.

For example, using sudo on the simple ls command gives the following log entry in secure:
Provided that your not dealing with deliberately malicious intentions, this should give you a better audit log of who is performing what. A potential danger is to use sudo to simply gain a root shell, via sudo -i. This will show up as executing COMMAD=/bin/bash (or whatever shell your using) and the subsequent commands will not be seen directly. However, it may still be enough to narrow the list or correlate against the action without requiring dedicated audit tools.