auditd disappeared

slimm609 · 11-29-2010, 10:43 PM

"Kernel panic - not syncing: audit: auditd disappeared"

got this on multiple systems over the past few months randomly. I have never seen this besides red hat 5. Google turns up NOTHING! There is no stack trace during the panic and there is nothing useful in the logs. Just auditd disappearing.

Not sure where to go from here but was asking to see if anyone had any ideas or suggestions

unSpawn · 11-30-2010, 03:20 PM

Quote:

Originally Posted by slimm609

got this on multiple systems over the past few months randomly.

What systems exactly (audit+kernel version)?
What does syslog and audit/audit.log say?
When did this start to happen?
Any events leading up to this when comparing systems?
Anything else these systems have in common?

slimm609 · 12-01-2010, 08:08 PM

we figured it out. Thanks Unspawn.

We have to audit almost everything so it produces a ton of logs. Logrotate was rotating the log while it was auditing a ton of stuff and ended up panicing because it was losing a message or 2. a daily audit log on the system is around 400mb with little to no users and so we just set auditd to rotate the logs rather then have logrotate do it.

we have to have the config set to panic if a single audit message is lost and that was what was happening. It was when cron.daily was running.

they are all red hat 5.5 with latest updates. (not around the machines for exact versions atm)

unSpawn · 12-02-2010, 01:06 AM

Good you found it but how did you remedy loosing messages during logrotation by auditd then if I may ask?

slimm609 · 12-02-2010, 07:12 AM

having logrotate not do the rotation will hopefully calm down the logs when auditd needs to rotate. With auditd rotating the logs rather then logrotate it fills the audit_backlog long enough to rotate the log. There is a chance that it could fill the log when the system is going crazy and we could lose some but not doing it when cron.daily runs hopefully we wont have as many logs going on. cron.daily generates a lot of audit logs since it is touching so much.

Also with auditd rotating the logs itself hopefully it will be able to handle it a little better then some outside program moving the log then sending a hup to it.

unSpawn · 12-02-2010, 12:00 PM

Quote:

Originally Posted by slimm609

With auditd rotating the logs rather then logrotate it fills the audit_backlog long enough to rotate the log. There is a chance that it could fill the log when the system is going crazy and we could lose some but not doing it when cron.daily runs hopefully we wont have as many logs going on. (..) Also with auditd rotating the logs itself hopefully it will be able to handle it a little better then some outside program moving the log then sending a hup to it.

Indeed the reason why I chose to let audit rotate its own logs. But then again I never generate the amount of logging you probably do, cron accounting for only 12K lines in 5 days...

Quote:

Originally Posted by slimm609

cron.daily generates a lot of audit logs since it is touching so much.

I can imagine that. For some tasks you could avoid logging if you could determine the job just isn't required to run. For instance if nothing gets installed without package manager prelink doesn't need to run. And that only requires checking if the hashes of the RPMDB files match.

slimm609 · 12-02-2010, 09:44 PM

we have removed most from cron so it only runs what is required.

Here is the auditd rules

Code:

# Remove any existing rules
-D

# Enable auditing
-e 1

# Increase buffer size to handle the increased number of messages.
-b 16384

# Failure of auditd causes a kernel panic
-f 2

-w /bin/login -p x
-w /bin/logout -p x

# DAC permission changes
-a exit,always -F arch=b32 -S chmod -S chown -S fchmod -S fchown -S lchown
-a exit,always -F arch=b64 -S chmod -S chown -S fchmod -S fchown -S lchown
# unauthorized file access attempts
-a exit,always -F arch=b32 -S open -F success=0
-a exit,always -F arch=b32 -F success=0 -S mknod -S pipe -S mkdir -S creat -S truncate -S ftruncate
-a exit,always -F arch=b64 -S open -F success=0
-a exit,always -F arch=b64 -F success=0 -S mknod -S pipe -S mkdir -S creat -S truncate -S ftruncate
# privileged commands
-w /usr/sbin/pwck
-w /bin/chgrp
-w /usr/bin/newgrp
-w /usr/sbin/groupadd
-w /usr/sbin/groupmod
-w /usr/sbin/groupdel
-w /usr/sbin/useradd
-w /usr/sbin/userdel
-w /usr/sbin/usermod
-w /usr/bin/chage
-w /usr/bin/setfacl
-w /usr/bin/chacl
-a exit,always -F arch=b32 -S chroot -S mount -S umount2 -S adjtimex -S kill
-a exit,always -F arch=b64 -S chroot -S mount -S umount2 -S adjtimex -S kill
# deleting files
-a exit,always -F arch=b32 -S unlink -S rmdir -S rename -S link -S symlink
-a exit,always -F arch=b64 -S unlink -S rmdir -S rename -S link -S symlink
# system administration actions
-w /var/log/audit/audit.log
-w /var/log/audit/audit[1-4].log
-w /var/log/messages
-w /var/log/lastlog
-w /var/log/faillog
-w /etc/audit/auditd.conf -p wa
-w /etc/audit/audit.rules -p wa
-w /etc/selinux/config -p wa
-w /etc/passwd -p wa
-w /etc/shadow -p wa -k shadow
-w /etc/group  -p wa
-w /etc/ld.so.conf -p wa
-w /etc/ld.so.conf.d -p wa
-w /etc/ssh/sshd_config
-w /etc/pam.d
-w /etc/login.defs
-w /etc/rc.d/init.d
-w /etc/inittab -p wa
-w /var/run/utmp
-w /var/run/wtmp
-a exit,always -F arch=b32 -S acct -S reboot -S sched_setparam -S sched_setscheduler -S setdomainname -S setrlimit -S settimeofday -S swapon
-a exit,always -F arch=b64 -S acct -S reboot -S sched_setparam -S sched_setscheduler -S setdomainname -S setrlimit -S settimeofday -S swapon
# security personnel actions
-a exit,always -F arch=b32 -S init_module -S delete_module -S setxattr -S lsetxattr -S fsetxattr -S removexattr -S lremovexattr -S fremovexattr
-a exit,always -F arch=b64 -S init_module -S delete_module -S setxattr -S lsetxattr -S fsetxattr -S removexattr -S lremovexattr -S fremovexattr
-w /bin/su

unSpawn · 12-04-2010, 07:56 PM

I modeled most of my defaults after the CAPP rules too but with a key for each group of rules for easier ausearch / aureport, without directory and inode creation rules and without root rules ("-F auid!=0 -F auid!=4294967295"). The latter two may clash with your "have to audit almost everything":

Code:

-D
-b 5120
-r 21000 
# System calls, exclusions first:
# Don't log UID 0 socket calls:
-a entry,never -S socketcall -F auid=0
-a exit,never -S socketcall -F auid=0
# Modifying the LDT and module syscalls:
-a entry,always -S modify_ldt -S create_module -S init_module -S delete_module -S query_module -k SYS_mod
# Privileges commands:
-a exit,always -S setxattr -S lsetxattr -S fsetxattr -S removexattr -S lremovexattr -S fremovexattr -k SYS_priv
-a entry,always -S mount -S umount -S umount2 -S acct -S reboot -k SYS_priv
-a entry,always -S adjtimex -S settimeofday -S clock_settime -k SYS_time
# Record failures for non-root users:
-a exit,always -S chmod -S fchmod -S fchmodat -S chown -S chown32 -S fchown -S fchown32 -S fchownat -S lchown -S umask -S creat -S open -S openat -S truncate -S ftruncate -F success=0 -F exit=-EACCES -F auid!=0 -F auid!=4294967295 -k FAIL
-a exit,always -S chmod -S fchmod -S fchmodat -S chown -S chown32 -S fchown -S fchown32 -S fchownat -S lchown -S umask -S creat -S open -S openat -S truncate -S ftruncate -F success=0 -F exit=-EPERM -F auid!=0 -F auid!=4294967295 -k FAIL

# SELinux:
-w /etc/selinux -p xwa -k CFG_selinux
-w /etc/selinux/config -p wa -k CFG_selinux
-w /etc/selinux/restorecond.conf -p wa -k CFG_selinux
-w /etc/selinux/semanage.conf -p wa -k CFG_selinux
# SELinux targeted policy:
-w /etc/selinux/targeted/setrans.conf -p wa -k CFG_selinux
-w /etc/selinux/targeted/modules/active/ -p wa -k CFG_selinux
-w /etc/selinux/targeted/contexts/files/file_contexts.local -p wa -k CFG_selinux

# The Auditd system itself:
-w /etc/audit/auditd.conf -p wa -k CFG_audit
-w /etc/audit/audit.rules -p wa -k CFG_audit
-w /etc/sysconfig/auditd  -p wa -k CFG_audit
-w /etc/libaudit.conf -p wa -k CFG_audit
-w /var/log/audit/ -p a -k LOG_audit
-w /var/log/audit/audit.log -p a -k LOG_audit.log
-w /var/log/audit/audit.log.[1-9] -p a -k LOG_audit.log
-w /etc/rc.d/init.d/auditd -p wa -k CFG_audit

# User accounts, login records and sudo:
-w /etc/group -p wa -k CFG_logins
-w /etc/passwd -p wa -k CFG_logins
-w /etc/gshadow -p wa -k CFG_logins
-w /etc/shadow -p wa -k CFG_logins
-w /etc/security/opasswd -p wa -k CFG_logins
-w /etc/login.defs -p wa -k CFG_logins
-w /etc/securetty -p wa -k CFG_logins
-w /etc/sudoers -p wa -k CFG_logins
-w /var/log/faillog -p a -k LOG_logins
-w /var/log/lastlog -p a -k LOG_logins
-w /var/log/tallylog -p a -k LOG_logins
-w /var/run/utmp -p a -k LOG_logins
-w /var/log/wtmp -p a -k LOG_logins
-w /var/log/btmp -p a -k LOG_logins

# Other /etc directories and files:
-w /etc/inittab -p wa -k CFG_etc
-w /etc/rc.d/rc.sysinit -p wa -k CFG_etc
-w /etc/rc.d/rc.local -p wa -k CFG_etc
-w /etc/rc.d/init.d/ -p xwa -k CFG_etc
-w /etc/sysconfig/ -p xwa -k CFG_etc
-w /etc/hosts -p wa -k CFG_etc
-w /etc/hosts.deny -p wa -k CFG_etc
-w /etc/hosts.allow -p wa -k CFG_etc
-w /etc/resolv.conf -p wa -k CFG_etc
-w /etc/nsswitch.conf -p wa -k CFG_etc
-w /etc/host.conf -p wa -k CFG_etc
-w /etc/fstab -p wa -k CFG_etc
-w /etc/ld.so.conf -p wa -k CFG_etc
-w /etc/ld.so.conf.d/ -p wa -k CFG_etc
-w /etc/localtime -p wa -k SYS_time
-w /etc/ntp.conf -p wa -k CFG_etc
-w /etc/sysctl.conf -p wa -k CFG_etc
-w /etc/modprobe.conf -p wa -k MOD_cfg
-w /etc/issue -p wa -k CFG_etc
-w /etc/issue.net -p wa -k CFG_etc
-w /etc/shells -p wa -k CFG_etc
-w /etc/profile -p wa -k CFG_etc
-w /etc/bashrc -p wa -k CFG_etc
-w /etc/csh.cshrc -p wa -k CFG_etc
-w /etc/csh.login -p wa -k CFG_etc
-w /etc/default/ -p wa -k CFG_etc
-w /etc/xinetd.d/ -p wa -k CFG_etc

# Syslog:
-w /etc/rsyslog.conf -p wa -k CFG_syslog
-w /etc/syslog.conf -p wa -k CFG_syslog

# PAM:
-w /etc/pam.d/ -p xwa -k CFG_pam
-w /etc/security/access.conf -p wa -k CFG_pam
-w /etc/security/limits.conf -p wa -k CFG_pam
-w /etc/security/pam_env.conf -p wa -k CFG_pam
-w /etc/security/namespace.conf -p wa -k CFG_pam
-w /etc/security/namespace.init -p wa -k CFG_pam

# System binaries:
-a always,exit -F path=/bin/kill -F perm=x -F auid!=0 -F auid!=4294967295 -k BIN_priv
-a always,exit -F path=/usr/bin/killall -F perm=x -F auid!=0 -F auid!=4294967295 -k BIN_priv
-a always,exit -F path=/bin/su -F perm=x -F auid!=0 -F auid!=4294967295 -k BIN_priv
-a always,exit -F path=/usr/bin/crontab -F perm=x -F auid!=0 -F auid!=4294967295 -k BIN_priv

Per system 'rpm -g' may be added (development tools), temp watches or problem indicators (Apache user accessing downloaders) and such.

slimm609 · 12-05-2010, 02:19 PM

what is the auid!=4294967295. I may be able to not audit that depending on what it is but i know i have to audit root for sure.