Hi,

Has anyone ever heard of the audit daemon was suspended due to space threshold being reached and possibly hang a server? How do you go about increasing the threshold or delete old files when the treshold is reached?

I am running RHAS 3.0 on HP Itanium servers.

Thank you in advance and have a great day.

When the filesystem containing the audit logs exceeds 80% utilization, things start randomly hanging on the system, auditd service will definitely get hanged up if the /var system reaches 100% on disk space. Normally audit services write save.* and bin.* files in /var/log/audit.d/ directory. If the /var file system is 100 % full then the audit services may hang up.

you can make changes to /etc/audit/audit.conf for the configuration and filesets.conf to specify the rulesets for the same.
Its better to turn off the notify logging as it puts unnecessary amount of logs in log files.

check out this statement in audit.conf

notify = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20%";

This will make audit services go into "suspend mode" or hanged up whenever the /var filesystem reaches 80% utilization. Change the above line to

notify = "/bin/true";
this will make sure that audbin command is not able to suspend audit services.

Anyways you can always stop the audit services if you are running a server. Make use of chkconfig command to turn audit service off.

Rgds,
Rahul.

Hi Rahul,

Thanks for the education on auditd. I am not sure if that was teh problem, because we had plenty of disk space in /var directory. I suspect something like raw devices and ocfs had a hicup or some sort, or even the SANs had some issues. At the moment, I am not able to pin-point exactly what was the caused of the issue we had that one particular day.

Please feel free to shout at me with more ideas.

Thanks,
J

firstly, m not shouting or educating anyone out here!! m just trying to provide you with the ideas or info. which i feel could have caused the problem, coz i m not the admin for your machine.

anyways, you mentioned that SAN is being used in your system along with raw devices which looks like a Database server. are your etc, bin and sbin are on SAN storage ? SAN issues are generally due to HBA card failure. but for audit services to hangup the most prominent reason should be some device failure. rawdevice failure ca atmost cause some of the processes to go into Disk sleep which might have caused cpu high. SAN failure is more of a permanent thing and not temporary one. Have a look at your audit.conf. were only audit services suspended or others too ??

I didn't mean you were shouting...And it's good to educate people like my self newbie into the Linux world.

No, our /etc, /bin, and /sbin is not on the SAN. But it's locally attched. I just have this funny feeling about the raw devices & ocfs, and Linux FS running on the same box and have Oracle RAC for clustering.

I don't know a whole lot about Linux at this time to even know where to look for problems and I was hoping to find some clues on where to look for. This symtom doesn't happen all the time, so make it very difficult to trace.

Any ideas, please don't hesitate to let me know.

Thanks,
J