Been using linux for years and came up on a new problem I have never encountered.

Please read carefully before jumping to conclusions.

OS: RHEL WS 3
System: HP Server Hyperthreading

Have had a system in place for about 6 months continuous uptime and no issues.

Issue: Can no longer use console to login nor use ssh into system (telnet disabled).

Symptoms:

ssh: ssh_exchange_identification: Connection closed by remote host

console: Type in any user name and password. Press enter, tty hangs forever.

Have not rebooted system yet, due to production system. Looking for advice of what to look for, here is what I was going to try tonight after reboot and hopefully login.

1. check for /etc/nologin.
2. check /etc/securetty (see if corrupt file)
3. rpm -Vf /etc/pam.d (don't feel this is going to help much)
4. rpm -Vf /etc/security (don't feel this is going to help much either)
5. rpm -V initscripts
6. check /etc/shadow and /etc/passwd for corrupt entries
7. rpm -Vf /bin/login
8. rpm -Vf /sbin/mingetty

If I can not login, will boot to cdrom with the install disk and mount the filesystems and perform the checks.

Any other ideas anyone may have???? Ideas or instances where this has happened to someone else?

Thanks,

Darren

All,

Found the problem after reboot.

Issue: /var had filled up. Therefore, neither lastlog nor wtmp could be written to.

Affecting program: auditd

Description: /var/log/audit.d directory was housing about (40) 20 megabyte files, filling up the /var filesystem preventing logging in. Apparently, the specific version with RHEL 3 update 4 does not clean up the files, but leaves them there. This is either a bug/configuration issue with auditd. Currently, just turned it off.

Hopes this helps others out there that may run across this same problem.

Darren

Thanks for your post; it helped a lot.

This was driving me crazy...I had noticed that /var seemed a little large but didn't think the problem could be there, but your post makes complete sense.

So besides turning auditd off what other solutions are you thinking of?

Cole

Glad this helped someone else. The fix for this is to have rotating logs.
I believe I saw a post that is is marked as a bug by Red Hat.

The /etc/audit/auditd.conf

# Standard output method is bin mode.
#
output {
mode = bin;
num-files = 4;
file-size = 20M;
file-name = "/var/log/audit.d/bin";
notify = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C";

# The following symlink is created whenever we switch to
# a new bin.
current = "/var/log/audit";

sync = no;

# uncomment these to cause audit records to be
# flushed to the disk after sync-after records
# are written to the log

# sync = yes;
# sync-after = 16;
error {
action {
type = suspend;
};
};
};

Notice, that the num-files is supposed to be only 4. This says to me it is a
bug. Hopefully, they have fixed this in an update from Red Hat.

Due to that we don't need these logs, since it is an internal computer, it was
not needed.


To fix the script, something like the following could be implemented in cron:

Put in a shell script to run from cron something similar.
I don't remember the exact layout of the filename. I remember
seeing one that was bin.1, so assuming the .1 is the day.


SEVEN_DAYS_OLD=`/usr/locale/ebin/date '+%d' --date '3 days ago'`

rm /var/log/audit/bin.${SEVEN_DAYS_OLD}
rm /var/log/audit/bin/<other_files>.${SEVEN_DAYS_OLD}


Of course, this does no checking, should have something similar to:


if [ -f /var/log/audit/bin/bin.${SEVEN_DAYS_OLD} ]
then
rm /var/log/audit/bin.${SEVEN_DAYS_OLD}
fi


So, if you need the audit logs, hopefully, the above helps if Red Hat and others
have not fixed the problem yet.


Darren

How about a few corrections to that post.

The shell script should read:



SEVEN_DAYS_OLD=`/usr/locale/ebin/date '+%d' --date '7 days ago'`

if [ -f /var/log/audit/bin/bin.${SEVEN_DAYS_OLD} ]
then
rm /var/log/audit/bin.${SEVEN_DAYS_OLD}
fi

Thanks again, Darren

I've just encountered this problem as well (on RHEL3), & there's another possible solution:

Edit audit.conf as follows:

output {
mode = bin;
num-files = 4;
file-size = 20M;
file-name = "/var/log/audit.d/bin";
notify = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20% -N 'rm -f %f'";

[etc...]

}

The notify line as above should (according to the docs) remove old 'save' files when the filesystem comes within 20% of full (change the figure after -T to specify how close to full you want to start deleting old files).

This should be of help if audit is filling up the log directory with 'save' files rather than 'bin' files - which is what was happening with me. As I understand it, the num-files option here refers to the number of 'bin' files that are in use. The 'notify' option states what should be done when a new 'bin' file is started (in the above config, when the previous one gets to 20MB). The config above will copy the old 'bin' file to a new file 'save.%u' (audbin uses %u to generate a number to make the filename unique), clear the file (-C flag), & then look to see if the filesystem threshold is reached (-T flag, within 20% of full here); if it is, then it runs the -N command (%f refers here to oldest file). The audbin man page is helpful.

Before making this change, the default was for the notify command simply to suspend the audit daemon, which caused the machine to hang as described at the top of this thread.

Hope that's helpful for anyone else encountering this problem! And thanks to Darren for the initial pointer.

Juliet

Thanks for posting this. I always forget about that sad little /var directory silently doing its job, and silently puking when full.. Booted with knoppix, deleted some files and I'm back in business.