LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 07-29-2005, 12:50 PM   #1
darrenprog
LQ Newbie
 
Registered: Jul 2005
Posts: 4

Rep: Reputation: 0
console login hangs RHEL 3


Been using linux for years and came up on a new problem I have never encountered.

Please read carefully before jumping to conclusions.

OS: RHEL WS 3
System: HP Server Hyperthreading

Have had a system in place for about 6 months continuous uptime and no issues.

Issue: Can no longer use console to login nor use ssh into system (telnet disabled).

Symptoms:

ssh: ssh_exchange_identification: Connection closed by remote host

console: Type in any user name and password. Press enter, tty hangs forever.

Have not rebooted system yet, due to production system. Looking for advice of what to look for, here is what I was going to try tonight after reboot and hopefully login.

1. check for /etc/nologin.
2. check /etc/securetty (see if corrupt file)
3. rpm -Vf /etc/pam.d (don't feel this is going to help much)
4. rpm -Vf /etc/security (don't feel this is going to help much either)
5. rpm -V initscripts
6. check /etc/shadow and /etc/passwd for corrupt entries
7. rpm -Vf /bin/login
8. rpm -Vf /sbin/mingetty

If I can not login, will boot to cdrom with the install disk and mount the filesystems and perform the checks.

Any other ideas anyone may have???? Ideas or instances where this has happened to someone else?

Thanks,

Darren
 
Old 07-30-2005, 05:11 AM   #2
darrenprog
LQ Newbie
 
Registered: Jul 2005
Posts: 4

Original Poster
Rep: Reputation: 0
Smile

All,

Found the problem after reboot.

Issue: /var had filled up. Therefore, neither lastlog nor wtmp could be written to.

Affecting program: auditd

Description: /var/log/audit.d directory was housing about (40) 20 megabyte files, filling up the /var filesystem preventing logging in. Apparently, the specific version with RHEL 3 update 4 does not clean up the files, but leaves them there. This is either a bug/configuration issue with auditd. Currently, just turned it off.

Hopes this helps others out there that may run across this same problem.

Darren
 
Old 08-16-2005, 03:59 PM   #3
colt151
LQ Newbie
 
Registered: Aug 2005
Posts: 2

Rep: Reputation: 0
Thumbs up

Thanks for your post; it helped a lot.

This was driving me crazy...I had noticed that /var seemed a little large but didn't think the problem could be there, but your post makes complete sense.

So besides turning auditd off what other solutions are you thinking of?

Cole
 
Old 08-24-2005, 01:25 AM   #4
darrenprog
LQ Newbie
 
Registered: Jul 2005
Posts: 4

Original Poster
Rep: Reputation: 0
Glad this helped someone else. The fix for this is to have rotating logs.
I believe I saw a post that is is marked as a bug by Red Hat.

The /etc/audit/auditd.conf

# Standard output method is bin mode.
#
output {
mode = bin;
num-files = 4;
file-size = 20M;
file-name = "/var/log/audit.d/bin";
notify = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C";

# The following symlink is created whenever we switch to
# a new bin.
current = "/var/log/audit";

sync = no;

# uncomment these to cause audit records to be
# flushed to the disk after sync-after records
# are written to the log

# sync = yes;
# sync-after = 16;
error {
action {
type = suspend;
};
};
};

Notice, that the num-files is supposed to be only 4. This says to me it is a
bug. Hopefully, they have fixed this in an update from Red Hat.

Due to that we don't need these logs, since it is an internal computer, it was
not needed.


To fix the script, something like the following could be implemented in cron:

Put in a shell script to run from cron something similar.
I don't remember the exact layout of the filename. I remember
seeing one that was bin.1, so assuming the .1 is the day.


SEVEN_DAYS_OLD=`/usr/locale/ebin/date '+%d' --date '3 days ago'`

rm /var/log/audit/bin.${SEVEN_DAYS_OLD}
rm /var/log/audit/bin/<other_files>.${SEVEN_DAYS_OLD}


Of course, this does no checking, should have something similar to:


if [ -f /var/log/audit/bin/bin.${SEVEN_DAYS_OLD} ]
then
rm /var/log/audit/bin.${SEVEN_DAYS_OLD}
fi


So, if you need the audit logs, hopefully, the above helps if Red Hat and others
have not fixed the problem yet.


Darren
 
Old 08-24-2005, 01:27 AM   #5
darrenprog
LQ Newbie
 
Registered: Jul 2005
Posts: 4

Original Poster
Rep: Reputation: 0
How about a few corrections to that post.

The shell script should read:



SEVEN_DAYS_OLD=`/usr/locale/ebin/date '+%d' --date '7 days ago'`

if [ -f /var/log/audit/bin/bin.${SEVEN_DAYS_OLD} ]
then
rm /var/log/audit/bin.${SEVEN_DAYS_OLD}
fi
 
Old 08-24-2005, 07:52 PM   #6
colt151
LQ Newbie
 
Registered: Aug 2005
Posts: 2

Rep: Reputation: 0
Thanks again, Darren
 
Old 08-31-2005, 07:41 AM   #7
julietkemp
LQ Newbie
 
Registered: Aug 2005
Distribution: RHEL/Debian
Posts: 4

Rep: Reputation: 0
I've just encountered this problem as well (on RHEL3), & there's another possible solution:

Edit audit.conf as follows:

output {
mode = bin;
num-files = 4;
file-size = 20M;
file-name = "/var/log/audit.d/bin";
notify = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20% -N 'rm -f %f'";

[etc...]

}

The notify line as above should (according to the docs) remove old 'save' files when the filesystem comes within 20% of full (change the figure after -T to specify how close to full you want to start deleting old files).

This should be of help if audit is filling up the log directory with 'save' files rather than 'bin' files - which is what was happening with me. As I understand it, the num-files option here refers to the number of 'bin' files that are in use. The 'notify' option states what should be done when a new 'bin' file is started (in the above config, when the previous one gets to 20MB). The config above will copy the old 'bin' file to a new file 'save.%u' (audbin uses %u to generate a number to make the filename unique), clear the file (-C flag), & then look to see if the filesystem threshold is reached (-T flag, within 20% of full here); if it is, then it runs the -N command (%f refers here to oldest file). The audbin man page is helpful.

Before making this change, the default was for the notify command simply to suspend the audit daemon, which caused the machine to hang as described at the top of this thread.

Hope that's helpful for anyone else encountering this problem! And thanks to Darren for the initial pointer.

Juliet
 
Old 06-11-2007, 05:43 PM   #8
ukulele
LQ Newbie
 
Registered: Jun 2007
Posts: 1

Rep: Reputation: 0
this worked for me

Thanks for posting this. I always forget about that sad little /var directory silently doing its job, and silently puking when full.. Booted with knoppix, deleted some files and I'm back in business.
 
  


Reply

Tags
login, nagios


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
lock root account after 3 login attempts - RHEL AS 3 jrparker2005 Red Hat 1 05-17-2005 01:43 PM
New RHEL WS 3 (RHPW) hangs when installing updates. halo14 Red Hat 2 05-03-2005 11:06 AM
logout from console hangs slice Linux - Newbie 8 02-22-2005 02:07 PM
Console repeat-hangs during Woody base-install alke Linux - Laptop and Netbook 0 02-15-2004 11:51 PM
Console mouse services just hangs up or dies!!! FuGeE Linux - Hardware 0 12-08-2002 04:43 AM


All times are GMT -5. The time now is 06:31 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration