Lost console login prompt--neither GUI or command line prompts display

JMCraig · 11-10-2008, 03:41 PM

Hi Folks,

RHEL 5.2:
Some while back, I did some updates and suddenly the console had no login prompt. I didn't have time to fuss with it at the time, so I just let it go. Now it's becoming more of an issue and I need to fix it. And, of course, I can't remember what I did, or even if I noticed it at the time (as I don't work from the console much). Everything but the login prompt has been working fine (it's mostly a web server). I can log in via an SSH client no problem (and run the GUI desktop via an SSH tunneled X-Server, but it's very clunky).

The GUI login wasn't coming up (just getting a plain red background) so I thought I'd just skip the GUI and have it boot up to run-level 3 (as I only use the GUI occasionally anyway)--well, I've got the same problem: no command line login prompt either.

The console shows all the services starting up (lots of [ OK ] entries) and concludes with:

Code:

Notice: nsd.tcl: finished reading config file.

I couldn't find anything related to "nsd.ctl" (note misspelling--should be nsd.tcl) on linxuquestion.org or the web. NSD is the name of an open source DNS server, but it's not installed (as far as yum can tell).

I do have Xen installed on the box and the messages before the final one relate to Xen, but they have the [ OK ] out there on the right side so I don't know that it could have anything to do with this.

I've seen a number of options for what this might be and none of them seem to apply:

o Bogus entry in /etc/shells
line that says /SENDMAIL/ANY/SHELL is reported in some forums/solutions.
Not there. /etc/shells appears clean (just lists likely looking shells: /bin/sh, /bin/bash, & so on)

o /etc/nologin
Not there. Shouldn't prevent display of the prompt anyway (if I understand how this works correctly).

My situation seems similar to the one reported in this thread:

http://www.linuxquestions.org/questi...r-boot-325005/

Unfortunately, there's no resolution on that thread....

Any suggestions gratefully accepted.

John

unSpawn · 11-10-2008, 05:51 PM

Quote:

Originally Posted by JMCraig

I can't find anything related to "nsd.ctl" on linxuquestion.org or the web.

Maybe because you spelled it wrong: "nsd.tcl" (AOL/ACS)?

Quote:

Originally Posted by JMCraig

Some while back, I did some updates and suddenly the console had no login prompt.

Shame you decided to let it go back then. If you know the aproximate start of the situation you could easily have listed packages and sorted them by %{installtime} to help track things down.

Quote:

Originally Posted by JMCraig

The GUI login wasn't coming up (..) no command line login prompt either.

For X11/Xorg there's separate logs, for the rest there's dmesg and syslog. Maybe post some.

JMCraig · 11-10-2008, 06:46 PM

Doh! Yes, the box does have the AOL http server, and yes, there is info out there if you search for "nsd.tcl".
If I can't find anything with that, I'll get back w/ some log goo.

>:~\

JMCraig · 11-11-2008, 09:15 AM

Well, the AOLserver (nsd) is working fine. There's nothing I can see in dmesg remotely resembling an error, so I'm back to square one.

unSpawn · 11-11-2008, 03:48 PM

Track back installed packages?

JMCraig · 11-12-2008, 07:45 AM

unSpawn,

Thanks for hanging in there on this one with me. Here's the list of packages from yum.log that might (timewise) have been the culprit:

[scroll]

Code:

Sep 15 11:56:11 Installed: kernel-module-ntfs-2.6.18-92.el5xen - 2.1.27-0.rr.10.11.x86_64
Nov 08 14:38:55 Updated: libxml2 - 2.6.26-2.1.2.6.x86_64
Nov 08 14:39:06 Updated: nss - 3.12.1.1-1.el5.x86_64
Nov 08 14:39:11 Updated: xulrunner - 1.9.0.2-5.el5.x86_64
Nov 08 14:39:14 Updated: krb5-libs - 1.6.1-25.el5_2.1.x86_64
Nov 08 14:39:18 Updated: initscripts - 8.45.19.1.EL-1.x86_64
Nov 08 14:39:20 Updated: libtiff - 3.8.2-7.el5_2.2.x86_64
Nov 08 14:39:22 Updated: bzip2-libs - 1.0.3-4.el5_2.x86_64
Nov 08 14:39:23 Updated: cups-libs - 1:1.2.4-11.18.el5_2.2.x86_64
Nov 08 14:39:26 Updated: net-snmp-libs - 1:5.3.1-24.el5_2.2.x86_64
Nov 08 14:39:29 Updated: libvorbis - 1:1.1.2-3.el5_1.2.x86_64
Nov 08 14:39:31 Updated: OpenIPMI-libs - 2.0.6-6.el5_2.2.x86_64
Nov 08 14:39:32 Updated: xen-libs - 3.0.3-64.el5_2.3.x86_64
Nov 08 14:39:33 Updated: pam_krb5 - 2.2.14-1.el5_2.1.x86_64
Nov 08 14:39:41 Updated: devhelp - 0.12-19.el5.x86_64
Nov 08 14:39:45 Updated: lvm2 - 2.02.32-4.el5_2.1.x86_64
Nov 08 14:39:45 Updated: dmraid - 1.0.0.rc13-14.el5_2.1.x86_64
Nov 08 14:39:54 Updated: perl - 4:5.8.8-15.el5_2.1.x86_64
Nov 08 14:39:55 Updated: libxml2 - 2.6.26-2.1.2.6.i386
Nov 08 14:39:56 Updated: bzip2-libs - 1.0.3-4.el5_2.i386
Nov 08 14:39:57 Updated: libtiff - 3.8.2-7.el5_2.2.i386
Nov 08 14:39:59 Updated: libvorbis - 1:1.1.2-3.el5_1.2.i386
Nov 08 14:40:00 Updated: krb5-libs - 1.6.1-25.el5_2.1.i386
Nov 08 14:40:01 Updated: bzip2 - 1.0.3-4.el5_2.x86_64
Nov 08 14:40:02 Updated: cups-libs - 1:1.2.4-11.18.el5_2.2.i386
Nov 08 14:40:06 Updated: nss - 3.12.1.1-1.el5.i386
Nov 08 14:40:07 Updated: libxml2-python - 2.6.26-2.1.2.6.x86_64
Nov 08 14:40:12 Updated: cups - 1:1.2.4-11.18.el5_2.2.x86_64
Nov 08 14:40:21 Updated: yelp - 2.16.0-21.el5.x86_64
Nov 08 14:40:22 Updated: OpenIPMI - 2.0.6-6.el5_2.2.x86_64
Nov 08 14:40:24 Updated: dhcpv6-client - 1.0.10-4.el5_2.3.x86_64
Nov 08 14:40:26 Updated: krb5-devel - 1.6.1-25.el5_2.1.x86_64
Nov 08 14:40:30 Updated: krb5-workstation - 1.6.1-25.el5_2.1.x86_64
Nov 08 14:40:31 Updated: nss-tools - 3.12.1.1-1.el5.x86_64
Nov 08 14:40:32 Updated: ed - 0.2-39.el5_2.x86_64
Nov 08 14:40:35 Updated: xulrunner - 1.9.0.2-5.el5.i386
Nov 08 14:40:36 Updated: xen-libs - 3.0.3-64.el5_2.3.i386
Nov 08 14:40:54 Installed: kernel-xen - 2.6.18-92.1.17.el5.x86_64
Nov 08 14:40:56 Updated: nss-devel - 3.12.1.1-1.el5.x86_64
Nov 08 14:41:03 Updated: devhelp - 0.12-19.el5.i386
Nov 08 14:41:04 Updated: pam_krb5 - 2.2.14-1.el5_2.1.i386
Nov 08 14:41:11 Updated: xulrunner-devel - 1.9.0.2-5.el5.x86_64
Nov 08 14:41:14 Updated: xen - 3.0.3-64.el5_2.3.x86_64
Nov 08 14:41:20 Updated: xulrunner-devel - 1.9.0.2-5.el5.i386
Nov 08 14:41:24 Updated: krb5-devel - 1.6.1-25.el5_2.1.i386
Nov 08 14:41:25 Updated: systemtap-runtime - 0.6.2-1.el5_2.2.x86_64
Nov 08 14:41:27 Updated: systemtap - 0.6.2-1.el5_2.2.x86_64
Nov 08 14:41:29 Updated: Virtualization-en-US - 5.2-11.noarch
Nov 08 14:41:30 Updated: sos - 1.7-9.2.el5_2.2.noarch
Nov 08 14:41:31 Updated: bzip2-devel - 1.0.3-4.el5_2.x86_64
Nov 08 14:41:33 Updated: libvorbis-devel - 1:1.1.2-3.el5_1.2.i386
Nov 08 14:41:35 Updated: kernel-headers - 2.6.18-92.1.17.el5.x86_64
Nov 08 14:41:37 Updated: libtiff-devel - 3.8.2-7.el5_2.2.i386
Nov 08 14:41:38 Updated: libvorbis-devel - 1:1.1.2-3.el5_1.2.x86_64
Nov 08 14:41:42 Updated: firefox - 3.0.2-3.el5.x86_64
Nov 08 14:41:43 Updated: libtiff-devel - 3.8.2-7.el5_2.2.x86_64
Nov 08 14:41:47 Updated: libxml2-devel - 2.6.26-2.1.2.6.x86_64
Nov 08 14:41:47 Updated: bzip2-devel - 1.0.3-4.el5_2.i386
Nov 08 14:41:57 Installed: kernel-xen-devel - 2.6.18-92.1.17.el5.x86_64
Nov 08 14:42:00 Updated: libxml2-devel - 2.6.26-2.1.2.6.i386
Nov 08 14:42:07 Updated: firefox - 3.0.2-3.el5.i386
Nov 08 14:42:10 Updated: tzdata - 2008i-1.el5.noarch
Nov 08 14:43:13 Updated: Virtualization-en-US - 5.2-11.noarch
Nov 08 14:43:13 Virtualization-en-US - 5.2-11.noarch: extra callback for package Virtualization-en-US - 5.2-11.noarch in state 10

[/scroll]

What's the deal with that last entry? Is that perhaps the culprit--some kind of error in the processing of that package?

So, I guess the thing to do is uninstall what I can and see if that helps. There isn't a slick way to undo/rollback an "Updated" operation is there? That is, I'll just have to look for an earlier version of the packages that have been updated and put that back, right? (Well, since I've got a Xen RHEL guest installed at the moment, maybe I can get some use out of that--as I can't get the network to do what I want there anyway.) It also appears that I might be able to intercept the bootup and choose the prior, non-Xen kernel to boot and see what that does....

If you see any red flags among this stuff, I'm very open to suggestions.

JMCraig · 11-13-2008, 07:39 PM

Hi Folks,

OK. Here's where the problem lay, apparently. I installed Project-Open and set it to run by adding the following to rc.local:

su -l projop -c "/usr/local/aolserver/bin/nsd -i -t etc/config.tcl"

The message that came up about "finished reading the nsd.tcl" file is just the message the OpenACS/nsd/AOL Server puts up as it starts--and, according to a friendly fellow on irc.freenode.net #openacs, the message is just referring to the wrong file name--it really means the config.tcl fine in the aolserver directory tree.

My testing in a bash session suggests I need to add an & so the su can return while the aolserver (OpenACS) process itself runs in the background:

su -l projop -c "/usr/local/aolserver/bin/nsd -i -t etc/config.tcl &"

Without the & to push the aolserver/nsd process to the background, it's got the console tied up so the prompt won't appear--until that process dies.

It hasn't been convenient for me to test this with a reboot, but it sure looks like this'll do it (based on extensive testing just in a bash session). (If this doesn't fix it, I'll post the results of whatever I figure out. I'm assuming it's fixed until a reboot proves me wrong.)

Thanks to all who took a look!

John

unSpawn · 11-14-2008, 12:15 PM

Well done fixing this yourself and posting the fix!