Hi,

Am having very strange problems with a CentOS 4 server. First thing to say is that I do not currently have physical access to the server (It's at work) and am hoping that I do not have to go there!

Got a call to say that server was down and had it re-booted.

On trying to SSH in I get username and password prompt as usual. It seems to log in successfully as I get the 'last logged in at X from Y' message; however, I get no command prompt.

I can get in using webmin, and can access file manager, logs etc that way.

I can also use SCP.

Strangely, when I try to reboot the machine via webmin, it claims to have excecuted the command but does not reboot.

This server is host to two Xen virtual machines which did not start at boot. I need to get in via SSH to find out what is happening.

Any help would be appreciated, I am beginning to contemplate travelling to the server which will not be good!

Tom

Hi.

Doesn't look good. Sounds like you've got a corrupted filesystem, as you can't run your login shell, or run commands to reboot the host.
Last time I saw something like this, it was from a corrupted /var FS on Solaris - I was getting a login prompt, but since /var wasn't writable, the 'last' log couldn't be written to so logins just hung.

Any errors in /var/log/messages?

Dave

P.S. Always on a Friday afternoon, eh?

Hmmm, don't like the sound of that!

Doesn't seem to be a problem with /var or logging. Last entries in /var/log/messages are as follows:

(for any users in different timezones, that was about 10 minutes ago!)

As you can see the session is opened successfully (secure log confirms this). The session is closed by me closing the PuTTY window.
Anything I type is echoed to the terminal, just no shell prompt.

The strange thing is that SCP works (which I think uses the bash shell to do stuff and will not work without a compatible shell).

Have been through other logs and cannot see anything out of the ordinary (although not altogether sure what I should be looking for).

Did think it might be a HDD space issue, but df reports only 60% of space in use.

Any ideas?


(You are right. These things ONLY happen on a Friday!)

OK,

Have made progress. Made a new user in webmin with a different default shell (tcsh). Now I get a shell prompt!
Running TOP shows several BASH instances which seem to have hung. I guess that BASH must have got corrupted somehow.

Any suggestions on the safest way to resurrect it? (will probably start off by trying to uninstall and reinstall using yum).

T

First you can verify the integrity of the bash package:
if something has changed and/or corrupted you have to investigate where the problem arised from. You can always try to reinstall it (maybe you get a newer version). However, it could be something else: has the /etc/bashrc or your ~/.bashrc be changed recently? Does it happen to every user on the system? Did you try to create a new user giving /bin/bash as shell? Has the /etc/passwd file been hacked?

Just some silly questions, trying to exclude the worst options...

It was affecting all users.

After changing default shell I managed to copy a bash excecutable from another machine. Bash is now working! (have kept a copy of the bad one, but am fairly sure that it must have been corrupted somehow).

My worry is that 'shutdown -r now' still does not work - it displays the normal messages but nothing happens.
This suggests that I may have further corruptions somewhere. Don't think I can run fsck from here as I would need to unmount the partition?

I think this is probably going to require a re install at some point, but at least I can access the filesystem now and copy data elsewhere :-)

Back in office on Thursday so guess it will be time for a boot CD and a large coffee!

T