Hi all,

Here is the issue:

Few days ago, the server did not respond to a ssh request from a user at night. A user tried to check what went wrong with computer and tried to login from terminal next morning. As the computer was unresponsive, he somehow decided to boot it by turning the power off. To make the story short, the server rebooted; however, he can't login to his account. Actually, the server could not start some processes; but was able to ask user to enter his account username. Even though, he enters the correct username and password, server does not accept the request. I also could not login as root.

I just checked the server logs by booting it in single user mode. Here are some interesting lines:

Before the reboot:
irqbalance : can't balance irqs on a uniprocessor system: failed

After the reboot:
irqbalance : can't balance irqs on a uniprocessor system: failed
fsck:
fsck /: (this is repeated 900+ times)
sshd: failed
ntpd: failed
httpd: httpd: bad user name apache
httpd: httpd startup failed
runuser: runuser:
runuser: user htt doesnot exist
iiim: htt startup failed
cannaserver: cannaserver : -u flag spesified, but can not run as root
cannaserver:
canna: failed
atd : atd startup failed
atd : failed
dbus-daemon-1: Could not get password database information for UID of current process user "???" unknown or no memory to allocate password entry
dbus-daemon-1: Unknown username "haldaemon" in message bus configuration file
dbus-daemon-1: Unknown username "root" in message bus configuration file


This might be something related with shadow file.
Here is part of /etc/shadow

root:$1$blah_blah_blah:14274:0:99999:7:::
bin:*:12761:0:99999:7:::
sshd:!!:12761:0:99999:7:::

I have to solve this issue as soon as possible. Any idea/help is appreciated.

First, thank you for not putting the word URGENT or anything similar in your post's title. Thank you. Thank you. Thank you. :-)

You didn't mention the hardware that you are running. It sounds like you have a substantial software environment. Are you using a RAID disk configuration? If you have a hardware-only RAID then you might want to check to see if the virtual disks are okay.

If you can boot into runlevel 1 and log on then can you fsck your root partition? If fsck says that the volume is clean AND refuses to perform the check then you can usually force a check using -f in the command but that depends on which file system type you are using.

Can you boot a live Linux CD? You could (hopefully) make a backup of your system and try to fix whatever is going wrong.

Look at your /var/log/messages file for pointers as to what is not working.

I appreciate your help. It is very valuable.

This seems to be a RAID issue. I'll go through the manuals and will resolve the issue.

I was able to boot in single user mode and run fsck. It showed 12 errors and fsck fixed them all. Then, I rebooted in run level 5, same processes failed to start. Later, I switched to single user mode and checked the /var/log/messages and boot.log, rerun fsck and got more errors which are all fized by fsck. I booted the machine in run level 5 one more time, the computer got frozen. I had to turn the power off. Now, it fails to boot in run level 5.

I can boot from live linux CD. I backed up the computer few days ago before starting to fix it. As I said before, this seems to be a hardware related problem. I have to learn more about RAID, especially get an understanding of how to check virtual disks.
I went over messages file. There is no major issue other than authentication errors.

THanks for your comments and time.