I am an enduser who now needs to do some sys admin! Only real past experience is having successfully installed Slackware back in 1995-96.

I have RHEL WS 4 on an HP XW6200 dual xeon system with 2 SATA (~75 GB each) and 1 IDE (160 GB) HDDs and Nvidia graphics card. Stable OS, no issues since 2005 till about 5 weeks ago. Abrupt power down brought up EXT3-fs error, system was hung, could remote login but not do a clean shutdown. Had to hard re-boot. Bootup normally happens from SATA drive.

Should have paid more attention, but all seemed well. Every few days the same error was repeated. Had to run fsck a couple of times. Last week, when a user tried to login, system got hung, did not allow remote login, had to switch off power. On restart showed following errors (could not get all messages as they were scrolling too fast):

(some ext3-fs errors in /dev/sda6 and /dev/sda2)
SCSI error <0 0 0 0> return code....
FMK EOM current sda sense= 70 cc
ASC=e2 ASCQ=cf
end request: I/O error, dev sda ...

On re-boot, errors were :

ata1 : called with no error (80)!
mkrootdev : label/ not found
mount: error 2 mounting ext3
mount: error 2 mounting none
switch root: mount failed: 22
umount /initrd/dev failed: 2
Kernel panic - not syncing: Attempted to kill init!

Tried F10 (system set-up). Found IDE and 1 SATA drive but not the 2nd SATA. System seemed to be trying to bootup from IDE, which is not the boot disk. Hence it would get hung, no messages.

Tried bootup with recovery DVD (Ubuntu). Many bootup commands showed [FAILURE], then showed :

INIT Id "x" respawning too fast:disabled for 5 minutes.
INIT Id "x" respawning too fast:disabled for 5 minutes.
INIT Id "x" respawning too fast:disabled for 5 minutes.
ATA: abnormal status 0x80 on port 0x34 F7
ATA: abnormal status 0x80 on port 0x34 F7
ATA: abnormal status 0x80 on port 0x34 F7
ata1: command 0x35 timeout, stat 0x80 host_stat 0x21
ata1: status 0x80 {Busy}
SCSI error: <0 0 0 0> return code=0x8000002
FMK EOM ILI current sda : sense 70 f0
ASC=2 ASCQ=a0
end_request: I/O error, dev sda, sector 129842497
Buffer I/O error on device sda6, logical block 2394324
Lost page write due to I/O error on sda6
(similar messages continued endlessly)

When I tried to switch off the CPU, it gave :
ACPI-0691: ***Error: acpi_ev_gpe_dispatch: No handles or method for GPE[1F], disabling event

then earlier ata errors continued.

Opened up the CPU unit, tightened the connections, then both SATA disks were recognized, normal boot-up was initiated, showed unclean shutdown, /usr unclean, and went to Repair mode. e2fsck did not work. Ran fsck. Filesystems were treated as ext2-fs. Showed a lot of inode errors, which I should probably not have cleaned. Finally, got :

EXT3-fs error (device sda6): ext3_find_entry: bad entry in directory # 163521:inod out of bounds - ffset=1148, inode=163599, rec_len=16, name_len=5

Read a post than partition should be re-mounted as ext2-fs, but did not know how to do this.

e2fsck /dev/sda6 gave :

couldn't find ext2 superblock, trying backup blocks ..
ext3 recovery flag clear, but journal has data
Recovery flag not set in backup superblock, so running journal anyway.
/: recovering journal
Backup journal inode block informatiom.

Pass1: checking inodes, blocks, and sizes
Root inode not a directory. Clear(y)?

That is where I am now. What should I do?

Honestly, what you should do is call a Linux pro - this may disappoint you since you'll have to pay for it but I believe your skills are not enough.
This looks like a serious problem which handled wrong might ruin every piece of data you've got.
Done correctly - well, everything except broken hardware can be fixed.
Of course it depends on the server, what's it used for, how many are depending on it?

If you want to try yourself, here's my "analysis":
1) At least one partition on one disk - but probably more - are faulty.
It could be a physically damaged hard disk or just a corrupt filesystem, impossible to tell by now.
One reason for a corrupt filesystem is an abrupt power down.
On the other hand, a faulty system might cause a powerdown.
2) The system seems to boot up from wrong disk/partition.
This can be because one disk is physically damaged and therefore not seen.
It could also be that the disks are numbered in a different order - that happens with anything but IDE-drives (AFAIK)

In order to help you along we need the following:
Post the systems /etc/fstab.
If you can't read that file write down what you remember/know of the partitioning & mounting scheme, including info about what filesystem is used.
Information about bootloader - lilo or grub?
Post bootloaders config file.

Do NOT!try to repare a filesystem if you don't know what you are doing! (Personally, I learned that the hard way, ruined a very pretty girls disk completely )

Or you could simply reinstall.
Depends of course on what it's used for, if you can set it up again with the used apps, and if you know where users data is. If the system is properly setup you can remove the system without touching users data.

Edit: I forgot something very important:
Don't do anything from the system itself! Shut it down if it's running and don't start it again!
Boot from a live-cd, then check out your disks & partitions.
Mount the partitions you need read-only!
Here are a few commands:
# fdisk -l (shows the disks & partitions)
# mount /dev/sda1 /mnt/sda1 -o ro (mounts /dev/sda1 readonly on directory /mnt/sda1)

1 members found this post helpful.

^agreed. Call a professional in attempt to save what can be saved. I hope you created backups every once a while.

If it is a production server, I would ditch all of the hardware and buy new hardware, just to be on the safe side. It could be a good time to upgrade to RHEL 5.5.

Thanks for all the suggestions.

Flip side : All data is safe thanks to automated daily backups
Flop side : The system that is down was via media for data backup from all linux workstations, and now I have to fix that.

Flop : Can't boot from Ubuntu live CD - explicitly specified CDROM for boot-up, but control gets transferred to HDD and ends up with Kernel panic.

Will try to get professional help.

If you need a Live CD for repairs, please use SystemRescueCDor one of the CentOS LiveCD's. Using other distros on business data is not something I would be comfortable with.

The BEST thing to do.

Knoppix was able to read all partitions except / and /user. Errors shown for accessing / and /usr were : wrong fs type, bad option, bad superblock. Called expert from HQ, whose SystemRescueCD didn't help gain access to /. As a last resort, he ran fsck -y, and said he lost /home. However I have backup on server and can retrieve all but 4 files. Finally, he re-installed OS on a new HDD. Thanks for all the inputs.