I noticed that I couldn't access our RH 7.3 server via Telnet this morning, and when I went to login at the actual server, it would let me type in a username, but then just hang, before the password prompt came up. So after a hard reboot, it starts booting, finds all the drives, then comes up with an error that says "Your system appears to have shut down uncleanly, Press Y.......(which I know isn't uncommon), but before I can do anything, it comes up saying "Checking root filesystem.....execvp: Permission Denied" FAILED
Can anyone diagnose this? I'm trying to find these errors in my Linux Bible, but am having no luck so far.....Thanks VERY MUCH in advance. If you need more info, please let me know.

First thing I would try would be to boot with a Linux CD distro such as BBC (Bootable Business Card) or Knoppix (or one of its clones). Once booted with one of these, you can fsck your file system to get past the unclean shutdown error.

A general user account can be prevented from logging in if the partition holding /var or /home is full. You should still be able to log in as root though. In any case, check the partitions capacity while still mounted from the boot CDROM session and address any issues.

Good luck.

Thanks idaho....I can type in the root password and fsck.ext3 all the filesystems just fine. They all came back fine except for one, which was supposedly "Fixed" after the first check....Btu then when I reboot, the same thing happens again and again. Now, instead of "execvp: Permission Denied" at the end of the error, I get "execvp: No such file or directory". I don't even know where to start.....

It is starting to sound like you have a HDD failure. Hopefully the partition that is failing is not a data directory for which you do not have recent backups.

If you don't have recent backups, both of the boot CDs I mentioned earlier are good tools to mount and copy over (via nfs, if no other option is available) the valid partitions to your backup media. You may even be able to copy over much of the data on the failed partition, depending on how localized the damage is.

Yes it is uncommon to find your machine shutdown uncleanly--that is what is known as a frigging disaster!!!!!!!!!!!! That is why we have things like uninterruptable power supplies and why there are IT STAFF.


It still sounds like a root kit to me. Google the name of the failed command. See what shows up.

Boot up on one of the rescue disks and search the hard drive using find for the letter combinations.

Hmmm.....Great.
1.) If it doesn't boot, then it seems that it's the root drive, the Hard drive that we have dedicated to the OS installation only, correct? If that's the case, then I do have very recent backups of it.
2.) It IS on a UPS
3.) I AM the IT Dept., however I'm still learning Linux and didn't set this server up....
4.) I've Googled the crizzzap out of my error(s) to no avail

SO.......What's the best way to repair this thing ASAP? Reinstall/upgrade to 9.0 and then restore from tape?

I would still boot up on a rescue disk and do fsck on the data drive.

You need to have accurate hardware logs for the machines in your care. With specific and accurate partition information. The gives you the possibilty of recovering from multiple nightmares. Dead drives often may be brought back long enough to read them once. Accurate logs will allow extraction from a binary image.

I hope you didn't get left with a "Psychic Administrator" situation.

If you have good recent back-ups you might want to go for the upgrade, especially if you have an O.S. drive. Fully patched and updated RedHat 9 is supposed to be pretty good. I got hit by one ofthe little bastards in RH-8, but haven't been hit since upgrade to RH-9.

Life is easier if you partition the O.S. drive into specific little chunks. There is usually some performance gain as well.

"/" should only have read-only, necessary for booting and nearly empty directories. /bin; /sbin; /etc; /dev; /initrd; /lib; /mnt; /misc; /opt; /proc; /Lost and Found.

All of the others should be in their own partitions. When you use df -h from the command-line with that set-up you know immediately if something is wrong, and you will know where to look. Things stand out and logs can't over-run everything else if they are confined to a partition.

With linux anything which is autonomously writing somewhere can be re-directed or even written to it's own partition to protect the system. You are supposed to schedule purges and clean-ups in the scripts, but stuff happens.

Linux Documentation Project-- HowTos: Performance Howto and Partitions HowTo.

You really should start with the Linux Installation Guide and other Documentation provided on the "DOCS" ISO you can download from RedHat.

One thing, nearly everything you learn for linux applies to advanced administration in windows 2000--2003 server and O.S. as well. The exact procedures will be different, but identifying the process makes the commands more intuitive for MS.

Thanks, that's good to hear. Actually the guy whos set this server up left the best set of hardware documentation I've ever seen. Where drives are physically in the server, what they're mounted as, RAID setups, etc.....My main problem, is that I don't know where to start troubleshooting because it seems that no one has ever seen this particular problem before.

From the symptoms you have posted (System became unresponsive, unrepairable HDD errors on reboot), you appear to have a HDD failure. At this point you need to replace your failed HDD, reinstall your OS, then apply your backups.

For my servers, I have been using systemimager to take a reinstallable snapshot of the system - after I have the OS configured, and before I start populating the data directories. I recently came across another tool (see sourceforge.net) called lifeboat. Lifeboat tries to snapshot an installable image of the system to CD - creating as many CDs as it takes to capture the snapshot. In either case, having an installable snapshot of just the OS is my recommendation to prepare for future disaster recoveries. Messing with the original distro installation CDs is not my preferred choice in a disaster recovery situation.

Hmmmm……Well, in repair mode I can successfully mount ALL filesystems with no issues whatsoever. I can list all the files and directories in each filesystem with no problem. I can start services from the command prompt. Cannot start X-server. I really think it's something minor, not a HDD failure. Any more ideas guys?

Sounds like the drive repair process involved trashing a critical file. You may not need to replace the drive, but my experience tells me to be very nervous about a drive once it has thrown an error.

If your backup tapes include /boot, /etc, /sbin, and /usr/sbin, then you can try recovering these directories and attempt a reboot.

Don't forget to check that none of the repaired partitions are at max capacity before rebooting.

Thanks idaho. I use tar for my backups....How can I mount my tape drive in Repair mode?

Nevermind. I know how to mount it, but to restore from a tar backup?

mt -f /dev/st0 rewind
?? ?? ??

http://freebooks.by.ru/view/RedHatLi...d/rhl6u215.htm

Don't worry about the version when it comes to the command line.

tar -xp /DirectoryName/*

****According to RedHat unleashed--I use Images so I really don't have first-hand experience.****

As far as media goes, there is nothing wrong with creating and keeping a KickStart file on hand to do a full custom install--it takes about the same time as a network restore, but more time than an Image. 12-25 minutes from "bare Iron".
Any hardware. And not just the kernel; we're talking a full truly custom install as fast as it can hammer the files onto the hard drive. (Including partitioning, filesystem, configuration of services, settings, and scripted restoration of non-system files and applications.)

It is the same process (but far easier) as the un-attended install in windows with all of the OEM add-ins and scripts pointing to files with configuration and settings.

The more specific the Kickstart file, the faster the install. It is plain text and editable in any Unix style editor (windows editors put extra crap in there and screw it up). You can set it up to do pre-install tasks and post-install tasks via simple scripts. You only have to do it once, then make copies and modify it at need. This means you never have to sit through a text or gui install again. (And with DVD boot-able, one disk should do it.)

Stripping the media of the things which you don't want or need and updating the media files with all of the newest updates is fairly simple. Again, the files which need to be edited are plain text. Then you can burn new "custom" CDs. You can also burn all of the updates to a different CD and set it up as a local source for up2date. (Then add it as a post-install script.)

If you can waste space on a hard drive you can put all of the files in a format similar to the CDs and keep a copy of the KickStart file there. Then stick a copy of the first installation CD or a rescue CD with a few scripts in it for automation and an empty floppy inside the case and forget it until you need it. Your script would ask for a floppy, copy KickStart onto it and then proceed to hammer a new install into the machine.

I didn't know that stuff about KickStart until I researched a new-be question and Read the install guide from RedHat. I took Advanced Admin for Linux/Unix and hadn't read the frigging Docs--what a moron.

**The other man could be right, the hard drive may not be trust worthy.

***I still think it was a failed "root-kit" attempt. I check out the Cracker sites on a regular basis; and find in their forums, little morons asking for directions on how to port scan, and how to install "root-kits".

/*
http://www.linuxselfhelp.com/gnu/tar...r/tar_toc.html

The Linux Self help Tar Guide.
*/
/*
http://www.tldp.org/LDP/lame/LAME/li...tml#TAR-BACKUP

Linux Administration Made Easy (LAME--found on the DOCS ISO on RedHat.)

This link is to the specific Chaper online at The Linux Documentation Project website.
*/

The usage for extracting from scsi tape device /dev/st0 is:

tar -xpvf /dev/st0 /what/you/want/extracted

To check what tar is going to find on your tape (so you get the /what/you/want/extracted down just right), you can run

tar -tvf /dev/st0

My recollection is that repair mode is really only intended for fscking your partitions. You may want to try to boot to single user mode, or use one of the bootable Linux CDs I mentioned earlier to mount your partitions and extract your tar backups.

Also, repeating myself here, check your partition usage (use the df command) before you start the tar backup extraction - no point in trying to extract to a partition that is full.