[SOLVED] Server Data Recovery Via Rescue Mode

seth.cannon · 08-28-2015, 01:19 PM

So my company has been working some time to recover data off a server that is no longer working on a Dell PowerEdge 2850. We have Red Hat Enterprise Linux 3 on the server. I've recently taken over the project and I've tried to use Rescue Mode which proved to be inadequate. By recommendation of Dell support, we are trying to get CentOS on the system to be able to recover the data. However, the system doesn't recognize the CentOS iso as a bootable device. We then tried upgrading BIOS to A07 rather than A02 to see if that helps, however, it requires and executable. Again we tried a few DOS isos that aren't recognized as bootable devices (to try to run the executable). Currently we are trying to run rescue mode to run the executable through rescue mode to flash the BIOS.

However, the system doesn't seem to recognize... well anything. Using fdisk -l yields no response (immediate return to input prompt). Trying to mount anything will come back to and error that /etc/fstab does not exist. I've seen some people with similar solutions running "sudo" commands which I'm not familiar with, and when tried the sudo command is not found.

We aren't trying to save the server, we plan on trashing it, but we need the information off it. Currently as I mentioned, we are trying to get the BIOS upgraded on a prayer that fixes some problems, but we have reason to believe that it won't even fix the problem. However, if someone has suggestions on how to get the BIOS flashed (or another avenue to extract the data) please let me know.

Side note: booting from a USB, CD, and external DVD has all yielded the same no bootable device problems. The only iso we've gotten to run is the actual RHEL 3 installation CD which is where we are running Rescue Mode from.

EDIT: SOLVED: Link to description of fix: http://www.linuxquestions.org/questi...ml#post5415296

Thanks to michaelk for the direction to the final solution!

yancek · 08-28-2015, 03:03 PM

Quote:

We have Red Hat Enterprise Linux 3 on the server

Which hasn't had any support for almost 5 years??

https://rhn.redhat.com/errata/RHSA-2010-0817.html

How did you try to use the CentOS iso? Put it on a DVD/flash drive? How? Have you tested it on another computer?

Are you saying when you run the RHEL 3 install CD, you get no output from fdisk and that /etc/fstab doesn't exist? How old is the hardware? Is it capable of booting usb/DVD? You might try downloading and using SystemRescueCD as it will fit on a CD.

seth.cannon · 08-28-2015, 04:08 PM

We tried to use CentOS using the ISO provided by the Dell support team located here: http://linux.dell.com/files/openmana...firmware-live/

We tried both using a DVD as well as a USB flash drive. Both returned a result of "No bootable device detected." For the USB we simply copied the iso file to the USB and plugged it in while disabling the BIOS ability to boot from hard drive. For the DVD, we burned the iso onto the CD.

As far as your RHEL question regarding the output of fdisk, yes, that is correct. Likewise for the fstab not existing. The Hardware is a Dell PowerEdge 2850, so it's a very old system. Yes It is capable of booting through a USB. It certainly has direct examples of that where other users were troubleshooting it with just a newer version of BIOS. We're not sure if the version we're currently running won't run from a USB (since we're on an older BIOS version than the examples we found). However, we are definitely confident that the PowerEdge 2850 is able to boot from a USB and DVD. However, as far as the DVD goes, the example we had found that verified that it was capable of using a DVD was by an example where the user had replaced the CD-only drive with a DVD compatible one, whereas we only have an external DVD we are using through a USB cable. The Dell support for the PowerEdge 2850 was guiding us down paths booting from USBs and DVDs, so I presume that it should work in theory.

I'll look into the SystemRescueCD in the meanwhile.

Edit: I'm currently downloading the RescueCD and will try to run it off that first thing Monday morning, and will post any results of that process then.

syg00 · 08-28-2015, 08:12 PM

An old server running non-supported distro and no backup. Some-one needs their arse kicked.
I don't think I've ever run RHEL 3, so this may be all waffle.

For the commands from a Redhat or CentOS system prefix them with "su -c" as follows. Each command will require the root password. There are better options, but this is simplest/safest.

Code:

su -c "fdisk -l"
df -hT

As for the on-disk fstab, that will be relative to where the partition is mounted - say /mnt/etc/fstab.

When you "copied" the iso to USB, how did you do it - it needs to be a "dd".

seth.cannon · 08-31-2015, 09:23 AM

Quote:

Originally Posted by syg00

For the commands from a Redhat or CentOS system prefix them with "su -c" as follows. Each command will require the root password. There are better options, but this is simplest/safest.

Code:

su -c "fdisk -l"
df -hT

As for the on-disk fstab, that will be relative to where the partition is mounted - say /mnt/etc/fstab.

As far as the commands go,

Code:

su -c "fdisk -l"

returns su: command not found.

Code:

df -ht

returns

Code:

df: option requires an argument --t
try 'df --help for more information.

Unfortunately, I'm not familiar with either command, so I'll have to do quite a bit of research to troubleshoot those commands.

Quote:

Originally Posted by syg00

When you "copied" the iso to USB, how did you do it - it needs to be a "dd".

Not sure what you mean by "dd". I simply drag-and-dropped the file to the USB. I'm not used to needing to do something more than that for USBs.

The rescueDVD is burned from yancek's post. I'm going to run that and see what happens.

EDIT: When I try to boot the SystemRescueCD.iso on a burned CD, I receive the same error as with others: "No boot device available"

chrism01 · 09-02-2015, 03:41 AM

This

Code:

df -ht

is not what syg00 said

Code:

df -hT

should work as suggested.

Also, could you not just move the disk to another system and mount it there?

michaelk · 09-02-2015, 06:42 AM

Typically an ISO file is a byte for byte copy of an optical disk. The files by them selves are not bootable so "drag and drop" copying will not work. For creating a bootable optical disk you need to select burn as image from either a windows 7/8 OS or using imgburn or isorecorder applications. For creating bootable flash drives you need to use unetbootin. If the ISO image is a hybrid then using dd will also work.

Typically you need to be root to get output from the fdisk command. You also need to be root to manually mount file systems. But should not be required in rescue mode and neither is using su (been awhile since I had to). I would probably stick with RH rescue mode since it has support for SCSI. systemrescuecd does support SCSI but do not know what other problems you might encounter.

Typically RH rescue mode will mount the root (/) file system to /mnt/sysimage. Other file systems will have to be manually mounted. What do you mean by no longer working? Do you know if the SCSI controller and drives are still functioning?

descendant_command · 09-02-2015, 08:57 AM

Quote:

Originally Posted by chrism01

could you not just move the disk to another system and mount it there?

^This.
In the time that it takes to write about doing it, it could be done.

seth.cannon · 09-02-2015, 10:21 AM

Quote:

Originally Posted by michaelk

The files by them selves are not bootable so "drag and drop" copying will not work. For creating a bootable optical disk you need to select burn as image from either a windows 7/8 OS or using imgburn or isorecorder applications. For creating bootable flash drives you need to use unetbootin. If the ISO image is a hybrid then using dd will also work.

The working computer I have available is a Mac, so while I "drag and dropped" to the CD, you still burn it by the in-built Mac burning feature. In the finder window when CD is selected there is a burn button at the top right that burns any files currently listed on the CD's window to the CD. As far as the flash drive goes, I'll try unetbootin.

Quote:

Originally Posted by michaelk

Typically you need to be root to get output from the fdisk command. You also need to be root to manually mount file systems. But should not be required in rescue mode and neither is using su (been awhile since I had to). I would probably stick with RH rescue mode since it has support for SCSI. systemrescuecd does support SCSI but do not know what other problems you might encounter.

I think I am at the root. I guess I can't guarantee that one, but, at least from what I can gather, I seem to be at the right place.

Quote:

Originally Posted by michaelk

Typically RH rescue mode will mount the root (/) file system to /mnt/sysimage. Other file systems will have to be manually mounted. What do you mean by no longer working? Do you know if the SCSI controller and drives are still functioning?

Not completely sure on which aspect of things not working you are asking about since there are a lot of things not working. If you are referring to the computer, then I meant that when the computer boots to the hard drive it gets stuck in an infinite loop of trying to check the disk. If you are referring to running commands from rescue mode such as fdisk or su, then I mean literally it returns blanks responses or commands not found. If you are referring to trying to boot to a .ISO file, on boot up the computer complains "No bootable device".

For the SCSI controller and drives, I don't know what that is. I know that it's listed in BIOS, but past that you have me lost on that.

Quote:

Originally Posted by chrism01

Also, could you not just move the disk to another system and mount it there?

I'll look into whether or not we can do this. Also will try the correct df command.
Edit: Seems we can't take out the drive because its on RAID-1 configuration.

Edit2: df -hT results:

Code:

Filesystem            Type         Size       Used          Avail     Use     Mounted on
rootfs               rootfs        6.0M       2.5M           3.3M      43     /
/dev/root.old          ext2        6.0M       2.5M           3.3M      43     /
/tmp/cdrom          iso9660        140M       140M              0     100     /mnt/source

michaelk · 09-02-2015, 10:39 AM

I do not use a Mac much but if you examine the disc and only see a single file then it was not burned correctly. The Mac has the dd command so a USB image or hybrid ISO file can be written in this manner. No need to use unetbootin.

http://hints.macworld.com/article.ph...60619181010389

Yes, I was asking about the errors encountered when trying to boot the Dell computer.

Are these the step taken when booting to rescue mode?
https://access.redhat.com/documentat...mode-boot.html

seth.cannon · 09-02-2015, 10:47 AM

Quote:

Originally Posted by michaelk

I do not use a Mac much but if you examine the disc and only see a single file then it was not burned correctly. The Mac has the dd command so a USB image or hybrid ISO file can be written in this manner. No need to use unetbootin.

http://hints.macworld.com/article.ph...60619181010389

I'll go ahead and look into creating USB.

Quote:

Originally Posted by michaelk

Are these the step taken when booting to rescue mode?
https://access.redhat.com/documentat...mode-boot.html

Yes, the first step that goes wrong is chroot /mnt/sysimage which yields

Code:

chroot: cannot change root direction to /mnt/sysimage: No such file or directory

michaelk · 09-02-2015, 11:05 AM

Some of the problems could be attributed to the SCSI module not being loaded so post the output of the command
lsmod

seth.cannon · 09-02-2015, 12:29 PM

Quote:

Originally Posted by michaelk

Some of the problems could be attributed to the SCSI module not being loaded so post the output of the command
lsmod

yields:

Code:

Module                  Size     Used by  Not tainted
parport                21504      0   
lvm-mod                46624      0 (unused)
ext3                   62692      0 (unused)
jbd                    47056      0 (ext3)
msdos                   7116      0 (unused)
raid5                  16872      0 (unused)
xor                    11728      0 (raid5)
raid1                  13644      0 (unused)
raid0                   3688      0 (unused)
md                     49536      0 [raid5 raid1 raid0]
sr_mod                 15832      0 (unused)
sd_mod                 12144      0 (unused)
scsi_mod               64088      2 [sr_mod sd_mod]
nfs                    77560      0 (unused)
lockd                  53744      0 [nfs]
sunrpc                 77244      0 [nfs lockd]
vga16fb                10944     63 
fbcon-vga-planes        4968      0 [vga16fb]

Sorry for the late reply, I had a meeting. Otherwise, the original reason for trying to flash bios was to see if that fixed problems that prevented things like CentOS from running (to be used to recover data). I went ahead and tried to put CentOS on a USB using the Disk Utility like you mentioned. Unlike before where I had a single file on the USB (.iso), I now had two folders (including one named boot). However, it still didn't boot, but I'm not sure if the BIOS was able to see a flash drive (though I'm positive at the least that some other versions of BIOS for the system are able to boot from USB). This leads me to thinking flashing BIOS is still necessary, but the only flash I've found so far is .exe not .iso, so I'll try to dig for a .iso flash to put on a CD now that I can properly burn. Unfortunately the CentOS iso is too big for a CD.

Edit: Well, I found an iso for the BIOS. I successfully flashed the BIOS. Unfortunately, other than the BIOS changing literally from the text A02 to A07, nothing seems to have changed. This includes the BIOS setup which does not match images I've found online of the BIOS config of the upgraded version. The reason that pointed us to upgrading BIOS to begin with was the fact that one of the step-by-step image based tutorials for booting from a USB was based in a new version of BIOS and their version had more config options related to the USB which still do not exist in my newly updated BIOS. Your comment on how to properly do ISOs to CDs has been enormously informative and helpful. It just seems that the avenue of updating BIOS didn't actually do the trick. Now to look into other avenues...

michaelk · 09-02-2015, 12:52 PM

I forgot to ask if the error was generated by the operating system or scsi card?

seth.cannon · 09-02-2015, 01:18 PM

Quote:

Originally Posted by michaelk

I forgot to ask if the error was generated by the operating system or scsi card?

Which error and how do I check?

Edit: Well the good news is that the BIOS flash allowed me to run an external DVD. This allowed me to run CentOS.iso (which was recommended to do by the Dell support team a long time ago to do data recovery on this machine). I couldn't do this before because of it not recognizing the USB drives. However, I have no idea what to do in CentOS. I'm not sure if it's different then RHEL Rescue Mode or not. Exploring around I found that dev is full of drives which didn't exist in RHEL Rescue Mode (yay I think!). However, mnt is empty, so there is no mnt/sysimage. I have no clue how I would go forward about doing data recovery. I'll do some googling research, but if you can guide me to use that system it would be awesome. Either way, you have been tremendously helpful so far.