LVM Reconstructions
 

Background:
We installed an OVA (vmware 6.0) provided by a manufacture that contained one LVM. The LVM contained only one volume initially (250gb). Through VMware three additional LUNS were mapped to appear at local drives. 26TB, 26TB, 34TB creating a total LVM of ~88TB. According to /etc/fstab the LVM is mounted on /var. After being damaged as listed below the system is no longer directly bootable to my knowledge.

The base system is RHEL 5.10 and utilizes iNodes v1.

What Happened:
Another company came in at some point an "unmounted" at least two of the PVs which caused the LVM to put out a missing flag. Due to the LVM not being activatable the system would no longer boot properly. The customer called me in after someone affected the system and has asked that I attempt to recover the data. When the drives were reconnected I dont know if the original order was maintained.

Data:
To access the data I have two choices, one is a Knoppix recovery CD or a CentOS Live Disk v5 (which is important because of iNodes v1). Here is tons of output from commands issues while on Knoppix as well as some action items we have taken.

I do have a copy of the backup LVM config file, which showed sdc and sde as missing. I attempted to repair by reloading the config file.

Here inline is the config file for the LVM that was loaded, it appears all the blkIDs lined up correctly.

After reloading the config I was able to activate the LVM.

I thought everything was great until I tried to mount it.

The above is where I'm stuck at this point. I tried running xfs_repair with both Knoppix and CentOS Live CD. Neither operation seems to have worked.

This is the output of Knoppix stating it doesn't support iNoveds v1
I do have hex dumps from the front of each device that I can share if that would help. Each block does show LVM data starting @ 0x3000.

Next Steps:
I'll take any chances in the dark that people think might work to mount and recover the data. I have full access to the system through the boot up of the rescue disk or Live disk.

Thanks,

VDI Engineering

Other Information that might be of use:
- We are able to mount the orginal boot drive through a 2nd virtual machine so the host O/S information is still accessable (thats how we got the LVM backup files)

The order of the disks doesn't matter. LVM assembles the array according to the UUID that is in the header of each physical volume. The header in each PV contains all the information that is in that /etc/lvm/backup or /etc/lvm/archive file (in ASCII, just not as nicely formatted), and the "id =" fields for the PVs are used to search for the devices. As stated in the file, the "device =" fields are just a hint to speed up the search.

It doesn't make a lot of sense that sdc and sde were missing when the array was being assembled, but magically reappeared for vgcfgrestore, which searches for those same UUIDs and will fail if any are not found.

Anyway, it does not appear that the problem is now with LVM, but rather with the XFS filesystem due to the drives being pulled while the filesystem was mounted. As the message from the Microknoppix xfsrepair states, you need to be using an older version that supports your V1 inodes. Does the CentOS 5 Live Disk not include xfsrepair ?

You are correct about the disks not just coming back into the set. We took the LVM config file listed below and removed the "flags = ["MISSING"]" so instead it was just "flags = []". Re-Ran the restore_cfg and that's when the missing disks showed back up. This is how we were able to re-add disks to the set.

We tried running xfs_repair from the CentOS Lice CD (v5) and it did run, but there was never any significant output on the command line. We let it run for about 5 days. We then ran the command again with the following added parms ( -v -i 10 ) after about 10 minutes, (not timed) the command line said, "Killed". I hope this added information provides some next steps.

This was the output from xfs_repair on CentOS 5 during the 5 day run period
We concluded that the xfs_repair program was not taking any action (staled) by looking at the disk I/OPS in VMWare, we were showing no activity to any disk for the VM.

Thanks

From what I can find, XFS was not a standard part of RHEL 5. The tools available then for it back then were perhaps not the best. Sorry about that. In any event, I can not help with XFS problems.

How much of that 88T was actual data ?. And if it's worth spending days on recovering, where are the backups ?. These are questions that the business should be answering, I'm not pissing on you.

What is the data they most care about - is it a proprietary format, or something "known" ?. Can they provision another 88T, preferably more, if you need to forensically scrape the data ?. Are they prepared to wait - maybe weeks ?. Can you re-construct directory structures and filenames as before if required by the software ?.
If you can't repair/mount the filesystem, there are probably no easy solutions unless you have backups.

syg00,

I completely understand what your saying and here's what I can tell you, I took nothing you said personally so no offense taken. The 88TB was a data tank for recorded videos. It contained the work of about 10 employees for the past 5 years! There was never a backup taken. The system is isolated (limited to about 1500sq ft in a single building), I know it's not an excuse but it is what it is unfortunatly. My team performed the build and installation in 2014. Another contractor who we will say was less than quailifed had O&M since turn over. I begged them to do back ups but they couldn't even be bothered to check for failed drives or validate daily operations.

There is no additional space to provision but if the LVM/XFS could be mounted I could/would attached 10TB external drives and scrape out the data one directory at a time. The files are all industry standard video formats (mp4, idx & some meta data). The system can not leave the building because of the data stored on it.

Turns out the system has been "broken" since Feb of 19 and they brought us in July of 19 to try and recover the data. Were at the point of complete (or partial) recovery or wipe the data tanks. Thats not what the customer wants but they know the O&M team failed miserably!

Any additional ideas on the XFS? Based on some previous remarks I'm going back overtoday and mounting the CentOS 5.10 disk to try and run xfs_check which predates xfs_repair. I also didn't realize you could run it against a single device (eg: /dev/sdc) I thought it was to be run against the entire VG.

Where did you hear that? You can't run xfs-repair against just part of a filesystem. If it tries to "fix" anything when run that way, it will just be doing further damage.

You have just a single ~97TB logical volume /dev/vg_dme/lv_var (also accessible as /dev/mapper/vg_dme-lv_var), and that is the "device file" for xfs-repair.

Thanks for letting me know. I only have done it against the entire LV todate. I came up with that information based on this thread I was reading last night.

https://forums.opensuse.org/showthre...needs-cleaning

I guess I made the assumption there was an underlaying LVM, but I guess it could also be XFS on a single drive.

Thanks for saving me some time.

Indeed, there was no LVM involved in that example.

UPDATE
 

UPDATE -
Based on the older style XFS I made one final attempt. Booted into CentOS 5.10 "linux recovery". Performed a scan and loaded up the disks. Utilited the old version of cfgrestore to reload the config.

Then the magic happened and I think it was due to an extra flag.

Trying to mount the LV provided the same result, "Structure Needs Cleaning"
xfs_check said that we needed to run xfs_repair with a -L

For some reason it mounted, I'm guessing because it bypassed the log tables that were not synced and allowed for a read-only copy. I now have access to 36TB of video files and my next deliema is how to get the data out of the system. I'm currently looking at external 10TB HDDs but then I have to split the data some how. I'm also looking at consumer grade NAS that would take like x6 HDD so that I could create a single large XFS to copy the data directly over. Any thoughts would be greatly appreciated as I have never done a recovery that didn't fix on a single external device.

First thing would be to have the company decide what they are going to use as a backup solution, then recover the files directly to that. Given the current scare, I suspect that having backups would now be a fairly high-priority item. Note that failure of any one of the 4 drives would make recovery of the filesystem impossible, and they would be reduced to using tools like photorec to dig files and file fragments out of what was left. Doing that for a ~97TB filesystem and figuring out what the original file names might have been would take months, if not years.

So you only need slightly more than 36T - hence my first question. Get the data copied - use the 10T drives if you have to. Copy it a file at a time -data are just data. Worry about getting it back together later, when you have repaired the filesystem. Get a list of the entire filesystem tree as well, along with ownership/permissions.