Corrupted LVM2

First I will start with the obligatory I am new and suck at Linux comment…

Okay now that we have that out of the way.

I have a test db server that has a LVM2 volume that was filled to capacity and corrupted. It would boot up and try to scan the volume but would hang forever. So I booted with a rescue cd and told the server to fast boot skipping the check. That worked and I was able to get into the server.

I ran a e2fsck –c /dev… and it ran for like 3 days (I was told by a Linux friend to let it run as long as it took) but after it was done I could no longer boot the server.

I booted back into the rescue cd and made a cifs mount to my windows box and copied the data off the lvm2 volume to my local hard drive.

I am told that I can now format the volume, add capacity and copy the data back. Then I should be all set. Is that true? Is there any other way to recover the volume as is? Without blowing anything away?

Thank you very much!

Originally Posted by InvertMe First I will start with the obligatory I am new and suck at Linux comment…

Welcome to LQ.

Originally Posted by InvertMe Is there any other way to recover the volume as is? Without blowing anything away?

Running fsck on a severely mutilated file system may result in a large portion of its contents being written to /lost+found by their inode number. If you run a Linux distribution with a package management that stores hashes of the contents of packages or have a backup you can check contents against then you could go that route but it requires effort (you have to know a bit about the shell and default tools), there's no guarantee (fsck dumping incomplete files), it's inefficient (if you have a backup then why not restore it?) and it's time consuming.

Originally Posted by InvertMe I am told that I can now format the volume, add capacity and copy the data back. Then I should be all set. Is that true?

No because you'll want to implement a backup strategy and resource monitoring before doing that, otherwise: yes.

We sadly don't have a back up, I am new to this company so I had nothing to do with this build/backup strategy. I think their thought process was "it's test, we don't need to back it up". I spend a week trying to get the server back in shape and have given up. I built a new server on our vm cluster and that is backed up. I had to buy more disks, ram, and an extra blade but in the end it's worth it to not have this happen again.

I'm sure I will have TONS of questions as time goes on though. I am sure I will frequent this site a lot.

The key thing is how was it corrupted. A full filesystem shouldnt become corrupted, it might make things hang but usually you are ok for a little bit as 5% of the space is reserved for root by default.

Was it corruption of the block device/raid array, physical volume, volume group, logical volume or the filesystem itself?

If it wasn't the filesystem and you just format it, that corruption could still be there and come back to haunt you later on.