Filesystem corrupt after VM lost it's VLAN

Mavman · 02-16-2020, 09:24 PM

I have a VMware VM running CentOS 5.11 (I know), where the ESXi host lost the VLAN for storage traffic, and now the VM dies on boot -

Code:

EXT3-fs error (device dm-): ext3_lookup: unlinked inode 2424833 in dir #2
EXT3-fs error (device dm-): ext3_lookup: unlinked inode 2785281 in dir #2
exec of init (/sbin/init) failed!!!: No such file or directory
Kernel panic - not syncing: Attempted to kill init!

This VM has one drive with two partitions. I grabbed the 5.11 CentOS rescue CD & booted into 'linux rescue' mode. fsck reports sda1 as /boot and clean, and I can mount it without issue. The second partition is an LVM with a few volumes, but is unable to detect anything on sda2-

Code:

fsck 1.38 (30-Jun-2005)
fsck: WARNING: couldn't open /etc/fstab: No such file or directory
e2fsck 1.38 (30-Jun-2005)
Couldn't find ext2 superblock, trying backup blocks...
fsck.ext2: while trying to open /dev/sda2

The superblock could not be read or does not describe a correct ext2 
filesystem. If the device is valid and it really contains an ext2 
filesystem (and not swap or ufs or something else), then the superblock 
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

fsck.ext2:

So I grabbed the list of superblocks with -

Code:

mke2fs -n /dev/sda2

Then for each superblock I tried running -

Code:

e2fsck -b <SuperBlock> /dev/sda2

And every time it errored with -

Code:

e2fsck 1.38 (30-Jun-2005)
e2fsck: while trying to open /dev/sda2

The superblock could not be read or does not describe a correct ext2 
filesystem. If the device is valid and it really contains an ext2 
filesystem (and not swap or ufs or something else), then the superblock 
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

e2fsck:

So...Is it lost? Thanks for your time in advance.

syg00 · 02-16-2020, 11:15 PM

/dev/sda2 is apparently your LVM pv - that means it is a (logical) device; it does not directly contain a filesystem, so it can't be fsck'd. You will have a selection of lv's within that pv, and these each do contain a filesystem. The CentOS liveCD should find and mount them all if possible at boot. Let's see what this returns - might need sudo.

Code:

lsblk -f

Mavman · 02-16-2020, 11:35 PM

Thanks for your response. Looks like lsblk wasn't available until CentOS/RHEL 6, was getting command not found. I mounted a CentOS 7 ISO I had readily available and ran it that way -

Code:

NAME	FSTYPE		LABEL	UUID					MOUNTPOINT
sda
-sda1	ext3		/boot	6df2b0d3-ef59-4af4-96fa-2e8add09bb74
-sda2	LVM2_member		0RvI3g-S8FU-KY9f-j7LO-gUIr-eg3Y-pv8h2b

syg00 · 02-17-2020, 04:19 PM

Hmmm - no lv's. What does lvscan return ?.

Mavman · 02-17-2020, 11:25 PM

Again, thank you for your time.

Code:

# lvscan
  inactive		'/dev/VolGroup00/LogVol00' [<17.88 GiB] inherit
  inactive		'/dev/VolGroup00/LogVol01' [2.00 GiB] inherit

syg00 · 02-18-2020, 03:07 AM

So at least they are recognisable. Before you start, note my sigline - get a backup of the current state of affairs.
I would simply activate the lvs, and fsck them. If you're lucky good things might happen and once they are mounted you can check them and hope the contents are valid.

FWIW, I rarely trust a filesystem after a major "event" - I reformat and restore from most recent valid backup. Given that we have progressed this far I'm presuming you don't have such.

Mavman · 02-18-2020, 10:40 AM

I do not have a good backup, no. Unfortunately I thought it had been backing up daily, found a set of VMs that weren't. I had snapshotted the machine in a working but broken state. You can't SSH/console into it because it throws inode errors, it doesn't even let you try. But the services on the box are still responding. So I had cloned that machine into a new one and snapshotted it's basic broken state and have been working on that cloned box and presented this thread as such for simplicity. But I can at least reset it to the basic broken image if need be.

Mavman · 02-18-2020, 06:18 PM

So, before I continue, a heads up that I've found a separate backup and so I'll be able to fully recover and have started going that route.

For the sake of this thread and continued learning, I'd like to see if it's still possible to fix this VM. Continuing with what you had said to try.

I ran -

Code:

# lvm vgscan
  Reading volume groups from cache.
  Found volume group "VolGroup00" using metadata type lvm2
# vchange -a y
  2 logical volume(s) in volume group "VolGroup00" now active

Then I mounted /dev/VolGroup00/LogVol00 to /tmp, and did an ls, which showed a few directories with '?' in place of values, including for the /etc directory. So I unmounted it and ran 'fsck /dev/VolGroup00/LogVol00 -y' against it. When that was done, /etc was no longer even listed. LogVol01 is swap. I'm going to revert to snapshot and prod at it some more, any other thoughts or does this look like the original was thoroughly boned?

jefro · 02-18-2020, 07:09 PM

testdisk may recover some stuff but the idea that vlan caused this is kinda low on my guessometer.

Some other issue happened.

Mavman · 02-18-2020, 10:48 PM

I'm sorry, but it was definitely the vlan. The event was correlated to user/time on top of them coming forward about it. You can't just drop the vlan passing storage traffic, it's similar to just pulling out SSD from a normal powered on machine- bad things can happen.

I haven't tried testdisk yet though, will let you know if that has any luck.