LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 02-16-2020, 10:24 PM   #1
Mavman
Member
 
Registered: Mar 2008
Distribution: Debian
Posts: 44

Rep: Reputation: 1
Filesystem corrupt after VM lost it's VLAN


I have a VMware VM running CentOS 5.11 (I know), where the ESXi host lost the VLAN for storage traffic, and now the VM dies on boot -
Code:
EXT3-fs error (device dm-): ext3_lookup: unlinked inode 2424833 in dir #2
EXT3-fs error (device dm-): ext3_lookup: unlinked inode 2785281 in dir #2
exec of init (/sbin/init) failed!!!: No such file or directory
Kernel panic - not syncing: Attempted to kill init!
This VM has one drive with two partitions. I grabbed the 5.11 CentOS rescue CD & booted into 'linux rescue' mode. fsck reports sda1 as /boot and clean, and I can mount it without issue. The second partition is an LVM with a few volumes, but is unable to detect anything on sda2-
Code:
fsck 1.38 (30-Jun-2005)
fsck: WARNING: couldn't open /etc/fstab: No such file or directory
e2fsck 1.38 (30-Jun-2005)
Couldn't find ext2 superblock, trying backup blocks...
fsck.ext2: while trying to open /dev/sda2

The superblock could not be read or does not describe a correct ext2 
filesystem. If the device is valid and it really contains an ext2 
filesystem (and not swap or ufs or something else), then the superblock 
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

fsck.ext2:
So I grabbed the list of superblocks with -
Code:
mke2fs -n /dev/sda2
Then for each superblock I tried running -
Code:
e2fsck -b <SuperBlock> /dev/sda2
And every time it errored with -
Code:
e2fsck 1.38 (30-Jun-2005)
e2fsck: while trying to open /dev/sda2

The superblock could not be read or does not describe a correct ext2 
filesystem. If the device is valid and it really contains an ext2 
filesystem (and not swap or ufs or something else), then the superblock 
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

e2fsck:
So...Is it lost? Thanks for your time in advance.
 
Old 02-17-2020, 12:15 AM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 19,397

Rep: Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439
/dev/sda2 is apparently your LVM pv - that means it is a (logical) device; it does not directly contain a filesystem, so it can't be fsck'd. You will have a selection of lv's within that pv, and these each do contain a filesystem. The CentOS liveCD should find and mount them all if possible at boot. Let's see what this returns - might need sudo.
Code:
lsblk -f
 
Old 02-17-2020, 12:35 AM   #3
Mavman
Member
 
Registered: Mar 2008
Distribution: Debian
Posts: 44

Original Poster
Rep: Reputation: 1
Thanks for your response. Looks like lsblk wasn't available until CentOS/RHEL 6, was getting command not found. I mounted a CentOS 7 ISO I had readily available and ran it that way -
Code:
NAME	FSTYPE		LABEL	UUID					MOUNTPOINT
sda
-sda1	ext3		/boot	6df2b0d3-ef59-4af4-96fa-2e8add09bb74
-sda2	LVM2_member		0RvI3g-S8FU-KY9f-j7LO-gUIr-eg3Y-pv8h2b
 
Old 02-17-2020, 05:19 PM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 19,397

Rep: Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439
Hmmm - no lv's. What does lvscan return ?.
 
Old 02-18-2020, 12:25 AM   #5
Mavman
Member
 
Registered: Mar 2008
Distribution: Debian
Posts: 44

Original Poster
Rep: Reputation: 1
Again, thank you for your time.

Code:
# lvscan
  inactive		'/dev/VolGroup00/LogVol00' [<17.88 GiB] inherit
  inactive		'/dev/VolGroup00/LogVol01' [2.00 GiB] inherit
 
Old 02-18-2020, 04:07 AM   #6
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 19,397

Rep: Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439Reputation: 3439
So at least they are recognisable. Before you start, note my sigline - get a backup of the current state of affairs.
I would simply activate the lvs, and fsck them. If you're lucky good things might happen and once they are mounted you can check them and hope the contents are valid.

FWIW, I rarely trust a filesystem after a major "event" - I reformat and restore from most recent valid backup. Given that we have progressed this far I'm presuming you don't have such.
 
Old 02-18-2020, 11:40 AM   #7
Mavman
Member
 
Registered: Mar 2008
Distribution: Debian
Posts: 44

Original Poster
Rep: Reputation: 1
I do not have a good backup, no. Unfortunately I thought it had been backing up daily, found a set of VMs that weren't. I had snapshotted the machine in a working but broken state. You can't SSH/console into it because it throws inode errors, it doesn't even let you try. But the services on the box are still responding. So I had cloned that machine into a new one and snapshotted it's basic broken state and have been working on that cloned box and presented this thread as such for simplicity. But I can at least reset it to the basic broken image if need be.
 
Old 02-18-2020, 07:18 PM   #8
Mavman
Member
 
Registered: Mar 2008
Distribution: Debian
Posts: 44

Original Poster
Rep: Reputation: 1
So, before I continue, a heads up that I've found a separate backup and so I'll be able to fully recover and have started going that route.

For the sake of this thread and continued learning, I'd like to see if it's still possible to fix this VM. Continuing with what you had said to try.

I ran -
Code:
# lvm vgscan
  Reading volume groups from cache.
  Found volume group "VolGroup00" using metadata type lvm2
# vchange -a y
  2 logical volume(s) in volume group "VolGroup00" now active
Then I mounted /dev/VolGroup00/LogVol00 to /tmp, and did an ls, which showed a few directories with '?' in place of values, including for the /etc directory. So I unmounted it and ran 'fsck /dev/VolGroup00/LogVol00 -y' against it. When that was done, /etc was no longer even listed. LogVol01 is swap. I'm going to revert to snapshot and prod at it some more, any other thoughts or does this look like the original was thoroughly boned?
 
Old 02-18-2020, 08:09 PM   #9
jefro
Moderator
 
Registered: Mar 2008
Posts: 20,464

Rep: Reputation: 3269Reputation: 3269Reputation: 3269Reputation: 3269Reputation: 3269Reputation: 3269Reputation: 3269Reputation: 3269Reputation: 3269Reputation: 3269Reputation: 3269
testdisk may recover some stuff but the idea that vlan caused this is kinda low on my guessometer.

Some other issue happened.
 
Old 02-18-2020, 11:48 PM   #10
Mavman
Member
 
Registered: Mar 2008
Distribution: Debian
Posts: 44

Original Poster
Rep: Reputation: 1
I'm sorry, but it was definitely the vlan. The event was correlated to user/time on top of them coming forward about it. You can't just drop the vlan passing storage traffic, it's similar to just pulling out SSD from a normal powered on machine- bad things can happen.

I haven't tried testdisk yet though, will let you know if that has any luck.

Last edited by Mavman; 02-19-2020 at 12:06 AM.
 
  


Reply

Tags
centos5, filesystem, virtual machine


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Creating a VLAN interface on top of another VLAN interface jbbroccard2 Linux - Networking 1 05-22-2013 02:25 AM
Route non-vlan packet to a vlan interface mic.sed Linux - Networking 2 04-23-2010 03:39 AM
VLan help on Cisco 870 to Linux vlan spide21 Linux - Networking 4 07-30-2009 09:20 AM
DHCP Config for VLAN's using 1NIC and non VLAN router. scottgutman Linux - Networking 1 07-22-2009 02:41 AM
VLAN configuration - native VLAN and setting PVID kumarwaiting Linux - Networking 0 07-24-2006 03:51 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 02:58 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration