LinuxQuestions.org - EXT4-fs group descriptors corrupted

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - EXT4-fs group descriptors corrupted (https://www.linuxquestions.org/questions/linux-newbie-8/ext4-fs-group-descriptors-corrupted-4175548263/)

EXT4-fs group descriptors corrupted

Hello All,

My hard drive recently failed on my raid and got a new drive and got the server booting again, but now my NFS 2TB drive won't mount.Can you help please.

Thank you

Code:

 lvs

  LV      VG    Attr    LSize  Pool Origin Data%  Move Log Copy%  Convert

  lv_root vg00  -wi-ao-- 35.57g

  lv_swap vg00  -wi-ao--  3.94g

  lvdata  vgdata -wi-a---  2.00t

Code:

mount -t ext4 /dev/vgdata/lvdata /data/

mount: wrong fs type, bad option, bad superblock on /dev/mapper/vgdata-lvdata,

      missing codepage or helper program, or other error

      In some cases useful info is found in syslog - try

      dmesg | tail  or so





EXT4-fs (dm-2): group descriptors corrupted!

device-mapper: ioctl: device doesn't appear to be in the dev hash table.

EXT4-fs (dm-2): ext4_check_descriptors: Checksum for group 3968 failed (16234!=0)

EXT4-fs (dm-2): group descriptors corrupted!

Code:

 mke2fs -n /dev/sdb1

mke2fs 1.41.12 (17-May-2010)

/dev/sdb1 is apparently in use by the system; will not make a filesystem here!



lsof /dev/mapper/vgdata-lvdata

[root@nfs01 ~]#

Code:

 fsck /dev/vgdata/lvdata

fsck from util-linux-ng 2.17.2

e2fsck 1.41.12 (17-May-2010)

fsck.ext4: Group descriptors look bad... trying backup blocks...

fsck.ext4: Bad magic number in super-block when using the backup blocks

fsck.ext4: going back to original superblock

fsck.ext4: Device or resource busy while trying to open /dev/mapper/vgdata-lvdata

Filesystem mounted or opened exclusively by another program?





lsof /dev/mapper/vgdata-lvdata

 fuser /dev/mapper/vgdata-lvdata

[root@nfs01 ~]#

Code:

--- Volume group ---

  VG Name              vgdata

  System ID

  Format                lvm2

  Metadata Areas        1

  Metadata Sequence No  3

  VG Access            read/write

  VG Status            resizable

  MAX LV                0

  Cur LV                1

  Open LV              0

  Max PV                0

  Cur PV                1

  Act PV                1

  VG Size              2.00 TiB

  PE Size              4.00 MiB

  Total PE              524286

  Alloc PE / Size      524286 / 2.00 TiB

  Free  PE / Size      0 / 0

  VG UUID              sq4utn-JLOM-j0Ho-h2io-l8hY-JmGO-vjNCLu



  --- Logical volume ---

  LV Path                /dev/vgdata/lvdata

  LV Name                lvdata

  VG Name                vgdata

  LV UUID                ilqGXq-wjf9-lXaR-WTce-SEgJ-T8wb-4NaArS

  LV Write Access        read/write

  LV Creation host, time nfs01.domain.com, 2014-08-29 13:46:16 -0400

  LV Status              available

  # open                0

  LV Size                2.00 TiB

  Current LE            524286

  Segments              1

  Allocation            inherit

  Read ahead sectors    auto

  - currently set to    256

  Block device          253:2

Code:

lvm lvs -a -o+devices

  LV      VG    Attr    LSize  Pool Origin Data%  Move Log Copy%  Convert Devices

  lv_root vg00  -wi-ao-- 35.57g                                            /dev/sda2(0)

  lv_swap vg00  -wi-ao--  3.94g                                            /dev/sda2(9106)

  lvdata  vgdata -wi-a---  2.00t                                           /dev/sdb1(0)

Did you google the error message "device doesn't appear to be in the dev hash table"?

Are you sure that lvdata has an ext filesystem? Not xfs or something else?

Try running blkid; it tries to figure out what is on that volume.

Also, this:

Code:

mke2fs -n /dev/sdb1

If you want to find the superblocks on lvdata, run this command with lvdata, not sdb1.

Finally, to clarify, is lvdata located on your RAID? Are there other volumes on the RAID that have problems? What kind of RAID, hardware or software? Any messages referring to the RAID and lvdata in the message buffer (dmesg or /var/log/[syslog|messages])?

Thank you for your assistance.Yes, I did an extensive search in google.

Yes i am sure it is EXT4

Code:

dev/mapper/vgdata-lvdata: UUID="f4803c5a-8149-438b-a16e-fbd2a73c9ffb" TYPE="ext4

I have hardware RAID, there is no reference to the device in dmesg

Code:

XT4-fs (dm-2): ext4_check_descriptors: Checksum for group 3992 failed (18287!=0)

EXT4-fs (dm-2): ext4_check_descriptors: Checksum for group 3993 failed (43630!=0)

EXT4-fs (dm-2): ext4_check_descriptors: Checksum for group 3994 failed (56686!=0)

EXT4-fs (dm-2): ext4_check_descriptors: Checksum for group 3995 failed (12399!=0)

EXT4-fs (dm-2): ext4_check_descriptors: Checksum for group 3996 failed (13166!=0)

EXT4-fs (dm-2): ext4_check_descriptors: Checksum for group 3997 failed (56943!=0)

EXT4-fs (dm-2): ext4_check_descriptors: Checksum for group 3998 failed (43375!=0)

EXT4-fs (dm-2): ext4_check_descriptors: Checksum for group 3999 failed (17518!=0)

EXT4-fs (dm-2): ext4_check_descriptors: Inode bitmap for group 4000 not in group (block 536868864)!

EXT4-fs (dm-2): group descriptors corrupted!

root@nfs01 ~]# mke2fs -n /dev/mapper/vgdata-lvdata
m

Code:

ke2fs 1.41.12 (17-May-2010)

Filesystem label=

OS type: Linux

Block size=4096 (log=2)

Fragment size=4096 (log=2)

Stride=0 blocks, Stripe width=0 blocks

134217728 inodes, 536868864 blocks

26843443 blocks (5.00%) reserved for the super user

First data block=0

Maximum filesystem blocks=4294967296

16384 block groups

32768 blocks per group, 32768 fragments per group

8192 inodes per group

Superblock backups stored on blocks:

        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,

        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,

        102400000, 214990848, 512000000

I'm not sure what else to try, any other suggestions?

Quote:

Originally Posted by cbtshare (Post 5393382)

I have hardware RAID, there is no reference to the device in dmesg

You mention it in your original posting. Is it relevant to the problem with lvdata?

Code:

/dev/sdb1 is apparently in use by the system; will not make a filesystem here!

That's because it's a physical volume, I would think.

Code:

fsck /dev/vgdata/lvdata

(...)

fsck.ext4: Device or resource busy while trying to open /dev/mapper/vgdata-lvdata

Filesystem mounted or opened exclusively by another program?

It seems that fsck never attempts to repair anything; is this correct? Perhaps you are luckier when running fsck in single-user mode.
You can also try fsck -b <one of the backup superblocks displayed by mkfs>, but it seems that it does it by itself already, with little success.

Yes it is relevant because the system is on a raid array and on that raid I use vcenter to make a host then from that create a vm, and assign hard drives.There are two hard drives in the system, /dev/sda the root and boot partition and /dev/sdb the nfs storage.

You are correct, fsck, never attempt to repair anything it just throws an error.I did try in single user mode, but will try again after the drive has finished being imaged.

I tried running FSCK in single user mode now, and I am getting this error in the screen shot.

I just ran fsck again, and again it deletes my data.why is that? there is nothing in lost+found except 2 files that are 400mb in total.The size of data on there was 2.TB.There is nothing in the logs either

Quote:

Originally Posted by cbtshare (Post 5397350)

I guess nothing?

Sorry I am on leave and spend most of the time in the real world rather than the usual cyberspace :)

It's also hard to say what's going on. Why lvdata is "busy" although you are in single user mode is a mystery to me.
Regarding the data: fsck repairs filesystem structures, not data. It identifies inconsistencies like wrong summary counters (benign), files that have no directory entries (will be moved to lost+found) and data blocks that belong to no files but are not marked free either (will be marked free). File content is not considered. So you may say that fsck has removed your data, but what it has really done is marking unaccounted-for data blocks as free, as it is unable to guess what files they might belong to.

So after fsck has done its work, you may end up with an empty, but consistent filesystem.

Your data is probably still there. With the help of a filesystem debugger like debugfs you might be able to salvage something, but it would be very tedious.

My feeling is that something is wrong with the way you defined the physical and/or logical volume(s) on top of your RAID, but I would have to play around with the disks to get clues.