Filesystem wont mount after resize operation.

Eusaphious · 12-16-2015, 03:28 PM

So, I was attempting to resize a large filesystem from 36T -> 50T. The raid expansion completed successfully. Extending the lvm partition went without error and running the fsck and the resize2fs commands completed without error.

Attempting to mount the filesystem fails with the following entry in dmseg output:

[4109179.786298] EXT4-fs (dm-0): no journal found

Code:

nas ~ # mount /dev/vg1/videos /mnt/videos -v
mount: wrong fs type, bad option, bad superblock on /dev/mapper/vg1-videos,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
nas ~ #
nas ~ # fsck.ext4 /dev/vg1/videos
e2fsck 1.42.12 (29-Aug-2014)
ext2fs_check_desc: Corrupt group descriptor: bad block for inode table
fsck.ext4: Group descriptors look bad... trying backup blocks...
Superblock has an invalid journal (inode 8).
Clear<y>? no
fsck.ext4: Illegal inode number while checking ext3 journal for videos

videos: ***** FILE SYSTEM WAS MODIFIED *****

videos: ********** WARNING: Filesystem still has errors **********

Any advice on next steps (without loosing data)? Should I allow the fsck to clear the block and continue the check?

Possible to roll back the resize operation so I can mount the filesystem?

Executing resize again shows nothing to do message:

Code:

nas ~ # resize2fs /dev/vg1/videos -p
resize2fs 1.42.12 (29-Aug-2014)
The filesystem is already 13670806528 (4k) blocks long.  Nothing to do!

dmesg output entries from detecting disk capacity.

Code:

[sdd] 78118891520 512-byte logical blocks: (39.9 TB/36.3 TiB)
[sdd] 78118912000 512-byte logical blocks: (39.9 TB/36.3 TiB)
[sdd] 109366456320 512-byte logical blocks: (55.9 TB/50.9 TiB)

Thanks for any input.

Eusaphious · 12-16-2015, 03:49 PM

Mounting without journal gives this error:

Code:

nas ~ # mount -o ro,noexec,noload /dev/vg1/videos /mnt/videos/
mount: mount /dev/mapper/vg1-videos on /mnt/videos failed: Stale file handle

smallpond · 12-16-2015, 04:18 PM

Did you do the resize with the filesystem mounted or unmounted? Are there any disk errors in your logs? Can you post the output of:

Code:

tune2fs -l /dev/vg1/videos

Note: The basic rule in recovering data is sit on your hands until you are sure you need to type a command.

Eusaphious · 12-16-2015, 04:20 PM

The filesystem was resized unmounted. Here is the output of tune2fs and dumpe2fs:

Code:

nas ~ # dumpe2fs /dev/vg1/videos 
dumpe2fs 1.42.12 (29-Aug-2014)
Filesystem volume name:   videos
Last mounted on:          /mnt/videos
Filesystem UUID:          016b6b54-7e76-415c-9d77-f30f486263fb
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr dir_index filetype extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              854425600
Block count:              13670806528
Reserved block count:     0
Free blocks:              4432041941
Free inodes:              854417552
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         2048
Inode blocks per group:   128
RAID stride:              32752
Flex block group size:    16
Filesystem created:       Sat Aug 16 01:39:52 2014
Last mount time:          Fri Oct 30 06:32:04 2015
Last write time:          Wed Dec 16 20:43:38 2015
Mount count:              0
Maximum mount count:      -1
Last checked:             Wed Dec 16 20:07:08 2015
Check interval:           0 (<none>)
Lifetime writes:          33 TB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      b23488ea-85c7-4739-9c2b-d260c7b0bfd6
Journal backup:           inode blocks
Journal superblock magic number invalid!
nas ~ # 
nas ~ # 
nas ~ # tune2fs -l /dev/vg1/videos
tune2fs 1.42.12 (29-Aug-2014)
Filesystem volume name:   videos
Last mounted on:          /mnt/videos
Filesystem UUID:          016b6b54-7e76-415c-9d77-f30f486263fb
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr dir_index filetype extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              854425600
Block count:              13670806528
Reserved block count:     0
Free blocks:              4432041941
Free inodes:              854417552
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         2048
Inode blocks per group:   128
RAID stride:              32752
Flex block group size:    16
Filesystem created:       Sat Aug 16 01:39:52 2014
Last mount time:          Fri Oct 30 06:32:04 2015
Last write time:          Wed Dec 16 20:43:38 2015
Mount count:              0
Maximum mount count:      -1
Last checked:             Wed Dec 16 20:07:08 2015
Check interval:           0 (<none>)
Lifetime writes:          33 TB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      b23488ea-85c7-4739-9c2b-d260c7b0bfd6
Journal backup:           inode blocks

Eusaphious · 12-16-2015, 04:25 PM

Should I just run the fsck and pray? haha

smallpond · 12-16-2015, 05:06 PM

Superblock looks like the resize succeeded. You can either run fsck -y and open a beverage to drown your sorrows, or if you are really brave start digging in with debugfs. I don't suppose you have a spare 50TB around to back this up on?

Eusaphious · 12-16-2015, 05:39 PM

Quote:

Originally Posted by smallpond

Superblock looks like the resize succeeded. You can either run fsck -y and open a beverage to drown your sorrows, or if you are really brave start digging in with debugfs. I don't suppose you have a spare 50TB around to back this up on?

Only 30T... so cant exactly make a image of it, I could backup the data though If I could access it. debugfs doesn't show any files.

I kicked off the fsck on it. It wiped the journal. At this point I will probably end up with all my files in lost&found with "random" names assigned.

I did this process once before when I expanded my RAID and had zero issues.

Eusaphious · 12-16-2015, 07:40 PM

The fsck finished and put everything in lost&found as suspected. It looks like it only renamed the top level directories though so that shouldn't be too difficult to fix.

syg00 · 12-16-2015, 11:26 PM

I don't want to rain on your party, but fsck fixes the sanity of the filesystem - not necessarily the files stored within.
Which is why you get "fragments" stored in lost+found.

I'm sure Ted and his mates try hard to keep everything as valid files, but it sure isn't guaranteed.

Did you happen to fill the filesystem before you embarked on this resizing ?.