mdadm: RAID5 Array crashed on reshape (shrink) due to file system error

Konstantin92 · 10-16-2021, 07:11 AM

Hello there!

Unfortunately I messed up things when I attempted to shrink my RAID5 array. Originally this array had 5 x 16TB drives, which I wanted to downscale to a 4 x 16TB array. Therefore, I followed this doc: https://documentation.suse.com/sles/...id-resize.html
Dump as shit, I did not adjust the file-system before using mdadm, which broke the file-system apparently. The reshape process is stuck at 0% and the array is not mountable.

I tried some things like testdisk (which tells me, that "The harddisk (48 TB / 43 TiB) seems too small! (< 64 TB / 58 TiB)"

Code:

dmesg

said:

Code:

EXT4-fs (md1): bad geometry: block count 15627725312 exceeds size of device (11720793984 blocks)

So: Is there a way to revert the reshape process? (

Code:

--update=revert-reshape

does not work)
(linked to that

Is there a way to extend the number of devices again? (so that the block count matches the device size)
Or is there a way to modify the file-system, so that reshape could continue?
Or is there a way to recover some of the data files?

Any suggestion is highly appreciated! Also, if you need more information, please let me know.

wpeckham · 10-16-2021, 07:59 AM

I would go to my most recent backup, clean up the devices and create a new array at the desired size, and then restore form backup.

Konstantin92 · 10-16-2021, 08:26 AM

Thank you for your reply!! I have a backup from June, but unfortunately this is missing some important data. Therefore it is critical for me to restore this data.
If nothing works, I will consider this solution. But as long as there is a glimmer of hope, I would like to work on a rescue.

Meanwhile I was able to use the command

mdadm --create /dev/md1 --level=5 --raid-devices=5 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sde1 /dev/md0 --assume-clean --readonly

to re-create the array. Nevertheless, it is still not mountable, since the filesystem is still broken.
Is there any way to restore the file-system?
I tried to use testdisk, which detects at GTP file-system (as it should), but could not detect a proper partition so far.

syg00 · 10-16-2021, 06:11 PM

Short answer - NO.

Quote:

Originally Posted by Linux_RAID_wiki

RAID is very good at protecting your data. In fact, NEARLY ALL data lost as reported to the raid mailing list, is down to user error while attempting to recover a failed array.

In particular NEVER NEVER NEVER use "mdadm --create" on an already-existing array unless you are being guided by an expert. It is the single most effective way of turning a simple recovery exercise into a major forensic problem - it may not be quite as effective as "dd if=/dev/random of=/dev/sda", but it's pretty close ...

That's from the home page. Your prospects were pretty grim before that, but doing a scrape of all that data and hoping to get any/all the data since the last backup looks pretty forlorn.

Konstantin92 · 10-18-2021, 07:04 AM

Thank you for your reply. Since this case seems pretty hopeless, I'll try to seek for professional assistance by a company or give up the data then...

wpeckham · 10-18-2021, 10:26 AM

Three lessons to take away from this:
1. If it is important enough to restore, it is important enough to have in a backup (or two, or three).
2. Never mess with your storage before backing up your data.
3. RAID IS NOT BACKUP! It is added availability or performance (or both), but not data security.