Can't Fix. massive ext3 file system corruption "directory entry across blocks"

legaultp1 · 10-06-2009, 07:00 AM

hello
here is some system specs.
I have a hardware raid controller providing me 12TB redundant disk space
16 disk@1TB per disk (adaptec SAS 31610 PCI-E)
raid 6 with 2 hot spare.

i have EFI support in the kernel with large disk support.
i am using gentoo linux X86_64 with 8GB of ram.

so, a hardware faillure is out of the question.
here is my main problem.

I used GPT label on the disk.
i created a 8TB raid volume.
on linux, it show's a 8TB disk.

created first 2TB GPT partition as EXT3. this one still works fine.
created 8 more GPT partitions as EXT3.

i used parted (GNU parted) 1.8.8

i was able to copy/read files from all the partitions until i restarted the system a few hours later.

after that, i am screwed-up on any access on partition 2 or more.
i get the different errors.

----------------
find /volume give me this error.

Code:

[11717.472272] EXT3-fs error (device sde2): htree_dirblock_to_tree: bad entry in directory #488289: directory entry across blocks - offset=0, inode=2112176422, rec_len=26760, name_len=241
[11717.712914] EXT3-fs error (device sde2): htree_dirblock_to_tree: bad entry in directory #455137: rec_len % 4 != 0 - offset=0, inode=3646194470, rec_len=7377, name_len=76
[11717.800651] EXT3-fs error (device sde2): htree_dirblock_to_tree: bad entry in directory #75905: rec_len % 4 != 0 - offset=0, inode=2474015448, rec_len=11753, name_len=237
[11718.006873] EXT3-fs error (device sde2): htree_dirblock_to_tree: bad entry in directory #313793: directory entry across blocks - offset=0, inode=3062395205, rec_len=19872, name_len=222

cp file1 /tmp/file2

Code:

[14154.650549] attempt to access beyond end of device
[14154.650555] sde2: rw=0, want=26632867872, limit=4095996660
[14154.650558] attempt to access beyond end of device
[14154.650560] sde2: rw=0, want=6323548784, limit=4095996660
[14154.650561] attempt to access beyond end of device
[14154.650563] sde2: rw=0, want=32894712208, limit=4095996660
[14154.651490] attempt to access beyond end of device
[14154.651495] sde2: rw=0, want=7104498096, limit=4095996660
[14154.651498] attempt to access beyond end of device
[14154.651500] sde2: rw=0, want=4737273160, limit=4095996660
[14154.651503] attempt to access beyond end of device
[14154.651504] sde2: rw=0, want=16362933096, limit=4095996660

-------------------
fsck /dev/sde2

Code:

Superblock has an invalid ext3 journal (inode 8).
Clear? no

fsck.ext3: Illegal inode number while checking ext3 journal for volume1

if i fix the file system, all the files are being deleted by fsck.ext3.

parted see and created the file system this way:

Code:

Disk /dev/sde - 8172 GB / 7611 GiB - CHS 993632 255 63

     Partition                  Start        End    Size in sectors
 1 P MS Data                       34 4095996659 4095996626 
 2 P MS Data               4095996660 8191993319 4095996660 
 3 P MS Data               8191993320 10239991649 2047998330
 4 P MS Data               10239991650 11263990814 1023999165
 5 P MS Data               11263990815 12287989979 1023999165
 6 P MS Data               12287989980 12902395904  614405925
 7 P MS Data               12902395905 13516801829  614405925
 8 P MS Data               13516801830 14540800994 1023999165
 9 P MS Data               14540800995 15962698079 1421897085

testdisk see the partition like this:

Code:

TestDisk 6.11, Data Recovery Utility, April 2009
Christophe GRENIER <grenier@cgsecurity.org>
http://www.cgsecurity.org

Disk /dev/sde - 8172 GB / 7611 GiB - CHS 993632 255 63

     Partition                  Start        End    Size in sectors

 1 P MS Data                       34 4095996657 4095996624
 2 P MS Data               4095996660 8191993315 4095996656
 3 P MS Data               8191993318 10239991645 2047998328
 4 P MS Data               10239991650 11263990809 1023999160
 5 P MS Data               11263990815 12287989974 1023999160
 6 P MS Data               12287989978 12902395897  614405920
 7 P MS Data               12902395905 13516801824  614405920
 8 P MS Data               13516801830 14540800989 1023999160
 9 P MS Data               14540800995 15962698074 1421897080

there appear to have a 2 sector differential between testdisk and parted. could this cause my problem and if so, how do i fix it?

is there a way to recover the data as i moved it instead of copied it and have no backup of it.
i have a 2nd volume of 2TB i can use to do tests.i can dd the partition on the 2nd disk array and try to recover it so i do not touch the original corrupted disk but work instead on a copy.

ramram29 · 10-07-2009, 10:09 PM

You can try to run foremost on the dd copy of each partition. It will recover most common files. But it is a lot of work afterward, sorting all the files. Did you backup your data before you did the move?

legaultp1 · 10-07-2009, 11:14 PM

hello ramram

i copied the data from a 2nd array to the new one.
open a few files then, once everything was working,
I reassigned the first array to a different task and moved my vmware images to it.
then, had to reboot and since then, it all went bad.

sad thing is that it did work for a while until i restart.

i will try the application you mentioned.
the other odd thing is that the first 2TB of that array is fine while anything 2TB+ from the start of the disk is bad.

Also, since i no longer have the backup of it. kinda hard to do a backup when you do not have the space since i am not a company here.

Thanks for the help. will try it.

legaultp1 · 10-07-2009, 11:56 PM

Hi RamRam
Ok tried it. was creating me chunk of files from 4K to 5MB. nothing close to the average file size i had
random content inside the files
I will just forget the idea to rescue my file system.
it was stuff i accumulated over the years so, it is not this critical.

Thanks for the help.