Hello, I have a weird puzzle which I don't understand. Here is a brief summary of my problem:

Synology-provided e2fsck is old (from 2012) and a bug in the 2012 version (fixed in 2014) prevents repair of my filesystem. However, when I compile e2fsck myself, the filesystem scan does not even begin due to a superblock problem. I am confused because the Synology-provided version has no problem with the superblock, so why would my compiled version work differently?

Here are some additional (more involved) details... I have a Synology NAS box, and uname -a displays:

Linux DS1812 3.10.105 #23824 SMP Tue Dec 25 18:26:34 CST 2018 x86_64 GNU/Linux synology_cedarview_1812+

The filesystem (ext4 on a Synology raid volume at /dev/vg1/volume_1) has errors after a power failure a while back. I can still boot and mount and access files, but there is definitely one folder which is corrupted, so I am eager to try to repair the filesystem using e2fsck / fsck.ext4.

I tried running e2fsck first in read-only mode (-n) to get a preview of problems and how long my system would need to be inaccessible when I do the real fsck repair. The read-only scan took nearly two weeks (!), which is weird.

I tried to do a real repair (after unmounting, etc), but it was getting stuck after 24 hours with very slow "clone_file_block: internal error: can't find dup_blk" errors when trying to fix multiply-claimed blocks during Pass 1D of the e2fsck scan.

I did a web search about dup_blk errors, and it looks like it was a bug which was fixed in 2014/2015: https://bugs.launchpad.net/ubuntu/+s...s/+bug/1321418

Synology's version of e2fsck is 1.42.6 (21-Sep-2012): affected by dup_blk bug
The bug was fixed in 1.42.12 (Aug 2014)
The current version is 1.44.5 (Dec 2018)

Since the bug was actually fixed by the e2fsprogs maintainers, it seems logical for me to get a more up-to-date version of e2fsck. I filed a ticket with Synology support, hoping they would update their official e2fsck, but if that happens, it will take quite a while to go through their approval/testing process. One suggestion was that I backup my system, wipe it, and start over and restore from backup. Another suggestion was to move all of the disks (six disks in the raid volume) to an empty computer with a Ubuntu Live CD, and run a more recent e2fsck provided by Ubuntu from there. This is a reasonable idea, but I don't have a spare computer with six empty drive bays. So I thought I could try to compile e2fsck myself (and I was able to), but this is where the puzzle starts....

When I compile e2fsck from source (./configure; make; make check), all tests pass in make check. But when I try to run the program on my filesystem, I get errors about: "Corruption found in superblock. (reserved_gdt_blocks = 6791)" and the fsck scan doesn't even start the first pass.

I read up on this and tried to use other backup superblocks, i.e. fsck.ext4 -n -C 0 -b 32768 /dev/vg1/volume_1 , but all attempts failed with the same superblock corruption message.

Normally, if you have a corrupted superblock, I think that means your system is unbootable unless you repair the superblock from a backup. However, in my case, the old e2fsck 1.42.6 provided in the Synology install does NOT report a superblock problem. It starts the scan, gets through a few passes, and then starts to slow down dramatically with the dup_blk errors described above. Also, I am able to boot, mount my filesystem, and open files on it, so wouldn't that be inconsistent with a superblock problem?

I even compiled an old version (1.42.6) of e2fsck matching the one on the Synology. The version I compiled myself also fails with the superblock problem.

So my confusion is: Synology's version of e2fsck is outdated and has a bug which prevents me from fixing my filesystem, but any version (including the same version that Synology has) of e2fsck which I compile myself reports a bad superblock. Why would this be?

Have you run SMART checks on all the disks to see that one or more of them aren't failing?

Very unlikely any of us have the knowledge to answer - open a bug against e2fsprogs and see if the devs can suggest a way forward. After checking the disk ...

It appears you have no problems running multiple versions of e2fsck without backing up anything, what's stopping you from trying e2fsck from the Ubuntu live CD?

It should be mentioned that fsck is for fixing filesystems, not necessarily looking after files ensconced.
After major events like this, I habitually just reformat and restore - no questions asked. And no, RAID has never been a backup replacement.

I would assume later versions of any software tighten up enforcement of "the rules" - often things initially "just work" even if not documented, or even not supported. Later on, these "features" disappear. Maybe something similar has occurred with ext4. All supposition of course.

Thanks for the replies!

@hydrurga: Yes, SMART checks are run automatically (short tests every day, extended tests every week) and the status of all disks is good.

@Brains: Since this is a Synology NAS unit, I don't know how to boot from a live CD. Unlike with a regular computer, there doesn't appear to be a way to set the boot drive on the NAS (for example by pressing a hotkey like F2 or by accessing BIOS). I did try copying a recent version of e2fsck from an Ubuntu distribution onto the Synology, and when I ran it, it failed with the same superblock error.

@syg00: I thought about contacting the e2fsprogs list, but it is a little intimidating. Their mailing list looks very technical and I barely understand any of it. But I may try that route.

Thanks also for the interesting point about tightening enforcement of the rules in later versions. That does make sense.

However, what I don't understand is that I also recompiled the same version of the software (1.42.6 from 2012). So the rules enforced for that version should be the same.

# SYNOLOGY VERSION
sh-4.3# /sbin/e2fsck -V

e2fsck 1.42.6 (21-Sep-2012)
Using EXT2FS Library version 1.42.6, 21-Sep-2012

# SELF-COMPILED VERSION
sh-4.3# /tmp/e2fsprogs/e2fsprogs-1.42.6/build/e2fsck/e2fsck -V

e2fsck 1.42.6 (21-Sep-2012)
Using EXT2FS Library version 1.42.6, 21-Sep-2012

The only thing I can imagine is something due to the kernel version it was compiled for as reported by file:

# SYNOLOGY VERSION
sh-4.3# /opt/bin/file /sbin/e2fsck

/sbin/e2fsck: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, stripped

# SELF-COMPILED VERSION
sh-4.3# /opt/bin/file /tmp/e2fsprogs/e2fsprogs-1.42.6/build/e2fsck/e2fsck

/tmp/e2fsprogs/e2fsprogs-1.42.6/build/e2fsck/e2fsck: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /opt/lib/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, with debug_info, not stripped

or the shared libraries it is linked to as reported by ldd:

# SYNOLOGY VERSION
sh-4.3# /opt/bin/ldd /sbin/e2fsck

linux-vdso.so.1 (0x00007ffc1bd66000)
libext2fs.so.2 => /usr/lib/libext2fs.so.2 (0x00007f0cd3d79000)
libcom_err.so.2 => /usr/lib/libcom_err.so.2 (0x00007f0cd3b75000)
libblkid.so.1 => /usr/lib/libblkid.so.1 (0x00007f0cd3932000)
libuuid.so.1 => /usr/lib/libuuid.so.1 (0x00007f0cd372d000)
libe2p.so.2 => /usr/lib/libe2p.so.2 (0x00007f0cd3525000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f0cd3180000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f0cd2f62000)
/lib64/ld-linux-x86-64.so.2 => /opt/lib/ld-linux-x86-64.so.2 (0x00007f0cd3fe3000)

# SELF-COMPILED VERSION
sh-4.3# /opt/bin/ldd /tmp/e2fsprogs/e2fsprogs-1.42.6/build/e2fsck/e2fsck

linux-vdso.so.1 (0x00007ffff7bde000)
libpthread.so.0 => /opt/lib/libpthread.so.0 (0x00007f2b07db7000)
libgcc_s.so.1 => /opt/lib/libgcc_s.so.1 (0x00007f2b07d9e000)
libc.so.6 => /opt/lib/libc.so.6 (0x00007f2b07beb000)
/opt/lib/ld-linux-x86-64.so.2 (0x00007f2b07dd9000)

did you compile it on the NAS??? i didn't know they come with a full development environment...
unsurprisingly, since that version was compiled (not by you, by the distro maintainers) for a different system altogether.
i think Brains meant that you run fsck from your ubuntu install, on the network drives (and NOT from the NAS OS).

Here's a Quick Search (tm) for you.

@ondoho: Yes, I compiled it on the NAS. You're right, the Synology NAS doesn't come with a full development environment, but I installed gcc, make, etc from Entware (https://github.com/Entware/Entware-n...n-Synology-NAS) which installs into /opt/. That's why I am executing file, ldd, etc from /opt/bin/

I don't think it's possible to run fsck on a network drive. Especially if you are going to repair errors, which would require the drive to be accessible by the OS, but the drive would have to be unmounted. For the Ubuntu version, I would have to move all six drives into a spare computer with six free bays (which I don't have)

^ i see.
i guess you're right.
i guess "something went wrong" with the compilation.
If this entware is a project to bring more gnu/linux functionality to NAS devices, maybe you should check out their documentation.
oh, and the search link i provided. can't hurt to see how others tackled similar problems.

If you managed to get that compiled, running a few commands to get the info they may require will be well within your skills. I'm sure they will appreciate the efforts you've already made.