The Latest Kernel Release.

Andypoo · 10-12-2019, 07:05 PM

Quote:

Originally Posted by rogan

It seems as an issue present on kernels with support for many file systems compiled in...
I'll try a huge without f2fs tomorrow.

Yes, the issue occurs because F2FS code changed so instead of returning -EINVAL during its superblock failure, it now returns something else to indicate filesystem corruption.

This meant that the code in init/do_mounts.c (mount_block_root function) did not iterate beyond F2FS filesystem to check if it was actually a different filesystem anymore.

It has actually bothered me for some time to see the superblock probing failures for ocfs2 and F2FS (because they look scary and I worried about what would happen if another filesystem decided to do some sort of repair move), but until now, it hasn't caused a problem. Now, I am even more worried by that scenario.

Basically, if you also see messages like:

Code:

[    4.425052] (swapper/0,1,1):ocfs2_fill_super:1023 ERROR: superblock probe failed!
[    4.425782] (swapper/0,1,1):ocfs2_fill_super:1225 ERROR: status = -22
[    4.426285] F2FS-fs (sda1): Magic Mismatch, valid(0xf2f52010) - read(0x0)
[    4.426698] F2FS-fs (sda1): Can't find valid F2FS filesystem in 1th superblock
[    4.427485] F2FS-fs (sda1): Magic Mismatch, valid(0xf2f52010) - read(0xa366f000)
[    4.428229] F2FS-fs (sda1): Can't find valid F2FS filesystem in 2th superblock
[    4.428949] F2FS-fs (sda1): Magic Mismatch, valid(0xf2f52010) - read(0x0)
[    4.429368] F2FS-fs (sda1): Can't find valid F2FS filesystem in 1th superblock
[    4.430094] F2FS-fs (sda1): Magic Mismatch, valid(0xf2f52010) - read(0xa366f000)
[    4.430816] F2FS-fs (sda1): Can't find valid F2FS filesystem in 2th superblock
[    4.431642] BTRFS: device fsid 045d1c63-e943-4345-a0b9-777c8c4813cc devid 1 transid 996985 /dev/root

before your root filesystem is mounted, then the new kernels will lead to a kernel panic and unable to load your root filesystem.

If you don't have F2FS compiled in, or are using an initrd with explicit root filesystem module bundled in (with a generic kernel), then I don't think the issue surfaces.

Also, if you are using ext4 or reiserfs (which I believe are likely the most common choices with Slackware at the moment), then the issue also does not surface as I believe these are attempted before F2FS in the filesystem order. (Still trying to find what defines that order).

LuckyCyborg · 10-13-2019, 12:13 AM

Quote:

Originally Posted by Andypoo

If you don't have F2FS compiled in, or are using an initrd with explicit root filesystem module bundled in (with a generic kernel), then I don't think the issue surfaces.

Well, considering that that "huge kernel" is used today only by Slackware, and everybody else uses "generic" kernels with initrds, is expected that sooner or later to appear issues like this.

Maybe sooner or later Mr. Volkerding will put an end to those "huge" kernels, which aren't really needed today and anyway looks like they aren't the way is supposed to go.

Of course, I've already learned that the Slackware community, at least from this forum, will oppose to any changes, like a bunch of grumpy old men, but still I hope that the common sense will prevail at least in some particular cases.

rogan · 10-13-2019, 12:54 AM

I can confirm that the issue is gone when booting "huge" without f2fs.

LuckyCyborg: It's a big fat bug and we caught it thanks to our unorthodox use of a support-it-all installation kernel. Imho a far safer and better approach than trying automatical generation of an initrd. Keeping the installation as uncomplicated as possible has it's merits.
As most people will never need anything anything else, is also simplifies maintenance and upgrades greatly. All for the cost of a little bit extra memory use.

Didier Spaier · 10-13-2019, 05:34 AM

Quote:

Originally Posted by Andypoo

I e-mailed the maintainers. Does anybody know the proper way to report major regressions in longterm?

This is not obvious to me looking at the list of lists: http://vger.kernel.org/vger-lists.html

However the fs2fs developers do use this list: https://sourceforge.net/projects/lin...nux-f2fs-devel

PS they have a private stable tree where they backport most upstreamed code here: https://git.kernel.org/pub/scm/linux...fs-stable.git/

rogan · 10-13-2019, 08:17 AM

I filed a bugzilla report: https://bugzilla.kernel.org/show_bug.cgi?id=205181
We'll see if they're interested I guess....

GazL · 10-13-2019, 08:34 AM

Quote:

Originally Posted by blancamolinos

This is a demonstration of the disastrous patch policy of LTS kernels. The people who maintain them do not even verify that the kernel can boot. Hopefully they will check that the kernel with the patches compiles without errors, but who knows. It is better to follow the stable kernel than this crap of LTS kernels.

To be honest, stable kernels aren't much better. Patches get taken from the main development branch as and when they're applied, so if a bad patch arrives in the merge/early release candidate stage then it can get applied to both stable and lts with very little real world exposure/testing. This is why, from time to time, we see bad patches that are applied to one release candidate and fixed in a subsequent one get back ported to the stable branches, while the bug was never present in one of linus' .0 mainline releases.

Basically, linux kernel release engineering is a mess, and always has been.

rogan · 10-13-2019, 11:32 AM

All kernels except 4.9 and downwards and 5.4 have the issue. So the only solution is to remain with 4.19.75 if you have btrfs on root and only use "huge".

cwizardone · 10-14-2019, 11:29 AM

5.4-rc3

The newest "mainline" development kernel, version 5.4-rc3, has been been made available for testing.

The tarball, https://git.kernel.org/torvalds/t/linux-5.4-rc3.tar.gz

Mr. Torvalds' announcement, http://lkml.iu.edu/hypermail/linux/k...0.1/05411.html

Poprocks · 10-14-2019, 01:09 PM

Quote:

Originally Posted by rogan

I filed a bugzilla report: https://bugzilla.kernel.org/show_bug.cgi?id=205181
We'll see if they're interested I guess....

Excellent. I was going to say, if this is a kernel bug, why are we placing blame on Slackware's slightly unorthodox use case. Isn't the whole POINT of the kernel to allow every unorthodox use case under the sun. This is a regression that theoretically shouldn't be too difficult to correct for those familiar with the codebase and I'm sure it will be fixed relatively expeditiously.

rogan · 10-14-2019, 02:01 PM

This is definitely a kernel bug. Still waiting for a reply though.

Edit: Seems like andypoo has gotten through to them, so the maintainers
know about the issue now.

cwizardone · 10-16-2019, 06:09 PM

Another round of updates has been scheduled for release Friday night, GMT.

If no problems are found while testing the release candidates, they might be available late Thursday or early Friday (depending on your time zone).

There will be 112 patches to the 5.3.7 kernel update, 81 to 4.19.80, 65 to 4.14.150, 92 to 4.9.197, and finally, 79 patches to 4.4.197 update.

The details:

5.3.7, http://lkml.iu.edu/hypermail/linux/k...0.2/01351.html

4.19.80, http://lkml.iu.edu/hypermail/linux/k...0.2/01507.html

4.14.150, http://lkml.iu.edu/hypermail/linux/k...0.2/01310.html

4.9.197, http://lkml.iu.edu/hypermail/linux/k...0.2/01633.html

4.4.197, http://lkml.iu.edu/hypermail/linux/k...0.2/01267.html

Jan K. · 10-17-2019, 01:45 PM

Quote:

Originally Posted by rogan

All kernels except 4.9 and downwards and 5.4 have the issue. So the only solution is to remain with 4.19.75 if you have btrfs on root and only use "huge".

There's a dozen references to btrfs fix in 5.4 changelog, but only 4 of them are carried "down" to 4.19?

Not that I understand much of these fantastic logs, but it amuses me to see how many words, I recognize!

rogan · 10-17-2019, 02:27 PM

The problem was a commit in f2fs that affected other filesystems as well, at least btrfs that we know of. It has been patched in the coming releases.

petejc · 10-17-2019, 04:30 PM

Quote:

Originally Posted by rogan

I can confirm that the issue is gone when booting "huge" without f2fs.

LuckyCyborg: It's a big fat bug and we caught it thanks to our unorthodox use of a support-it-all installation kernel. Imho a far safer and better approach than trying automatical generation of an initrd. Keeping the installation as uncomplicated as possible has it's merits.
As most people will never need anything anything else, is also simplifies maintenance and upgrades greatly. All for the cost of a little bit extra memory use.

I've not been able to boot with the 5.3 series. Recompiling 5.3.6 with F2FS disabled did not seem to make a difference. Still getting a kernel panic on booting with initrd (via syslinux). 5.2.17 with essentially the same config is fine. Not sure if 5.3.7, which is just out, will help.

Andypoo · 10-17-2019, 05:41 PM

Quote:

Originally Posted by petejc

I've not been able to boot with the 5.3 series. Recompiling 5.3.6 with F2FS disabled did not seem to make a difference. Still getting a kernel panic on booting with initrd (via syslinux). 5.2.17 with essentially the same config is fine. Not sure if 5.3.7, which is just out, will help.

Interesting. What filesystem do you use? Mind sharing your mkinitrd line? Happy to look into it with you.

Have you managed to capture the kernel panic text? There was actually a hint in it for me prior to mounting, as it logs out the error from do_mounts. It could be a similar issue to the one that was backported if other modules are also cleaning up/changing their superblock detection behaviour.

I haven't tried any of the 5.x series yet personally.

Andypoo.