[SOLVED] Dracut strangeness

Dogmatix · 03-05-2021, 04:44 PM

So, I've been beating my head against this for too long. I found a fix, but I don't like it because it requires modifying scripts that probably will get updated. Maybe someone can tell me what I can't see. This is a long post, but the solution is complicated.

I've scoured the 'net for information on dracut and how to deal with a luks-crypt partition that holds the LVM that contains root, swap, and home, and little of it has been useful.

I have several computers running Leap 15.2 with different hard disk configurations, and only one has a dracut problem when it creates the new initrd after a kernel update. I have two computers with the exact same hard disk partitioning below, but only one has the dracut problem, and I can't figure out why.

After a kernel update and YaST runs dracut, there is no crypt module in any initrd on /boot, so the computer won't boot. My solution has been to boot off the distro DVD into the installation section, then after it asks for the decryption password, run through these steps from a console (Ctrl-Alt-F2):

Code:

1. mkdir ttt
2. mount /dev/mapper/ZuleSys-root /ttt
3. mount --bind /proc /ttt/proc
4. mount --bind /dev /ttt/dev
5. mount --bind /sys /ttt/sys
6. mount /dev/sda1 /ttt/boot
7. chroot /ttt
8. dracut --regenerate-all --force
9. exit
10. reboot

That process works every time - dracut properly includes what's needed for my luks crypt partition, and I'm prompted for the password at boot time.

Here's my lsblk:

Code:

NAME               MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                  8:0    0 298.1G  0 disk  
├─sda1               8:1    0   250M  0 part  /boot
└─sda2               8:2    0 297.9G  0 part  
  └─cr_ZuleSysLVM  254:0    0 297.9G  0 crypt 
    ├─ZuleSys-Root 254:1    0    30G  0 lvm   /
    ├─ZuleSys-Home 254:2    0 259.9G  0 lvm   /home
    └─ZuleSys-Swap 254:3    0     8G  0 lvm   [SWAP]
sr0                 11:0    1  1024M  0 rom

Here's the blkid:

Code:

/dev/sda1: LABEL="ZuleBoot" UUID="480bf83c-0505-4405-adbb-a9be37f32baf" TYPE="ext2" PTTYPE="dos" PARTUUID="000057e4-01"
/dev/sda2: UUID="f820119e-0e89-4743-ba34-e0d0172b2251" TYPE="crypto_LUKS" PARTUUID="000057e4-02"
/dev/mapper/cr_ZuleSysLVM: UUID="yGxlJr-wBM7-sth8-zxFj-7m1t-GhHO-3KLdcD" TYPE="LVM2_member"
/dev/mapper/ZuleSys-Root: LABEL="ZuleRoot" UUID="0bf2b9fc-d734-49c5-b682-628b7af9e7d4" TYPE="ext4"
/dev/mapper/ZuleSys-Home: LABEL="ZuleHome" UUID="435dd7fa-b73f-434a-b4f3-8c06404db37a" TYPE="ext4"
/dev/mapper/ZuleSys-Swap: UUID="717c0dd2-2032-4286-a908-296726be6f01" TYPE="swap"

Here's the /etc/crypttab:

Code:

cr_ZuleSysLVM /dev/disk/by-uuid/f820119e-0e89-4743-ba34-e0d0172b2251 none luks

Here's the relevant section of fstab:

Code:

/dev/ZuleSys/Root    /                    ext4       acl,user_xattr        1 1
LABEL=ZuleBoot       /boot                ext2       acl,user_xattr        1 2
/dev/ZuleSys/Home    /home                ext4       acl,user_xattr        1 2
/dev/ZuleSys/Swap    swap                 swap       defaults              0 0
/dev/dvd  /mnt/dvd iso9660 ro,users,nofail,noatime,utf8 0 0

My "fix" involves making a new file /etc/dracut.conf.d/myconf.conf with these contents:

Code:

add_dracutmodules+=" crypt "
install_items+=" /etc/crypttab /etc/cmdline.d/90crypt.conf "

Then, I edit /usr/lib/dracut/modules.d/90crypt/module-setup.sh as described here.

The line numbers don't match up with that link, but the idea is the same. The snippet of my module-setup.sh is this:

Code:

            if [ "${forceentry}" = "yes" ]; then
                echo "$_mapper $_dev $_luksfile $_luksoptions"
            else
                for _hdev in "${!host_fs_types[@]}"; do
#                    [[ ${host_fs_types[$_hdev]} == "crypto_LUKS" ]] || continue
#                    if [[ $_hdev -ef $_dev ]] || [[ /dev/block/$_hdev -ef $_dev ]]; then
                        echo "$_mapper $_dev $_luksfile $_luksoptions"
                        break
#                    fi
                done
            fi

The first commented line is line 109 in Leap 15.2.

Best I can figure from dracut's debug output, dracut only looks at the swap partition inside the LVM. It never looks at ZuleSys-Root. With the default setup, dracut decides that there are no luks-crypt partitions so it does not include anything related to decrypting it at boot time. The modification to module-setup.sh and myconf.conf force it to include the crypt stuff and create etc/crypttab and etc/cmdline.d/90crypt.conf in the initrd.

I have tried different versions for /etc/crypttab with no effect. I've tried forcing dracut to include the crypt module and include crypttab and 90crypt.conf, but it makes its own crypttab unless 90crypt/module-setup.sh is modified as shown.

Can anyone see what's wrong with this setup? Why does it not work right when others do?

Thanks for your help.

mrmazda · 03-06-2021, 01:26 AM

Code:

# inxi -SM
System:    Host: ab85m Kernel: 5.3.18-lp152.63-default x86_64 bits: 64 Desktop: KDE 3 Distro: openSUSE Leap 15.2
Machine:   Type: Desktop System: ASUS product: All Series v: N/A serial: N/A
           Mobo: ASUSTeK model: B85M-E v: Rev X.0x serial: MB-1234567890 UEFI: American Megatrends v: 3602
           date: 04/04/2018
# df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/sda1         327024    5016    322008   2% /boot/efi
/dev/sda4        4046394 1578777   2258721  42% /usr/local
/dev/sda5        6467641  568632   5567233  10% /home
/dev/sda8        8028736 4696490   2918550  62% /     # includes 6 installed kernels!

As you can see, a 15.2 installation need not take a whole lot of disk space. During software selection, deselect the option to install recommends, and with a basic role selection, one can get by with a seriously modest / filesystem. 8GB is actually one of my larger / filesystem sizes here. Some are on 4.8GB, others on 5.6GB, 6.4GB or 7.2GB. Only /home/ and /usr/local/ get discrete native filesystem space here on my (many) test installations.

You could temporarily replace your swapper with another EXT4, install another 15.2, or 15.3 or TW, and diff some of their files until you find something different, if the fresh installation works as expected. If it doesn't, you've a reproducible scenario that warrants a bug report.

Dogmatix · 03-06-2021, 10:24 AM

FWIW, mrmazda, my no-GUI file server uses 3.7GB in /, while my main "work" desktop uses 24GB. The main desktop has multiple DEs and tons of development files loaded up.

Sure, I could try a fresh installation, but only at a very last resort. This is a fairly minor annoyance, and surely there's some solution without wiping the whole thing away and starting over. The part I don't get is why this one installation has this issue while 3 others do not. One installation that does not have this problem has the exact same partitioning, luks crypt, and LVM scheme. That's the puzzle.

mrmazda · 03-06-2021, 11:45 AM

Where did you see a suggestion to wipe away "the whole thing" or "starting over"? A temporary extra installation using your normal swap space is neither, and presents an opportunity to diff a whole lot of files that should be exact matches or immaterially different.

That 4696490 blocks representing 62% usage on / above would change to about 3086515 blocks representing about 38.4% usage if I removed all but one kernel. Instead of temporarily re-purposing swap, you could shrink /home by 4GB, temporarily or otherwise, to make space for a direct comparison installation.

Dogmatix · 03-06-2021, 06:39 PM

Sorry, mrmazda, I misread what you meant.

I will point out that to make my computer bootable, I boot from the distro DVD into installation, then chroot to the installed system. Unless I'm mistaken, that means I'm using what's installed on the hard disk, not the DVD, to run dracut and fix things so I can boot from the installed system.

I've compared some of the dracut scripts on the problem computer with those on another and they're exactly the same.

mrmazda · 03-06-2021, 11:27 PM

There are regulars on the openSUSE forums with a lot more familiarity with LVM and crypto than I. Consequently, the only other suggestion I can make is to try there, or the regulars on the openSUSE support mailing list.

Dogmatix · 03-07-2021, 09:39 AM

Yes, I will post on the openSUSE forums once I can. I've registered and can get to everything but the forums. I posted here because I could and it seemed like a good place to start.

Dogmatix · 03-07-2021, 12:24 PM

Well, I found the problem.

After comparing dracut's output booting from the installation DVD and the output booting from the installed system, the only difference was the device name I chose in crypttab.

In my first post, crypttab started with "cr_ZuleSysLVM". That name is reflected everywhere, like in lsblk, blkid, and even device names sprinkled throughout /dev.

Booting into the DVD, that installer doesn't read my crypttab - it makes up its own name, "cr_auto-2".

The real bug is string comparisons in sh. The dracut debug output has lines like this:

Code:

[[ CRYPT-LUKS1-f820119e0e894743ba34e0d0172b2251-cr_ZuleSysLVM != LVM-* ]]

and this:

Code:

[[ CRYPT-LUKS1-f820119e0e894743ba34e0d0172b2251-cr_ZuleSysLVM =~ LVM-* ]]

The code that does this is in /usr/lib/dracut/dracut-functions.sh

If the made-up name includes LVM, those string comparisons don't work as the coder expected.

Once I lopped off the LVM from that name in crypttab, booted with the DVD, and rebuilt my initrd, everything is working properly. In my installed system, dracut now finds the luks crypt partition and makes initrd properly.

tl;dr: do not use "LVM" in your crypttab name.

syg00 · 03-07-2021, 07:25 PM

Nice find. That's a bug - report it.

mrmazda · 03-07-2021, 07:42 PM

Quote:

Originally Posted by syg00

Nice find. That's a bug - report it.

https://en.opensuse.org/openSUSE:Submitting_bug_reports

Dogmatix · 03-08-2021, 05:39 PM

I'm gonna report it on the dracut github page, as it's not specific to openSUSE.
https://github.com/dracutdevs/dracut