[SOLVED] Systemd dependency cycle

displace · 03-12-2022, 10:29 AM

Hello.

I am sorry because I am relatively new to systemd and apparently I don't understand how it works. I'm on Xubuntu 20.04 where I'm trying to utilize systemd to decrypt and mount a certain encrypted partition upon boot. I followed some tutorials online and I ended up adding the following two files to /etc/systemd/system folder. These two are supposed to decrypt the specified partition and then mount it after it has been opened by dm-crypt (/dev/mapper/).

/etc/systemd/system/mnt-windows.mount

Code:

[Unit]
Requires=unlock-windows-veracrypt.service
After=unlock-windows-veracrypt.service                

[Mount]
What=/dev/mapper/windows
Where=/mnt/windows
Options=nosuid,nodev,noexec

[Install]
WantedBy=local-fs.target

/etc/systemd/system/unlock-windows-veracrypt.service

Code:

[Unit]
Description=Open encrypted windows partition
Requires=cryptsetup.target                
After=cryptsetup.target
StopWhenUnneeded=true

[Service]
Type=oneshot
ExecStart=/bin/sh -c '/usr/bin/cat /boot/cryptpass.bin | /sbin/cryptsetup -v open --type tcrypt --tcrypt-system --veracrypt /dev/disk/by-partlabel/VERA windows'
ExecStop=/sbin/cryptsetup -v close windows
RemainAfterExit=true

I then activated this by running systemctl enable mnt-windows.mount and rebooted. This mostly worked fine, but at some point the mount started to fail and I noticed errors in the syslog. Upon further inspection it turns out that certain services were failing to run due to systemd dependency loops. I tried to debug the issue, but was unsuccessful. When running the command systemd-analyze verify default.target I get lots of entries like these:

Code:

sockets.target: Found ordering cycle on snapd.socket/start
sockets.target: Found dependency on sysinit.target/start
sockets.target: Found dependency on snapd.apparmor.service/start
sockets.target: Found dependency on apparmor.service/start
sockets.target: Found dependency on local-fs.target/start
sockets.target: Found dependency on mnt-windows.mount/start
sockets.target: Found dependency on unlock-windows-veracrypt.service/start
sockets.target: Found dependency on basic.target/start
sockets.target: Found dependency on sockets.target/start
sockets.target: Job snapd.socket/start deleted to break ordering cycle starting with sockets.target/start
sockets.target: Found ordering cycle on uuidd.socket/start
sockets.target: Found dependency on sysinit.target/start
sockets.target: Found dependency on snapd.apparmor.service/start
sockets.target: Found dependency on apparmor.service/start
sockets.target: Found dependency on local-fs.target/start
sockets.target: Found dependency on mnt-windows.mount/start
sockets.target: Found dependency on unlock-windows-veracrypt.service/start
sockets.target: Found dependency on basic.target/start
sockets.target: Found dependency on sockets.target/start
sockets.target: Job uuidd.socket/start deleted to break ordering cycle starting with sockets.target/start
sockets.target: Found ordering cycle on cups.socket/start
sockets.target: Found dependency on sysinit.target/start
sockets.target: Found dependency on snapd.apparmor.service/start
sockets.target: Found dependency on apparmor.service/start
sockets.target: Found dependency on local-fs.target/start
sockets.target: Found dependency on mnt-windows.mount/start
sockets.target: Found dependency on unlock-windows-veracrypt.service/start
sockets.target: Found dependency on basic.target/start
sockets.target: Found dependency on sockets.target/start
sockets.target: Job cups.socket/start deleted to break ordering cycle starting with sockets.target/start

If I disable the mnt-windows.mount then the above command returns no errors. It seems that the two files I added are somehow creating a dependency cycle (or several). I found a document that explains the systemd execution tree quite nicely. The two files are linked to local-fs.target and cryptsetup.target in that tree, but according to the above error messages the unlock-windows-veracrypt.service appears to be pulling in the basic.target dependency for some reason. And since this target is way below in the execution tree this creates a cycle and causes systemd to randomly stop execution of some targets.

Any idea how to solve this?
Thanks in advance.

ondoho · 03-13-2022, 04:01 AM

Don't youneed to enable both units?

Is the problematic behavior 100% reproducible?

Please show us

Code:

systemctl -n9999 status unlock-windows-veracrypt.service mnt-windows.mount

Generally speaking, remove config options you aren't sure about. There seems to be too much going on in your unit files.

MadeInGermany · 03-13-2022, 05:30 AM

Run

Code:

systemctl list-dependencies

Or try to get the relevant part

Code:

systemctl list-dependencies mnt-windows.mount

Regarding a mount, you can try a x-systemd.automount option.
See

Code:

man systemd.mount

or
https://www.freedesktop.org/software...emd.mount.html

The automount option delays the mount until an access occurs.

displace · 03-13-2022, 11:21 AM

Hey, thank you for your answers.

Quote:

Originally Posted by ondoho

Don't youneed to enable both units?

As far as I am aware, no because the mnt-windows.mount automatically pulls the unlock-windows-veracrypt.service as a dependency. If I have mnt-windows.mount disabled and manually start it with systemctl start mnt-windows.mount it will run fine and the partition will be decrypted & mounted correctly. When I manually stop it the dm-crypt device /dev/mapper/windows will be closed automatically. You think it would be better to have them run independently?

Quote:

Originally Posted by ondoho

Is the problematic behavior 100% reproducible?

Yes. Although each time I run the systemd-analyze command a different set of errors is displayed.

Quote:

Originally Posted by ondoho

Please show us

Code:

systemctl -n9999 status unlock-windows-veracrypt.service mnt-windows.mount

I'm assuming the output of this command isn't dependent on whether the mnt-windows.mount service is enabled or disabled. I have run the command with the service disabled and stopped. Here is the output you requested:

Code:

root@predator:/etc/systemd/system# systemctl -n9999 status unlock-windows-veracrypt.service mnt-windows.mount
● unlock-windows-veracrypt.service - Open encrypted windows partition
     Loaded: loaded (/etc/systemd/system/unlock-windows-veracrypt.service; static; vendor preset: enabled)
     Active: inactive (dead)

mar 13 16:59:17 predator systemd[1]: Starting Open encrypted windows partition...
mar 13 16:59:19 predator sh[13571]: Command successful.
mar 13 16:59:19 predator systemd[1]: Finished Open encrypted windows partition.
mar 13 16:59:49 predator systemd[1]: Stopping Open encrypted windows partition...
mar 13 16:59:49 predator cryptsetup[14086]: device-mapper: remove ioctl on windows  failed: Device or resource busy
mar 13 16:59:49 predator cryptsetup[14086]: Command successful.
mar 13 16:59:49 predator systemd[1]: unlock-windows-veracrypt.service: Succeeded.
mar 13 16:59:49 predator systemd[1]: Stopped Open encrypted windows partition.

● mnt-windows.mount - /mnt/windows
     Loaded: loaded (/etc/systemd/system/mnt-windows.mount; disabled; vendor preset: enabled)
     Active: inactive (dead)
      Where: /mnt/windows
       What: /dev/mapper/windows

mar 13 16:59:19 predator systemd[1]: Mounting /mnt/windows...
mar 13 16:59:19 predator ntfs-3g[14077]: Version 2017.3.23AR.3 integrated FUSE 28
mar 13 16:59:19 predator ntfs-3g[14077]: Mounted /dev/mapper/windows (Read-Write, label "System", NTFS 3.1)
mar 13 16:59:19 predator ntfs-3g[14077]: Cmdline options: rw,nosuid,nodev,noexec
mar 13 16:59:19 predator ntfs-3g[14077]: Mount options: nosuid,nodev,noexec,allow_other,nonempty,relatime,rw,fsname=/dev/mapper/windows,blkdev,blksize=4096
mar 13 16:59:19 predator ntfs-3g[14077]: Ownership and permissions disabled, configuration type 7
mar 13 16:59:19 predator systemd[1]: Mounted /mnt/windows.
mar 13 16:59:49 predator systemd[1]: Unmounting /mnt/windows...
mar 13 16:59:49 predator ntfs-3g[14077]: Unmounting /dev/mapper/windows (System)
mar 13 16:59:49 predator systemd[1]: mnt-windows.mount: Succeeded.
mar 13 16:59:49 predator systemd[1]: Unmounted /mnt/windows.

Quote:

Originally Posted by MadeInGermany

Run

Code:

systemctl list-dependencies

Yes, I get this output:

Code:

root@predator:/etc/systemd/system# systemctl list-dependencies mnt-windows.mount
mnt-windows.mount
● ├─-.mount
● ├─dev-mapper-windows.device
● ├─system.slice
● └─unlock-windows-veracrypt.service

root@predator:/etc/systemd/system# systemctl list-dependencies unlock-windows-veracrypt.service
unlock-windows-veracrypt.service
● ├─system.slice
● ├─cryptsetup.target
● │ └─systemd-cryptsetup@cryptroot.service
● └─sysinit.target
●   ├─apparmor.service
●   ├─dev-hugepages.mount
●   ├─dev-mqueue.mount
●   ├─keyboard-setup.service
●   ├─kmod-static-nodes.service
●   ├─plymouth-read-write.service
●   ├─plymouth-start.service
●   ├─proc-sys-fs-binfmt_misc.automount
●   ├─setvtrgb.service
●   ├─sys-fs-fuse-connections.mount
●   ├─sys-kernel-config.mount
●   ├─sys-kernel-debug.mount
●   ├─sys-kernel-tracing.mount
●   ├─systemd-ask-password-console.path
●   ├─systemd-binfmt.service
●   ├─systemd-boot-system-token.service
●   ├─systemd-hwdb-update.service
●   ├─systemd-journal-flush.service
●   ├─systemd-journald.service
●   ├─systemd-machine-id-commit.service
●   ├─systemd-modules-load.service
●   ├─systemd-pstore.service
●   ├─systemd-random-seed.service
●   ├─systemd-sysctl.service
●   ├─systemd-sysusers.service
●   ├─systemd-timesyncd.service
●   ├─systemd-tmpfiles-setup-dev.service
●   ├─systemd-tmpfiles-setup.service
●   ├─systemd-udev-trigger.service
●   ├─systemd-udevd.service
●   ├─systemd-update-utmp.service
●   ├─cryptsetup.target
●   │ └─systemd-cryptsetup@cryptroot.service
●   ├─local-fs.target
●   │ ├─-.mount
●   │ ├─boot-efi.mount
●   │ ├─mnt-windows.mount
●   │ ├─systemd-fsck-root.service
●   │ ├─systemd-remount-fs.service
●   │ └─tmp.mount
●   └─swap.target

Hmm, interesting. It seems that unlock-windows-veracrypt.service is pulling in sysinit.target dependency for some reason.

displace · 03-13-2022, 11:35 AM

Question: If I install a systemd service onto a certain target (i.e. WantedBy=local-fs.target), does that mean that my service has to start and complete before the given target (local-fs.target) is considered complete and the execution can continue or is my service simply started when the execution "cursor" reaches the given target (local-fs.target) in the execution tree?

EDIT: Interesting. I created an empty test service and listed its dependencies. It depends on sysinit.target even when it's empty.
Is this a thing that all services automatically depend on sysinit.target??

SOLVED: Yeah, that seems to be the case. I took a look at how existing systemd servcies work, and I found that some of them that load before sysinit.target have a "DefaultDependencies=no"clause in the [Unit] section. I added it to unlock-windows-veracrypt.service and now the errors are gone!

Code:

[Unit]
Description=Open encrypted windows partition
Requires=cryptsetup.target
After=cryptsetup.target
StopWhenUnneeded=true
DefaultDependencies=no

[Service]
Type=oneshot
ExecStart=/bin/sh -c '/usr/bin/cat /boot/cryptpass.bin | /sbin/cryptsetup -v open --type tcrypt --tcrypt-system --veracrypt /dev/disk/by-partlabel/VERA windows'
ExecStop=/sbin/cryptsetup -v close windows
RemainAfterExit=true