[SOLVED] 3 minutes boot delay due to udev worker timeout in linux-5.6-bpo kernel

deepclutch · 06-23-2020, 11:53 AM

With linux 5.6.0-0.bpo.2-amd64, while booting it is stuck at "waiting for /dev to be fully populated...(time out)".

There is no such issue with kernel -4.19.0-9-amd64. I am just trying the latest kernel and wonder if any module is missing? Attached is the diff result comparing 4.19.0-9 and 5.6.0-0.bpo.2 kernels.
System running on openRC + eudev etc.

This is the message when I shutdown the system:

Code:

Currently running processes (pstree): init-*-rc--openrc---openrc-run---sendsigs---pstree       *-udevd---3*[udevd]  Killing all processes... : worker [xyzab] 
failed while handling '/devices/pci0000:00/0000:00:1c.3/0000:09:00:0/usb4/4-0' udevd worker [xxxx] 
failed while handling '/devices/pci0000:00/0000:00:1c.3/0000:09:00:0/usb3/3-0:1-0'

I am not sure where to look into, to find the problem. BTW, I've tried other 5.x kernels and the problem persists.

deepclutch · 06-25-2020, 06:37 PM

I tried booting with rw init=/bin/bash and started services manually. /etc/rcS.d/S02eudev takes a minute to finish with "done. (time out)" message shown.

deepclutch · 06-26-2020, 11:32 AM

I installed recent versions of some packages from ceres (untable) repo.But, the problem of 3 minute delay with udevd worker persists at boot.

Code:

:~# apt list |grep ,local

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

crda/now 4.14+git20191112.9856751-1~bpo10+1 amd64 [installed,local]
elogind-build-deps/now 243.8-1 all [installed,local]
elogind/now 243.8-1~bpo10+1 amd64 [installed,local]
eudev/now 3.2.9-7~bpo10+2 amd64 [installed,local]
init-system-helpers/now 1.57+devuan1~bpo10+1 all [installed,local]
init/now 1.57+devuan1~bpo10+1 amd64 [installed,local]
libeinfo1/now 0.42-1~bpo10+1 amd64 [installed,local]
libelogind-dev-doc/now 243.8-1~bpo10+1 all [installed,local]
libelogind-dev/now 243.8-1~bpo10+1 amd64 [installed,local]
libelogind0/now 243.8-1~bpo10+1 amd64 [installed,local]
libeudev-dev/now 3.2.9-7~bpo10+2 amd64 [installed,local]
libeudev1/now 3.2.9-7~bpo10+2 amd64 [installed,local]
libpam-elogind/now 243.8-1~bpo10+1 amd64 [installed,local]
librc1/now 0.42-1~bpo10+1 amd64 [installed,local]
openrc/now 0.42-1~bpo10+1 amd64 [installed,local]
refind/now 0.12.0-1 amd64 [installed,local]
wireless-regdb/now 2020.04.29-1~bpo10+1 all [installed,local]

deepclutch · 06-30-2020, 09:18 AM

Installed haveged. But, udev waiting & time out continues. What is least helpful with this fresh Beowulf install is, I cannot find any logs about what is actually failing or "hangs" while booting or in any log files. In the previous installation (from ascii to beowulf), when I boot with linux-5.6.x, I can see some usb id's related error messages can be seen. (I guess a bluetooth adapter and wireless adapter).

Right now, I saw some message while shutting down. I think it's complaining about usb device and udevd worker. (will confirm)

Where do I start with? I tried starting init scripts manually and even eudev started fine. If I run "udevadm settle" it takes few minutes.

deepclutch · 07-07-2020, 11:38 AM

@those who are following this issue:

I ended up removing most of the packages by removing linux-image-5.6.x, eudev, elogind, apt (involuntary) and other critical packages. This removed whole of X11, wayland and Gnome/GTK packages as well. Around 515 packages. Then, reinstalled kernel-5.6, eudev, elogind, apt (it was tricky for novices). On next reboot, it went smooth. No error messages.

Yet to find out which package(s) is breaking the system.

deepclutch · 07-08-2020, 11:54 PM

Found out "modemmanager" package is messing with eudev rules. I removed it and problem resolved. There are some rough edges remaining, though.