I just performed a dist-upgrade to debian wheezy, which included switching over to systemd for init. After the upgrade, the system stalls out around the time that it's mounting the disks, but before networking is enabled.

I've managed some troubleshooting, including enabling the debug shell, but I can't figure out what's going on. Each of the currently active processes appear to have completed successfully, but it doesn't seem to go on to the next step.

From the debug shell, I cannot run any systemd commands; They just hang until I press control+C. I'm not using LVM, but I have a software raid mounted on /samba (md0). I can mount all disks manually, but they are not present at the point where systemd stalls.

I've been struggling on this for a few weeks and now I'm out of ideas. Any suggestions are appreciated.

kernel arguments: dmesg, systemd --test & ps output attached.

Attached Files

	dmesg.txt (61.6 KB, 11 views)
	ps_auxOp.txt (4.8 KB, 9 views)
	systemd-test.txt (57 Bytes, 7 views)

How long are you waiting?
On my system when changed to systemd, boot logging only showed fsck, to get more info, I edited line in /etc/default/grub.
Changed command line from quiet to verbose, which enabled me to view everything at boot.
My boot time is approximately 2mins. Quite long if you ask me, however the shoutdown is a matter of seconds now.

I use both Debian wheezy with sysVinit and Debian jessie with systemd. I had the bug you described in jessie about a month or so ago. The system initialization would stall for a long time while populating /dev. Eventually the problem would resolve itself and jessie would continue on to a KDM login screen. The bug was eventually fixed during a routine apt-get download of package upgrades.

Since wheezy is always behind jessie in package versions you probably have hit the same bug in wheezy's systemd that jessie used to have. You might be able to fix the bug in wheezy by installing jessie's current version of systemd but that is risky.

My opinion of systemd is that it is too buggy and has too many design flaws to be using it on wheezy. I suggest that you go back to sysVinit.

----------------------
Steve Stites

I stopped troubleshooting last Wednesday and picked it up again on Saturday night. It was still at the same screen. I don't reboot this machine too often, but if it's going to take days to come up, that's going to be a problem :-D . I feel like there's one small thing that I'm missing, then it'll all be okay, but I'm starting to think I need to start the process of reverting back to SystemV, or restoring to the backup I actually had the foresight to make before the update.

Wheezy does not have systemd in it natively. Jessie, which is the current testing, is the first version of Debian to have systemd natively. So now that is out of the way, why did you install systemd on a system that was never designed to use it?

Also, please just use normal fonts and type setting in the forum.

It could be any number of things. It could be a corrupted journal, bad settings in fstab (systemd can't read "default" settings well), or even any other issue.

systemd starts various initialization routines in parallel which under systemVinit are run sequentially. These threads have to have exclusive control of various resources for a while so they issue exclusive mutexes for those resources and then release them when they are finished with the resource. You can get mutex interlocks or not depending on the timing of how fast the various routines run on your machine and the order in which they ask for exclusive control of the resources they are initializing. Problems in which independent threads handle mutexes in ways which are incompatible with the way other threads running in parallel handle the same mutexes are extremely hard to debug and are usually design flaws rather than bugs.

So your delay could be a thread being held up for a long time until another thread finally finishes with a mutex they both use. Or your delay could be a mutex interlock between two or more threads in which case Linux will wait forever. The bug I described on my machine was simply a long wait that eventually resolved itself.

-------------------
Steve Stites

Thanks for all the suggestions. To follow up, I did not intentionally install systemd, it came along with a dist-update. I figured Debian was moving in that direction. I ended up appending an "init=" line to my grub config and it's running good enough for now. I have systemd on a few other boxes and it works quite well, but this box has been migrated through quite a few debian releases and may have developed some quirks along the way that didn't work well with systemd.

systemd is in backports do you have backports in sources.list?
Also it says that you can either use init=/systemd/bin or install systemd-sysv
https://packages.debian.org/wheezy-backports/systemd

It's also a package in wheezy, although it's not installed on my wheezy machine.
https://packages.debian.org/wheezy/systemd

The only reason you would need to use dist-upgrade is if you are upgrading from Squeeze to Wheezy, Wheezy to Jessie, or have enabled backports and need to install something from backports that is installed on your system from a 3rd party repo (e.g. MATE). For normal usage you should only need to use apt-get upgrade. If you have backports enabled you installed systemd by calling dist-upgrade.

From memory the systemd in Wheezy was a really old version and was there as a technology preview. The version in Jessie is in the version number 2 hundred something while Wheezy is very low and in the 40s I think.