Debian Jessie systemd and display manger unmaintanable since ~2014/07/20
 

First of all , sorry formy english, I am french speaking and moreover working on a bad environment, I cannot make good formatting , my mouse cursor moving very slowly.

The problem:
since last Sunday (2014/07/20) dist-upgrade, the init system in Jessie is broke, be it on i386 or amd64.

]It is clearcly suspeced that systemd is the node of the problem.
Aftezr checking the root system, it enters an infiite loop saying
[FONT="Fixedsys"Welcome to emergency model! After logging in type "journalctl -xb" to view system logs, system default to try again to boot into default mode[/FONT]

Of course, I am unable to log in in any way. This phenomena appeared on three machines (amd64 and i386) after reboot.
On the machine I am working on now, I could boot a SystemRescueCD and after change rooting on the fixed drive, installe (re?)systemvinit-core.

I could finaly boot this machine in text mode.
OTOH, I could by no mean, start gdm3, (re) installing mate-desktop-environment-extra force dpkg to remove systemvinit-core

I installed good old xdm , and could finally log in and launch an xterm. Moreover the mouse is moving exessively slow.
root@josee:~# LANG=C apt-get -s install mate-desktop-environment-extra
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
accountsservice alacarte apg argyll colord-data cracklib-runtime dnsmasq-base gir1.2-accountsservice-1.0 gir1.2-clutter-gst-2.0 gir1.2-gck-1 gir1.2-gconf-2.0 gir1.2-gcr-3 gir1.2-gdm3 gir1.2-gkbd-3.0
gir1.2-gmenu-3.0 gir1.2-gnomebluetooth-1.0 gir1.2-gtkclutter-1.0 gir1.2-gtop-2.0 gir1.2-ibus-1.0 gir1.2-mutter-3.0 gir1.2-networkmanager-1.0 gir1.2-nmgtk-1.0 gir1.2-panelapplet-4.0 gir1.2-polkit-1.0
gir1.2-telepathyglib-0.12 gir1.2-telepathylogger-0.2 gir1.2-upowerglib-1.0 gir1.2-xkl-1.0 gkbd-capplet gnome-applets-data gnome-contacts gnome-control-center-data gnome-media gnome-menus gnome-online-accounts
gnome-packagekit-data gnome-panel gnome-panel-data gnome-session-common gnome-shell gnome-system-monitor hplip-data hyphen-en-us libaccountsservice0 libcolord-gtk1 libcolorhug2 libcrack2 libgdm1
libgnome-bluetooth11 libgnome-media-profiles-3.0-0 libgnome-menu-3-0 libgoa-backend-1.0-1 libgtkmm-3.0-1 libgusb2 libicc2 libimdi0 libmozjs185-1.0 libmutter0d libndp0 libnetfilter-conntrack3 libnl-route-3-200
libnm-glib-vpn1 libnm-gtk-common libnm-gtk0 libpanel-applet-4-0 libpwquality-common libpwquality1 libreoffice libreoffice-help-en-us librest-extras-0.7-0 libsane-hpaio libsocialweb-client2 libsocialweb-common
libsocialweb-service libsocialweb0 mobile-broadband-provider-info mousetweaks mutter-common mythes-en-us obexd-client packagekit-backend-aptcc python-imaging python-pexpect python-renderpm python-reportlab
python-reportlab-accel python3-packagekit realmd
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
caja gvfs gvfs-daemons libpam-systemd mate-applets mate-desktop-environment mate-desktop-environment-core mate-desktop-environment-extras mate-panel mate-polkit policykit-1 policykit-1-gnome systemd-sysv
udisks2
Suggested packages:
gstreamer0.10-tools meld gvfs-backends network-manager-gnome mate-netbook mate-user-guide xfsprogs reiserfsprogs exfat-utils btrfs-tools mdadm
The following packages will be REMOVED:
sysvinit-core
The following NEW packages will be installed:
caja gvfs gvfs-daemons libpam-systemd mate-applets mate-desktop-environment mate-desktop-environment-core mate-desktop-environment-extra mate-desktop-environment-extras mate-panel mate-polkit policykit-1
policykit-1-gnome systemd-sysv udisks2
0 upgraded, 15 newly installed, 1 to remove and 99 not upgraded.



I presume more people encounterded this bug.
Has someone clues about it.
Thanks in advance.

Johan Daine , Brussels Belgium

What version of systemd did you install? I'm also running Debian Jessie (with some Sid packages) and my systemd version is 204-14. I have not encountered any problems of the kind you describe. All appears to be running normally.

I don't use dist-upgrade but rather only the safe-upgrade option which seems to keep me out of too much trouble.
jdk

I finally could resolve the problem by

* installing systemd-sysv
* removing all 'auto' options in the mount options (in /etc/fsatb)of devices not permanently attached to the system (leaving this option did seem to hang the init process.
Removing the quiet option on the kernel command-line helped for this.

Everything seems now to work fine for now on two of my bikes. I will reiterate the trick on my main system, hoping it also works.

Thanks anyway.

Johan Daine

Best of luck with that. Please post your solution when you have one and mark this thread as [SOLVED] as a service to others.
jdk

This is most likely the problem, systemd will stop the boot process when encountering incorrect fstab entries (and an entry with the auto option for devices that are not connected at boot time is incorrect).

"Incorrect" or not, sysvinit seems to have handled auto fstab stanzas in a more forgiving manner, where the device was not connected - and seems to have worked well enough for the last 30+ years. sysvinit did not break user space, systemd did.

Though that is not the topic of this thread, sysvinit did not handle incorrect fstab entries "in a more forgiving manner", it simply ignored the errors, a behavior that should not ever be the norm, especially on production systems. IMHO, it is far more reasonable to pull the emergency brake in that case, rather than running the system in an undefined state.

It's the topic of thread: systemd broke a working system.

I would not call those "errors", more like "warnings".

What you're missing here is that an init system just breaking user space - breaking something which worked before is irresponsible and bad practice.

Usually when something like this changes - we would see warnings on boot (foo is deprecated - change your /etc/bar), but the system would still boot. You don't just break user space on a whim because you believe how it's been done since time immortal is "incorrect"... that certainly isn't (wasn't?) the Debian way either.

Then you will have to complain to Debian developers to change this behavior. I see nothing wrong with an init system refusing to boot a system to an undefined state.

Regarding "it's been done since time immortal", IMHO it doesn't matter how long a wrong behavior lasts (and simply ignoring errors is wrong in my eyes), if it is necessary to break userspace to correct it that is a good thing (not that this would be the case here, since the init system is in userspace). Change for change's sake is not good, but change to make systems more consistent (read: do not allow them to boot to an undefined state) is good. If you have to break something in the way then do it.
AFAIK, there was and is no Debian policy to make new releases backwards compatible with older ones, changes and breakages between releases are allowed.

Actually, Tobi you might not be correct with your information:

https://help.ubuntu.com/community/Fstab
http://www.linfo.org/etc_fstab.html
http://wiki.linuxquestions.org/wiki//etc/fstab

Each of these lists "auto" as a valid flag and auto is the equivalent of manually running "mount -a".

If systemd is hanging on "auto", it should also hang on "defaults" because "defaults" loads options: rw, suid, dev, exec, auto, nouser, and async

This isn't something the Debian developers may be able to fix without a patch to their implementation. This is an upstream issue from the systemd project. Unless a newer version of systemd has fixed this, the current implementation is faulty and broken.

The "auto" flag is meant for devices that should automatically mount when the mount -a is called, as you describe. That usually happens the first time during boot. An entry that is meant for removable devices/media, like USB storage devices or optical discs, which are not guaranteed to be present at boot time, is in fact incorrect when it contains the "auto"-flag. Sysvinit (or better, Debian's scripts) simply ignores the errors caused by those incorrect entries, which may be fine when it comes to trivial devices, like a CDROM or an USB stick, but the init system has no way to determine if the device mentioned in an entry is actually one that is essential for the systems purpose (like an internal HDD mounted as /var, but "removed" at boot due to hardware failure) or configured for non-essential tasks that can lead to system failure over time when the device is not present (like copying whatever to an external disk without checking if it is actually mounted, polluting the root filesystem with those actions).

This ignoring of not present devices that are configured by the admin to be mounted at boot is IMHO not something any init system, regardless if it is systemd, sysvinit, bsdinit, s6 or whatever else, should do, it comes down to guessing "hey, this might not be essential, so lets just ignore it and start the system in a possibly undefined state anyways". Guessing is something that no OS that deems itself to be stable should do, instead in case of unrecoverable errors (and a missing filesystem that is configured by the admin to be there at boot time using the "auto"-flag is exactly that, an unrecoverable error) the system should, as gracefully as possible, come to a halt and inform the admin what has happened.
Granted, when it comes to that there has still some work to be done in systemd, so that it provides clear error messages why it halted the system, but it still should halt the system instead of starting it in an undefined state.

In short: It is the admin's job to provide correct configuration. If the admin is not doing that, for whatever reason, the init system should point out that there is incorrect configuration, but it should never just guess, ignore errors or try to compensate for those errors without admin interaction.

Yes, but it also could simply issue a warning of:

"Warning: Invalid flag on /dev/sdX in /etc/fstab."

and continue onward with the boot process.

The mount utility should be the software issuing the warning through the system log, not the init system. The init system is meant to only process the boot scripts and pass commands to the system utilities (util-linux) which do all the remainder of getting the system running in the correct order. The init system should not hanging the system when util-linux's mount is handling the mount points and bringing disks online.

In our Runit implementation we have the stage-1 script pass commands to mount (util-linux) to bring all the disks online with the /etc/fstab configuration. If mount detects a problem usually it'll display a warning in the boot-log, but the boot continues regardless.

In fact you really shouldn't use fstab for removeable drives. Udisks and udisks2 should be used for those types of drives. You can use fstab but you are correct in the sense it should be tuned correctly in fstab, but it should only spit out a warning and continue onward, not stop everything.

I would prefer if it drops me to a shell to fix it before continuing the boot process, which may fail anyways if an essential filesystem is missing. Continuing just because it might work is unacceptable, init systems should not guess.
On the contrary, the init system should do exactly that: stop the boot when such en essential service as boot time mounting fails for whatever reason. Imagine this scenario: You are running a headless server and have mounted /var/log to a partition (probably because one or more of your services produce large amounts of data that has to be logged) on a disk that has failed. Since mounting this partition has failed and the init system did not halt this problem might go unnoticed (seriously, who reads the logs after every boot?) until the /-partition got filled up. Now you have two problems instead of one, just because the init system decided/was designed not to halt the machine when encountering unrecoverable error, but boot into an undefined state. This is just one possible type of failures, it is pretty easy to imagine more of those scenarios and likely you will bve hit by one that the admin did not consider.

Then your stage-1 script makes the same mistake that the current Debian scripts for sysvinit makes: It just guesses that the failed mounts are not essential and leaves the system in an undefined state. This is not acceptable on production systems.
I totally agree with that.
And here I totally disagree. An init system should never do guess work, nor should it just ignore unrecoverable errors.

That's still not the job of init. The job should be util-linux's job, and util-linux's job only.

You're right that you can write in a if/else command in the boot script which runs mount, but that's the job of the system administrator, not init, and that's still the job of util-linux (mount) only.

Init's job is to boot the system then pass commands via the bootscript(s), nothing else and nothing more, then sit quietly in the background waiting for kill, run, halt, etc. system mode signals.

The problem is Tobi if a command to check the file system options for validity and state isn't written into the Stage-1 or file system mounting bootscript, the system only spits out a warning and continues. It can be written and should be written in as part of the mount command structure, but not as part of the init codebase. That's the job of the system administrator and the author of the bootscripts.

If those aren't the ones drafting bootscripts correctly, then it's their fault, not init's fault.

OK, it seems that we disagree what an init systemd should actually do.
For me an init system (and an init system includes the actual binary and the boot scripts, hence init system, not init binary) should boot the system into a defined state. That includes error handling of the tools it has to start to achieve that job. IMHO, an init system that is not able to boot the system to a defined state (and a halt due to unrecoverable errors is also a defined state) is simply failing its job. I will not say useless here, it still will function most of the time, but most of the time is not enough, it should especially handle the situation well in a worst case scenario.

Anyways, I think we should just agree to disagree here.