We have a problem with systemd not starting things in the right order, but the configuration seems accurate, so I thought maybe there would be an systemd expert around here that could provide some insight.

Basically we have an embedded Linux box (running an offshoot of Debian), and two of the things that need to happen during the systemd-controlled startup are: 1) set the system clock with "hwclock -s", and 2) bring up the Ethernet interface with "ifup". It's important that hwclock be fully completed before ifup runs. Right now, that ordering is not ensured by the systemd configuration and I don't understand why. Instead what's happening is that hwclock and ifup are launched and run simultaneously, and the system clock gets set just as dhclient is waiting for a DHCP offer. Since the system clock changes, dhclient gets confused on its timeout and instantly and gives up waiting for the DHCP offer, and then 5 minutes passes before trying again.

The ifup command is being run by a systemd service called "networking.service". The networking.service has "ExecStart=/sbin/ifup". Then ifup spawns dhclient because the network interface is configured to use DHCP.

The hwclock command is being run by a systemd service called "hwclock.service". The hwclock.service has ExecStart=/sbin/hwclock -s. In the [Unit] section of the hwclock.service, there is a directive specifying "Before=networking.service", which should guarantee that the hwclock command completely finishes *before* networking.service is launched. But that is not what ends up happening in reality.

I grabbed a snippet of the syslog to show the sequence of events:
Nov 3 17:16:45 sdc daemon.info dhclient[314]:
Nov 3 17:16:45 sdc daemon.info dhclient[314]: Listening on LPF/eth0/b8:27:eb:1e:87:94
Nov 3 17:16:45 sdc daemon.info dhclient[314]: Sending on LPF/eth0/b8:27:eb:1e:87:94
Nov 3 17:16:45 sdc daemon.info dhclient[314]: Sending on Socket/fallback
Nov 3 17:16:45 sdc daemon.info dhclient[314]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 4
Dec 13 16:03:14 sdc daemon.info dhclient[314]: No DHCPOFFERS received.
Dec 13 16:03:14 sdc daemon.info dhclient[314]: No working leases in persistent database - sleeping.

Note above how dhclient starts running on the date "Nov 3" (this is incorrect value of the system clock before hwclock is run). Then after dhclient does a DHCP discover (still on Nov 3), the clock suddenly changes to Dec 13 (the correct date) due to hwclock being run. Of course this causes dhclient to instantly timeout waiting for DHCP offers.

The current configuration for hwclock.service is as follows (in a file named "hwclock.service"):
[Unit]
Description=Real-time Clock Service
Before=networking.service

[Service]
Type=oneshot
ExecStartPre=/usr/local/bin/pcf8523
ExecStart=/sbin/hwclock -s
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

What is the best way to configure the hwclock.service so that the hwclock command "completely finishes and the system time is fully set" *before* the networking.service is started?

The "Before=networking.service" only means that the network will start after your this process is called doesn't matter if it has finished or not. My guess is the clock setting program doesn't have time to finish before the the network is being called.

In your networking.service under [Unit] add
This should allow for the clock to finish before the network is started.

Appreciate the suggestion, but it did not seem to help. Is there a way I can get systemd to dump all the services that are running? I ask because now I'm not 100% sure that the networking.service is the one that's doing the "ifup".

There's another service called "ifup@.service" that also does an "ifup". I guess this is a template service with the @ symbol representing the interface name. But I don't see any references to the "ifup@.service" in any other service.

What I'd really like is a dump of all the services that are running, along with some information about what caused each service to be activated. "systemctl list-dependencies" doesn't really give much information other than what's in the systemd files.

seems to work, but there's probably something better in
I'm still learning how to work with systemd, too.

Edit: maybe ??

Well that helps, and it looks like my ifup command is in fact being run by the ifup@.service. So now the question becomes more complicated. I don't want hwclock.service to run before ifup@eth0.service (what if there's no eth0?) I don't want to hard-code any interface name into my dependencies of the hwclock.service.

I don't know what is causing the ifup@.service to activate. It's not referenced anywhere by any other service. Nothing "Requires=ifup@.service" and ifup@.service doesn't have a "RequiredBy=". So how do I figure out what is causing this ifup@.service to get activated?

Basically I just want to set the damn clock very early in the boot process, before a clock change can interfere with processes that depending on the clock for their function. Surely this cannot be that difficult. systemd seems to have gone out of its way to make this as convoluted as possible, requiring you to reverse-engineer and fully comprehend the entire architecture of 100 service inter-dependencies before you can safely make a single sane modification.

To make matters worse, setting the clock sometimes screws up systemd, because systemd depends on the clock to know what it's doing, and changing it mid-process sometimes causes it to do strange things.

Ok rant over, open to any suggestions. Thanks!

If the ifup@.service is enabled, it's going to start at boot. In other words, it's "activated" by being there.

Did you try also adding per lazydogs suggestion to that service as well as to the networking.service?

Now you've introduced another service? ifup@eth0.service ??
Why would eth0 not being present affect the service waiting for ifup? ifup happens and eth0 then fails? So what?


Aside:
May I ask why you're concerned about the hardware clock?
Are you not running ntpd to keep the system time correct?

Yes ifup@.service is enabled simply because it exists, but how does it know to start "ifup@eth0.service"? Something must be telling systemd to instantiate the "template service" ifup@.service with the parameter "eth0". What is doing that?

Yes I did, but it makes no difference. But this is not surprising after discovering that networking.service is not the unit that is doing the "ifup".

But it's not "another" service, "ifup@eth0.service" is an instantiation of the ifup@.service "template" service. This leads to the $64,000 question which is... How do you make a template service be a prerequisite for another service? If I just add "Before=ifup@.serivce" to the hwclock.service, then will it just "Do-What-I-Want (tm)"? Or would I need to add "Before=ifup@eth0.serivce"? The systemd docs aren't clear on these kinds of nuances with template services. I don't want to mention "eth0" in the hwclock service because the name "eth0" could change if someone swaps in a different USB-to-Ethernet dongle.

As I said, this is in an embedded device. Running ntpd is not helpful because there is no access to the Internet. Also, all filesystems are mounted read-only to handle power loss without corruption. Our embedded device creates logs (among other things) that should have reasonably accurate timestamps associated with them. To maintain the clock during power cycling, we have a PCF8523 chip with an I2C interface. Setting the system clock from a PCF8523 chip during boot requires some initialization and then ultimately running "hwclock -s". Unfortunately, radically changing the clock while the system is booting seems to have catastrophic effects on some processes (i.e. dhclient).

It seems everyone is moving towards NetworkManager.service to run the networking part. Do you have this?

No, this is a tiny embedded device. We don't need a lot of fancy functionality. I just want to set the damn clock before the network tries to start. It's astonishing to me that systemd, in all it's complexity, doesn't have a way to say "do this first". Instead, everything is managed from the *end*, so you have to work backward through zillions of inter-dependencies to get something to start early.

What flavor of Linux is it using? Could be a badly trimmed down version of something.

It's Raspbian. Yes Raspbian does have a package for NetworkManager that could be installed. However, we do not want to do that because we have a lot of existing code that programmatically modifies the network configuration, and all this code is written against the old Debian-style network configuration files (/etc/network, etc.). Switching to NetworkManager will render all this code non-functional, and it will all need to be rewritten.

I appreciate the thoughts and ideas, but if the best-practice solution to the problem of "How do I get systemd to set my clock early" is: "Oh if you want to set your clock early then first you have to switch to NetworkManager and rewrite all your network config code," then I guess I'll just figure something else out. Surely there's a better answer?

No, I'm not saying that at all. I gave you what I believe is need to allow things to boot in the order you want them to.

Sure and I did appreciate the suggestion, but as I've already said in my previous reply, adding "Requires=hwclock.service" to networking.service and/or ifup@.service doesn't change the behavior of systemd. It still runs the two networking services while the hwclock.service is finishing.

I'm certainly no expert, but isn't what the OP is asking for done with After?
In the ifup and/or networking service(s)
...to get it/them to wait for the hwclock.service.

Here's a page I found that might help you track down your issues.

How to Manage ‘Systemd’ Services and Units Using ‘Systemctl’ in Linux