[SOLVED] How to get systemd run a script AFTER network is up

First off, I already looked on the forum (and elsewhere) for answers. None seem to reliably work for me.

I want to run a script at boot on an Armbian install. Since Armbian uses systemd, I made a Unit file for said script. But no matter what I try, the script seems to run before the network is up, causing it to fail.

Here's an example of what my unit file looks like:

[Unit]
 Description=Run my script
 ConditionPathExists=/etc/init.d/script.sh
 After=network-online.target
 Wants=network-online.target

[Service]
 Type=oneshot
 ExecStart=/etc/init.d/script.sh
 TimeoutSec=0
 StandardOutput=tty
 RemainAfterExit=yes

[Install]
 WantedBy=multi-user.target

I can tell from systemctl status script.service that the script runs at boot, but it fails to reach its destinations on the network.
I tried: After=network.target, network-online.target, NetworkManager-wait-online.service, default.target

Also tried: Type=oneshot, Idle, Forking

It seems no matter what I try, it will always start before the network is up. It did work *once* when I used Type=Idle, but it failed again on the next reboot ...

I tried various "solutions" which I found online, but none seem to work.

systemd passes any sysv initscripts it finds through a generator. Maybe your initial script is getting run before your unit - try moving the script somewhere else.

The script executed by a systemd service should definitely not reside in /etc/init.d.

Then you need to ask yourself what "network up" you require.
When the network is up, doesn't necessarily mean it's connected to the internet.
When it's connected to the internet, doesn't necessarily mean it can resolve addresses.

Best bet is to add a waiting loop to your script that tries to ping the address you need to reach and continues only when that is possible.

Thanks for the replies.
"Network up" for me means the ethernet interface is up and has an ip address. In this particular case, I don't need name resolution.
After posting the above, I found out there really isn't a systemd solution to having something execute last. I found out through this bug report from 2017 (which was closed without a real solution, the "solution" being listing it under "incompatibilities" on a webpage): https://github.com/systemd/systemd/issues/7703

In the replies to the bug report:

"systemd doesn't really have any way to ensure that something executes after everything else, without explicitly defining "everything else".", and

"Looking at code we again have usual case of (intentionally?) incomplete documentation. idle is not really "delayed until all jobs are dispatched" as documentation makes users to believe; it is delayed up to the fixed non-configurable timeout which is hard-coded at 5 seconds (it is up to 6 seconds in total due to implementation).",

and then this from Lennart Poettering:

"So this is indeed one place where we simply aren't compatible with SysV's definition, because we cannot provide conceptually for a "last service" in our boot process, since we are heavily parallelized, event- and request-based. We provide similar functionality, but not with the exact same semantics, and that's simply something we have to accept and document.

I have added this to this list now:

https://www.freedesktop.org/wiki/Sof...mpatibilities/

And I figure we should add something similar to the rc-local-generator man page.

Or in other words, I figure this issue is a documentation issue, and not something we can or should conceptually fix."

Saying it's semantics, what a joke. "we cannot provide conceptually for a "last service" in our boot process" is Poettering-speak for "I don't want to". Because surely it can be done. The more I learn about systemd, the more I understand the haters. Anyway ...

I "solved" it by using a "latish" target in the Unit file, i.e. After=basic.target
Then I looked at how many seconds before boot was complete my script ran (still failing), which was about 1.5 seconds. Then I added 'sleep 2' at the top of my script. Not happy with the solution, but it works for now.

Hope this helps someone running into similar issues.

Originally Posted by jem777 First off, I already looked on the forum (and elsewhere) for answers. None seem to reliably work for me. I want to run a script at boot on an Armbian install. Since Armbian uses systemd, I made a Unit file for said script. But no matter what I try, the script seems to run before the network is up, causing it to fail.

Have you seen the solutions that purport to run something after everything else has been done by systemd?

Some of these are (if memory serves) are called something like "run last" and they are supposed to be a solution to systemd not running its "rc-local" service last as SysV's init would do. The one I tinkered with was supposed to work by creating a new target for the system replacing "multiuser" and "graphical" as final target for systemd. Sadly, I never got any of these to work. (It was a late night effort and I bailed worried that being tired may have introduced an error that broke things in a big way.) If you want to try something like this, I'd suggest making two versions: one to be the last step for each of the traditional target: "last_multiuser" and "last_graphical", perhaps, to allow better control when booting to different "levels" (given my old-timer-ness, maybe "last3" and "last5" would be better choices. :^D )

I'd try a search for those type of solutions and see if you can make one of those work.

Pity that we have to re-engineer the wheel because systemd doesn't understand the purpose of "rc.local" but here we are.

HTH...

Originally Posted by jem777 and then this from Lennart Poettering: "So this is indeed one place where we simply aren't compatible with SysV's definition, because we cannot provide conceptually for a "last service" in our boot process, since we are heavily parallelized, event- and request-based. We provide similar functionality, but not with the exact same semantics, and that's simply something we have to accept and document.

If he actually said that [exaggerated eye-roll], it appears to me that even he doesn't really even understand how systemd works given that others have been able to engineer a kludge of a mechanism for running tasks after everything else has been completed (in parallel or not)[1]. The rc-local service could have been the second-to-last step in the boot process where all the parallelism completed before proceeding to multiuser or graphical. Someone needs to present a paper at a Linux conference about these solutions that others have come up with and hope that his manager at Red Hat is in attendance.

[1] -- But given how frequently systemd gets patched/updated, it makes me wonder how often these custom mechanisms are breaking.

Thanks for the responses rnturn.

The one I tinkered with was supposed to work by creating a new target for the system replacing "multiuser" and "graphical" as final target for systemd.

I was thinking along these lines as a solution too. Unfortunately, I ran out of time trying things, I needed to ship this system to a customer, hence my hacky 'solution'. I might give that a go in the future though! If that doesn't work, I might try a systemd-less distro. I already moved to Devuan (basically Debian without systemd) for my firewall. I reinstalled it last year with Ubuntu initially, but ran into a number of bugs in systemd, as well as the horrible netplan, which just didn't want to play nice with the four network interfaces. (After each boot, one of the four, seemingly at random, refused to work). After trying for three whole days, I installed Devuan and had everything up and running within an hour.

If he actually said that

You can see that he did at the link I provided. If you look at the open bug reports for systemd, it's pretty clear they aren't really on top of things. Or they are prioritizing "new features" (i.e. usurping existing Linux functionality) over closing bugs. If you graph the lines of open bugs vs. closed bugs, the open bugs line is steeper, which never bodes well. I think they have around 1200 open bugs at the moment. Which obviously doesn't include this one, since it was closed as a "documentation issue".

Someone needs to present a paper at a Linux conference about these solutions that others have come up with and hope that his manager at Red Hat is in attendance.

That someone will just get accused of being "resistant to change". (I'm somewhat of an oldtimer myself, but I actually do like trying new things, especially if they work well )