Question: systemd: Coordinating root and user jobs

haertig · 02-11-2018, 12:27 PM

I have a setup where I use "nginx" webserver to front-end a "Calibre" ebook server. I use nginx to support SSL, and also to require client certs to access Calibre.

I have a primary Calibre instance on one computer that nobody gets to access except me. I have a nightly job that rsyncs my primary Calibre data over to a secondary Calibre instance (this runs on a different computer). It is this second instance that users can access, provided I have given them an appropriate client cert to use for authentication through the nginx front-end.

The daily rsync follows up with a restart of the secondary Calibre instance. This is required to re-index any new data that may have come across.

I have found that restarting the secondary Calibre instance breaks the nginx front-end access to it. The fix is simple - restart nginx. And then everything works fine again.

My issue is that the rsync and Calibre restart systemd stuff runs as userid "calibre". However, the nginx systemd stuff runs as "root".

What I would like to do, if possible in systemd, is have a root job that restarts nginx trigger off of the completion of the user job that does the rsync/Calibre_restart. Is this possible? How?

An alternate, of course, is to run all jobs - the rsync, the Calibre restart, and the nginx restart as root. I can do that. But I would like to know if it is even possible to do this the way I describe above, as a root systemd job that triggers off of a user systemd job.

Thanks!

==================================================

As userid "calibre" (e.g., "systemctl --user ..."):

Code:

[calibre@rpi3b user]$ cat rsync_calibre.timer
[Unit]
Description=Timer to Rsync Calibre books from 10.192.0.2
RefuseManualStart=no
RefuseManualStop=no

[Timer]
Persistent=true
OnBootSec=120
OnCalendar=daily
Unit=rsync_calibre.service

[Install]
WantedBy=timers.target
[calibre@rpi3b user]$

Code:

[calibre@rpi3b user]$ cat rsync_calibre.service
[Unit]
Description=Rsync Calibre books from 10.192.0.2 to /var/opt/calibre/

[Service]
Type=simple
ExecStart=/home/calibre/bin/rsync_calibre_books.sh

[Install]
WantedBy=default.target
[calibre@rpi3b user]$

Code:

[calibre@rpi3b bin]$ cat rsync_calibre_books.sh
#!/usr/bin/bash

/usr/bin/rsync -avzh --password-file=/home/calibre/bin/rsync.calibre.credentials rsync://calibre@10.192.0.2:873/files/ /var/opt/calibre/

/usr/bin/sleep 5

/usr/bin/systemctl --user restart calibre-server
[calibre@rpi3b bin]$

Code:

[calibre@rpi3b user]$ cat calibre-server.service
[Unit]
Description=Calibre content server
After=network.target

[Service]
Type=simple
ExecStart=/home/calibre/bin/start_calibre.sh

[Install]
WantedBy=multi-user.target
[calibre@rpi3b user]$

Code:

[calibre@rpi3b bin]$ cat start_calibre.sh
#!/usr/bin/bash

/usr/bin/calibre-server --url-prefix /calibre --num-per-page 999 --port 8080 --userdb /srv/calibre/users-sqlite --enable-auth /var/opt/calibre
[calibre@rpi3b bin]$

==================================================

As userid "root":

Code:

[root@rpi3b]# cat nginx.service
[Unit]
Description=A high performance web server and a reverse proxy server
After=network.target network-online.target nss-lookup.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
PrivateDevices=yes
SyslogLevel=err

ExecStart=/usr/bin/nginx -g 'pid /run/nginx.pid; error_log stderr;'
ExecReload=/usr/bin/nginx -s reload
KillSignal=SIGQUIT
KillMode=mixed

[Install]
WantedBy=multi-user.target
[root@rpi3b]#

MensaWater · 02-13-2018, 02:39 PM

Is your "nightly job" in cron? If so is it in the crontab for the calibre user?

Have you thought of moving it from the calbre user's crontab to instead be in the root user's crontab and modify the job so it runs the resync as calibre (su - calibre -C <commands>) then do the restart of nginx without the su?

haertig · 02-13-2018, 03:52 PM

Quote:

Originally Posted by MensaWater

Is your "nightly job" in cron?

Thanks for the reply.

I'm actually using "systemd timers" instead of cron. But I could do the same thing as you suggest (functionally) in systemd. Run everything as "root" with selective su's to "calibre".

Since I am less knowledgeable in systemd than I am in cron, that's why I decided to use systemd. To practice and learn more about it. A good brain exercise. cron is definitely simpler, systemd is more flexible/configurable. All that code I posted above, in my initial post, is systemd configuration. You can see it's nowhere near as simple as cron!

That's why I was asking about how to do this job coordination using systemd. So I could increase my knowledge (decrease my ignorance!)

My question was not really "I don't know any way to accomplish this task by any means". I do. My question was more like "I want to learn how to accomplish this task in a new way (using systemd). Can anyone give me any pointers?"

( I view systemd vs. cron similarly to how I view legacy grub vs. grub2. I can do everything I need in legacy grub (cron), but I learned grub2 (systemd) so I wouldn't fall too far behind on the technology curve. systemd is a lot more than just a cron replacement. It's an init replacement, and all kinds of other stuff too. )

ondoho · 02-14-2018, 01:02 AM

fwiw, archwiki has been very helpful in systemd matters:
https://wiki.archlinux.org/index.php/Systemd/User
hope it helps.

MensaWater · 02-14-2018, 12:12 PM

So I hadn't been aware of systemd timers as an alternative to cron. On our systemd based servers we've continued to use cron.

I just asked a co-worker about this and he said he'd played with timers but didn't like them. Of course that's an opinion and I found others on the internet.

Also I found mention of the need to specify user with in the unit file but see you using "systemctl --user". On my RHEL7 system I found that "--user" flag doesn't work at all which led to this from RedHat saying they didn't implement it on RHEL7 because it was listed as not stable and they weren't sure it wouldn't be removed later:
https://bugzilla.redhat.com/show_bug.cgi?id=1198655

All that leads to the question: Which distro and version of Linux are you using?

haertig · 02-14-2018, 12:57 PM

I am running this on an Arch ARM installation (Raspberry Pi3 hardware).

The command that reportedly does not work correctly in the RedHat bug report you mentioned, works just fine under Arch. See below. Note that you have to manually enable the systemd stuff to run in the user environment ... it is not enabled by default. It is only enabled by default in the root environment.

Code:

[calibre@rpi3b ~]$ uname -a
Linux rpi3b 4.14.13-1-ARCH #1 SMP Wed Jan 10 18:35:39 MST 2018 aarch64 GNU/Linux
[calibre@rpi3b ~]$

Code:

[calibre@rpi3b ~]$ systemctl --user status
● rpi3b
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Mon 2018-02-12 20:44:12 MST; 1 day 15h ago
   CGroup: /user.slice/user-1001.slice/user@1001.service
           ├─calibre-server.service
           │ ├─1131 /usr/bin/bash /home/calibre/bin/start_calibre.sh
           │ └─1132 python2 /usr/bin/calibre-server --url-prefix /calibre --port 8080 --userdb /srv/calibre/users-sqlite --enable-auth /var/opt/calibre
           └─init.scope
             ├─328 /usr/lib/systemd/systemd --user
             └─331 (sd-pam)
[calibre@rpi3b ~]$

Code:

[calibre@rpi3b ~]$ systemctl --user
UNIT                                                                                         LOAD   ACTIVE SUB       DESCRIPTION                                                                           
sys-devices-platform-serial8250-tty-ttyS0.device                                             loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS0                                            
sys-devices-platform-serial8250-tty-ttyS2.device                                             loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS2                                            
sys-devices-platform-serial8250-tty-ttyS3.device                                             loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS3                                            
sys-devices-platform-soc-3f201000.serial-tty-ttyAMA0.device                                  loaded active plugged   /sys/devices/platform/soc/3f201000.serial/tty/ttyAMA0                                 
sys-devices-platform-soc-3f202000.mmc-mmc_host-mmc0-mmc0:aaaa-block-mmcblk0-mmcblk0p1.device loaded active plugged   /sys/devices/platform/soc/3f202000.mmc/mmc_host/mmc0/mmc0:aaaa/block/mmcblk0/mmcblk0p1
sys-devices-platform-soc-3f202000.mmc-mmc_host-mmc0-mmc0:aaaa-block-mmcblk0-mmcblk0p2.device loaded active plugged   /sys/devices/platform/soc/3f202000.mmc/mmc_host/mmc0/mmc0:aaaa/block/mmcblk0/mmcblk0p2
sys-devices-platform-soc-3f202000.mmc-mmc_host-mmc0-mmc0:aaaa-block-mmcblk0.device           loaded active plugged   /sys/devices/platform/soc/3f202000.mmc/mmc_host/mmc0/mmc0:aaaa/block/mmcblk0          
sys-devices-platform-soc-3f215040.serial-tty-ttyS1.device                                    loaded active plugged   /sys/devices/platform/soc/3f215040.serial/tty/ttyS1                                   
sys-devices-platform-soc-3f300000.sdhci-mmc_host-mmc1-mmc1:0001-mmc1:0001:1-net-wlan0.device loaded active plugged   /sys/devices/platform/soc/3f300000.sdhci/mmc_host/mmc1/mmc1:0001/mmc1:0001:1/net/wlan0
sys-devices-platform-soc-3f902000.hdmi-sound-card0.device                                    loaded active plugged   /sys/devices/platform/soc/3f902000.hdmi/sound/card0                                   
sys-devices-platform-soc-3f980000.usb-usb1-1\x2d1-1\x2d1.1-1\x2d1.1:1.0-net-eth0.device      loaded active plugged   SMSC9512/9514 Fast Ethernet Adapter                                                   
sys-devices-virtual-misc-rfkill.device                                                       loaded active plugged   /sys/devices/virtual/misc/rfkill                                                      
sys-module-configfs.device                                                                   loaded active plugged   /sys/module/configfs                                                                  
sys-subsystem-net-devices-eth0.device                                                        loaded active plugged   SMSC9512/9514 Fast Ethernet Adapter                                                   
sys-subsystem-net-devices-wlan0.device                                                       loaded active plugged   /sys/subsystem/net/devices/wlan0                                                      
-.mount                                                                                      loaded active mounted   Root Mount                                                                            
boot.mount                                                                                   loaded active mounted   /boot                                                                                 
dev-hugepages.mount                                                                          loaded active mounted   /dev/hugepages                                                                        
dev-mqueue.mount                                                                             loaded active mounted   /dev/mqueue                                                                           
run-user-1001.mount                                                                          loaded active mounted   /run/user/1001                                                                        
sys-kernel-config.mount                                                                      loaded active mounted   /sys/kernel/config                                                                    
sys-kernel-debug.mount                                                                       loaded active mounted   /sys/kernel/debug                                                                     
tmp.mount                                                                                    loaded active mounted   /tmp                                                                                  
init.scope                                                                                   loaded active running   System and Service Manager                                                            
calibre-server.service                                                                       loaded active running   Calibre content server                                                                
-.slice                                                                                      loaded active active    Root Slice                                                                            
dbus.socket                                                                                  loaded active listening D-Bus User Message Bus Socket                                                         
dirmngr.socket                                                                               loaded active listening GnuPG network certificate management daemon                                           
gpg-agent-browser.socket                                                                     loaded active listening GnuPG cryptographic agent and passphrase cache (access for web browsers)              
gpg-agent-extra.socket                                                                       loaded active listening GnuPG cryptographic agent and passphrase cache (restricted)                           
gpg-agent-ssh.socket                                                                         loaded active listening GnuPG cryptographic agent (ssh-agent emulation)                                       
gpg-agent.socket                                                                             loaded active listening GnuPG cryptographic agent and passphrase cache                                        
basic.target                                                                                 loaded active active    Basic System                                                                          
default.target                                                                               loaded active active    Default                                                                               
paths.target                                                                                 loaded active active    Paths                                                                                 
sockets.target                                                                               loaded active active    Sockets                                                                               
timers.target                                                                                loaded active active    Timers                                                                                
rsync_calibre.timer                                                                          loaded active waiting   Timer to Rsync Calibre books from 10.192.0.2                                          

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

38 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
[calibre@rpi3b ~]$

haertig · 02-14-2018, 01:18 PM

Quote:

Originally Posted by ondoho

fwiw, archwiki has been very helpful in systemd matters...

Thanks. Actually the Arch wiki is helpful for just about everything! Even if you are running some distro other than Arch. Some of the best, if not THE best, documentation out there for Linux. I've been using Arch's wiki for quite a while. Even before I started running Arch. I think Arch ARM is perfect for less-than-powerful hardware like the Raspberry Pi3 that I am running this application on. You can make your system as light as you need it to be without having to jump through a bunch of hoops, like if you were using Linux From Scratch. Sure, you could potentially get even lighter with LFS, but the time/effort that would take is a little long - unless your goal is to learn LFS (not a bad goal to have, however!) Arch is really good. I have two GoTo Linux distros: For a desktop system that can be configured and up and running with most everything you could ever need very quickly (or for a system for beginners), I go with LinuxMint. For servers, and for low-power hardware, I go with Arch (sometimes choosing Slackware as an alternate). I've tested/played with most everything else over the years, but have settled on these two as my GoTo's.

Some old sayings hold very true: If you want highly technical/competent help with Linux, any Linux distro, ask a Slackware guru. If you want good documentation, look to the Arch stuff. If you want highly active forums with lots of places to find answers to simple and/or beginner questions, go with Ubuntu (or something built on top of Ubuntu, like LinuxMint).

MensaWater · 02-14-2018, 02:38 PM

Quote:

Originally Posted by haertig

Note that you have to manually enable the systemd stuff to run in the user environment ... it is not enabled by default. It is only enabled by default in the root environment.

Yep but the bug report for RHEL7 says they (RedHat) don't and won't allow it to be enabled for users. I just noted it as it might be possible other distros/versions made similar decisions (or derivatives like CentOS, Scientific Linux & OEL would have inherited the decision).

I wasn't suggesting Arch didn't support it. I was just asking which distro/version you were using which I now know is Arch so the Arch Wiki is going to be more helpful than I am.