[SOLVED] Debugging systemctl start hang

MALDATA · 06-02-2020, 06:07 PM

I have an application that needs to run as a service. So, I threw together a quick service file (myapp.service) that looks like this:

Code:

[Unit]
Description=My application
After=multi-user.target

[Service]
ExecStart=/usr/local/bin/MyApp
Type=simple
User=myuser
WorkingDirectory=/home/myuser
Restart=always
RestartSec=5
TimeoutStartSec=30

[Install]
WantedBy=multi-user.target

When I log in as myuser and run /usr/local/bin/MyApp from /home/myuser, it runs fine.

When I do "systemctl start myapp" however, it hangs until I ctrl-c. It doesn't print any output, and nothing goes to journalctl. It just sits there, well past the 30 second timeout. When I boot the system and SSH into it, the service is always listed as enabled but dead.

Anyone have some tips to debug this?

Here's the added weirdness... without getting too far into the weeds, this system gets created from a disk image. When I take that image and run it, this app and service file work fine! But if I take that image and run it through mender-convert, then the service behaves this way. So, I know the app and the service file are fine... but I have no way of debugging to figure out what systemctl is choking on in the mender image...

pan64 · 06-04-2020, 08:48 AM

I would add some logging into MyApp (into a specific file) and you will see if it was created at all.
Or just use strace on that app, not on systemctl.

MALDATA · 06-04-2020, 09:44 AM

I actually tried going even simpler and removing my application from the situation entirely. Now I just run a script:

Code:

#!/bin/bash

echo "This should go to stdout"
echo "This should go to myfile.txt" > ${HOME}/myfile.txt

The service file is pretty much the same:

Code:

[Unit]
Description=My Application
After=multi-user.target

[Service]
ExecStart=/home/myuser/testscript.sh
Type=simple
User=myuser
WorkingDirectory=/home/myuser
Restart=always
RestartSec=5
StandardOutput=file:/home/myuser/stdout.log
StandardError=file:/home/myuser/stderr.log

[Install]
WantedBy=multi-user.target

Still nothing. Running `systemctl start myapp.service` just sits there. No files get created in /home/myuser, stdout.log, stderr.log, myfile.txt, nothing.

MALDATA · 06-10-2020, 10:34 PM

Hi all,

After doing some digging, I found out the cause, but I'm still kinda surprised that there was no way to get more debugging information except guessing.

Basically, the image creation process ended up changing the owner of a bunch of directories. So, the default debian user became the owner of /, /lib, /lib/systemd, /lib/systemd/system, etc. I would expect that that would either affect all services or none of them, but that was what happened. Just finding those directories and setting them back to being owned by root fixed things up.

But to circle back to the real meat of the issue here... there was no information from systemd that anything was wrong, and trying to start the service just hung. I don't necessarily expect an error message from systemd that says "hey, a bunch of directories have weird ownership," but I'm surprised that there wasn't any output at all and that trying to start that service didn't outright fail. If something has gone haywire with systemd, is there any way to increase the verbosity or get more output, or anything?

JZL240I-U · 06-11-2020, 05:19 AM

Kudos to you for finding that. I think you should mark this thread as solved and possibly start a new thread concerning output / verbosity of systemd...