systemd unit does not load at startup. manual start shows no such file/dir failure
 

On Debian Jessie.

systemd unit does not load at startup. attempting to manually start the service results in:

but

If
is run, then the service starts fine (when started manually).

The unit file:
Any help would be appreciated. Please let me know if any more information would be helpful.

P.S. The machine is set up to automatically log in as a (non-privileged) user in graphical mode.

Sounds similar to a dependency requirement not specified...

Very likely it will need an After (or Wants, or maybe Requires - as in "Requires=local-fs.target") entry related to the filesystem.

Assuming the filesystem is local maybe the "Requires=local-fs.target" would work. If it is NFS then "Requires=remote-fs.target" might do it.

Thank you, jpollard, for your reply. I've tried adding "Requires=local-fs.target" without any luck.

As a workaround, removing the symbolic link from /etc/systemd/system/graphical.target.wants/ and copying the actual service file from /usr/lib/systemd/system/ worked. Not sure why the workaround should be necessary considering the other service (accounts-daemon) is up and running and is a symbolic link to the actual file. Any suggestions on how to get the service running using the symbolic links that systemctl (re)enable uses?

There's some pieces missing from your original post which may or may not help:
- with what commands did you install and enable STAFProc.service?
- what does 'stat /usr/lib/systemd/system/STAFProc.service /usr/local/staf/STAFEnv.sh /usr/local/staf/startSTAFProc.sh' return?
- when 'systemctl start STAFProc' does not start, what are all the lines systemctl returns? (Same for using the "status" argument.)
- when 'systemctl start STAFProc' does not start, what are all the lines 'journalctl -xn' returns with respect to this service?
- are there any related lines in /var/log/{messages,secure} or any other log file?
- since /usr/local/staf/startSTAFProc.sh sources /usr/local/staf/STAFEnv.sh always, does incorporating that into /usr/local/staf/startSTAFProc.sh help?
Finally, please describe in detail what startSTAFProc.sh and STAFEnv.sh do or just list (or attach) their full contents if unsure.

There's some pieces missing from your original post which may or may not help:

• with what commands did you install and enable STAFProc.service?
  
  Code:

  ---

  # systemctl enable STAFProc

  ---

• what does 'stat /usr/lib/systemd/system/STAFProc.service /usr/local/staf/STAFEnv.sh /usr/local/staf/startSTAFProc.sh' return?
  
  Code:

  ---

    File: ‘/usr/lib/systemd/system/STAFProc.service’
  Size: 436            Blocks: 8          IO Block: 4096  regular file
Device: 807h/2055d      Inode: 277719      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)  Gid: (    0/    root)
Access: 2015-07-12 19:29:18.834799802 -0700
Modify: 2015-07-12 19:28:47.550800262 -0700
Change: 2015-07-12 19:28:47.574800262 -0700
 Birth: -
  File: ‘/usr/local/staf/STAFEnv.sh’
  Size: 545            Blocks: 8          IO Block: 4096  regular file
Device: 80eh/2062d      Inode: 434178      Links: 1
Access: (0777/-rwxrwxrwx)  Uid: (    0/    root)  Gid: (  50/  staff)
Access: 2015-07-12 04:03:47.640947304 -0700
Modify: 2015-07-12 03:53:13.270430560 -0700
Change: 2015-07-12 03:53:13.274430560 -0700
 Birth: -
  File: ‘/usr/local/staf/startSTAFProc.sh’
  Size: 196            Blocks: 8          IO Block: 4096  regular file
Device: 80eh/2062d      Inode: 434179      Links: 1
Access: (0777/-rwxrwxrwx)  Uid: (    0/    root)  Gid: (  50/  staff)
Access: 2015-07-12 04:03:47.632947305 -0700
Modify: 2015-07-12 03:53:13.274430560 -0700
Change: 2015-07-12 03:53:13.278430560 -0700
 Birth: -

  ---

• when 'systemctl start STAFProc' does not start, what are all the lines systemctl returns? (Same for using the "status" argument.)
  
  Code:

  ---

  # systemctl start STAFProc
Failed to start STAFProc.service: Unit STAFProc.service failed to load: No such file or directory.
# systemctl -o verbose -l status STAFProc
● STAFProc.service
  Loaded: not-found (Reason: No such file or directory)
  Active: inactive (dead)

  ---

• when 'systemctl start STAFProc' does not start, what are all the lines 'journalctl -xn' returns with respect to this service?
  
  Code:

  ---

  # journalctl -xn
-- Logs begin at Mon 2015-07-13 02:25:31 PDT, end at Mon 2015-07-13 02:27:30 PDT
Jul 13 02:27:10 debian systemd[2187]: Starting Basic System.
-- Subject: Unit UNIT has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit UNIT has begun starting up.
Jul 13 02:27:10 debian systemd[2187]: Reached target Basic System.
-- Subject: Unit UNIT has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit UNIT has finished starting up.
--
-- The start-up result is done.
Jul 13 02:27:10 debian systemd[2187]: Starting Default.
-- Subject: Unit UNIT has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit UNIT has begun starting up.
Jul 13 02:27:10 debian systemd[2187]: Reached target Default.
-- Subject: Unit UNIT has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit UNIT has finished starting up.
--
-- The start-up result is done.
Jul 13 02:27:10 debian systemd[2187]: Startup finished in 105ms.
-- Subject: System start-up is now complete
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- All system services necessary queued for starting at boot have been
-- successfully started. Note that this does not mean that the machine is
-- now idle as services might still be busy with completing start-up.
--
-- Kernel start-up required KERNEL_USEC microseconds.
--
-- Initial RAM disk start-up required INITRD_USEC microseconds.
--
-- Userspace start-up required 105720 microseconds.

  ---

• are there any related lines in /var/log/{messages,secure} or any other log file?
  - nothing in messages. Relevant lines from daemon.log:
  
  Code:

  ---

  # grep STAF daemon.log
:Jul 13 02:23:18 debian sh[856]: 20150713-02:23:18;3072780032;00000100;Caught STAFException in JSTAF.STAFServiceTerm(), Service: STAX, Exception: STAFConnectionIOException, Location: /opt/dev/autobuild/build/src/staf/stafif/STAFConnectionProviderInlImpl.cpp(280), Text: STAFConnectionReadUInt: Error reading from socket: other side closed socket, Error code: 22
daemon.log:Jul 13 02:23:18 debian sh[856]: 20150713-02:23:18;3072780032;00000100;Error terminating service, STAX, RC: 6, Result: Error terminating service, JSTAF, Result:

  ---

  Code:

  ---

  # dmesg | grep STAF
[    5.939139] systemd[1]: Cannot add dependency job for unit STAFProc.service, ignoring: Unit STAFProc.service failed to load: No such file or directory.
[    5.939277] systemd[1]: Cannot add dependency job for unit STAFProc.service, ignoring: Unit STAFProc.service failed to load: No such file or directory.

  ---

  in the preceding line:
  
  Code:

  ---

  [    4.922790] systemd[1]: /usr appears to be on its own filesytem and is not al
ready mounted. This is not a supported setup. Some things will probably break (s
ometimes even silently) in mysterious ways. Consult http://freedesktop.org/wiki/
Software/systemd/separate-usr-is-broken for more information.

  ---

  Code:

  ---

  # grep STAF syslog
Jul 13 02:23:17 debian sh[856]: STAFProc ending normally
Jul 13 02:23:18 debian sh[856]: 20150713-02:23:18;3072780032;00000100;Caught STAFException in JSTAF.STAFServiceTerm(), Service: STAX, Exception: STAFConnectionIOException, Location: /opt/dev/autobuild/build/src/staf/stafif/STAFConnectionProviderInlImpl.cpp(280), Text: STAFConnectionReadUInt: Error reading from socket: other side closed socket, Error code: 22
Jul 13 02:23:18 debian sh[856]: 20150713-02:23:18;3072780032;00000100;Error terminating service, STAX, RC: 6, Result: Error terminating service, JSTAF, Result:

  ---

• since /usr/local/staf/startSTAFProc.sh sources /usr/local/staf/STAFEnv.sh always, does incorporating that into /usr/local/staf/startSTAFProc.sh help?
  - startSTAFProc.sh already sources STAFEnv.sh :o, removed the extra source command.
• Finally, please describe in detail what startSTAFProc.sh and STAFEnv.sh do or just list (or attach) their full contents if unsure.
  - STAFEnv.sh just sets some environment variables so that startSTAFProc.sh can find everything.
  - startSTAFProc.sh sources STAFEnv.sh and then runs nohup /usr/local/staf/bin/STAFProc &
  - STAFProc (the process that gets started in the background) listens for requests and executes any commands received. example:
  Code:

  ---

  # staf 10.0.0.3 ping ping
Response
--------
PONG

  ---

  sends a request to STAFProc which in turn executes the ping command (part of the ping service, hence the "ping ping") to target-machine.

Thanks. So there's: 0) uncertaintly if /usr is mounted when this service runs, 1) some error terminating STAX right as it starts, and 2) a "Cannot add dependency job for unit" error. #0 could be fixed by not using a separate /usr (or somehow ensuring it gets mounted before any service runs? Don't ask me.), #1 I don't know the cause and #2 could be countered adding some "Wants=" and "After=" stanzas to the service file to ensure Systemd gets your dependency ordering. Also what happens if you fill in the env vars and run Execstart like this?:

Hi I got it running after reinstalling the OS without the separate /usr (0). (1) is no longer a problem. And (2) I tried messing with Wants and Afters (before the reinstall, without any luck).

Instead of
I tried setting variables using
(also without any luck).

Anyway, aside from not having the separate /usr, I moved the unit file to /etc/systemd/system (where "local" unit files are supposed to go anyway; /usr/lib/systemd/system is for unit files from "installed packages"). Given the workaround (removing the symbolic link from /etc/systemd/system/graphical.target.wants/ and copying the actual service file from /usr/lib/systemd/system/), I'd guess that this was the correct fix.

Thanks jpollard and unSpawn for your help.

I think that you should only have "multi-user.target" in the [Install] section as this is reached *before* graphical.target and having both may cause a conflict or race condition,

Documentation indicates that more than one may be specified. Only one target is activated at boot (in my case graphical.target is the default) and the corresponding target.wants are run.