LinuxQuestions.org
Help answer threads with 0 replies.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Debian
User Name
Password
Debian This forum is for the discussion of Debian Linux.

Notices


Reply
  Search this Thread
Old 03-23-2022, 08:41 AM   #1
pepperslq
LQ Newbie
 
Registered: Feb 2018
Posts: 18

Rep: Reputation: Disabled
How to diagnose systemd start job problems?


Every 10 boots or so, in the bootup messages, there is: "A startup job is running on /dev/sda4", followed by a timer. Then the timer runs out at 1m30s, and it dumps into emergency mode. The only fix I found is to reboot 3 or 4 times, then the message doesn't appear for a while.

/dev/sda4 is /home. I assume the start job is a fsck, but I can't figure out exactly what is happening, or which "services" are running at startup and why it's hanging. And why does it go away after a few reboots?

Anyone know how to diagnose this? I've been searching, but the systemd documentation and nomenclature is beyond confusing.

I tried to make the start job timeout longer, but editing the timeout in /etc/systemd/user.conf had no effect. Is there another place to set the timeout (if that's the problem)?
 
Old 03-23-2022, 02:46 PM   #2
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,837

Rep: Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308Reputation: 7308
did you check the logs? (in /var/log and also in dmesg/journalctl). Did you try to run fsck manually? Did you check the services?
 
Old 04-22-2022, 03:54 AM   #3
tomwest
Member
 
Registered: Aug 2013
Distribution: debian
Posts: 63

Rep: Reputation: Disabled
Maybe running: systemd-analyze blame, might show something in the boot up that the "startup job" runs. I've had this issue on powering off when the same message and time delay of one and a half minutes runs. That was usually I found because the network hadn't quite finished some action. After I came to close the browser and ensure no network interactivity was occurring, then that message and wait didn't reoccur, but it didn't happen at boot up, only at poweroff.
 
Old 04-22-2022, 12:19 PM   #4
jailbait
LQ Guru
 
Registered: Feb 2003
Location: Virginia, USA
Distribution: Debian 12
Posts: 8,337

Rep: Reputation: 548Reputation: 548Reputation: 548Reputation: 548Reputation: 548Reputation: 548
Quote:
Originally Posted by pepperslq View Post
Every 10 boots or so, in the bootup messages, there is: "A startup job is running on /dev/sda4", followed by a timer. Then the timer runs out at 1m30s, and it dumps into emergency mode. The only fix I found is to reboot 3 or 4 times, then the message doesn't appear for a while.

/dev/sda4 is /home. I assume the start job is a fsck, but I can't figure out exactly what is happening, or which "services" are running at startup and why it's hanging. And why does it go away after a few reboots?
I had a similar problem once when I rearranged my partition tree making it more complex.

The problem might be in /etc/fstab. One of the options in /etc/fstab is the pass number which controls the order in which partitions are mounted. As part of mount fsck checks the device/partition for errors. The root device should be 1. Other partitions can be set to 0 to disable checking, 2 to check after the root partition, or 3 to check after the 2 partitions, etc. If the pass numbers are set incorrectly then it is often a matter of timing as to whether the partitions are checked by fsck in the correct order or not. If a partition is checked too early fsck may not be able to find that partition.

Last edited by jailbait; 04-22-2022 at 12:29 PM.
 
Old 04-22-2022, 09:07 PM   #5
tomwest
Member
 
Registered: Aug 2013
Distribution: debian
Posts: 63

Rep: Reputation: Disabled
jailbait wrote:
Quote:
The problem might be in /etc/fstab. One of the options in /etc/fstab is the pass number which controls the order in which partitions are mounted. As part of mount fsck checks the device/partition for errors. The root device should be 1. Other partitions can be set to 0 to disable checking, 2 to check after the root partition, or 3 to check after the 2 partitions, etc. If the pass numbers are set incorrectly then it is often a matter of timing as to whether the partitions are checked by fsck in the correct order or not. If a partition is checked too early fsck may not be able to find that partition.
That's an interesting theory. A few thoughts come to mind. Firstly, we don't know if it's fsck that is holding up the boot. If it is, this "hold up" would likely not occur if the user has only one disk drive, unless there was some anomalous numbering in the sixth field of fstab. The info from the fstab man page is:
Quote:
Filesystems within a drive will be checked sequentially, but filesystems on different drives will be checked at the same time to utilize parallelism available in the hardware.
On one of my machines the fsck is very brief, at 3ms, as shown by the output from the command: systemd-analyze blame:
Code:
755ms systemd-fsck@dev-disk-by\x2duuid-6d2620b\x2d3a4\x2d4ec\x2dab4\x2d97e7300ef0.service
752ms colord.service
If /etc/fstab is stable, and the machine has a single disk, and the problem is intermittent, how likely is it that fsck is the issue? I think more information is needed.

Last edited by tomwest; 04-22-2022 at 09:11 PM.
 
Old 04-23-2022, 10:01 AM   #6
jailbait
LQ Guru
 
Registered: Feb 2003
Location: Virginia, USA
Distribution: Debian 12
Posts: 8,337

Rep: Reputation: 548Reputation: 548Reputation: 548Reputation: 548Reputation: 548Reputation: 548
Quote:
Originally Posted by tomwest View Post
I think more information is needed.
My theory could be checked out by pepperslq posting his /etc/fstab and explaining what file systems pepperslq wants to mount on what directories.
 
Old 04-24-2022, 02:29 AM   #7
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 19,872
Blog Entries: 12

Rep: Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053Reputation: 6053
First of all, pertinent information will have been be logged, and you can use journalctl and/or systemctl to retrieve it.

'journalctl -b' is everything for the current boot.
'journalctl -b -1' is everything for the previous boot.

You need to know what exactly has been happening there.

Quote:
Originally Posted by pepperslq View Post
I tried to make the start job timeout longer, but editing the timeout in /etc/systemd/user.conf had no effect. Is there another place to set the timeout (if that's the problem)?
I don't think the timeout is the problem.
Increasing the timeout will do just that (if it doesn't you have a different problem to solve) - it won't solve the underlying problem.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
systemd start job fails without leaving reason Turbocapitalist Linux - Server 5 10-17-2019 02:29 PM
unkown cause for systemd failure state - systemd-logind cannot start org.freedesktop.systemd1 after server reboot mario.p Linux - Server 5 07-30-2018 02:11 AM
systemd start job Times Out and Fails pmouse Linux - Software 5 05-06-2016 05:20 AM
Can you start a job without getting the job number spit back out? BrianK Linux - General 5 12-13-2007 08:09 PM
Diagnose virus or hack, where do you start? davidar Linux - Security 5 10-16-2006 03:22 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Debian

All times are GMT -5. The time now is 06:14 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration