LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Software (https://www.linuxquestions.org/questions/linux-software-2/)
-   -   FSCK past "next check after" date_not running? (https://www.linuxquestions.org/questions/linux-software-2/fsck-past-next-check-after-date_not-running-4175608353/)

LenHoff 06-21-2017 09:18 PM

FSCK past "next check after" date_not running?
 
Using Mint - Cinnamon 18.1., Ext4 on all 'Nix partitions; dual booted w/ Windows (on a 2nd HDD).

I read a good bit on the need (or lack there of) for running fsck if no big system problems. Especially on Ext4 using journaling.
That's one debate. I don't have boot issues or system crashes, but I see a few oddities that could be system related, or not.

1) But I can't tell it's fsck'd the root / partition, since 1st installing LM. That's the last check time tune2fs shows. The time to check again is way past, per tune2fs settings & syslog entries.
Whether I really *need* to run "full" fsck on / partition is a separate matter.

2) I know how to force FSCK to run next boot. The question is whether there's an obvious reason why it's ignoring Tune2fs settings to run on /, at the set time?

3) In current Linux, is it standard that FSCK does "brief" checks on all NON-root partitions, on every boot? If "normal", fine. If not, something's off.
Syslog shows each boot, it does a brief? check of every partition - except / (sdf1). This doesn't take long (whatever it's doing), but seems odd it's not checking the / partition.

I'd appreciate any ideas.

Some info & some puzzling things:
* Tune2fs settings, data for root / device (sdf1) shows it's been > 3 mo, since last check.
* Check interval is set @ 3 mo (7776000s) [was changed ~ 3 days ago, from default of '0' to 3 mo.)

* Actual Time since last check ~ 95 days (8,223,209 sec).
* Maximum mount count set @ -1 (ignore mount count, for running fsck).

* Every boot, a msg flashes maybe 1 sec., "fsckd-cancel-msg: Press Ctrl+c to cancel all file system checks in progress."
* Yet, no evidence that fsck ever checks the root / partition (sdf1, uuid e47b7349...). Not in any logs, not syslog, or fsck: checkfs.txt, checkroot.txt, etc.

* Syslog shows FSCK doing some check on the other 5 partitions,
but NOT on the / partition (sdf1). Syslog reports the / (sdf1) checktime is reached, but doesn't run FSCK on it.
Code:

SYSLOG:
"Jun 20 08:35:01 d5200 systemd-fsck[745]: /dev/sdf8: clean, 48/305216 files, 54355/1220352 blocks"

"[  10.235641] EXT4-fs (sdf1): warning: checktime reached, running e2fsck is recommended"

Below are Tune2fs settings / data:

Code:

sudo tune2fs -l /dev/sdf1
tune2fs 1.42.13 (17-May-2015)
Filesystem volume name:  <none>
Last mounted on:          /
Filesystem UUID:          e47b7349-... [note: this is the correct / partition uuid]
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:        signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:        clean >> [note: this is likely from check on Mar 18, when installed LM]
Errors behavior:          Continue
Filesystem OS type:      Linux
Inode count:              610800
Block count:              2441216
Reserved block count:    122060
Free blocks:              726505
Free inodes:              352902
First block:              0
Block size:              4096
Fragment size:            4096
Reserved GDT blocks:      595
Blocks per group:        32768
Fragments per group:      32768
Inodes per group:        8144
Inode blocks per group:  509
Flex block group size:    16
Filesystem created:      Sat Mar 18 19:59:03 2017
Last mount time:          Wed Jun 21 17:53:01 2017
Last write time:          Wed Jun 21 17:52:58 2017
Mount count:              125
Maximum mount count:      -1
Last checked:            Sat Mar 18 19:59:03 2017
Check interval:          7776000 (3 months)
Next check after:        Fri Jun 16 19:59:03 2017
Lifetime writes:          15 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:                  256
Required extra isize:    28
Desired extra isize:      28
Journal inode:            8
Default directory hash:  half_md4
Directory Hash Seed:      9790e18f-5de8-42c0-a691-7d07811db459
Journal backup:          inode blocks


hydrurga 06-21-2017 10:27 PM

Have a look at this bug report

https://bugzilla.redhat.com/show_bug.cgi?id=1105877

with particular regard to setting broken_system_clock to 0.

syg00 06-21-2017 11:04 PM

The e2fsprogs devs removed the periodic fsck years ago. Checks are forced at mount if necessary, else they are skipped. People seem to want their boots to be fast(er).
Then along came systemd. They should have left well-enough alone.

The short answer is the fsck is always skipped unless over-ridden in the mke2fs.conf. systemd then goes and uses the fstab to decide if fsck should be invoked (from systemd).

hydrurga 06-22-2017 03:28 AM

Quote:

Originally Posted by syg00 (Post 5725432)
The e2fsprogs devs removed the periodic fsck years ago. Checks are forced at mount if necessary, else they are skipped. People seem to want their boots to be fast(er).
Then along came systemd. They should have left well-enough alone.

The short answer is the fsck is always skipped unless over-ridden in the mke2fs.conf. systemd then goes and uses the fstab to decide if fsck should be invoked (from systemd).

Interesting. As far as I can see, however, my use of the tune2fs -c option to check every n boots *does* work (Mint 18.1):

Code:

sudo tune2fs /dev/sda5 -c 10

syg00 06-22-2017 04:23 AM

Quote:

Originally Posted by syg00 (Post 5725432)
... unless over-ridden in the mke2fs.conf.

...

hydrurga 06-22-2017 05:40 AM

Quote:

Originally Posted by syg00 (Post 5725546)
...

Yes, but OP is saying that the tune2fs -i option is being ignored in their specific case, while my observations are that the -c option at least is being respected.

Setting -i or -c does not affect mke2fs.conf, apart from overriding it. The two options set the relevant fields, s_checkinterval and s_max_mnt_count, in the superblock.

tytso 06-22-2017 06:57 AM

The "e2fsprogs devs" changed mke2fs so that the periodic fsck settings are left disabled when an ext4 file system is formatted. Older ext4 file systems or a file system which has the periodic fsck period settings set _should_ be honored.

If this message shows up when you mount the file system:

"[ 10.235641] EXT4-fs (sdf1): warning: checktime reached, running e2fsck is recommended"

That means that either (a) /dev/sdf1 wasn't in /etc/fstab, or the settings in /etc/fstab indicate that fsck should not be run for that file system (the fsck pass number field is set to zero), or (b) the init scripts and/or the systemd unit files are't doing the right thing.

LenHoff 06-23-2017 01:25 AM

OK, thanks all. Very interesting.

The bug linked was from several yrs ago. It mentioned the broken_time flag in e2fsck.conf file - as a possibility.
In Mint 18 (probably / maybe other recent distros), there's not an active e2fsck.conf. There's an example / man file in /usr/share/man, but it's not a populated file, ready to be extracted if e2fsck runs. Just explains options. Not sure which "system time" they're referring to that may be broken, but my current system time seems OK. Date cmd shows same time as clock in system tray.

@syg00 - I see mke2fs.conf & it has a line, "enable_periodic_fsck = 0." Is this the setting you mean being overridden? If what you say is true, that would appear to be disabling periodic checks. Odd, in ALL my searching & reading, I didn't see one thing about periodic chks now disabled by default, or e2fsck.conf no longer used - or anything remotely like what you guys are saying.

I think tune2fs & whatever it controls is messed up. Syslog for today (6/22) - NO record of fsck being run on ANY of the partitions, as it was doing y'day - for all partitions except root (every reboot). But, tune2fs -l /dev/sdf1 (same as used y'day) still has all the same settings, same "last check 3/18/2017." Maybe it only changes the "last checked" if it fsck's root partition?

Next question: What causes "mke2fs.conf to override fsck being skipped?"
Perhaps a Tune2fs setting to run after x time period (not # of boots) has no effect, but a setting to run after x boots does work?

That's the difference in what I did vs. hydrurga. He used "check after x mounts." Also, I changed Tune2fs settings only 3 - 4 days ago. Though it's been > 3mo from "last check," I changed it only 3 days ago, vs. changing it then actually waiting 3 mo. No, shouldn't matter, but I could see something like that triggering a bug.

@ hydrurga - where are the settings held - you speak of, "The two options set the relevant fields, s_checkinterval and s_max_mnt_count, in the superblock ."

@ tytso - yes, I get (got) that notice in Syslog, every reboot y'day - maybe 4 boots. Today, nothing about it. As said above, tune2fs -l still showing last checked over 3 mo. ago, and "next check" = Jun 16. Like I said, kinda messed up.
Below is my fstab file. Looks like it should've FSCK'd on root / (sdf1). Seems to have right settings. Unless I'm misunderstanding, and syslog really shows root mounts(ed) root / before FSCK ran on the other 5 partitions.

From Syslog y'day, I'm not sure if the 1st line (below), w/ "BOOT_IMAGE", indicates it's mounting root; or the 2nd line - that says "REmount Root;
*OR* if the line (way down), after syslog showed all other partitions were mounted, is where it's mounting root - the line says, "EXT4-fs (sdf1): mounted filesystem".
That last line definitely came after mounting other partitions. It's a matter of which entry indicates when root was mounted.

Code:

Jun 20 11:23:44 d5200 kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-53-generic root=UUID=e47b7349-cf47-4a46-9d1e-4c8e0377e29e ro quiet splash
.......

Jun 20 11:23:44 d5200 systemd[1]: Starting Remount Root and Kernel File Systems...
.......

Jun 20 11:23:44 d5200 systemd[1]: Starting File System Check on /dev/disk/by-uuid/f3f0933d-ecb1-44f4-...
Jun 20 11:23:44 d5200 kernel: [    0.016768] Initializing cgroup subsys io
Jun 20 11:23:44 d5200 kernel: [    0.016775] Initializing cgroup subsys memory
Jun 20 11:23:44 d5200 kernel: [    0.016787] Initializing cgroup subsys devices
Jun 20 11:23:44 d5200 kernel: [    0.016792] Initializing cgroup subsys freezer
Jun 20 11:23:44 d5200 kernel: [    0.016796] Initializing cgroup subsys net_cls
Jun 20 11:23:44 d5200 kernel: [    0.016799] Initializing cgroup subsys perf_event
Jun 20 11:23:44 d5200 kernel: [    0.016803] Initializing cgroup subsys net_prio
Jun 20 11:23:44 d5200 systemd[1]: Activated swap /dev/disk/by-uuid/0f973315-83fe-4a8f-8b0a-8dfed35307c6.
.....

Jun 20 11:23:44 d5200 kernel: [    6.786789] EXT4-fs (sdf1): mounted filesystem with ordered data mode. Opts: (null)


Code:

#
# <file system>......................... <mount point>  <type>  <options>      <dump>  <pass>
# / was on /dev/sdf1 during installation
UUID=e47b7349-cf47-...................... /               ext4    errors=remount-ro 0      1
# /home was on /dev/sdf10 during installation
UUID=f61a519e-........................... /home          ext4    defaults        0      2
# /opt was on /dev/sdf9 during installation
UUID=4984c2e2-151b-......................./opt            ext4    defaults        0      2
# /tmp was on /dev/sdf12 during installation
UUID=f3f0933d-........................... /tmp            ext4    defaults        0      2
# /usr/local was on /dev/sdf8 during installation
UUID=899da343-........................... /usr/local      ext4    defaults        0      2
# /var was on /dev/sdf11 during installation
UUID=8f899476-........................... /var            ext4    defaults        0      2
# swap was on /dev/sdf7 during installation
UUID=0f973315-........................... none            swap    sw              0      0


LenHoff 06-23-2017 11:10 PM

I entered Upstart mode from Advanced boot options. FSCK apparently ran on root - I believe the msg stated scheduled check time was past. I didn't select anything or provide options, to begin the check. So apparently it will run in Upstart mode, based on set time since last check.

After it ran, found 2 mentions that it ran on root. Most didn't show results.

1) in /var/run/initramfs/fsck.log (not a place I'd expect to see results logged). It ran w/ preset options.
Code:

Log of fsck -C -a -T -t ext4 /dev/sdf1
Fri Jun 23 15:11:57 2017

/dev/sdf1: recovering journal
/dev/sdf1: clean, 257899/610800 files, 1714712/2441216 blocks


2) in /var/log/boot.log
Code:

Scanning for Btrfs filesystems
/dev/sdf1: recovering journal
/dev/sdf1: clean, 257899/610800 files, 1714712/2441216 blocks.

However, boot.log doesn't show dates - no way to know when it ran.


Syslog had no record of FSCK running on root - after UPstart, but tune2fs list showed "last checked" had changed to Fri Jun 23. A lot of hoop jumping to get it to run and / or see IF it ran.
I've seen no documentation that explains an easier way to do this & see results. The tune2fs doc discusses it, but seems out of date / deprecated.

Maybe it will run automatically or prompt to (force) running it when Linux catches file system errors. If you need to run it manually, a fair waste of time looking for results.

tytso 06-24-2017 06:10 AM

You can dump out the superblock settings using "dumpe2fs -h /dev/sdXX". There are a couple of fields which are displayed that are especially relevant to this discussion:

Code:

Last mount time:          Sat Jun 24 00:22:14 2017 <---- time the file system was last mounted
Last write time:          Sat Jun 24 06:53:53 2017 <---- time the superblock was last modified
Mount count:              0                        <---- how many times the file system has been mounted since the last fsck
Maximum mount count:      -1                      <---- set by tune2fs -c or by mke2fs
Last checked:            Sat Jun 24 06:53:53 2017 <---- time the file system was last checked
Check interval:          0 (<none>)              <---- set by tune2fs -C or by mke2fs

If you check the mke2fs.conf man page, you will see the documentation for enable_periodic_fsck. If this is false (which is now the default) the Maximum mount count and Check Interval fields will be -1 and 0, respectively. If this is true, then mke2fs will set the check interval to 180 days, and the maximum mount count to a random value between 20 and 40.

If these fields are set to an enabled value, then there are two effects. The first is that the kernel will warn (via a syslog) when the file system is mounted and the mount counts and/or the time between the current time and the last fsck time exceeds the maximum mount count and/or check interval. The second is that when e2fsck (aka fsck.ext[234]) is run, usually by the /sbin/fsck program, but I believe systemd directly runs fsck.<fstype> via generated unit files, the e2fsck program will force a full check of the file system instead of just checking to see if the "file system corrupt" flag is set (and only do a full check if the "corrupt" bit is set).

Everything else is up to the init system --- e.g., sysvinit, upstart or systemd. So if you see a kernel warning message, that means the superblock fields are set to enable periodic fsck checks. It also means that your init system is for some reason not running fsck -A (see the fsck man page; this will use fstab to run the fsck.<fstype> for those devices that have a non-zero fsck pass number), or running fsck.<fstype> via some kind of systemd auto-generated unit file. It's actually REALLY BAD if that is true, since if it is not running fsck, then not only will the periodic checks are not getting done, but if the file system gets corrupted, the full fsck check won't get done on the next reboot.

The important thing to remember here is that tune2fs has a very simple job. All it needs to do is to modify the superblock setting. And the kernel will warn if those settings are apparently not being honored by the init system, and fsck will do a full check if necessary. And the init system's responsibility is to run the fsck.ext4 program, either directly, or indirectly via fsck -A.

I know that my Laptop with Debian Stretch (the new Debian stable) is doing things correctly. As for other distributions, and other init systems, your mileage may vary. As the ext4 and e2fsprogs maintainer, I don't control the init system. $DEITY knows the systemd developers don't pay attention to me (and many other kernel developers). So if dumpe2fs shows that periodic fsck's are supposed to be happening, and you see the kernel warnings showing that the file system is getting mounted without the file system check happening --- take it up with your distribution and/or the init system developers. It's not anything I can fix.

tytso 06-24-2017 06:19 AM

Quote:

Originally Posted by LenHoff (Post 5726463)

Maybe it will run automatically or prompt to (force) running it when Linux catches file system errors. If you need to run it manually, a fair waste of time looking for results.

Where the fsck output will be logged is up to the init scripts and/or systemd unit files. Again, not under my control. If you are using systemd, you might want to check if it was logged via journald and possibly accessible via journalctl. If you find journalctl hopeless complicated, and unreliable because it uses a binary format that can get corrupted on a crash --- complain to the systemd developers. I have no control over it. (Note: Debian currently does not enable persistent logs using journald by default; it prefers to stick with syslog and other explicit text log files instead of journald's "advanced" binary log format. You can enable it, though.)

If you don't want to reply on your distribution to save the fsck output, you can enable e2fsck to save its logs in a location of that you specify, by configuring /etc/e2fsck.conf. See the e2fsck.conf man page for the details. It is quite a bit more sophisticated than your distribution logging scheme. If you use it though you will be responsible for deleting or archive old logs over time. (One of the reasons why we don't enable it by default; the other reason is because the distribution usually does take care of this for you.) Also note that if your distribution happens to try to run fsck from the init script, you may need to make sure /etc/e2fsck.conf is installed in the ramdisk image. Again, these are all distribution issues, and what applies for one distribution setup and/or init system setup and/or initramfs setup may be different for another.


All times are GMT -5. The time now is 04:56 PM.