Discussion with all linux champions and guru on How fsck works???

squiz · 08-01-2009, 01:03 AM

Quote:

Originally Posted by vap16oct1984

Yes you are shutdown -h cleanly unmounts the filesystem before powering off. Well let me clear few things.
1)Can you share your scripts with us?
2)Here the problem comes in which system i mean either in remote system or in a system in which you runs the script?
3)In which file system you have write backup data?
4)What are the file system that you are using in both systems?
5)You are taking about fsck fails to the check test on boot up. This happens in which system out of two? And how you know that fsck is not
working while booting?

Well its hard to say much more about this before knowing these facts?
So please let me know all these.

Here is my script, for what it is worth.

Code:

#!/bin/bash

#Day of month
DOM=`'date' "+%d"`

#Day of week. 1 is Monday, 0 is Sunday.
DOW=`'date' "+%w"`

if [ "$DOM" == "01" ]; then
	nice -19 ionice -n7 rsnapshot monthly
fi

if [ "$DOW" == "0" ]; then
	nice -19 ionice -n7 rsnapshot weekly
fi


nice -19 ionice -n7 rsnapshot sync && nice -19 ionice -n7 rsnapshot daily
sync
if [ ! -f /tmp/noautopoweroff ]; then
	/sbin/shutdown -h now
fi

This script is run on the machine on which the backups are being stored (the local machine) and not on the machine to be backed up (the remote machine).

The backed up data is written to an ext4 filesystem.

I'm not sure that it is relevant, but the remote filesystems (containing data to be backed up) are reiserfs. The remote system has lots of small files too.

On the local machine after 30 mounts, fsck checks the integrity of my local filesystem. This integrity check often fails and I have to manually repair the filesystem.

To check this, I just unmounted the filesystem and ran "fsck -C -f <dev>" and got the following:

Code:

.
.
Problem in HTREE directory inode 3178937: node (613) has invalid depth (2)
Problem in HTREE directory inode 3178937: node (613) has bad max hash
Problem in HTREE directory inode 3178937: node (613) not referenced
Problem in HTREE directory inode 3178937: node (614) has invalid depth (2)
Problem in HTREE directory inode 3178937: node (614) has bad max hash
Problem in HTREE directory inode 3178937: node (614) not referenced
Problem in HTREE directory inode 3178937: node (615) has invalid depth (2)
Problem in HTREE directory inode 3178937: node (615) has bad max hash
Problem in HTREE directory inode 3178937: node (615) not referenced
Problem in HTREE directory inode 3178937: node (616) has invalid depth (2)
Problem in HTREE directory inode 3178937: node (616) has bad max hash
Problem in HTREE directory inode 3178937: node (616) not referenced
Invalid HTREE directory inode 3178937 (/.snapshots/daily.1/modolingo/home/sekretariat/.maildir/shared-folders/training/spam/cur).  Clear HTree index<y>? yes

vap16oct1984 · 08-03-2009, 12:48 AM

A power interruption caused an HTREE problem which appears to upset ext3
fsck.

Googling for '"Problem in HTREE directory inode" "has invalid depth"'
produces relatively few hits, and none where fsck dies.

Log below.

Running fsck manually offered the option of clearing 2 HTREE directory
inodes, which then produced a clean file system, which checks OK with
"fsck -f".

Problem in HTREE directory inode 3178937: node (613) has invalid depth (2)
Problem in HTREE directory inode 3178937: node (613) has bad max hash
Problem in HTREE directory inode 3178937: node (613) not referenced
Problem in HTREE directory inode 3178937: node (614) has invalid depth

It just means that the interior nodes in the HTREE were corrupt. If
you give permission to clear the htree index, e2fsck put the inode on
the list of directories that need to have their HTREE indexes rebuilt,
and a "Pass 3A" will rebuild the directory's (or directories') HTREE
indexes. This is similar to what "e2fsck -fD" does, except it only
rebuilds directories whose HTREE indexes were corrupted, instead of
rebuilding and optimize all of the directories in the system.

So if that was the only message you received, and there were no other
reports of damage to the directory, you wouldn't have lost any
directory names. It's in all likelihood "good as new".

squiz · 08-03-2009, 04:28 AM

Quote:

Originally Posted by vap16oct1984

A power interruption caused an HTREE problem which appears to upset ext3
fsck.

I don't think the power interruption should have caused it in my case, as the filesystem was cleanly unmounted.

Quote:

It just means that the interior nodes in the HTREE were corrupt.

Yes, but what can cause that to happen with a cleanly unmounted filesystem?

Quote:

If you give permission to clear the htree index....

How do I set this so that this is done automatically on boot time. I don't want the system to hang on boot up due to such an fsck check failing.

I think the problem might have been to do with the directory having a very large number of files (>20k). I have no idea why this causes problems though.

unSpawn · 08-03-2009, 05:35 AM

Quote:

Originally Posted by vap16oct1984

A power interruption caused an HTREE problem which appears to upset ext3 fsck (..)

Your reply appears to be a copy 'n paste of parts of http://www.linux-archive.org/ext3-us...orruption.html and http://www.linux-archive.org/ext3-us...tory-node.html. What you're trying to accomplish doing that?

Quote:

Originally Posted by squiz

The filesystem to which I write the backup data often fails the fsck test on boot up. Why would this be? Surely "shutdown -h" cleanly unmounts the filesystem before powering off? (..) The backed up data is written to an ext4 filesystem. (..) I don't want the system to hang on boot up due to such an fsck check failing.

Wrt "expected behaviour" AFAIK fsck erroring out is grave enough to warrant human attention so IMNSHO you should't search for ways to override it but investigate instead (symptoms vs cause). Unless backup integrity mean nothing to you (but then again why make backups in the first place?). Does it always fail on fsck? Did you or can you try (another disk in combination with) ext3? And since ext4 is still under development, have you considered getting on the ext4 mailing list?

squiz · 08-04-2009, 07:31 AM

Quote:

Originally Posted by unSpawn

Your reply appears to be a copy 'n paste of parts of http://www.linux-archive.org/ext3-us...orruption.html and http://www.linux-archive.org/ext3-us...tory-node.html. What you're trying to accomplish doing that?

I also noticed that. Strange.

Quote:

Wrt "expected behaviour" AFAIK fsck erroring out is grave enough to warrant human attention so IMNSHO you should't search for ways to override it but investigate instead (symptoms vs cause). Unless backup integrity mean nothing to you (but then again why make backups in the first place?). Does it always fail on fsck? Did you or can you try (another disk in combination with) ext3? And since ext4 is still under development, have you considered getting on the ext4 mailing list?

It seems to be ok now that I have reduced the number of files in this directory. It was also failing occasionally with ext3, I believe. Maybe I will join the mailing list and post this problem.

unSpawn · 08-04-2009, 08:15 AM

Do let us know if you do. Not only because it would be interesting to read what diagnostics they use and what the outcome is, but since ext4 is still under development I'd guess any help by testing the developers can get is welcome.

vap16oct1984 · 08-04-2009, 11:35 PM

Yes i think its all because of the file system you are using .ext4 file system in still under development the ext4 file system, released on December 25, 2008 and its comes with kernal version 2.6.28.

sqiz can you tell me what kernal version you are using in which ext4 file system running.

The ext4 file system is backward compatible with ext3 and ext2, making it possible to mount ext3 and ext2 file systems as ext4. But i am not sure how much its compatible with reiserfs
file system. You are using it in one of your system.

The most important think that is to be noticed is the two file system that you are using ext4 and reiserfs.

karamarisan · 08-05-2009, 01:39 AM

Quote:

Originally Posted by vap16oct1984

But i am not sure how much its compatible with reiserfs
file system. You are using it in one of your system.

The most important think that is to be noticed is the two file system that you are using ext4 and reiserfs.

About as compatible as it is with NTFS, at least from a filesystem structure standpoint. This shouldn't be even relevant; the whole point of POSIX interfaces to files is that you can read and write data from/to wherever and it'll get there fine because the details of the FS are completely hidden behind standard calls. Did you read those links that were provided for you earlier?