I have a heavily used file server that I want to restart, then if it requires e2fsck's on any volume to run them after it restarts. The only problem is that the server is rarely rebooted, and they said it might kernel panic because its been so long.

I've heard there's a way to have it go past the kernel panic if it does happen, but I'm not sure how to do that or the other stuff.

If it was a Windows server, I would schedule a shutdown with the force switch, and have the chkdsk's already scheduled for each volume on reboot. But for RHEL, I really don't know.

I'm hoping this can be done, so that way I can have it kick off at say 7am, then when I get in at 8am it will probably be near the end of the e2fsck's so I can see what's going on.

any help/advice/opinions are welcome. thanks.

I don't know about "will kernel panic" if it hasn't been booted in
a long time - sounds like FUD to me.


And ext2/3 have a mechanism to check after xx days or yy reboots (whichever
comes first). That said: if fsck finds something WRONG with the file-system
there's no way to actually get past that in an automation process, and in
reality you wouldn't want that - tedious as it is.

That all said: what makes you think you should reboot the box?



Cheers,
Tink

The server has 10's of GB's of movies and jpgs moved on/off it a month, and we've had file system problems a few times in the past year, so we're going to institute regular reboots of it. I want to have the fsck's run if necessary, do they run by default when the system is rebooted? if so that's what I want. But how do I know if it will run th checks on reboot or not? I just want to avoid having problems like that again, as there's a few TBs of data total so the checks take a long time to complete.

I'm not sure about the kernel panic stuff, but I have seen a server just sit there when you issue a remote reboot because of one, which is what i was trying to avoid.

Hmmmm ... I'll hazard the guess that there must be some other problem with the
machine if it has fs problems just from use.

I think that (you're not very specific in your description) that what you see
is NOT a kernel panic, but one of the instances where there was FS corruption,
and the server needs some TLC.

As for the "automated fsck", have a read of "man tune2fs".

The server hadn't been restarted in over 2 years before the last time we had to restart it due to the fs corruption. not sure if the kernel panic they're talking about happened on another server or this one, maybe people are just paranoid. I'm just more worried of having it restart, then sit idle for an hour before i get in to manually power it down, in the event it did kernel panic.

I see the max mount counts in tune2fs, but I guess I don't understand how to check what mine's set to or how many that counter is at to see if it will do it or not. maybe i'm missing something in the man page

w/o knowing the current count (and me not knowing how to determine it) you
could always just set (temporarily?) max_reboot_count to 1.



Cheers,
Tink

I guess my question is if the server doesn't think it needs one, should i manually force one to check like you say, or is that unnecessary?

I figured out how to view the tune2fs info, here's the info from one of my partitions, so if i'm reading this right since it says "filesystem state: not clean" it will try to do a e2fsck, even though the mount count or date to check hasn't come up yet, right?

Also, when it reboots, will it automatically run the e2fsck and automatically repair all the problems it finds, or do i need to hit Y to get it going or Y every time it finds an error?

Filesystem volume name: <none>
Last mounted on: <not available>
Filesystem UUID: d48bfd94-1e9c-44ee-8ab8-1e4f23bf8de1
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: filetype sparse_super large_file
Default mount options: (none)
Filesystem state: not clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 107446272
Block count: 214887424
Reserved block count: 10744371
Free blocks: 60902910
Free inodes: 107301748
First block: 0
Block size: 4096
Fragment size: 4096
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 16384
Inode blocks per group: 512
Filesystem created: Fri Jun 12 16:22:27 2009
Last mount time: Tue Jan 12 13:30:19 2010
Last write time: Tue Apr 20 11:21:01 2010
Mount count: 2
Maximum mount count: 27
Last checked: Tue Jan 12 13:27:52 2010
Check interval: 15552000 (6 months)
Next check after: Sun Jul 11 14:27:52 2010
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128

I am not familiar with Red Hat, but I don't like the Filesystem state: not clean in the above output.

The longer you leave this, the worse it is going to get. So please get started now.

As the output of tune2fs shows that the filesystem is unclean, it should do a fsck at the next boot regardless. But you say your server hasn't been rebooted in 2Y, and I forget the details of how things worked then (time for an update maybe?).

If it doesn't fsck automatically:

There used to be the option to shutdown -rF to reboot and Force a filesystem check at the next boot, but this has been removed in the last few years (and /but maybe it'll still work for you because you are way out of date).

Now you are supposed to create the (empty) file /forcefsck and then reboot however you like.

Do it like this (as root)

fsck will run at the next boot. It will do its best to fix things up.

You should not have to put a paperweight on the "Y" key for either "solution".

thanks tredegar. Looking in my version of shutdown, the -F still exists so I could use that. I also read somewhere that it will say "not clean" if the volume is mounted as read/write, but it wasn't for RHEL, so not sure that even applies here. I think when I do restart the server, I'll do the shutdown -rF and force the check on all volumes. Glad to know no paperweight is needed :-)

I think my big concern now is if I issue the shutdown -rF remotely, if it kernel panics, the server will sit there until i can manually power down and bring it back up, then start the time consuming disk checks. its always something.

I have never met a kernel that panicked at a fsck. If it does, then things are very seriously broken, and I think you (or your predecessor) should have addressed this some years ago.

I suspect you do not want to do this, but it really is time you "took the bull by the horns" and addressed this out-of-date-server and no-fsck-for-2-YEARS problem.

I suggest you take a backup (do you have any already?) and then start cleaning up.

Meanwhile, your unclean filesystem will be busy messing things up further. The longer you leave it in an unclean state, the worse the mess will become, and the longer it will take you to recover, if that is possible at all.

I wish you luck.

The server has only been up for 98 days since the last reboot. I meant to say the last time it was restarted before that last reboot was over 2 years ago.

What I meant about the kernel panic, was that I have seen where servers have been issued a shutdown -r now, but there has been a kernel panic during the shutdown, so it would just sit there in a mostly shutdown state until it was manually powered down and brought back up. So if i remotely issued the shutdown command, I'd hate for the server to be just sitting like that with no way to tell.

I already have a backup, and all the volumes were fsck'd during the last reboot.

I thought I remember reading somewhere that you could set a timer or something so if it kernel panic'd it would sit at the screen for X seconds then continue with the shutdown/reboot you were doing, but I'm not having any luck finding it yet.