LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices



Reply
 
Search this Thread
Old 03-25-2008, 05:11 PM   #1
Nothsa
LQ Newbie
 
Registered: Nov 2002
Posts: 22

Rep: Reputation: 15
Thousands of "unattached inode" entries freezing fsck.


We have some CentOS systems with ext3 filesystems that (on occasion) experiences long power failures that are longer than the UPS can handle. We run an fsck on the file systems at every boot, and sometimes they will come back on after a power failure, and when fsck runs there are tens of thousands of "unattached inode" entries:

Code:
Inode ##### ref count is 1, should be 2. Fix? yes
Unattached inode ##### Connect to /lost+found? yes
Where ##### is a different number for each entry. (I've set fsck to answer 'yes' to all questions, hence the "yes"es for the two lines, but I have also tried setting it to answer "no" to all questions). The problem is that it will go through about 7000-8000 of these entries, and then freeze, like it's reached some kind of limit and doesn't want to process any more entries. At this point, someone has to reboot it and it goes to another 7000-8000 before it has to be rebooted again. I'm pretty certain that this is not a hard drive fault because this has happened across 20 different systems and 20 different hard drives.

Does anyone have any ideas:
a) what might be causing the problem, and how to get around it, and
b) what I can do to fix/avoid it without any human intervention? Possible change filesystems, or something else?

I don't want any of these unattached files as I am sure they are just temporary files, so I don't have a problem with just dumping them all. I just can't find a way to do that =/
 
Old 03-26-2008, 10:20 AM   #2
tronayne
Senior Member
 
Registered: Oct 2003
Location: Northeastern Michigan, where Carhartt is a Designer Label
Distribution: Slackware 32- & 64-bit Stable
Posts: 3,120

Rep: Reputation: 818Reputation: 818Reputation: 818Reputation: 818Reputation: 818Reputation: 818Reputation: 818
Ouch!

Well, a power fail crash leaves lots of things hanging open and that's pretty much that. You might be able to cut that down by periodic file system sync (just run sync from a cron every so often; sync flushes the file system buffers and that'll cut a lot of that down). This may be your best option without having to do a lot more work.

You might want to try one of the journaling file systems (which means you have to unload everything, reinitialize the partition and reload). I've been using Reiser for some years and have had zero problems with it -- on rare occasions I've had similar outages, the systems came back up clean (of course there is an automagic check on reboot, but I do come up clean). There are other journaling file systems; take a look at http://en.wikipedia.org/wiki/Comparison_of_file_systems for a discussion.

One suggestion, though, is get your UPS to shut down the systems when the batteries are about to go? Most of them will do that.
 
Old 03-26-2008, 11:44 AM   #3
Nothsa
LQ Newbie
 
Registered: Nov 2002
Posts: 22

Original Poster
Rep: Reputation: 15
Thanks for the tips! I didn't know about the sync command, so that might be all I need, but I'll do some reading up on it.

Also, isn't ext3 already a journaling file system?
 
Old 03-26-2008, 01:19 PM   #4
tronayne
Senior Member
 
Registered: Oct 2003
Location: Northeastern Michigan, where Carhartt is a Designer Label
Distribution: Slackware 32- & 64-bit Stable
Posts: 3,120

Rep: Reputation: 818Reputation: 818Reputation: 818Reputation: 818Reputation: 818Reputation: 818Reputation: 818
Well, duh, yeah, it is (cripes I hate getting old).

It does have some advantages and disadvantages, though, discussed at http://en.wikipedia.org/wiki/Ext3

Seems like, if you're getting that many inodes all over the place that you might be doing a whole lot of file creation, updates and the like? Might also be worth a look at how applications are doing things; for example, do applications stay open for a long, long time and do lots and lots of reads and writes to files? Be worth a look at flushing after a or a few writes in the application itself (like a call to fflush()) if you can. I'm talking about stuff users start up and leave running for hours (or sometimes days) -- I've seen more than a few instances of files open in editors for three or four days. Even something as simple as a scheduled reboot at, oh, 0330 on Sunday can alleviate a lot of that nonsense. It doesn't hurt, either, to "bounce" a DBMS server in the middle of the night sometimes -- just stop and restart the DBMS server makes it flush and clean up after itself (things like pending updates to tables, logs, locks, all that stuff).

And, syncing every hour or so can't hurt either.

You can see if sync will help by just logging in as root on a given server, enter sync and hit the return -- it is takes a while (more than a second or two), that tells you you've got a lot of stuff hanging out there. I generally run that in threes (sync;sync;sync); first one goes slow, second and third usually go really, really fast.

Anyway, sorry about being old and dumb and hope some of the above helps a little.
 
Old 05-11-2008, 05:56 PM   #5
archtoad6
Senior Member
 
Registered: Oct 2004
Location: Houston, TX (usa)
Distribution: MEPIS, Debian, Knoppix,
Posts: 4,727
Blog Entries: 15

Rep: Reputation: 231Reputation: 231Reputation: 231
I like the cron sync idea.

If you want to be a little more about how long sync is taking & how much repeating speeds it up, try:
Code:
time sync; time sync; time sync
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Courious "unknown" entries in "netstat" output Sheridan Linux - Networking 5 09-01-2009 10:09 AM
Unattached Inode? Killean Slackware 8 05-17-2006 05:38 PM
FSCK.EXT3 "error allocating inode bitmap" arfon Linux - Software 6 05-15-2006 11:48 AM
Conditional display of "HCL Entries" and "Reviews" ? J.W. LQ Suggestions & Feedback 1 09-01-2005 09:53 AM
"computers that are thousands of times more powerful than those that exist today." rksprst General 12 02-03-2005 08:25 PM


All times are GMT -5. The time now is 10:53 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration