LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 09-09-2013, 04:57 AM   #1
BoraxMan
Member
 
Registered: Apr 2010
Posts: 84

Rep: Reputation: 8
Most reliable filesystem?


I have an external 1.5T hard drive that connects via a USB 2 connection.

This hard drive will be used for storing photos, music (midis, mods), text files and generally used for archiving purposes.

I want to know what is the most reliable filesystem to use in this case?

By reliable, I mean

1) Least likely to lose data if not unmounted cleanly

2) Has a good FSCK program which works well in recovering data

3) Doesn't have unexpected bugs or issues which may bite me when using it normally

4) Can cope with potential minor hardware issues without seizing up.

Note: PERFORMANCE IS NOT AN ISSUE! I've web searched and found some recommendations for reliability and speed, but in this case, speed is not a major concern. I wouldn't mind some efficiency.

My suspicion is that it is either EXT3 or EXT4.
 
Old 09-09-2013, 05:31 AM   #2
zhjim
Senior Member
 
Registered: Oct 2004
Distribution: Debian Squeeze x86_64
Posts: 1,436
Blog Entries: 11

Rep: Reputation: 181Reputation: 181
If you want to make extra sure use ext3 over ext4. Also ext4 is stable and working good its realative young compared to ext3. So there might be some bugs. Ext3 is around dunno how long but I'd say that there are no more bugs for this one. Also there might be I guess they are so seldom and random. Else they would have been squashed a long time ago.

Considering feature ext4 does not have so much more. The only I can really think of is the online resize. Which now also is in ext3 according to the man page. ext4 will be a tid faster but as you said speed is not to be concernd.

Beeing unplugged while beeing mounted or not unmounted before hand IS ALWAYS dangerous under linux. No matter what filesystem you have. Data will not get written to the disk directly but only on sync. But I guess this can also be tuned either in the kernel or withtin the filesystem.
 
Old 09-09-2013, 05:56 AM   #3
TobiSGD
Moderator
 
Registered: Dec 2009
Location: Hanover, Germany
Distribution: Gentoo
Posts: 15,407
Blog Entries: 2

Rep: Reputation: 3994Reputation: 3994Reputation: 3994Reputation: 3994Reputation: 3994Reputation: 3994Reputation: 3994Reputation: 3994Reputation: 3994Reputation: 3994Reputation: 3994
Quote:
Originally Posted by BoraxMan View Post
1) Least likely to lose data if not unmounted cleanly
Any filesystem that gives you mount options to disable caching, have a look at mount's sync option. Will seriously affect performance.
Quote:
2) Has a good FSCK program which works well in recovering data
Anything besides btrfs, I would think.
Quote:
3) Doesn't have unexpected bugs or issues which may bite me when using it normally
Impossible to answer by definition. If we would know of any unexpected bugs they wouldn't be unexpected and most likely already fixed. Even filesystems that are seasoned occasionally are bitten by bugs. That is why it is important to have a good backup scheme.
Quote:
4) Can cope with potential minor hardware issues without seizing up.
Hardware issues are always unpredictable, so this is also not easy to answer. A bitflip in the content of a sector that contains data will be unnoticed by any file-system that doesn't use checksums on the content, a bitflip on a metadata sector might be noticed and correct easily by fsck. A unreadable sector (especially on disks with 4K sectors) that contains metadata may seriously disrupt the filesystem so that you loose data. Another reason for having a good backup scheme.
 
Old 09-09-2013, 11:07 AM   #4
DavidMcCann
Senior Member
 
Registered: Jul 2006
Location: London
Distribution: CentOS, Salix
Posts: 2,955

Rep: Reputation: 767Reputation: 767Reputation: 767Reputation: 767Reputation: 767Reputation: 767Reputation: 767
This computer has had ext3 since 2005 and it's still going strong, for what that's worth. (I hope that's not going to be a case of "famous last words"!)
 
Old 09-09-2013, 01:15 PM   #5
cjcox
Member
 
Registered: Jun 2004
Posts: 305

Rep: Reputation: 42
There is no "panacea" filesystem. So.. a filesystems that is dropped abruptly could have corruption issues that are not easily fixed... that's just the chance you take.

With that said, the more complex filesystems, like reiserfs (not use by many anymore) should be avoided. The more complex things are, the easier it is to corrupt.

Ext3 has maturity. So you're probably right on it being the "best bet"... emphasis on "bet", since you should NEVER clip the connection to a contemporary filesystems abruptly. There's a reason why file access protocols exist that people use in place of direct filesystem access. But of course, these are not going to be fast (in general).

Hope you find what you are looking for.
 
Old 09-09-2013, 02:58 PM   #6
jefro
Guru
 
Registered: Mar 2008
Posts: 11,086

Rep: Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362Reputation: 1362
Using a filesystem to secure data isn't a great idea. Use a backup plan instead. Checking a filesystem doesn't repair or return lost data.

I might be tempted to suggest zfs but I'm sure there would be folks who'd disagree. I'd use pools.

I think they have a check utility for btrfs now.

Last edited by jefro; 09-09-2013 at 03:01 PM.
 
Old 09-10-2013, 03:37 AM   #7
BoraxMan
Member
 
Registered: Apr 2010
Posts: 84

Original Poster
Rep: Reputation: 8
Thanks for the responses.

I wanted to ask around, as EXT3 is what I was using, but for some reason results in quite a bit of file fragmentation (the drive is less than 50% capacity so free space isn't an issue). My question really is "Which filesystem/filesystem tools give me the best chance when there IS a problem (due to corruption, faulty/failing hardware etc)". After all, using reliable hardware, clean unmounts, there shouldn't be a difference. It's when there are adverse circumstances (ie, I lost a whole FAT32 partition due to a dodgy IDE ribbon cable) where the issue lies.

The real important stuff (photos of family) are backed up, but the rest is too volumous and for archiving purposes (ie, could be redownloaded, recreated). It's just that everyone focuses on which one gives the best performance and benchmarks for speed, but benchmarks purely for error handling, and reliability in the face of issues seem virtually non existent. It seems odd that people benchmark fsck speed, but not ability to recover.
 
Old 09-10-2013, 04:13 AM   #8
zhjim
Senior Member
 
Registered: Oct 2004
Distribution: Debian Squeeze x86_64
Posts: 1,436
Blog Entries: 11

Rep: Reputation: 181Reputation: 181
Quote:
Originally Posted by BoraxMan View Post
I wanted to ask around, as EXT3 is what I was using, but for some reason results in quite a bit of file fragmentation (the drive is less than 50% capacity so free space isn't an issue). My question really is "Which filesystem/filesystem tools give me the best chance when there IS a problem (due to corruption, faulty/failing hardware etc)". After all, using reliable hardware, clean unmounts, there shouldn't be a difference. It's when there are adverse circumstances (ie, I lost a whole FAT32 partition due to a dodgy IDE ribbon cable) where the issue lies.
To circumvent faulty IDE cables use USB cables . Just kidding
Hardware faults and unproper usage are very hard to be handled by software. Also ext family has the journal which copes with most of read/write error and unplanned outages. But this also is no cure all. Depending on the moment of the process when the fault happens its able to recover the "new" data or it just does not have the new data. Last case is clearly a fault but the rest of the data is not effected by it.
Hardware failures just can't be hold back by software. You just sometimes heave be daring and kick the devils butt.


Quote:
Originally Posted by BoraxMan View Post
The real important stuff (photos of family) are backed up, but the rest is too volumous and for archiving purposes (ie, could be redownloaded, recreated). It's just that everyone focuses on which one gives the best performance and benchmarks for speed, but benchmarks purely for error handling, and reliability in the face of issues seem virtually non existent. It seems odd that people benchmark fsck speed, but not ability to recover.
There are benchmarks which observe if having the journal of a filesystem on another physical device that cleary show that this approach is faster. (Journals for me are error handling processes). Also think about incremental vs full backups. Also it goes into the direction of saving not recovering its just that recovery mostly depends on medium speed which are handle by many a benchmarks.

I guess you are just in a mind swirl to squash every little thing that might come up but as I learned its near to impossible and you just have to rely on the folks writing the filesystem and the users to have it debugged to a point where it is considered "safe". Or you go all out and get a second device and go up the raid mirroring road.
 
Old 09-10-2013, 05:16 AM   #9
BoraxMan
Member
 
Registered: Apr 2010
Posts: 84

Original Poster
Rep: Reputation: 8
Quote:
Originally Posted by zhjim View Post
To circumvent faulty IDE cables use USB cables . Just kidding
Hardware faults and unproper usage are very hard to be handled by software. Also ext family has the journal which copes with most of read/write error and unplanned outages. But this also is no cure all. Depending on the moment of the process when the fault happens its able to recover the "new" data or it just does not have the new data. Last case is clearly a fault but the rest of the data is not effected by it.
Hardware failures just can't be hold back by software. You just sometimes heave be daring and kick the devils butt.



There are benchmarks which observe if having the journal of a filesystem on another physical device that cleary show that this approach is faster. (Journals for me are error handling processes). Also think about incremental vs full backups. Also it goes into the direction of saving not recovering its just that recovery mostly depends on medium speed which are handle by many a benchmarks.

I guess you are just in a mind swirl to squash every little thing that might come up but as I learned its near to impossible and you just have to rely on the folks writing the filesystem and the users to have it debugged to a point where it is considered "safe". Or you go all out and get a second device and go up the raid mirroring road.
It's more that I found the lack of any thorough analysis interesting, and wanted to ask if people had experience or data otherwise. One would think that data recoverability would be as equal of a concern, a file-system speed, but I guess not.

The only thing I found was this,
http://elinux.org/images/2/26/Evalua...ty-ELC2010.pdf

Which seems to indicate, kernel version perhaps matters more.
 
Old 09-10-2013, 06:13 AM   #10
TobiSGD
Moderator
 
Registered: Dec 2009
Location: Hanover, Germany
Distribution: Gentoo
Posts: 15,407
Blog Entries: 2

Rep: Reputation: 3994Reputation: 3994Reputation: 3994Reputation: 3994Reputation: 3994Reputation: 3994Reputation: 3994Reputation: 3994Reputation: 3994Reputation: 3994Reputation: 3994
Quote:
Originally Posted by BoraxMan View Post
One would think that data recoverability would be as equal of a concern, a file-system speed, but I guess not.
For a simple reason: Backups. Data recoverability in case of sudden blackouts is pretty good nowadays, since that case is not so uncommon (I use ext4 and never had a problem with power outages), but there is no reliable way to prevent data loss in case of hardware failure, that is why backups are the accepted solution for this type of error.
1.5TB disks are not so expensive and with USB 3.0 or eSATA connections backup speed is as good as with internal disks, so the size of backups should at least in your case not really be a concern.
 
1 members found this post helpful.
Old 09-10-2013, 07:09 AM   #11
zhjim
Senior Member
 
Registered: Oct 2004
Distribution: Debian Squeeze x86_64
Posts: 1,436
Blog Entries: 11

Rep: Reputation: 181Reputation: 181
Intresting paper you brought up there. It looks promising to say that ext family is "save". Also I'm not quite sure if the numbers for journal fss are only when writing to the journal or if the journal writes to the disk itself. But I guess it's when writing to the journal.

Another thing we have to seperate here is outtake while one is working on something or how data destructs over time. LIke a 10 year old hd lossing some blocks due to whatever. And the data loose over time can not really be helped. Okay one could do some checksum or crc but what happens if the crc or checksum data is in the faulty block? For a superblock of a fs its okay to have it twice or even more times but for normal data?

If in the middle of writing an error occurs it has to be handled by the software which shows the user something went wrong. If the error is due to a power outtake you acknowledge right away knowing your data is lost. Also this sucks big time you know it and can handle it in a many ways. Safety copies every 5 minutes, UPS, etc. So I'd say that is also one big reason why few have analysed recoverability in depth cause it can be quite easily avoided (also sometimes costly). And as the source of the fault can be so many folded that it really is hard to take on every little bit. Just for a show case a loose sata cable and an earthquake. AFter the quake only half the connection remains which might lead to not so easy to predict errors.
 
Old 09-10-2013, 08:00 AM   #12
Shadow_7
Senior Member
 
Registered: Feb 2003
Distribution: debian
Posts: 1,506

Rep: Reputation: 233Reputation: 233Reputation: 233
I corrupted an ext filesystem on a 1TB drive. The USB bus on my old laptop was dodgy and did the damage to the disk in the docking station. Photorec from package testdisk seems to be able to recover some of those files. Although meaningful names on them are not recovered. I only ran it on the disk a short while, because I don't have the space elsewhere for the data that resides on that disk.

The ext2 filesystem is the goto for limited write USB storage because of the wear leveling perks. As far as I know anyway. But ext2 is pre-journaling AFAIK. So fsck on that takes a long long time. There's pros and cons with most filesystems. Ext3 and ext4 adds a journal to an ext2 filesystem AFAIK. Which allows fsck to cut it short and check just the last things touched. Which is much faster and almost unnoticed after a power loss fail. I've had two of those in the past three days. Be we've finally had weather lately.
 
Old 09-10-2013, 08:22 AM   #13
Pastychomper
Member
 
Registered: Sep 2011
Location: Scotland
Distribution: Debian
Posts: 51

Rep: Reputation: 14
Quote:
Originally Posted by BoraxMan View Post
Thanks for the responses.

I wanted to ask around, as EXT3 is what I was using, but for some reason results in quite a bit of file fragmentation (the drive is less than 50% capacity so free space isn't an issue).
That's good going, ext2+ filesystems are not normally prone to fragmentation. I've never managed to get more than about 11% fragmentation, which made no discernable difference in my case, but I understand that even extreme fragmentation doesn't significantly degrade performance on an ext filesystem. It might be more of a problem if the disk is very close to capacity, but even there it should only affect speed, not reliability.
 
Old 09-10-2013, 09:23 AM   #14
PrinceCruise
Member
 
Registered: Aug 2009
Location: /Universe/Earth/India/Pune
Distribution: Slackware64 14.1/Current, CentOS 6.5/7.0
Posts: 713

Rep: Reputation: Disabled
So far I have survived a couple of black outs and unclean unmounts with ext4 since last 3-4 years without any data loss.
On most occasions fsck did its job well.


Regards.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Most reliable filesystem subi211 Linux - General 7 08-01-2013 03:44 AM
Most reliable filesystem for embedded purposes top_hill Linux - Server 1 07-01-2011 09:22 AM
filesystem check :- xfs_check un reliable? khaleel5000 Linux - General 3 05-29-2008 10:53 PM
DISCUSSION: Virtual Filesystem: Building a Linux Filesystem from an Ordinary File mchirico LinuxAnswers Discussion 0 10-28-2004 10:35 PM
Best and most reliable filesystem for "/" and swap destin Slackware 2 04-09-2004 05:31 PM


All times are GMT -5. The time now is 09:24 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration