LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 01-31-2011, 11:35 AM   #1
King_DuckZ
Member
 
Registered: Nov 2009
Location: Rome, IT
Distribution: Sabayon
Posts: 61

Rep: Reputation: 2
Filesystem for flash memories


Hello everyone, I've been looking on the internet on what filesystem is best suited for flash memories like USB keys, mmc, SD cards and the likes, but I'm not feeling quite satisfied with the answers I got.

There are two use cases I'm dealing with:
1) I have an SD card that I want to use for storing some data I'm working on. I need to preserve file permissions, and I need to maximize seek and read speed, but the filesystem is accessed for writes quite often too.
2) I want to install a linux distribution on a USB key

Now, I've been reading around that said memories are block devices, and that I should stick to regular filesystems. I went for ext2 for the first case, but I experienced data loss on a brand new PNY SD card, which is unacceptable.
Also, I suspect ext2 is striving for writing files consecutively or in some other harddisk-optimized way, which looks like a complete waste to me.
In addition, other filesystems would flood the memory with journal crap.

I looked into JFFS2, UBIFS, LogFS and here's what I came up with (some statements are contraddictory):
- journaling might shorten the memory's life, and it will definitely slow things down
- wear leveling is always hardware-implemented, and a software wear leveling could cancel out with the hardware's effort
- hardware wear leveling has often a cheap implementation, and a better software wear leveling is desirable
- JFFS2 doesn't store any index on the device; this comes at the cost of a slow mount but the flash memory is accessed less often
- LogFS should work well on memories up to 64mb
- MTD-oriented filesystems can be used on block devices
- ext3 = ext2 + journaling
- ext3 is not just ext2 with journaling

So, is there anyone who can add some clarification to all this? Should I go ext2 everywhere, or ext4, or should I try one of those MTD filesystems? If so which one? What's the risk/drawback of each solution?

Thanks to anyone who can help!
 
Old 01-31-2011, 11:45 AM   #2
silvyus_06
Member
 
Registered: Oct 2010
Distribution: Ubuntu 10.04 , Linux Mint Debian Edition , Microsoft Windows 7
Posts: 386

Rep: Reputation: 49
flash memories do have a specific lifetime, which is very long by the way..

i kinda don't get it why journaling slows things down because it's only a few kilobytes of read write i guess..

logFS is definitely something you don't want... 64MB is a very small quantity....

ext3 is ok for both linux and the files. you better use ext4 because it is said to have faster boot times and also faster filesystem checking...
 
Old 01-31-2011, 12:26 PM   #3
King_DuckZ
Member
 
Registered: Nov 2009
Location: Rome, IT
Distribution: Sabayon
Posts: 61

Original Poster
Rep: Reputation: 2
Quote:
Originally Posted by silvyus_06 View Post
i kinda don't get it why journaling slows things down because it's only a few kilobytes of read write i guess..
I don't know... That's what I understood from browsing the net... those statements are not necessarily true, again, it's just what I understood.
 
Old 01-31-2011, 03:52 PM   #4
arizonagroovejet
Senior Member
 
Registered: Jun 2005
Location: England
Distribution: SLED, openSUSE
Posts: 1,032

Rep: Reputation: 174Reputation: 174
This guy (http://joggler.exotica.org.uk/ubuntu/) seems to think BRFS with a relevant mount option is a good way to go for running Linux from a USB flash drive.

I'm using his BRFS image and this is what the relevant line from /etc/fstab looks like:
Code:
 LABEL=linux-root /		btrfs	noatime,compress,ssd	0       1

The contents of this post represents the sum of my knowledge regarding which filesystems are good for flash drives.
 
Old 02-02-2011, 06:31 AM   #5
King_DuckZ
Member
 
Registered: Nov 2009
Location: Rome, IT
Distribution: Sabayon
Posts: 61

Original Poster
Rep: Reputation: 2
I didn't know about btrfs! It seems promising although not very mature yet, I'll keep an eye on that!
 
Old 02-02-2011, 07:13 AM   #6
salasi
Senior Member
 
Registered: Jul 2007
Location: Directly above centre of the earth, UK
Distribution: SuSE, plus some hopping
Posts: 3,916

Rep: Reputation: 777Reputation: 777Reputation: 777Reputation: 777Reputation: 777Reputation: 777Reputation: 777
I've been looking at this for some time (note; this is from the stuff that I've read, not from stuff that I've tried, given that I am getting ready to try) and here is the list of clarifications that seem 'knowable'; there is a lot on the 'net that is either rubbish (insert your own, more extreme word for 'rubbish', if desired), or only applicable to some of the situations, without saying which the author is talking about.

Quote:
- wear leveling is always hardware-implemented, and a software wear leveling could cancel out with the hardware's effort
- hardware wear leveling has often a cheap implementation, and a better software wear leveling is desirable
This is probably the single most important thing, because several sites commonly don't say what type of device they are talking about, and that makes a night-and-day difference.

SSDs (things that look to the system like a hard drive) have a controller with wear levelling, USB thumb drives, Compact Flash cards don't. So, if you don't know whether the author is talking about an SSD or not, it is probably best to disregard what they say.

Some controllers are better than others (broadly, the more recent the better), but, if you've got a controller, you've probably got to trust the controller, even if you know that its not the finest controller in the world.

Quote:
Originally Posted by King_DuckZ View Post
Also, I suspect ext2 is striving for writing files consecutively or in some other harddisk-optimized way, which looks like a complete waste to me.
Yesno. You probably are seeing an attempt to optimise the performance of a real hard disk, but it probably isn't really ext2 that is to blame. If the IO scheduler attempts to do stuff like 'elevator seeks' that is really only a waste of time on a flash device, although it would be sensible on a real hard disk.

Quote:
Originally Posted by King_DuckZ View Post
Now, I've been reading around that said memories are block devices,
Yes, but you have to remember that the block size will be quite different from the block sizes that you see on conventional hard disks. So, anything that you see that uses the conventional block sizes will be wrong. According to Ted T'so, ext4 (and ext3??) don't make any such assumptions anywhere in the code.

Quote:
Originally Posted by King_DuckZ View Post
- ext3 = ext2 + journaling
- ext3 is not just ext2 with journaling
Yes, this is a common confusion. If you disable all of the features that ext2 doesn't have, the format on disk of ext3 is the same as ext2. This is not quite the same as saying that exactly the same code is used, or that the same bugs (or, undocumented features), if any, exist. And journalling is not the only feature that ext3 supports that isn't supported by ext2, so maybe you can make an ext2-with-additional-features if you really use ext3 in non-journalling mode, I don't know.

Quote:
I didn't know about btrfs! It seems promising although not very mature yet
By now, BTRFS is probably just about usable (YMMV) in terms of maturity, but there must be at least an order of magnitude less testing done on BTRFS-on-SSD than BTRFS-on-hard-disk. make of that what you will.

In testing, BTRFS's SSD mode didn't seem to do much. I'd guess that at this point it just tells the system that unconventional block sizes are about to be encountered, but I don't know. Certainly the performance change is anything other than dramatic, but maybe it helps device life more than performance...
 
Old 06-02-2011, 11:53 AM   #7
King_DuckZ
Member
 
Registered: Nov 2009
Location: Rome, IT
Distribution: Sabayon
Posts: 61

Original Poster
Rep: Reputation: 2
Hi salasi, thanks for replying. By now I got convinced that my cheap USB stick is already doing wear leveling, so I just went for ext4. This old thread from 2003, specifically, talks about DataTraveler having wear level technology (my memory stick is a DataTraveler 8GiB), and as for SD cards, it seems that wear leveling is part of the standard.
Besides, I did try BTRFS, and I really noticed some speed gain with my usb stick, but unfortunately free space detection is unpredictable: for example, even if I still had 1.5GiB of free space reported, the copy of 700-1000MiB files couldn't be completed. I assume that either the reported free space is wrong, either the allocation is defective as memory starts to run low. I will certainly give it another try in the future.
If anyone has anythong more to say, I will keep on reading this thread from time to time (and hopefully gathering informations here is helping someone else as well).
 
Old 06-03-2011, 05:38 AM   #8
salasi
Senior Member
 
Registered: Jul 2007
Location: Directly above centre of the earth, UK
Distribution: SuSE, plus some hopping
Posts: 3,916

Rep: Reputation: 777Reputation: 777Reputation: 777Reputation: 777Reputation: 777Reputation: 777Reputation: 777
Quote:
Originally Posted by King_DuckZ View Post
By now I got convinced that my cheap USB stick is already doing wear leveling, so I just went for ext4. This old thread from 2003, specifically, talks about DataTraveler having wear level technology (my memory stick is a DataTraveler 8GiB), and as for SD cards, it seems that wear leveling is part of the standard.
I think I saw, in the last week or so, an announcement of an SSD with a controller, which implied that it was the first. Of course, wear levelling could be done in the controller, or by system software on the host computer, so, if you have some kind of 'wear levelling driver' the device itself wouldn't have to do it. There would be a speed impact with this approach, but it may not be too severe...

Quote:
Originally Posted by King_DuckZ View Post
Besides, I did try BTRFS, and I really noticed some speed gain with my usb stick, but unfortunately free space detection is unpredictable: for example, even if I still had 1.5GiB of free space reported, the copy of 700-1000MiB files couldn't be completed. I assume that either the reported free space is wrong...
That may well be correct; BTRFS is CoW, and does snapshots and that makes free space/space in use more complex and it is quite possible that 'breaks' the old, reliable du/df utils, unless you have a recent version (don't know this; it is just speculation on my part).

There is some benchmarking over at phoronix, eg:
http://www.phoronix.com/scan.php?pag...s_nilfs2&num=1
http://www.phoronix.com/scan.php?pag...tbook_fs&num=1
http://www.phoronix.com/scan.php?pag..._zfs_ssd&num=1

but broadly, the message is a bit complex. You can quite easily turn around the 'which is fastest, which is slowest' list by tweaking the test conditions. nilfs2 is the other filesystem that seems to worth consideration, but how it would work in practice (does it make the wear issue worse (it shouldn't, but does it deliver, in practice, the potential), does it play nicely both with and without controllers? what about the compressing (SF) controllers?) is another matter.
 
  


Reply

Tags
filesystem


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] What flash filesystem to choose? alephan Linux - Kernel 10 05-01-2011 03:30 PM
[SOLVED] Can we have JFFS2 Filesystem on NOR Flash. Niraj Kulkarni Linux - Newbie 0 04-28-2010 08:28 AM
Encrypted filesystem and boot from flash Chinook06 Linux - Security 1 01-13-2007 09:59 PM
Linux filesystem on a flash rom eddycur Linux - Software 1 03-24-2004 06:09 AM
Required files for my Flash Filesystem? radhapriya Linux - Software 0 10-01-2003 02:16 AM


All times are GMT -5. The time now is 05:31 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration