LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - General (http://www.linuxquestions.org/questions/linux-general-1/)
-   -   Anyone know of a Linux filesystem that supports on the fly compression? (http://www.linuxquestions.org/questions/linux-general-1/anyone-know-of-a-linux-filesystem-that-supports-on-the-fly-compression-4175461571/)

Arcosanti 05-11-2013 05:34 PM

Anyone know of a Linux filesystem that supports on the fly compression?
 
Anyone know of a good Linux filesystem that supports on the fly compression the way btrfs does but is not experimental?

kooru 05-12-2013 03:32 AM

Only ZFS and Reiser4 if I remember..

jpollard 05-12-2013 05:11 AM

The usual problem is that file reads/writes can be random - and compression usually causes severely slow responses.

273 05-12-2013 05:18 AM

I hate to be facetious but I thought most Linux file systems were "experimental"? Since you have a good backup strategy why not use btrfs?

Arcosanti 05-13-2013 03:18 AM

Quote:

Originally Posted by jpollard (Post 4949435)
The usual problem is that file reads/writes can be random - and compression usually causes severely slow responses.

I've seen some compression methods where the file access frequency dictated the level of compression used. Highly accessed files are lightly compressed and seldom accessed files are highly compressed. The idea obviously is to work around this problem of sluggish file system response.

Quote:

Originally Posted by 273 (Post 4949436)
I hate to be facetious but I thought most Linux file systems were "experimental"? Since you have a good backup strategy why not use btrfs?

I don't consider EXT3 AND EXT4 to be experimental. I prefer a file system that is rock solid in it's performance. Btrfs is not yet there, so I am not using it yet.

I have Rieser support built in to my kernel but I'm not sure if it is Reiser 4. I'll look into ZFS as I have never heard of it.

jpollard 05-13-2013 03:25 AM

Quote:

Originally Posted by Arcosanti (Post 4949950)
I've seen some compression methods where the file access frequency dictated the level of compression used. Highly accessed files are lightly compressed and seldom accessed files are highly compressed. The idea obviously is to work around this problem of sluggish file system response.

The problem with most compression methods is that they work on the file level to achieve the compression. File access, being random, much prefers a block level encryption... Unfortunately, block level encryption tends not to save that much space unless you have a filesystem that supports such partial blocks... I believe that is where zfs/btrfs try to put the compression.

The only delays then is the decompression of the block. With the others you have to decompress the entire file first so that you can find the block boundary.

273 05-13-2013 07:32 AM

Quote:

Originally Posted by Arcosanti (Post 4949950)
I don't consider EXT3 AND EXT4 to be experimental. I prefer a file system that is rock solid in it's performance. Btrfs is not yet there, so I am not using it yet.

I have Rieser support built in to my kernel but I'm not sure if it is Reiser 4. I'll look into ZFS as I have never heard of it.

EXT4 is still in beta as far as I am aware -- it has been adopted by many distributions as default but is still very much "the new kid on the block". Any Reiser file system is still in beta also last time I checked and with doubts over any support.
That is why I was being a little cheeky about Linux file systems being experimental -- if you look into it only EXT3 from the list is currently proven and out of beta.
Edit: Just to clarify my claims:
http://en.wikipedia.org/wiki/Ext4#De...tial_data_loss
http://en.wikipedia.org/wiki/Reiser4
I know Wikipedia is not the most reliable of sources but the claims made there can be found in other places too.

jefro 05-13-2013 03:46 PM

Looks like there are many ways available.

One might be able to use squashfs or aufs or fuse.

As above ZFS is a good choice.

The Unsorted Block Image File System (UBIFS) is a successor to JFFS2, and competitor to LogFS,[1

chrism01 05-13-2013 08:14 PM

ext4 is the default for RHEL; not something they do lightly, so I'd say its definitely not beta, even unofficially ;)

ZFS: you can get a native Linux kernel port here http://zfsonlinux.org/

273 05-14-2013 04:16 AM

Quote:

Originally Posted by chrism01 (Post 4950490)
ext4 is the default for RHEL; not something they do lightly, so I'd say its definitely not beta, even unofficially ;)

Well, yeah, it has been six months since the last major bug was found I suppose ;). What I was getting at is it has only just reached maturity and could still have the odd problem. I know calling it still in beta is overstating it but trusted and proven is a sliding scale and being 100% confident in EXT4 but not trusting BTRFS at all seems potentially a little picky to me.

TobiSGD 05-14-2013 05:31 AM

Quote:

Originally Posted by 273 (Post 4950672)
I know calling it still in beta is overstating it but trusted and proven is a sliding scale and being 100% confident in EXT4 but not trusting BTRFS at all seems potentially a little picky to me.

Ext4 has a very widespread usage now (almost any distro uses it as default filesystem) and the number of reported bugs is very small. The conclusion is that ext4 is pretty stable. Of course no software ever can reach the 100%, but ext4 is pretty close.
Btrfs, in opposite, has a very small number of installations (I don't know of any distro using it as default, but there may be a few), so it can be as stable as ext4, but no one knows, since it is not well tested. This is usually a showstopper for production environments.

syg00 05-14-2013 05:46 AM

If it ain't tested, it won't ever get (widely) accepted - hopefully sites are trialling it in non-prod. I had some issues with RAID10 recovery testing on btrfs early on (pulling a drive out mid-flight), but haven't re-tested that. Other than that I try to have at least one btrfs f/s per system on non-prod boxes. Love it.

Ext4 has been solid for quite a while as per @TobiSGD comment re acceptance as the default f/s on enterprise distros.

TobiSGD 05-14-2013 06:26 AM

Quote:

Originally Posted by syg00 (Post 4950732)
If it ain't tested, it won't ever get (widely) accepted - hopefully sites are trialling it in non-prod.

Indeed people do that, since BTRFS has advantages over ext4: http://www.anchor.com.au/blog/2013/0...up-experiment/

Arcosanti 05-14-2013 10:00 AM

I've been using both ext3 and ext4 since the summer of 2011 and have had no problems with either. They are both rock solid as far as I can tell. Much better than ext2 which had a tendency to unravel much faster than the FAT32 filesystem that was used under Win95 and 98 if the system ended up getting shut down in a non proper way. The problem that apparently plagued ext4 have apparently been addressed were as for btrfs, the problems have not yet been addressed. Also when you go to make a btrfs a warning comes up telling you that it is an experimental filesystem. Not so with ext4.

Looks like I won't be able to use ZFS. It requires a 64 bit system and I only have a 32 bit system. Apparently ZFS is not stable under 32 bits at this time. I am not really looking to use the read only and flash based file systems either as I am dealing with a hard drive. Fusion-zip might be a way to go though. Some other ideas have since come to mind such as using the UPX binary compression tool. I've used this years ago under Win95 and DOS with great success. I think I'll give the Linux version a try. I have also moved my /tmp folder to tempfs as I noticed nothing was using it and I really was wanting a fast way of clearing out this folder anyway. Now I don't have that stuff on my hard drive anymore. I am also experimenting with moving small stuff over to tmpfs as well and having them restored from a .tar.xz archive at boot time. So far I have only moved the BSD games to tmpfs.


All times are GMT -5. The time now is 12:35 AM.