There's a spare hard disk I have at home. I'd like to use it for storing mainly anime episodes and streaming them to a Raspberry Pi.
Would be nice, if I could compress it, as I don't care for computational overhead, or random access.
I looked into btrfs, because of compression. But I then learned about deduplication. I think deduplication would be ideal for storing anime series.
I haven't found any FUSE solutions. But I can imagine something that basically writes a giant compressed archive to the disk might be effective. (And effectively "deduplicate".)
Also, it would only be necessary to present the files as normally uncompressed files to other computers.
A problem is, that I'm currently running Debian Squeeze (old-stable). I've wanted to try Arch, or Debian Sid, so this compression idea might be a good reason as any to do so. I enjoy making weird shizzle work, so I'm willing to do some experimenting and learn.
Could anyone nudge me into the right direction, please?

Here is a paper in Linux Journal which give the results from several compression programs used on different types of data.

http://www.linuxjournal.com/article/8051

For myself, disks are so cheap that I just buy more disks rather than using compression.

---------------------
Steve Stites

Many modern video formats will not compress well. I kind of doubt you will save much space. I forget what the current best compression method is.



Btrfs and zfs have the ability to compress data on the fly as well as maybe one or two others.

Another way might be to use some on the fly compression. Fuse, or squashfs or aufs maybe.

As jefro already pointed out, usually video files are already compressed and will not further compress. They will even get larger if you try that, due to the added overhead. You will possibly save space if you convert them to a different compression format, for example if they are MPEg-2 encoded you might convert them to H.264, but this will also result in a decreased quality (re-encoding will always cause qulaity loss)

Yes, that is true. But the main problem with compressed file-systems seems to be they perform per file compression. But if you have a collection of files with identical parts, disk block deduplication, or solid block compression might work.
The problem with compressed archives seems to be the dictionary is based on a "local window"; it might not look back far enough to identify repetition across large files. But this is just from very shallow research into the matter.
Per block deduplication, as opposed to per file, could save space, and I believe significantly. Mainly because the opening sequences of anime.
I'll look into zfs and the exact deduplication methods of btrfs.

I was getting ready to dig into the source, but btrfs has updated their documentation to clearly state bedup provides per file deduplication and not per block.
Also I remembered that a block compressed archive has horrendous random access.
This means that as of yet there are no space saving solutions for my case.
This probably makes me look more naive, but I'm convinced a per-block deduplicating open-source file system would be a great idea.

ZFS deduplication.

1 members found this post helpful.