Hi all,

I have a 4-5yr old box that I'm retrofitting to act as a home media server that will occasionally double as a spare gaming machine for co-op LAN games. I was planning to wipe everything and start from scratch with the latest stable version of Mint x64 (v17.1 at time of writing). When I'm done it will have one 200GB drive for OS/swap/etc and several 2+ TB drives for data. I have some familiarity with bash and ext4, but I'm not too savvy on the ins and outs of Linux filesystems/kernels, etc. So I wanted to get some feedback / opinions regarding experiences with the different filesystem unioning options.

Primarily, I just want to be able to merge several physical disks/partitions into one virtual filesystem without too bad of a performance hit. I plan to stream some files from the server to laptop/htpc on my LAN and I might also play some co-op LAN games under wine occasionally. I'd prefer something that was pretty easy to use/maintain but I'll probably need some flexibility w/r/t configuration. That said, I am also trying to consider any pitfalls in the event that I run into disk failures, etc. I had originally been considering using LVM or doing a software RAID (as I don't have the $$$ to invest in hardware one). But IIRC I think I was reading that a single physical disk failure requires you to replace the disk and then do some repair/maintenance operations before you are able to access any of the data again (including data on healthy HDDs). I think I also remember reading (ok...skimming) a whitepaper some time ago and seeing something about disks in a RAID being more prone to failure than disks that are NOT in a RAID. To be fair, I think the paper was focusing on a business environment (24/7/365 availability and accessed by a large number of people). In contrast, my server would probably only be on as needed (maybe a day or so at a time) and only accessed by at most 4 devices at a time. Anyway, I get the impression that LVM/RAID also have a bit more work when it comes to adding new disks (as opposed to simply formatting it then adding to fstab).

Most of the info is already backed up or not too hard to re-obtain (e.g. mp3s from Amazon/Google, backups of CDs/DVDs that I have physical copies of, etc). That said, it would still be time consuming and I'd like to avoid having everything else unavailable until I get around to restoring things. So moving past RAID and LVM, I saw a lot of other options for unioning various physical disks/partitions into a single virtual partition such as aufs, unionfs, mhddfs, and overlayfs. I had also read that some of these were implemented in FUSE (mhddfs and the unionfs-fuse variant of unionfs) which may or may not have its own performance problems.

Based on my comments above, I'm leaning towards ex4 and either unionfs or aufs. But if you have a different recommendation that you think would be appropriate for my scenario, I'd still be interested to hear about it. Anyway, if anyone can answer any of the below questions it would be much appreciated. Or failing that if you have experiences setting up something similar and can share pros/cons/alternate approaches. Thanks in advance.

1. Is there any truth to the FUSE-based options being terrible on performance? If so, how bad is it under modern versions? Is it going to be noticeable for streaming large files (SATA HDDs + gigabit LAN on wired cat6)? Or playing games (assuming that the games are on the OS drive and that is NOT part of the virtual/unioned fs)?

2. Mint 17.1 should be using Ubuntu 14.04 Trusty Tahr as a base and I see that the manpages have listings for aufs and union-fuse ... does this mean they are available ootb/via default sources? or would I still need to patch the kernel (for aufs)? I'm assuming I would have to patch the kernel for the fuse-less version of unionfs regardless.

3. I've never messed with kernels but from what I am reading here and here, getting the unionfs kernel module going seems somewhat simple. Is it really that easy or should I plan to spend some time reading up on kernel patching first?

4. How configurable are unionfs/aufs? Is it possible to configure either of these so that different virtual paths will write to different physical disks? For instance, it would be really cool if I could have different paths setup to write to different drives (e.g. /media/virtualfs/music goes to /sda while /media/virtualfs/videos goes to /sdb) but still provide a fall-back rule to cover storage space (e.g. /media/virtualfs/videos goes to /sdb, but if /sdb is full then put it on /sda).

With software RAID using mdadm if a disk fails in, say, a RAID5 configuration, then your disk array goes into a "degraded" state. That is you can still read and write data to your array but one more disk failure means you'll lose all of your data. Greyhole is also a fairly decent solution which allows you to pool your disks together.

Sorry, for the slow reply. I appreciate the mention of greyhole; I hadn't heard of that one yet and it sounds interesting. I had previously been looking at AUFS and overlayfs. In terms of greyhole, I found this while trying to read up on it:

this page (second post):

I have some questions about it though...
1) Since I don't need any redundancy, is it possible to disable/turn off the redundancy in greyhole or would it just be better to use something like aufs/etc + samba?
2) On the main page, it states "Using Greyhole to store a lot of small files is not-a-good-idea"... what would be considered small here? Guessing that pictures would be a no-no. What about a fairly large mp3 collection (no clue on actual size, but let's say 20+ GB)?
3) Does it have any option similar to the pmfs (parent/most-free-space) option in AUFS? ("Selects a writable branch where the parent dir exists, such as tdp mode. When the parent dir exists on multiple writable branches, aufs selects the one which has most free space, such as mfs mode.")

The other main options I have been reading about are AUFS/OverlayFS. Seems like OverlayFS has been added to mainline kernel and I was reading that Ubuntu (and thus likely Mint as well) may stop adding AUFS in favor of OverlayFS. From what I have read, AUFS sounds much more appealing to me than OverlayFS. So if I end up not going the greyhole route, I had some questions on those as well:
1) Does OverlayFS being merged with mainline kernal indicate that AUFS will be phased out / new development may stop?
2) Assuming that AUFS will stay around but that Ubuntu/Mint no longer have AUFS integrated in their next releases, would that significantly change the install process (which as described here and here sounds to just involve running "sudo apt-get install aufs-tools")?
3) If OverlayFS is replacing AUFS, will it support the feature set of AUFS? For example, being able to union three or more writable filesystems/directories into a pool or options like round robin/most free space/parent+most free space, etc. So far from my reading, I get the impression that OverlayFS is not nearly as versatile as AUFS and seems to only shine in its simplicity and handling of the live disc filesystems (but would not be nearly as good an option for NAS/NFS).

The answer to that one is in this article.
http://zornsoftware.codenature.info/...or-mhddfs.html

Thanks. Sounds like that answers one of my questions about greyhole: -- Like since it lists 'one' as an option, I would assume that it is possible to disable the redundancy. But yeah, I am starting to get leery about the issues it apparently has with smaller files. I'm still a bit curious on the whole Overlayfs vs aufs thing, but I can ask that as a side question later/in another thread since that's only really relevant in terms of "how much of pita will it be to get aufs going next time I upgrade Mint" lol.

I thought I had read somewhere that mhddfs and fuse-based systems in general were always going to be worse on performance, so it was surprising to see it someone saying performance was only about 5 slower. Maybe I wrote it off too soon but I had read a few places about performance issues and that had had me worried about it.

I might play with mhddfs and aufs on my test box when I get some time and see which one works better. Also, I noticed this article puts the unioning command for mhddfs in fstab (as many aufs guides did as well). I thought I remembered reading somewhere that fstab doesn't necessarily mount things in the order specified and that it was better to put the actual union/mount command (for either) in a .rc file etc (or om the future, I suppose it will be in whatever the systemd equivalent is). Have you ever run into any issues having it in fstab?

If you don't need redundancy then you could just use LVM with a logical volume JBOD. Then you could use any file system you want.

Yes, LVM was the first thing I had looked at after deciding RAID wouldn't do for me. Unfortunately, LVM didn't seem right for what I wanted either. My biggest issue with it was that it seemed similar to RAID in that it wouldn't let you add/remove hdd's without rebuilding the virtual group or whatever it was called. Plus it sounded like if one hard drive failed, you were SOL / had to replace it before you could access data on ANYTHING in the LVM group again...

I don't need redundancy, but my preference for the drive failure scenario is that the healthy drives would continue to be accessible in the pool and I would only need to restore the one hdd's data from backup.

You could check out gluster or ceph. Although they're designed to be clustered you can use them with only one machine. And it does exactly what you want: keep non-failed drive data available when a drive fails.