Filesystem for a 30-40TB drive
 

If I use MDADM RAID 6 (or a hardware RAID card) and create a 30-40TB drive, which file system should I use and why? (Drives will be 4TB spindles.)

OS will probably be Debian 9 (once released) as this will be a server and I want rock solid stability above all else. This is a future build (year or so from now, maybe longer) so maybe by then, BTRFS will finally fix all their RAID 5/6 issue and that FS could be a possibility for me to look at.

PS I probably won't be using ECC RAM.

Not sure about the filesystem, but if you want rock-solid stability, why not go for Centos? Its EOL is 10 years, instead of the 5 Debian offers.

You pick a filesystem based on a number of features. What features of the common file systems do you think you may need?

When I was looking for a distro I could call home, I downloaded a ton of them. CentOS was one of the distros that I could not figure out how to make a program auto-start at boot. User error, I'm sure.

I'm too new to Linux to intelligently answer that question. I just want a FS that scales well when it comes to handling such a large partition. If it helps, this is just a Plex server build.

I want the FS to "heal" itself in the event of a spontaneous reboot or a power failure. (Yes, the server will have a UPS attached to it but you never know.) I've read that EXT4 isn't the best when it comes to really large partitions like what I'm talking about and XFS would be better. Also read that EXT4 wastes lots of space for meta-data compared to XFS.

BUT, if BTRFS can get their RAID 6 issues squared away during my planning phase, I'm fairly certain I'll go with that when it's time to build this new server. However, if they can't, I need to think about alternative FS's. :)

I'm looking into building a new server with 12tb of storage with snapraid on Ubuntu with 2-3 parity disks. Then using something to pool them into a single location. Seems like a good system to me.

The underlying filesystem would be probably be xfs since most of my data is large mkv video files for media serving around the house. But I wouldn't be against regular ext4. I don't know enough about the differences other than "xfs is better for bigger files" or something like that.

Well this has got a bit more interesting than most similar threads.

I have used btrfs RAID5 for my photos for years (yes, I know, but I'm anal about backups) - couldn't live without snapshot.
Been thinking of switching to LVM for RAID control (no explicit mdadm setup) as it offers intelligent spreading of data and metadata across the disks, and most importantly automatic failure policies.
FWIW I'll probably use ext4 because I'm comfortable with it - for a media server I might be inclined to try XFS; I've been to a couple of conference talks by the developer and things seem to be continually improving.

But that snapraid looks a good candidate - thanks @jmgibson1981.

I use XFS on all large partitions. It's a phenomenal file system with excellent tools. I've never had one fail. But you have to read the man page for mkxfs to figure out which options to use. It's a vastly flexible and powerful system. But ext4 is better for a typical installation on a PC, and XFS must be specially configured for SELinux if you're using that.

I have XFS systems that have been without a single problem for 10 years or more.

There are tons of web pages devoted to testing filesystem on various kernels. I say kernels because it is very important to know that the kernel level or version impacts how the filesystem is supported and speeds.

My guess is that you should consider ZFS, Btrfs, XFS and even Ext4. ZFS and Btrfs have built in raid ability. XFS and ext4 on LVM is very common. I am growing fond of XFS in my use.

Ext4 may have the most documentation and user knowledge and tools. It is not too hard to set it to check filesystems on boot.

What is the purpose of getting a 30~40TB RAID? That seems excessively large for common usage and so must assume there is a specific purpose in mind. Without knowing what you are intending to do it is a bit harder to recommend. If I had to say then I'd probably go RAID -> LVM -> XFS as a personal preference and is in fact what I do have set-up for my personal KVM host/home server.

CentOS 7 uses systemd, so the same methods you use for the latest versions of Debian and Ubuntu which are also systemd based, previously in CentOS 6 when it was still sysvinit based, you'd use chkconfig.

I was about to say the same thing, but I had given up :)

1. Why are you essentially excluding ZFS by imposing either mdadm or hardware raid, and to a lesser extent by avoiding ECC-RAM?

2. Many rock solid solutions eventually crash. How do you plan to backup your 30-40TB data?

When I build my "2.0" media server..... if BTRFS has addressed all the RAID 5/6 issues, I'm fairly certain that will be the FS I go with. I just need to have a backup FS to go with if BTRFS isn't ready when I am.

As for Kernel, since I'll be using Debian 9, whatever that is going to ship with. (4.9 I think?)

Based on my experience.....especially when building out media servers...... I ALWAYS seem to run into the problem of running out of space on an array after X months and adding new drives and re-balancing takes F_O_R_E_V_E_R. This is why I like to build huge RAID 6 arrays and have a single partition.

I also never fully understood LVM. Is it like Drivepool in Windows land? (Combining multiple drives into a "single" drive?)

1. It appears that expanding a ZFS RAID setup is a bit more of a "pain in the butt" vs. expanding a hardware, MDADM or BTRFS RAID.

2. My existing media server will act as a backup to my new server. I have 8, 4TB drives in there now running in RAID 6. When I build the new server, I'll use RAID 6 in that one and my old server will either be setup as RAID 0 or utilize a drive 'pooling' type setup.