LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-11-2017, 02:21 PM   #1
road hazard
Member
 
Registered: Nov 2015
Posts: 145

Rep: Reputation: Disabled
Question Filesystem for a 30-40TB drive


If I use MDADM RAID 6 (or a hardware RAID card) and create a 30-40TB drive, which file system should I use and why? (Drives will be 4TB spindles.)

OS will probably be Debian 9 (once released) as this will be a server and I want rock solid stability above all else. This is a future build (year or so from now, maybe longer) so maybe by then, BTRFS will finally fix all their RAID 5/6 issue and that FS could be a possibility for me to look at.

PS I probably won't be using ECC RAM.
 
Old 04-11-2017, 02:40 PM   #2
vincix
Senior Member
 
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,032

Rep: Reputation: 76
Not sure about the filesystem, but if you want rock-solid stability, why not go for Centos? Its EOL is 10 years, instead of the 5 Debian offers.
 
Old 04-11-2017, 03:14 PM   #3
jefro
Moderator
 
Registered: Mar 2008
Posts: 19,154

Rep: Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921
You pick a filesystem based on a number of features. What features of the common file systems do you think you may need?
 
Old 04-11-2017, 04:32 PM   #4
road hazard
Member
 
Registered: Nov 2015
Posts: 145

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by vincix View Post
Not sure about the filesystem, but if you want rock-solid stability, why not go for Centos? Its EOL is 10 years, instead of the 5 Debian offers.
When I was looking for a distro I could call home, I downloaded a ton of them. CentOS was one of the distros that I could not figure out how to make a program auto-start at boot. User error, I'm sure.
 
Old 04-11-2017, 04:47 PM   #5
road hazard
Member
 
Registered: Nov 2015
Posts: 145

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by jefro View Post
You pick a filesystem based on a number of features. What features of the common file systems do you think you may need?
I'm too new to Linux to intelligently answer that question. I just want a FS that scales well when it comes to handling such a large partition. If it helps, this is just a Plex server build.

I want the FS to "heal" itself in the event of a spontaneous reboot or a power failure. (Yes, the server will have a UPS attached to it but you never know.) I've read that EXT4 isn't the best when it comes to really large partitions like what I'm talking about and XFS would be better. Also read that EXT4 wastes lots of space for meta-data compared to XFS.

BUT, if BTRFS can get their RAID 6 issues squared away during my planning phase, I'm fairly certain I'll go with that when it's time to build this new server. However, if they can't, I need to think about alternative FS's.
 
Old 04-11-2017, 05:44 PM   #6
jmgibson1981
Member
 
Registered: Jun 2015
Location: Tucson, AZ USA
Distribution: Debian
Posts: 362

Rep: Reputation: Disabled
I'm looking into building a new server with 12tb of storage with snapraid on Ubuntu with 2-3 parity disks. Then using something to pool them into a single location. Seems like a good system to me.

The underlying filesystem would be probably be xfs since most of my data is large mkv video files for media serving around the house. But I wouldn't be against regular ext4. I don't know enough about the differences other than "xfs is better for bigger files" or something like that.

Last edited by jmgibson1981; 04-11-2017 at 06:04 PM.
 
1 members found this post helpful.
Old 04-11-2017, 06:31 PM   #7
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 18,086

Rep: Reputation: 2906Reputation: 2906Reputation: 2906Reputation: 2906Reputation: 2906Reputation: 2906Reputation: 2906Reputation: 2906Reputation: 2906Reputation: 2906Reputation: 2906
Well this has got a bit more interesting than most similar threads.

I have used btrfs RAID5 for my photos for years (yes, I know, but I'm anal about backups) - couldn't live without snapshot.
Been thinking of switching to LVM for RAID control (no explicit mdadm setup) as it offers intelligent spreading of data and metadata across the disks, and most importantly automatic failure policies.
FWIW I'll probably use ext4 because I'm comfortable with it - for a media server I might be inclined to try XFS; I've been to a couple of conference talks by the developer and things seem to be continually improving.

But that snapraid looks a good candidate - thanks @jmgibson1981.

Last edited by syg00; 04-11-2017 at 09:36 PM. Reason: cleanup
 
Old 04-12-2017, 12:18 AM   #8
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,513

Rep: Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004Reputation: 1004
I use XFS on all large partitions. It's a phenomenal file system with excellent tools. I've never had one fail. But you have to read the man page for mkxfs to figure out which options to use. It's a vastly flexible and powerful system. But ext4 is better for a typical installation on a PC, and XFS must be specially configured for SELinux if you're using that.

I have XFS systems that have been without a single problem for 10 years or more.
 
Old 04-12-2017, 03:29 PM   #9
jefro
Moderator
 
Registered: Mar 2008
Posts: 19,154

Rep: Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921Reputation: 2921
There are tons of web pages devoted to testing filesystem on various kernels. I say kernels because it is very important to know that the kernel level or version impacts how the filesystem is supported and speeds.

My guess is that you should consider ZFS, Btrfs, XFS and even Ext4. ZFS and Btrfs have built in raid ability. XFS and ext4 on LVM is very common. I am growing fond of XFS in my use.

Ext4 may have the most documentation and user knowledge and tools. It is not too hard to set it to check filesystems on boot.

Last edited by jefro; 04-13-2017 at 03:03 PM.
 
Old 04-12-2017, 04:35 PM   #10
r3sistance
Senior Member
 
Registered: Mar 2004
Location: UK
Distribution: CentOS 6/7
Posts: 1,375

Rep: Reputation: 217Reputation: 217Reputation: 217
What is the purpose of getting a 30~40TB RAID? That seems excessively large for common usage and so must assume there is a specific purpose in mind. Without knowing what you are intending to do it is a bit harder to recommend. If I had to say then I'd probably go RAID -> LVM -> XFS as a personal preference and is in fact what I do have set-up for my personal KVM host/home server.

Quote:
Originally Posted by road hazard View Post
When I was looking for a distro I could call home, I downloaded a ton of them. CentOS was one of the distros that I could not figure out how to make a program auto-start at boot. User error, I'm sure.
CentOS 7 uses systemd, so the same methods you use for the latest versions of Debian and Ubuntu which are also systemd based, previously in CentOS 6 when it was still sysvinit based, you'd use chkconfig.
 
Old 04-13-2017, 03:12 AM   #11
vincix
Senior Member
 
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,032

Rep: Reputation: 76
I was about to say the same thing, but I had given up
 
Old 04-13-2017, 08:57 AM   #12
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris 11.4, Oracle Linux, Mint, Tribblix, Ubuntu/WSL
Posts: 9,764

Rep: Reputation: 463Reputation: 463Reputation: 463Reputation: 463Reputation: 463
Quote:
Originally Posted by road hazard View Post
If I use MDADM RAID 6 (or a hardware RAID card) and create a 30-40TB drive, which file system should I use and why? (Drives will be 4TB spindles.)
...
PS I probably won't be using ECC RAM.
1. Why are you essentially excluding ZFS by imposing either mdadm or hardware raid, and to a lesser extent by avoiding ECC-RAM?

2. Many rock solid solutions eventually crash. How do you plan to backup your 30-40TB data?
 
Old 04-14-2017, 12:14 PM   #13
road hazard
Member
 
Registered: Nov 2015
Posts: 145

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by jefro View Post
There are tons of web pages devoted to testing filesystem on various kernels. I say kernels because it is very important to know that the kernel level or version impacts how the filesystem is supported and speeds.

My guess is that you should consider ZFS, Btrfs, XFS and even Ext4. ZFS and Btrfs have built in raid ability. XFS and ext4 on LVM is very common. I am growing fond of XFS in my use.

Ext4 may have the most documentation and user knowledge and tools. It is not too hard to set it to check filesystems on boot.
When I build my "2.0" media server..... if BTRFS has addressed all the RAID 5/6 issues, I'm fairly certain that will be the FS I go with. I just need to have a backup FS to go with if BTRFS isn't ready when I am.

As for Kernel, since I'll be using Debian 9, whatever that is going to ship with. (4.9 I think?)
 
Old 04-14-2017, 12:30 PM   #14
road hazard
Member
 
Registered: Nov 2015
Posts: 145

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by r3sistance View Post
What is the purpose of getting a 30~40TB RAID? That seems excessively large for common usage and so must assume there is a specific purpose in mind. Without knowing what you are intending to do it is a bit harder to recommend. If I had to say then I'd probably go RAID -> LVM -> XFS as a personal preference and is in fact what I do have set-up for my personal KVM host/home server.



CentOS 7 uses systemd, so the same methods you use for the latest versions of Debian and Ubuntu which are also systemd based, previously in CentOS 6 when it was still sysvinit based, you'd use chkconfig.
Based on my experience.....especially when building out media servers...... I ALWAYS seem to run into the problem of running out of space on an array after X months and adding new drives and re-balancing takes F_O_R_E_V_E_R. This is why I like to build huge RAID 6 arrays and have a single partition.

I also never fully understood LVM. Is it like Drivepool in Windows land? (Combining multiple drives into a "single" drive?)
 
Old 04-14-2017, 12:35 PM   #15
road hazard
Member
 
Registered: Nov 2015
Posts: 145

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by jlliagre View Post
1. Why are you essentially excluding ZFS by imposing either mdadm or hardware raid, and to a lesser extent by avoiding ECC-RAM?

2. Many rock solid solutions eventually crash. How do you plan to backup your 30-40TB data?
1. It appears that expanding a ZFS RAID setup is a bit more of a "pain in the butt" vs. expanding a hardware, MDADM or BTRFS RAID.

2. My existing media server will act as a backup to my new server. I have 8, 4TB drives in there now running in RAID 6. When I build the new server, I'll use RAID 6 in that one and my old server will either be setup as RAID 0 or utilize a drive 'pooling' type setup.
 
  


Reply

Tags
file system, raid


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Need RAID5 Web Server with 40TB, suggestion MrUmunhum Linux - Server 1 03-27-2010 04:43 PM
Filesystem for SSD drive Changes Linux - Software 5 09-02-2009 10:18 AM
filesystem for 500G drive ufmale Linux - Newbie 6 05-31-2008 06:49 PM
kernel accessing pata cd rom drive instead of sata OS hard drive for filesystem elaps2007 Linux - Kernel 0 03-19-2008 09:39 AM
How to get the filesystem for an unmounted drive? Tortanick Linux - General 8 03-12-2007 06:20 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 04:07 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration