The Plan: RAID 5 + LUKS + ext4 (with stride and stripe_width)

digdogger · 07-29-2015, 02:52 AM

Hello everyone. I have a system with four 1.5 TB disks. I plan to create a RAID 5 array out of them, create one large partition on it, encrypt it with LUKS, and create one large ext4 filesystem on it. This filesystem would be used for storing personal files and won't contain any boot or OS files. I thought I would layout my plan here, and see if anybody has any advice on how I might do things better.

I've already checked all four disks with badblocks and smartctl:

Code:

badblocks -vws -o /root/badblocks-sdf.txt /dev/sdf
smartctl -t long /dev/sdf

and they all have no bad blocks on them and passed their SMART tests.

PART I: Create Partitions and RAID

First, I've got to create partitions on all the disks that will be used for RAID. I want to use parted to create one partition that uses the entire disk, but even at this stage, I'm not sure if I should use gpt or an msdos partition table. I'll likely go with gpt unless anybody knows of any reason I shouldn't.

I've already gone ahead and done this with the disks:

Code:

parted -a optimal /dev/sdc
(parted) mklabel gpt
(parted) mkpart primary 0% 100%
(parted) set 1 raid on
(parted) print                                                            
Model: ATA ST31500541AS (scsi)
Disk /dev/sdc: 1500GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      1049kB  1500GB  1500GB               primary  raid

Thanks: http://blog.yo61.com/creating-an-opt...n-with-parted/

Next, is to create the RAID array itself. I think the command I have below should work, but I'm wondering about the --layout option , which has this in the mdadm man page:

Quote:

The layout of the RAID5 parity block can be one of left-asymmetric, left-symmetric, right-asymmetric, right-symmetric, la, ra, ls, rs. The default is left-symmetric.

Based on some research I've done, the default option is the best one for normal use, so I'll likely go with this command:

Code:

mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 --spare-devices=0 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1

PART II: Create Encryption Layer

I have done some googling, and found this:

Code:

cryptsetup -y -v luksFormat /dev/xvdc

From: http://www.cyberciti.biz/hardware/ho...setup-command/

and this:

Code:

cryptsetup --verify-passphrase --hash=sha256 --cipher=aes-cbc-essiv:sha256 --key-size=256 luksFormat /dev/hda3

From: https://help.ubuntu.com/community/En...lesystemHowto3

I'm wondering if I should be using any of the "--hash=sha256 --cipher=aes-cbc-essiv:sha256 --key-size=256" options, or anything along those lines, or if I should just keep it default?

PART III: Create Filesystem

I'll most likely create an ext4 filesystem on top of the encryption layer, but I am open to the possibility of other options as well, like xfs or btrfs .

I found http://busybox.net/~aldot/mkfs_stride.html, and if I enter:
RAID Level: 5
Number of physical disks: 4
RAID chunk size (in KiB): 512
number of filesystem blocks (in KiB): 4

I chose 512 for the RAID chunk size since, according to the mdadm man page:

Quote:

-c, --chunk=
Specify chunk size of kibibytes. The default when creating an array is 512KB. To ensure compatibility with earlier versions, the default when Building and array with no persistent metadata is 64KB. This is only meaningful for RAID0, RAID4, RAID5, RAID6, and RAID10.

I believe that "number of filesystem blocks (in KiB)" is actually supposed to be the block size, so I've selected 4, to match the "-b 4096" option for mke2fs . The results I got are:

Code:

mkfs.ext3 -b 4096 -E stride=128,stripe-width=384

But I'm wondering, with the encryption layer in between, are the stride and stripe-width calculations even valid any more? Am I better off just to go without any -E options and just use the mke2fs defaults?

Thanks for your time. Let me know if you have any suggestions.

Keruskerfuerst · 07-30-2015, 01:13 AM

What data will be stored on the raid array?
As a result, you need to adjust the chunk size.

With

Quote:

mkfs.ext3 -b 4096 -E stride=128,stripe-width=384

you will create a ext3 filesystem.

If you want to create a ext4 filesystem, the command

Quote:

mkfs.ext4

is right.

digdogger · 07-30-2015, 01:42 AM

Thanks for your reply, Keruskerfuerst.

I will store all sorts of files like, movies, music, web site files, source code and scripts, email, and some large archive files. It's all over the map, unfortunately. But there won't be any boot or OS files, files won't change very often, and there will be a fair bit more reading than writing. Parts of the filesystem will be shared by Samba (read only) for movie and music playback, so fast reading would be desirable.

The "mkfs.ext3" bit was just copied and pasted from another web site. I would actually use:

Code:

mke2fs -n -j -t ext4 -L Storage -m 1 -T ext4,big /dev/md0

I'll run it with -n first and see what it would do. It should report the optimal block size that it provides for me, but looking at /etc/mke2fs.conf on my system, it will likely be 4096.

Still not sure about the stripe and stride-width options , and how the encryption layer in between might effect them. From what little I know about encryption, the encrypted version of something is usually larger than the unencrypted version, so possibly providing those options might actually slow things down, and if that's the case, I'll leave them out.

Keruskerfuerst · 07-30-2015, 09:55 AM

Raid 5 is usually used for a database.
In your case, a different Raid Level might a good choice.

digdogger · 07-31-2015, 07:12 PM

I'm fairly certain I can use a RAID 5 array for pretty much any files. RAID 5 just means you have have redundancy and can lose one physical disk without losing any data (but if you lose two disks or lose another disk while the array is rebuilding, you lose all the data).

Anyway, I think I've found what I'm looking for here:

http://wiki.drewhess.com/wiki/Creati...on_a_partition

syg00 · 07-31-2015, 09:19 PM

Nice link - hope it all works as you plan.
I stayed out of this discussion as I don't use LVM enough, and I have my doubts about this mad rush to encrypt everything.

RAID5 is fine for what you want to do - I use it on btrfs for things I care about. A side benefit of btrfs is that all the functionality is part of the design of the filesystem layer rather than offloaded to one or more emulated device layers that get can in the way (ZFS first got me interested in this). If want to add a disk, I add a disk and re-balance. KISS.
And the data is CRC checked automagically; a big plus for me. And ...

digdogger · 08-04-2015, 08:05 PM

Actually, I didn't mention LVM, but it was something I was considering. But I think I've made things complicated enough, and since I just want one large partition I don't think I need an LVM layer. btrfs is one thing I was considering as well, but I thought I would stick to ext4 since I have experience with it.

syg00 · 08-04-2015, 08:45 PM

How about that ... my bad

Must have been using btrfs too long - now it seems when I read mdadm, I think LVM.