raid of encrypted drives

chrismiceli · 08-24-2008, 05:48 PM

I've been trying to create an encrypted raid setup, but with a slight variation from the usual setups. I have successfully set up raid and then encrypted /dev/md0 using dm-crypt and the device mapper.

However, using hdparm to measure the drive speed yielded results in the range of 50MB/sec or so, in fact no faster than a single nonraided encrypted device. I suspect because there is some sort of bottleneck in the decryption process.

So then I decided to test a different configuration. I created 3 encrypted devices /dev/mapper/system(1,2,3) and raided them together. I got results as high as 96MB/sec, surpassing the performance of a single nonencrypted drive, and approaching half the performance of plain raid.

With these results in mind, I set out to install linux to a raid array of encrypted devices.

Code:

cryptsetup luksOpen  /dev/sda1 system1
cryptsetup luksOpen /dev/sdb1 system2
cryptsetup luksOpen /dev/sdc1 system3

mdadm --create /dev/md0 --level=1 -n 3 /dev/mapper/system1 /dev/mapper/system2 /dev/mapper/system3

But the installer didn't see the device /dev/md0. What's more, reassembling the raid array between reboots proved problematic, frequently giving out device busy errors.

I tried installing into a separate partition and copying the install over and changing the boot scripts. Needless to say, it hasn't worked. At boot the scripts attempt to mount /dev/mapper/system1 to /, rather than unlocking all 3 encrypted volumes and then starting the raid device for mounting.

Even if fiddling with the scripts yields a positive result, I'm still uneasy about the myriad of errors the above causes. Sometimes the encrypted partitions fail to unlock (or taking minutes to do so) and luksClose has trouble removing the devices. I've had to issue the stop raid command several times before being able to close the devices.

Does anyone know how to proceed with the configuration described, or perhaps know of a tutorial outlining it?

Bruce Hill · 08-25-2008, 12:53 AM

This box has the following RAID arrays, encrypted with LUKS, and
/ and ~/ are also LVMs.

Code:

root@silas:~# mdadm -D /dev/md0
/dev/md0:
        Version : 00.90.03
  Creation Time : Wed Jul  2 21:35:05 2008
     Raid Level : raid1
     Array Size : 979840 (957.04 MiB 1003.36 MB)
  Used Dev Size : 979840 (957.04 MiB 1003.36 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Mon Aug 25 01:16:35 2008
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : <munged>
         Events : 0.4

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1

root@silas:~# mdadm -D /dev/md1
/dev/md1:
        Version : 00.90.03
  Creation Time : Wed Jul  2 21:35:21 2008
     Raid Level : raid1
     Array Size : 96320 (94.08 MiB 98.63 MB)
  Used Dev Size : 96320 (94.08 MiB 98.63 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Mon Aug 25 04:41:09 2008
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : <munged>
         Events : 0.4

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       8       18        1      active sync   /dev/sdb2

root@silas:~# mdadm -D /dev/md2
/dev/md2:
        Version : 00.90.03
  Creation Time : Mon Jun 30 04:24:32 2008
     Raid Level : raid0
     Array Size : 195318016 (186.27 GiB 200.01 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Sat Aug 23 23:40:48 2008
          State : active
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

     Chunk Size : 64K

           UUID : <munged>
         Events : 0.25

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       1       8       19        1      active sync   /dev/sdb3

Code:

root@silas:~# pvdisplay
  --- Physical volume ---
  PV Name               /dev/mapper/root
  VG Name               cryptvg
  PV Size               186.27 GB / not usable 3.25 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              47684
  Free PE               40004
  Allocated PE          7680
  PV UUID               <munged>

root@silas:~# vgdisplay
  --- Volume group ---
  VG Name               cryptvg
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               186.27 GB
  PE Size               4.00 MB
  Total PE              47684
  Alloc PE / Size       7680 / 30.00 GB
  Free  PE / Size       40004 / 156.27 GB
  VG UUID               <munged>

root@silas:~# lvdisplay
  --- Logical volume ---
  LV Name                /dev/cryptvg/root
  VG Name                cryptvg
  LV UUID                <munged>
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                10.00 GB
  Current LE             2560
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

  --- Logical volume ---
  LV Name                /dev/cryptvg/home
  VG Name                cryptvg
  LV UUID                <munged>
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                20.00 GB
  Current LE             5120
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

Code:

root@silas:~# fdisk -l

Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x681acfcf

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1         122      979933+  fd  Linux raid autodetect
/dev/sda2             123         134       96390   fd  Linux raid autodetect
/dev/sda3             135       12292    97659135   fd  Linux raid autodetect

Disk /dev/sdb: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x18142ccc

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1         122      979933+  fd  Linux raid autodetect
/dev/sdb2             123         134       96390   fd  Linux raid autodetect
/dev/sdb3             135       12292    97659135   fd  Linux raid autodetect

Disk /dev/md0: 1003 MB, 1003356160 bytes
2 heads, 4 sectors/track, 244960 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

Disk /dev/md1: 98 MB, 98631680 bytes
2 heads, 4 sectors/track, 24080 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000

Disk /dev/md1 doesn't contain a valid partition table

Disk /dev/md2: 200.0 GB, 200005648384 bytes
2 heads, 4 sectors/track, 48829504 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x08040000

Disk /dev/md2 doesn't contain a valid partition table

Code:

root@silas:~# mount
/dev/mapper/cryptvg-root on / type jfs (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/mapper/cryptvg-home on /home type jfs (rw)
/dev/md/1 on /boot type jfs (rw)
tmpfs on /dev/shm type tmpfs (rw)
192.168.1.11:/home on /serverhome type nfs (rw,rsize=8192,wsize=8192,hard,intr,addr=192.168.1.11,nfsvers=3,proto=udp)
192.168.1.11:/backup on /server1 type nfs (rw,rsize=8192,wsize=8192,hard,intr,addr=192.168.1.11,nfsvers=3,proto=udp)
192.168.1.11:/backup2 on /server2 type nfs (rw,rsize=8192,wsize=8192,hard,intr,addr=192.168.1.11,nfsvers=3,proto=udp)

With swap on /dev/md0.

And though I prefer bonnie++ to hdparm:

Code:

root@silas:~# hdparm -tT /dev/md2

/dev/md2:
 Timing cached reads:   2274 MB in  2.00 seconds = 1137.68 MB/sec
 Timing buffered disk reads:  446 MB in  3.01 seconds = 148.33 MB/sec

If that is something like you have in mind, post back and perhaps
we can get some HOW-TO up for you. This was first create the RAID
arrays, then the LVMs in /dev/md2, then encrypt / and ~/ - iirc.