LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Slackware (http://www.linuxquestions.org/questions/slackware-14/)
-   -   RAID 5 with 4 hard disks... array started with only 3 out of 4 disks (http://www.linuxquestions.org/questions/slackware-14/raid-5-with-4-hard-disks-array-started-with-only-3-out-of-4-disks-4175421379/)

kikinovak 08-10-2012 07:36 AM

RAID 5 with 4 hard disks... array started with only 3 out of 4 disks
 
Hi,

I'm currently installing Slackware 13.37 on a small HP Proliant ML36 server with four hard disks, 4x250GB. Each one of the disks has three RAID type (FD) partitions.

- one for /boot (4 disks, RAID 1)
- one for swap (4 disks, RAID 1)
- one for / (4 disks, RAID 5)

I created the RAID arrays with mdadm --create, and everything went OK... except something seems puzzling to me. The raid array for the / partition started only with 3 out of 4 disks. Here's how it looks:

Code:

# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md3 : active raid5 sda3[0] sdd3[4] sdc3[2] sdb3[1]
      729364992 blocks level 5, 512k chunk, algorithm 2 [4/3] [UUU_]
      [>....................]  recovery =  3.5% (8652032/243121664) finish=166.5min speed=23463K/sec
     
md2 : active raid1 sda2[0] sdd2[3] sdc2[2] sdb2[1]
      979840 blocks [4/4] [UUUU]
     
md1 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
      96256 blocks [4/4] [UUUU]
     
unused devices: <none>

# mdadm --detail /dev/md3
/dev/md3:
        Version : 0.90
  Creation Time : Thu Aug  9 12:43:55 2012
    Raid Level : raid5
    Array Size : 729364992 (695.58 GiB 746.87 GB)
  Used Dev Size : 243121664 (231.86 GiB 248.96 GB)
  Raid Devices : 4
  Total Devices : 4
Preferred Minor : 3
    Persistence : Superblock is persistent

    Update Time : Fri Aug 10 13:39:27 2012
          State : clean, degraded, recovering
 Active Devices : 3
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 1

        Layout : left-symmetric
    Chunk Size : 512K

 Rebuild Status : 7% complete

          UUID : 2f5b58e4:d1cc9b55:208cdb8d:9e23b04b
        Events : 0.3345

    Number  Major  Minor  RaidDevice State
      0      8        3        0      active sync  /dev/sda3
      1      8      19        1      active sync  /dev/sdb3
      2      8      35        2      active sync  /dev/sdc3
      4      8      51        3      spare rebuilding  /dev/sdd3

Until now, I've only ever used RAID level 5 with a maximum of 3 disks. This is the first time I use RAID 5 with 4 disks.

Is this behaviour normal ? Will the fourth disk eventually be included in the RAID 5 array in what looks like two and a half hours ?

I prefer to ask, since I intend to load quite some important data onto that server.

Cheers,

Niki

lithos 08-10-2012 07:41 AM

Quote:

Originally Posted by kikinovak (Post 4750801)
Hi,

I'm currently installing Slackware 13.37 on a small HP Proliant ML36 server with four hard disks, 4x250GB. Each one of the disks has three RAID type (FD) partitions.

- one for /boot (4 disks, RAID 1)
- one for swap (4 disks, RAID 1)
- one for / (4 disks, RAID 5)

I created the RAID arrays with mdadm --create, and everything went OK... except something seems puzzling to me. The raid array for the / partition started only with 3 out of 4 disks. Here's how it looks:

Code:

# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md3 : active raid5 sda3[0] sdd3[4] sdc3[2] sdb3[1]
      729364992 blocks level 5, 512k chunk, algorithm 2 [4/3] [UUU_]
      [>....................]  recovery =  3.5% (8652032/243121664) finish=166.5min speed=23463K/sec
     
md2 : active raid1 sda2[0] sdd2[3] sdc2[2] sdb2[1]
      979840 blocks [4/4] [UUUU]
     
md1 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
      96256 blocks [4/4] [UUUU]
     
unused devices: <none>

Until now, I've only ever used RAID level 5 with a maximum of 3 disks. This is the first time I use RAID 5 with 4 disks.

Is this behaviour normal ? Will the fourth disk eventually be included in the RAID 5 array in what looks like two and a half hours ?

I prefer to ask, since I intend to load quite some important data onto that server.

Cheers,

Niki

What do you mean started with 3 out of 4 - IT IS USING 4 but RAID 5 uses 1 disk for checksum so you have total space 3* size of drive.
You see your md uses:
Code:

sda, sdb, sdc and sdd drives (4)
active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]

Code:

Update Time : Fri Aug 10 13:39:27 2012
          State : clean, degraded, recovering
 Active Devices : 3
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 1

 Rebuild Status : 7% complete


    Number  Major  Minor  RaidDevice State
      0      8        3        0      active sync  /dev/sda3
      1      8      19        1      active sync  /dev/sdb3
      2      8      35        2      active sync  /dev/sdc3
      4      8      51        3      spare rebuilding  /dev/sdd3

this is normal for RAID5
it states that one drive was somehow faulty and is now rebuilding the array with that 4th spare drive to include it in the array.

Celyr 08-10-2012 07:42 AM

Yes it's normal, the linux kernel is now creating the raid5.
You can use it right now but if a drive fail you will lose everything because the ditribuited parity isn't build yet.

kikinovak 08-10-2012 07:58 AM

Quote:

Originally Posted by Celyr (Post 4750806)
Yes it's normal, the linux kernel is now creating the raid5.
You can use it right now but if a drive fail you will lose everything because the ditribuited parity isn't build yet.

OK thanks very much!

Celyr 08-10-2012 09:05 AM

If you have time issue maybe you can get it faster
Code:

# sysctl dev.raid.speed_limit_min
# sysctl dev.raid.speed_limit_max

speed is often limited by those parameters, you can change it using
Code:

sysctl -w dev.raid.speed_limit_max=value

foobarz 08-10-2012 09:05 AM

Yes, this is solved already, but as I understand it, this is normal right after create of raid5/6. For raid5, 1 disk is parity that is initially added as a spare and immediately begins to rebuild. Same for raid6, but 2 disks are initially added as spares and they rebuild one at a time.

When running these mdadm raids, be aware of the udev rules, /lib/udev/rules.d/64-md-raid.rules that udev will run on type "linux_raid_member" block devices: it will run "mdadm -I $tempnode" on them to try to incrementally assemble the raid array and start it if all members are added incrementally. Then, in /boot/initrd-tree/init, slackware will do this stuff after udev has been settled and done its mdadm incremental assmebly attempt:

mdadm -E -s > /etc/mdadm #(this will overwrite whatever /etc/mdadm.conf you might have copied into your initrd-tree)
mdadm -S -s # stop all detected arrays
mdadm -A -s # start all detected arrays if they can be started, even if they will start degraded (it will start them degraded at boot, be aware of that)


I have been looking into this stuff some because I will maybe use mdadm (or zfs; not decided yet). The udev mdadm incremental assembly can do the assemble correctly if you copy a good /etc/mdadm.conf into your initrd-tree/etc and comment out the mdadm runs in the initrd-tree/init script. And, the incremental assemble method when you have it assemble your arrays will not start them degraded at boot, which can be important potentially to protecting your data. Starting the array degraded because a drive is missing by some accident will result in the missing drive to need a rebuild, and you want to avoid that happening unless a drive as really failed while the system was up and running, not because a drive went missing at boot for some weird reason like you pulled the power cable on the drive or somehow the drive port assignment changed and the drive is somehow not found or it failed to get power spin-up (using staggered spin-up jumper on the hard drive).

Slax-Dude 08-10-2012 09:09 AM

your create command should be include the "-f" flag.

as per mdadm's manual:
Quote:

-f, --force
Insist that mdadm accept the geometry and layout specified without question. Normally mdadm will not allow creation of an array with only one device, and will try to create a RAID5 array with one missing drive (as this makes the initial resync work faster). With --force, mdadm will not try to be so clever.

kikinovak 08-10-2012 06:44 PM

raid
 
Several hours later.

Code:

[root@nestor:~] # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md3 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1]
      729364992 blocks level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
     
md2 : active raid1 sda2[0] sdd2[3] sdc2[2] sdb2[1]
      979840 blocks [4/4] [UUUU]
     
md1 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
      96256 blocks [4/4] [UUUU]
     
unused devices: <none>
[root@nestor:~] # df -h
Sys. de fichiers Taille Util. Disp. Uti% Monté sur
/dev/md3          685G  39G  612G  6% /
/dev/md1            92M  29M  59M  33% /boot
tmpfs              436M    0  436M  0% /dev/shm

No need to use any extra options here. Apparently, all I had to do was to wait until the last disk was synchronised. Took a couple of hours.

wildwizard 08-10-2012 07:39 PM

Quote:

Originally Posted by Slax-Dude (Post 4750855)
your create command should be include the "-f" flag.

as per mdadm's manual:

Your reading the sentence backwards.

If you use force it takes longer.

Slax-Dude 08-11-2012 07:33 AM

So, does [CODE]mdadm --detail /dev/md3[/QUOTE] now show the array with all 4 disks as opposed to the 3 disks + 1 spare?
I didn't know that, sorry :(


All times are GMT -5. The time now is 04:05 AM.