software RAID: very slow mdamd resync

mattca · 08-13-2009, 04:24 PM

I hope I'm posting this to the right place..

I set up a RAID5 device using 6 SATA hard drives.. it's been resyncing for a couple hours now and is only at 13%. It says it will take another 1200 minutes and the speed has been bouncing around between 10,000 - 13,000 K/sec. I've already changed /proc/sys/dev/raid/speed_limit_min to 20,000 and speed_limit_max to 50,000 but that hasn't changed anything.

Any ideas as to why it's so slow? These drives are pretty big.. is it just the sheer amount of space that needs to be managed on the initial sync? Can I expect future syncs to be this slow? What about reading and writing to the device?

Thanks.

Code:

[root@localhost matt]# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sdf1[6] sde1[4] sdd1[3] sdc1[2] sdb3[1] sda1[0]
      4775963520 blocks level 5, 64k chunk, algorithm 2 [6/5] [UUUUU_]
      [==>..................]  recovery = 13.4% (128325508/955192704) finish=1226.5min speed=11235K/sec
      
unused devices: <none>

Code:

[root@localhost matt]# iostat -x
Linux 2.6.29.6-desktop-1mnb (localhost) 	14/08/09 	_i686_	(1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.40    0.04   93.08    0.30    0.00    6.17

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda             182.98     0.00 2445.83    0.00 21030.16     0.01     8.60     0.40    0.16   0.16  38.43
sdb             169.70     0.23 2460.11    0.22 21063.18     3.67     8.56     0.36    0.15   0.13  31.91
sdc             165.32     0.00 2463.47    0.00 21030.06     0.01     8.54     0.41    0.17   0.16  39.67
sdd             145.80     0.00 2482.99    0.00 21030.05     0.01     8.47     0.33    0.13   0.13  32.26
sde             134.87     0.00 2493.92    0.00 21030.04     0.01     8.43     0.32    0.13   0.13  31.22
sdf               0.05   226.70    0.00 2402.04     0.16 21029.87     8.76     0.29    0.12   0.12  28.55
md0               0.00     0.00    0.01    0.00     0.10     0.00     8.00     0.00    0.00   0.00   0.00

Code:

[root@localhost matt]# mdadm -D /dev/md0 
/dev/md0:
        Version : 0.90
  Creation Time : Thu Aug 13 22:12:18 2009
     Raid Level : raid5
     Array Size : 4775963520 (4554.71 GiB 4890.59 GB)
  Used Dev Size : 955192704 (910.94 GiB 978.12 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Fri Aug 14 01:14:13 2009
          State : clean, degraded, recovering
 Active Devices : 5
Working Devices : 6
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 64K

 Rebuild Status : 13% complete

           UUID : 80e75a59:b20117a6:36987a58:9ca5603c
         Events : 0.6

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       19        1      active sync   /dev/sdb3
       2       8       33        2      active sync   /dev/sdc1
       3       8       49        3      active sync   /dev/sdd1
       4       8       65        4      active sync   /dev/sde1
       6       8       81        5      spare rebuilding   /dev/sdf1

mesiol · 08-14-2009, 12:50 PM

Hi,

resync performance is around 10 -15 Mb/s, this is a ok.

Calculate the size of your RAID/SYNC_performance and you know how long it takes.

There is nothing wrong when you try to sync 4,5TB RAID.

Read performance is okay on RAID5 depending on the disk performance, write performance will be slower, because RAID5 parity has to be calculated for each block.