RAID10 write speed decreased to normal SSD speed after rebuilding the array.

mke2k · 07-07-2014, 05:27 AM

OS: Ubuntu 14.04 LTS
Software Raid10: 4x120Gb Intel 530 SSDs
Server: HP ProLiant DL160 G6
FS: md0: ext4/ md1:swap

Test is made on fresh new install, before making any change to some package or reboot.

Writing speed decreased to the a normal SSD write speed (in my case).

Bellow I add the stats/commands I performed.

Quote:

dd if=/dev/zero of=test bs=1048576 count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 1.79013 s, 1.2 GB/s

dd if=test of=/dev/null bs=1048576
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 0.376622 s, 5.7 GB/s

Code:

mdadm --fail /dev/md0 /dev/sda1
mdadm --remove /dev/md0 /dev/sda1
mdadm --fail /dev/md1 /dev/sda2
mdadm --remove /dev/md1 /dev/sda2
mdadm --add /dev/md0 /dev/sda1
mdadm --add /dev/md1 /dev/sda2

After rebuilding the array:

Quote:

dd if=/dev/zero of=test bs=1048576 count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 9.5621 s, 225 MB/s

dd if=test of=/dev/null bs=1048576
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 0.375301 s, 5.7 GB/s

Quote:

cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md0 : active raid10 sdc1[2] sdb1[1] sda1[4] sdd1[3]
195177472 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]

md1 : active raid10 sdd2[5] sda2[4] sdc2[2] sdb2[1]
39095296 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]

I don't know where to start now. Do I miss to set some flag or something?

mke2k · 07-07-2014, 08:41 AM

It seems that it's the "overwrite" speed that I was reading.

Quote:

rm test

dd if=/dev/zero of=test bs=1048576 count=2048
2048+0 records out
2048+0 records in
2147483648 bytes (2.1 GB) copied, 1.62431 s, 1.3 GB/s

I did not know that.
Puzzled.

yo8rxp · 07-11-2014, 04:15 AM

Well SSD drives are defined by MTBF ( Mean Time Between Failure ) , depending by how many times you write in same sector.
In order to avoid this , SSD does not delete the information at once , but when the drive is full and/or the space is reclaimed .

So in your case dd-ing same test file , data has to be deleted first by controller and then written onto disk.

Real speed can be read from mdstat while rebuilding raid structure until is resynced

cat /proc/mdstat during rebuild will show the resync speed.

While on read speed can rely on about 500-600 MB/s per drive , on write sometimes cannot.
One ideea is to use only 80 % from the drives , and leave about 20% unallocated space as over provisioning.