mdadm Rebuilding Raid 1 on 2 new disks problems

Ezplan · 09-05-2018, 11:19 AM

Hi All,
I had a mdadm raid-1 with 4 devices (md0, md1, md2 and md3) on two 300GB disks. I failed, and removed 1 disk and replaced it with a 1Tb disk. I partitioned the disk the same as the original and re-built (added) the array one device at a time with the new disk. Everything seemed to go OK.

I went ahead and failed and removed the other 300GB disk and added the 2nd 1Tb disk. I partitioned the disk the same as the original and rebuilt the array one device at a time. Again, went OK but the md3 array was not using the full size of the 1TB disks.

On the original disks md3 was built on top of an extended partition, (sda4 and sdb4) with a logical partition on top, (sda5 and sdb5). I tried to replicate that on the new disks, but it appears I didn't. mdstat and parted are not showing the same info, see below:

Code:

[root@pequod ~]# cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sdb5[3]
      265135104 blocks super 1.1 [2/1] [_U]
      bitmap: 2/2 pages [8KB], 65536KB chunk

md0 : active raid1 sda1[2] sdb1[3]
      102388 blocks super 1.0 [2/2] [UU]

md127 : active raid1 sdd[1] sdc[0]
      976631488 blocks super 1.2 [2/2] [UU]
      bitmap: 5/8 pages [20KB], 65536KB chunk

md2 : active raid1 sdb3[3] sda3[2]
      8190968 blocks super 1.1 [2/2] [UU]

md1 : active raid1 sdb2[3] sda2[2]
      20478908 blocks super 1.1 [2/2] [UU]
      bitmap: 1/1 pages [4KB], 65536KB chunk

unused devices: <none>

Code:

[root@pequod ~]# parted -l
Model: ATA ST1000NM0055-1V4 (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  106MB   105MB   primary  ext2         raid
 2      106MB   21.1GB  21.0GB  primary               raid
 3      21.1GB  29.5GB  8399MB  primary               raid
 4      29.5GB  1000GB  971GB   primary               raid


Model: ATA ST1000NM0055-1V4 (scsi)
Disk /dev/sdb: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  106MB   105MB   primary  ext2         raid
 2      106MB   21.1GB  21.0GB  primary               raid
 3      21.1GB  29.5GB  8399MB  primary               raid
 4      29.5GB  1000GB  971GB   primary               raid

Model: Unknown (unknown)
Disk /dev/md1: 21.0GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End     Size    File system  Flags
 1      0.00B  21.0GB  21.0GB  ext4

Model: Unknown (unknown)
Disk /dev/md2: 8388MB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End     Size    File system     Flags
 1      0.00B  8388MB  8388MB  linux-swap(v1)

Model: Unknown (unknown)
Disk /dev/md0: 105MB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End    Size   File system  Flags
 1      0.00B  105MB  105MB  ext2

Model: Unknown (unknown)
Disk /dev/md3: 271GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End    Size   File system  Flags
 1      0.00B  271GB  271GB  ext3

During the final rebuild the md3 device would not allow me to add the sda disk. So how do I fix this: add the sda disk/partition and use the full size of the new disks? I have a backup of the data on md3 so I can wipe it if necessary. But I'd like to keep md0, md1 and md2 intact.

Sorry for the length! Any help is greatly appreciated! Thanks!

syg00 · 09-05-2018, 08:49 PM

Software RAID doesn't automatically fill the devices it is built on - you have to explicitly tell mdadm to grow it once everything is in place. Then you have to expand the filesystem to use the extra space as well.

I don't know how you managed to convince mdadm that /dev/sdb5 (still) exists, but I'd be inclined to just delete md3 and rebuild it from scratch using /dev/sda4 and /dev/sdb4 and restore your data. That way you'll be using all the space, and have a working array.
KISS.

Ezplan · 09-06-2018, 09:07 PM

Thanks for the reply syg00!

Yes, I've done some more research and was introduced to mdadm --grow and resize2fs.

I did not completely wipe md3, rather I removed sda4 as a primary partition then recreated it as an extended and built sda5 on top as a logical partiton - the same as on the original 300GB disk. I added sda5 to the array and it rebuilt! I then failed and removed sdb5 and then tried to remove the sdb5 partition with parted - it reported it didn't exist! I went ahead and removed sdb4 then recreated sdb4 as extended and sdb5 as logical as above. I added sdb5 to the array and it rebuilt.

I then did mdadm --grow and it completed successfully with the full size of the disk. Then unmounted the md3 drive and used resize2fs which completed and filled the disk. Mounted md3 and all data was there and the new larger disk size is shown!

I realize this is probably not "best practice" and I'm very lucky, but so far everything is running ok and after a day of reads and writes to the array no problems. However, I will keep an eye on this array and maintain regular backups.

Thanks again!

syg00 · 09-06-2018, 09:45 PM

Whatever works is good. If you are happy to play around, you'll never stop learning something.