Hi,

I have a physical linux system running Centos 6.7 kernel 2.6.32-573.12.1.el6.x86_64. It has 4 SATA disks configured into 3 MD RAID 1 mirrors, md0 for /boot, md1 for the OS and md2 for my data.

Recently I found issues with /dev/sda causing high iowait due to journalling. I checked SMART and it found issue with sda which was then failed in md0 and md1.

I rebooted to boot from sdb and got a GRUB error:

grub error 17 Cannot mount selected partition

Now, everything I have read about this error points to incorrect grub devices for the partitions. However, I have checked these thoroughly and they appear correct.

I set up md0 as metadata version 0.90 for grub compatibility so that rules out that. The remaining devices are md1 version 1.0 and md2 version 1.1.

The system was booted using BIOS not UEFI. Currently /dev/sdb is set as the first boot disk since sda failed.

Can anyone suggest anything? Until I get the system booted from /dev/sdb I cannot replace the failed disk.

MDSTAT:
[root@praetorian ~]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[1] sda1[2](F)
524224 blocks [2/1] [_U]

md2 : active raid1 sdc1[1] sdd1[0]
976751872 blocks [2/2] [UU]

md1 : active raid1 sdb2[1]
976234364 blocks super 1.1 [2/1] [_U]
bitmap: 3/8 pages [12KB], 65536KB chunk

MDADM:
[root@praetorian ~]# mdadm -D /dev/md0
/dev/md0:
Version : 0.90
Creation Time : Wed Jan 13 15:27:27 2016
Raid Level : raid1
Array Size : 524224 (512.02 MiB 536.81 MB)
Used Dev Size : 524224 (512.02 MiB 536.81 MB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Wed Jan 13 17:03:11 2016
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 1
Spare Devices : 0

UUID : 1ca3ca64:f8155cd7:33e245c8:023725b8 (local to host praetorian)
Events : 0.20

Number Major Minor RaidDevice State
0 0 0 0 removed
1 8 17 1 active sync /dev/sdb1

2 8 1 - faulty /dev/sda1

GRUB.CONF:
default=0
timeout=30
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.32-573.12.1.el6.x86_64) - sda
root (hd0,0)
kernel /vmlinuz-2.6.32-573.12.1.el6.x86_64 ro root=/dev/mapper/vg_sys-lv00 rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_sys/lv06 rd_MD_UUID=cd696e97:ea0d2ba6:55ee4691:8e65ab1a SYSFONT=latarcyrheb-sun16 crashkernel=128M rd_LVM_LV=vg_sys/lv00 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM
initrd /initramfs-2.6.32-573.12.1.el6.x86_64.img
title CentOS (2.6.32-573.12.1.el6.x86_64) - sdb
root (hd1,0)
kernel /vmlinuz-2.6.32-573.12.1.el6.x86_64 ro root=/dev/mapper/vg_sys-lv00 rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_sys/lv06 rd_MD_UUID=cd696e97:ea0d2ba6:55ee4691:8e65ab1a SYSFONT=latarcyrheb-sun16 crashkernel=128M rd_LVM_LV=vg_sys/lv00 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM
initrd /initramfs-2.6.32-573.12.1.el6.x86_64.img

GRUB FIND:
grub> find /grub/grub.conf
find /grub/grub.conf
(hd0,0)
(hd1,0)

GRUB BOOTLOADERS (confirmed present):
[root@praetorian ~]# dd if=/dev/sda bs=1024b count=1|strings|head -n 15
1+0 records in
1+0 records out
524288 bytes (524 kB) copied, 0.00581775 s, 90.1 MB/s
ZRrI
D|f1
GRUB
Geom
Hard Disk
Read
Error
pPf1
Loading stage1.5
Geom
Read
Error
0.97
/grub/stage2 /grub/grub.conf
^_[]
[root@praetorian ~]# dd if=/dev/sdb bs=1024b count=1|strings|head -n 15
1+0 records in
1+0 records out
524288 bytes (524 kB) copied, 0.00919304 s, 57.0 MB/s
ZRrI
D|f1
GRUB
Geom
Hard Disk
Read
Error
pPf1
Loading stage1.5
Geom
Read
Error
0.97
/grub/stage2 /grub/grub.conf
^_[]

I generally believe the software (author).
Most likely you setup grub on the second drive whilst booted from the "first" drive. And didn't make any allowances for it by, for example, adjusting the map file.
Grub knows which drive is which - 0x80 for the first, 0x81 second ... This info gets written into the stage 1 (and 1.5) - hence the message I would guess. From the grub screen the following should re-write the stage 1/1.5 to allow you to boot

I am booting from the first drive, and then running grub followed by:

root (hd0,0)
setup (hd0)
root (hd1,0)
setup (1,0)

What am I missing out? Can you provide me an example of a map file change appropriate to what I am doing please?