[SOLVED] software raid1 MBR after drive replacement

beowulfnode · 04-06-2011, 03:21 AM

I have a ubuntu server 10.10 nas box with the / and /boot on a mdadm software raid1 on the first partition of 2 usb sticks as /dev/md0. There are no LVMs. The sata drives are for actual data storage. I've just had to replace one of the sticks, and I'm concerned about the boot process.

In particular I want to know how to install grub2 to both MBRs of the raid after replacing a disk?

looking at various instructions it says to run the command "grub" however the installed apt package grub-pc 1.98 doesn't have that command and conflicts with the package "grub". Things in the man docs seem to point to use the command grub-install but they all seem to be about building the /boot contents and not the MBR.

Some info on the system
/dev/md0 is made from
/dev/sdc1
/dev/sdb1

Code:

cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid1 sdb2[0] sdc2[1]
      997312 blocks [2/2] [UU]

md0 : active raid1 sdb1[0] sdc1[1]
      14647232 blocks [2/2] [UU]

unused devices: <none>

a small section of /boot/grub/grub.cfg contains

Code:

insmod raid
insmod mdraid
insmod part_msdos
insmod part_msdos
insmod ext2
set root='(md0)'

and

Code:

root@nas:/#grub-probe -t device /boot/grub
/dev/md0

widget · 04-07-2011, 12:51 PM

The command to install grub on any device is, as root;

Code:

grub-install /dev/<device>

I do not run servers or usb sticks or raid so I am not sure what your designation for the sticks are but going by your post I would guess that you need to do this or something very similar as root;

Code:

grub-install /dev/md1

if 1 is the correct identifier for the other stick.

It is best when asking about boot questions to get and run this script;
http://bootinfoscript.sourceforge.net/
and post the entire results file. This give all the info needed to figure out any boot problem assuming the guy reading it is bright enough.

I like to run it every once in a while on my box just to check things out. Handy little bugger.

beowulfnode · 04-07-2011, 07:15 PM

thanks for the pointer to that script.

since my last post I ran

Code:

grub-install /dev/sda
grub-install /dev/sdb

the usb sticks appear as normal /dev/sd* drives, software raids appear as /dev/md*
as drives are inserted and removed the /dev/sd* letters change but currently the sticks are /dev/sda and /dev/sdb.

The reason for running the OS from usb sticks is that all 6 motherboard sata ports are reserved entirely for data storage with zfs-fuse.

the contents of the results of the script are

Code:

                Boot Info Script 0.55    dated February 15th, 2010                    

============================= Boot Info Summary: ==============================

 => Grub 2 is installed in the MBR of /dev/sda and looks for (md0)/boot/grub.
 => Grub 2 is installed in the MBR of /dev/sdb and looks for (md0)/boot/grub.
 => Windows is installed in the MBR of /dev/sdd

sda1: _________________________________________________________________________

    File system:       linux_raid_member
    Boot sector type:  -
    Boot sector info:  

sda2: _________________________________________________________________________

    File system:       linux_raid_member
    Boot sector type:  -
    Boot sector info:  

sdb1: _________________________________________________________________________

    File system:       linux_raid_member
    Boot sector type:  -
    Boot sector info:  

sdb2: _________________________________________________________________________

    File system:       linux_raid_member
    Boot sector type:  -
    Boot sector info:  

md0: _________________________________________________________________________

    File system:       ext3
    Boot sector type:  -
    Boot sector info:  
    Operating System:  Ubuntu 10.10
    Boot files/dirs:   /boot/grub/grub.cfg /etc/fstab /boot/grub/core.img

md1: _________________________________________________________________________

    File system:       swap
    Boot sector type:  -
    Boot sector info:  

sde: _________________________________________________________________________

    File system:       zfs
    Boot sector type:  -
    Boot sector info:  
    Mounting failed:
mount: unknown filesystem type 'zfs'

sdf: _________________________________________________________________________

    File system:       zfs
    Boot sector type:  -
    Boot sector info:  
    Mounting failed:
mount: unknown filesystem type 'zfs'
mount: unknown filesystem type 'zfs'

sdg: _________________________________________________________________________

    File system:       zfs
    Boot sector type:  Unknown
    Boot sector info:  
    Mounting failed:
mount: unknown filesystem type 'zfs'
mount: unknown filesystem type 'zfs'
mount: unknown filesystem type 'zfs'

sdh: _________________________________________________________________________

    File system:       zfs
    Boot sector type:  -
    Boot sector info:  
    Mounting failed:
mount: unknown filesystem type 'zfs'
mount: unknown filesystem type 'zfs'
mount: unknown filesystem type 'zfs'
mount: unknown filesystem type 'zfs'

sdj: _________________________________________________________________________

    File system:       zfs
    Boot sector type:  -
    Boot sector info:  
    Mounting failed:
mount: unknown filesystem type 'zfs'
mount: unknown filesystem type 'zfs'
mount: unknown filesystem type 'zfs'
mount: unknown filesystem type 'zfs'
mount: unknown filesystem type 'zfs'

=========================== Drive/Partition Info: =============================

Drive: sda ___________________ _____________________________________________________

Disk /dev/sda: 16.0 GB, 16022241280 bytes
255 heads, 63 sectors/track, 1947 cylinders, total 31293440 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes

Partition  Boot         Start           End          Size  Id System

/dev/sda1    *          2,048    29,296,639    29,294,592  fd Linux raid autodetect
/dev/sda2          29,296,640    31,291,391     1,994,752  fd Linux raid autodetect


Drive: sdb ___________________ _____________________________________________________

Disk /dev/sdb: 16.0 GB, 16039018496 bytes
75 heads, 40 sectors/track, 10442 cylinders, total 31326208 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes

Partition  Boot         Start           End          Size  Id System

/dev/sdb1    *          3,000    29,297,999    29,295,000  fd Linux raid autodetect
/dev/sdb2          29,298,000    31,325,999     2,028,000  fd Linux raid autodetect


Drive: sdd ___________________ _____________________________________________________

Disk /dev/sdd: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes

Partition  Boot         Start           End          Size  Id System



blkid -c /dev/null: ____________________________________________________________

Device           UUID                                   TYPE       LABEL                         

/dev/md0         290e3c87-4889-4f21-b19e-9167a7a5e3f2   ext3       sys                           
/dev/md1         1d86812e-4bcc-4b77-9735-50ff615115c0   swap                                     
/dev/sda1        166d6ae4-4eac-ea87-61d5-599cc7ed6bd3   linux_raid_member                               
/dev/sda2        1e2fe92f-7399-fcc9-c4cc-e6f1620a581e   linux_raid_member                               
/dev/sda: PTTYPE="dos" 
/dev/sdb1        166d6ae4-4eac-ea87-61d5-599cc7ed6bd3   linux_raid_member                               
/dev/sdb2        1e2fe92f-7399-fcc9-c4cc-e6f1620a581e   linux_raid_member                               
/dev/sdb: PTTYPE="dos" 
/dev/sdd                                                zfs                                      
/dev/sde                                                zfs                                      
/dev/sdf                                                zfs                                      
/dev/sdg                                                zfs                                      
/dev/sdh                                                zfs                                      
/dev/sdj                                                zfs                                      

============================ "mount | grep ^/dev  output: ===========================

Device           Mount_Point              Type       Options

/dev/md0         /                        ext3       (rw,noatime,errors=remount-ro)


=========================== md0/boot/grub/grub.cfg: ===========================

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
  set have_grubenv=true
  load_env
fi
set default="0"
if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function recordfail {
  set recordfail=1
  if [ -n "${have_grubenv}" ]; then if [ -z "${boot_once}" ]; then save_env recordfail; fi; fi
}

function load_video {
  insmod vbe
  insmod vga
}

insmod raid
insmod mdraid
insmod part_msdos
insmod part_msdos
insmod ext2
set root='(md0)'
search --no-floppy --fs-uuid --set 290e3c87-4889-4f21-b19e-9167a7a5e3f2
if loadfont /usr/share/grub/unicode.pf2 ; then
  set gfxmode=640x480
  load_video
  insmod gfxterm
fi
terminal_output gfxterm
if [ "${recordfail}" = 1 ]; then
  set timeout=-1
else
  set timeout=2
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/05_debian_theme ###
set menu_color_normal=white/black
set menu_color_highlight=black/light-gray
### END /etc/grub.d/05_debian_theme ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Ubuntu, with Linux 2.6.35-23-server' --class ubuntu --class gnu-linux --class gnu --class os {
	recordfail
	insmod raid
	insmod mdraid
	insmod part_msdos
	insmod part_msdos
	insmod ext2
	set root='(md0)'
	search --no-floppy --fs-uuid --set 290e3c87-4889-4f21-b19e-9167a7a5e3f2
	linux	/boot/vmlinuz-2.6.35-23-server root=UUID=290e3c87-4889-4f21-b19e-9167a7a5e3f2 ro   quiet
	initrd	/boot/initrd.img-2.6.35-23-server
}
menuentry 'Ubuntu, with Linux 2.6.35-23-server (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os {
	recordfail
	insmod raid
	insmod mdraid
	insmod part_msdos
	insmod part_msdos
	insmod ext2
	set root='(md0)'
	search --no-floppy --fs-uuid --set 290e3c87-4889-4f21-b19e-9167a7a5e3f2
	echo	'Loading Linux 2.6.35-23-server ...'
	linux	/boot/vmlinuz-2.6.35-23-server root=UUID=290e3c87-4889-4f21-b19e-9167a7a5e3f2 ro single 
	echo	'Loading initial ramdisk ...'
	initrd	/boot/initrd.img-2.6.35-23-server
}
menuentry 'Ubuntu, with Linux 2.6.35-22-server' --class ubuntu --class gnu-linux --class gnu --class os {
	recordfail
	insmod raid
	insmod mdraid
	insmod part_msdos
	insmod part_msdos
	insmod ext2
	set root='(md0)'
	search --no-floppy --fs-uuid --set 290e3c87-4889-4f21-b19e-9167a7a5e3f2
	linux	/boot/vmlinuz-2.6.35-22-server root=UUID=290e3c87-4889-4f21-b19e-9167a7a5e3f2 ro   quiet
	initrd	/boot/initrd.img-2.6.35-22-server
}
menuentry 'Ubuntu, with Linux 2.6.35-22-server (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os {
	recordfail
	insmod raid
	insmod mdraid
	insmod part_msdos
	insmod part_msdos
	insmod ext2
	set root='(md0)'
	search --no-floppy --fs-uuid --set 290e3c87-4889-4f21-b19e-9167a7a5e3f2
	echo	'Loading Linux 2.6.35-22-server ...'
	linux	/boot/vmlinuz-2.6.35-22-server root=UUID=290e3c87-4889-4f21-b19e-9167a7a5e3f2 ro single 
	echo	'Loading initial ramdisk ...'
	initrd	/boot/initrd.img-2.6.35-22-server
}
### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/20_memtest86+ ###
menuentry "Memory test (memtest86+)" {
	insmod raid
	insmod mdraid
	insmod part_msdos
	insmod part_msdos
	insmod ext2
	set root='(md0)'
	search --no-floppy --fs-uuid --set 290e3c87-4889-4f21-b19e-9167a7a5e3f2
	linux16	/boot/memtest86+.bin
}
menuentry "Memory test (memtest86+, serial console 115200)" {
	insmod raid
	insmod mdraid
	insmod part_msdos
	insmod part_msdos
	insmod ext2
	set root='(md0)'
	search --no-floppy --fs-uuid --set 290e3c87-4889-4f21-b19e-9167a7a5e3f2
	linux16	/boot/memtest86+.bin console=ttyS0,115200n8
}
### END /etc/grub.d/20_memtest86+ ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###

================================ md0/etc/fstab: ================================

# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
# / was on /dev/md0 during installation
UUID=290e3c87-4889-4f21-b19e-9167a7a5e3f2 /               ext3    noatime,errors=remount-ro 0       1
# swap was on /dev/md1 during installation
# UUID=1d86812e-4bcc-4b77-9735-50ff615115c0 none            swap    sw              0       0
#
# File systems for storage in ram
#
none	/tmp		tmpfs	defaults,noatime	0	0
none	/var/run	tmpfs	defaults,noatime	0	0
none	/var/lock	tmpfs	defaults,noatime	0	0
none	/var/tmp	tmpfs	defaults,noatime	0	0
none	/var/log	tmpfs	defaults,noatime	0	0

==================== md0: Location of files loaded by Grub: ====================


  10.0GB: boot/grub/core.img
   9.9GB: boot/grub/grub.cfg
  10.0GB: boot/initrd.img-2.6.35-22-server
   9.9GB: boot/initrd.img-2.6.35-23-server
   9.9GB: boot/vmlinuz-2.6.35-22-server
   9.9GB: boot/vmlinuz-2.6.35-23-server
   9.9GB: initrd.img
  10.0GB: initrd.img.old
   9.9GB: vmlinuz
   9.9GB: vmlinuz.old
=========================== Unknown MBRs/Boot Sectors/etc =======================

Unknown BootLoader  on sdg

00000000  33 c0 8e d0 bc 00 7c 8e  c0 8e d8 be 00 7c bf 00  |3.....|......|..|
00000010  06 b9 00 02 fc f3 a4 50  68 1c 06 cb fb b9 04 00  |.......Ph.......|
00000020  bd be 07 80 7e 00 00 7c  0b 0f 85 10 01 83 c5 10  |....~..|........|
00000030  e2 f1 cd 18 88 56 00 55  c6 46 11 05 c6 46 10 00  |.....V.U.F...F..|
00000040  b4 41 bb aa 55 cd 13 5d  72 0f 81 fb 55 aa 75 09  |.A..U..]r...U.u.|
00000050  f7 c1 01 00 74 03 fe 46  10 66 60 80 7e 10 00 74  |....t..F.f`.~..t|
00000060  26 66 68 00 00 00 00 66  ff 76 08 68 00 00 68 00  |&fh....f.v.h..h.|
00000070  7c 68 01 00 68 10 00 b4  42 8a 56 00 8b f4 cd 13  ||h..h...B.V.....|
00000080  9f 83 c4 10 9e eb 14 b8  01 02 bb 00 7c 8a 56 00  |............|.V.|
00000090  8a 76 01 8a 4e 02 8a 6e  03 cd 13 66 61 73 1e fe  |.v..N..n...fas..|
000000a0  4e 11 0f 85 0c 00 80 7e  00 80 0f 84 8a 00 b2 80  |N......~........|
000000b0  eb 82 55 32 e4 8a 56 00  cd 13 5d eb 9c 81 3e fe  |..U2..V...]...>.|
000000c0  7d 55 aa 75 6e ff 76 00  e8 8a 00 0f 85 15 00 b0  |}U.un.v.........|
000000d0  d1 e6 64 e8 7f 00 b0 df  e6 60 e8 78 00 b0 ff e6  |..d......`.x....|
000000e0  64 e8 71 00 b8 00 bb cd  1a 66 23 c0 75 3b 66 81  |d.q......f#.u;f.|
000000f0  fb 54 43 50 41 75 32 81  f9 02 01 72 2c 66 68 07  |.TCPAu2....r,fh.|
00000100  bb 00 00 66 68 00 02 00  00 66 68 08 00 00 00 66  |...fh....fh....f|
00000110  53 66 53 66 55 66 68 00  00 00 00 66 68 00 7c 00  |SfSfUfh....fh.|.|
00000120  00 66 61 68 00 00 07 cd  1a 5a 32 f6 ea 00 7c 00  |.fah.....Z2...|.|
00000130  00 cd 18 a0 b7 07 eb 08  a0 b6 07 eb 03 a0 b5 07  |................|
00000140  32 e4 05 00 07 8b f0 ac  3c 00 74 fc bb 07 00 b4  |2.......<.t.....|
00000150  0e cd 10 eb f2 2b c9 e4  64 eb 00 24 02 e0 f8 24  |.....+..d..$...$|
00000160  02 c3 49 6e 76 61 6c 69  64 20 70 61 72 74 69 74  |..Invalid partit|
00000170  69 6f 6e 20 74 61 62 6c  65 00 45 72 72 6f 72 20  |ion table.Error |
00000180  6c 6f 61 64 69 6e 67 20  6f 70 65 72 61 74 69 6e  |loading operatin|
00000190  67 20 73 79 73 74 65 6d  00 4d 69 73 73 69 6e 67  |g system.Missing|
000001a0  20 6f 70 65 72 61 74 69  6e 67 20 73 79 73 74 65  | operating syste|
000001b0  6d 00 00 00 00 62 7a 99  2a fd 33 e9 00 00 00 00  |m....bz.*.3.....|
000001c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200

the windows mbr is only a left over from that disk being in another system. From the looks of that script things look fine now for a restart

widget · 04-07-2011, 08:42 PM

I am still not clear as to how you are set up, ignorance of servers is almost total.

However, If your usb devices are seen by grub as sda and sdb, and you (for some reason) are taking them out an putting them back in, if either of them are seen as sda, it should boot as long as it can read what ever is seen as md0 (what ever that may be).

beowulfnode · 04-10-2011, 11:00 PM

well it seems that doing grub-install as above was what was needed for installing grub to the mbr when a disk was replaced in a raid array. No other grub commands were required to get the mbr on to the new usb disk.

For Widget
Something not immediately obvious is that between my first and second posts the letters for the usb sticks changed from
/dev/sbc to /dev/sda
/dev/sdb stayed the same

I'm taking usb sticks out because they are broken. Because it is using a raid 1 array, where each usb stick is a copy of the other, it doesn't matter if one of the usb sticks fails, the system keeps working. This gives me time to replace the broken usb stick.

USB and sata disks are represented as /dev/sd* while a combination of partitions on those devices in a software raid configuration is represented as /dev/md* and is treated like a partition.

My system is made up of
2 usb sticks
6 sata drives

The 2 usb sticks are combined together to be copies of each other in a raid 1 configuration for redundancy for when one fails, the system keeps working. Ubuntu is installed to these USB sticks, specifically to the raid 1 'partition' /dev/md0 that is made up of sda1 and sdb1 (the usb sticks first partitions). Since only the partitions sda1 and sdb1 are part of the raid 1 array the mbr needs to be put on to any new replacement disk that is supposed to be able to boot the system.

The 6 sata drives are used for data storage only. They are combined with a storage system called zfs-fuse, to be presented to the network as a Network attached storage. The sata disks have nothing to do with the boot sequence.

widget · 04-10-2011, 11:17 PM

Well, as old as I am, I learned something new. mdx neat.

I also was of the impression that it was safer to have a separate /boot partition in cases of non standard installs. Must just need that for encrypted stuff now. I know that some earlier grub2 versions were a little weak on raid too. Good to know that has been fixed.