[SOLVED] RAID 1 will not boot unless both disks connected

frznchckn · 05-19-2015, 09:45 PM

I installed Debian Jessie using the graphical installer and created a RAID 1 partition for / and one for swap. The install proceeded fine and I was able to boot.

I decided to try out the RAID 1 by disconnecting one of the drives and attempting to boot. Regardless of which drive is disconnected, I receive the following

Code:

 — Boot args (cat /proc/cmdline)

— Check rootdelay= (did the system wait long enough?)

— Check root= (did the system wait for the right device?)

— Missing modules (cat /proc/modules; ls /dev)

ALERT! /dev/disk/by-uuid/xxxxxxxxx does not exist. Dropping to a shell! 
BusyBox v.1.21.1 (ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash) 
Enter 'help' for list of built-in commands .

(initramfs)..

I originally thought that boot code was not installed on the second hard drive, but that would not explain why the "first" hard drive will not boot by itself.

Suggestions on where to start? The system boots fine with both hard drives installed and RAID1 partition is verified to be running based on /proc/mdstat.

Thanks.

smallpond · 05-20-2015, 04:13 PM

Please post contents of grub.conf and the datails of your raid volume:

Code:

mdadm --detail /dev/md1

syg00 · 05-20-2015, 08:12 PM

I imagine you will have to compel the array to run degraded. See "--run" in the mdadm manpage.

You may be able to fiddle the initrd to do it right (I don't use Debian or mdadm, so my treat guesswork for what it is).

jefro · 05-20-2015, 08:49 PM

When you removed this drive did you go to the raid controller bios and correct the config there?

frznchckn · 05-20-2015, 10:22 PM

I'm also looking into what initramfs is doing.

Raid Controller BIOS? I don't think that applies since it's a software RAID?

Raid volume specifics

Code:

$ mdadm --detail /dev/md1

/dev/md1:
        Version : 1.2
  Creation Time : Sun May 17 21:33:59 2015
     Raid Level : raid1
     Array Size : 945380352 (901.58 GiB 968.07 GB)
  Used Dev Size : 945380352 (901.58 GiB 968.07 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Wed May 20 21:57:38 2015
          State : clean 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : box01:1  (local to host box01)
           UUID : fd5ebc37:a023aa8f:bdb1c261:978f8969
         Events : 2373

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       8       18        1      active sync   /dev/sdb2

grub.cfg

Code:

# cat /boot/grub/grub.cfg 
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
  set have_grubenv=true
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="0"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}
function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod part_msdos
insmod part_msdos
insmod diskfilter
insmod mdraid1x
insmod ext2
set root='mduuid/fd5ebc37a023aa8fbdb1c261978f8969'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root --hint='mduuid/fd5ebc37a023aa8fbdb1c261978f8969'  b6adab9a-f85d-4902-b883-5c3176ca2d30
else
  search --no-floppy --fs-uuid --set=root b6adab9a-f85d-4902-b883-5c3176ca2d30
fi
    font="/usr/share/grub/unicode.pf2"
fi

if loadfont $font ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=en_US
  insmod gettext
fi
terminal_output gfxterm
if [ "${recordfail}" = 1 ] ; then
  set timeout=-1
else
  if [ x$feature_timeout_style = xy ] ; then
    set timeout_style=menu
    set timeout=5
  # Fallback normal timeout code in case the timeout_style feature is
  # unavailable.
  else
    set timeout=5
  fi
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/05_debian_theme ###
insmod part_msdos
insmod part_msdos
insmod diskfilter
insmod mdraid1x
insmod ext2
set root='mduuid/fd5ebc37a023aa8fbdb1c261978f8969'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root --hint='mduuid/fd5ebc37a023aa8fbdb1c261978f8969'  b6adab9a-f85d-4902-b883-5c3176ca2d30
else
  search --no-floppy --fs-uuid --set=root b6adab9a-f85d-4902-b883-5c3176ca2d30
fi
insmod png
if background_image /usr/share/images/desktop-base/lines-grub.png; then
  set color_normal=white/black
  set color_highlight=black/white
else
  set menu_color_normal=cyan/blue
  set menu_color_highlight=white/blue
fi
### END /etc/grub.d/05_debian_theme ###

### BEGIN /etc/grub.d/10_linux ###
function gfxmode {
	set gfxpayload="${1}"
}
set linux_gfx_mode=
export linux_gfx_mode
menuentry 'Debian GNU/Linux' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-b6adab9a-f85d-4902-b883-5c3176ca2d30' {
	load_video
	insmod gzio
	if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
	insmod part_msdos
	insmod part_msdos
	insmod diskfilter
	insmod mdraid1x
	insmod ext2
	set root='mduuid/fd5ebc37a023aa8fbdb1c261978f8969'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint='mduuid/fd5ebc37a023aa8fbdb1c261978f8969'  b6adab9a-f85d-4902-b883-5c3176ca2d30
	else
	  search --no-floppy --fs-uuid --set=root b6adab9a-f85d-4902-b883-5c3176ca2d30
	fi
	echo	'Loading Linux 3.16.0-4-amd64 ...'
	linux	/boot/vmlinuz-3.16.0-4-amd64 root=UUID=b6adab9a-f85d-4902-b883-5c3176ca2d30 ro initrd=/install/gtk/initrd.gz quiet
	echo	'Loading initial ramdisk ...'
	initrd	/boot/initrd.img-3.16.0-4-amd64
}
submenu 'Advanced options for Debian GNU/Linux' $menuentry_id_option 'gnulinux-advanced-b6adab9a-f85d-4902-b883-5c3176ca2d30' {
	menuentry 'Debian GNU/Linux, with Linux 3.16.0-4-amd64' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-3.16.0-4-amd64-advanced-b6adab9a-f85d-4902-b883-5c3176ca2d30' {
		load_video
		insmod gzio
		if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
		insmod part_msdos
		insmod part_msdos
		insmod diskfilter
		insmod mdraid1x
		insmod ext2
		set root='mduuid/fd5ebc37a023aa8fbdb1c261978f8969'
		if [ x$feature_platform_search_hint = xy ]; then
		  search --no-floppy --fs-uuid --set=root --hint='mduuid/fd5ebc37a023aa8fbdb1c261978f8969'  b6adab9a-f85d-4902-b883-5c3176ca2d30
		else
		  search --no-floppy --fs-uuid --set=root b6adab9a-f85d-4902-b883-5c3176ca2d30
		fi
		echo	'Loading Linux 3.16.0-4-amd64 ...'
		linux	/boot/vmlinuz-3.16.0-4-amd64 root=UUID=b6adab9a-f85d-4902-b883-5c3176ca2d30 ro initrd=/install/gtk/initrd.gz quiet
		echo	'Loading initial ramdisk ...'
		initrd	/boot/initrd.img-3.16.0-4-amd64
	}
	menuentry 'Debian GNU/Linux, with Linux 3.16.0-4-amd64 (recovery mode)' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-3.16.0-4-amd64-recovery-b6adab9a-f85d-4902-b883-5c3176ca2d30' {
		load_video
		insmod gzio
		if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
		insmod part_msdos
		insmod part_msdos
		insmod diskfilter
		insmod mdraid1x
		insmod ext2
		set root='mduuid/fd5ebc37a023aa8fbdb1c261978f8969'
		if [ x$feature_platform_search_hint = xy ]; then
		  search --no-floppy --fs-uuid --set=root --hint='mduuid/fd5ebc37a023aa8fbdb1c261978f8969'  b6adab9a-f85d-4902-b883-5c3176ca2d30
		else
		  search --no-floppy --fs-uuid --set=root b6adab9a-f85d-4902-b883-5c3176ca2d30
		fi
		echo	'Loading Linux 3.16.0-4-amd64 ...'
		linux	/boot/vmlinuz-3.16.0-4-amd64 root=UUID=b6adab9a-f85d-4902-b883-5c3176ca2d30 ro single initrd=/install/gtk/initrd.gz
		echo	'Loading initial ramdisk ...'
		initrd	/boot/initrd.img-3.16.0-4-amd64
	}
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###

### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/20_memtest86+ ###
menuentry "Memory test (memtest86+)" {
	insmod part_msdos
	insmod part_msdos
	insmod diskfilter
	insmod mdraid1x
	insmod ext2
	set root='mduuid/fd5ebc37a023aa8fbdb1c261978f8969'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint='mduuid/fd5ebc37a023aa8fbdb1c261978f8969'  b6adab9a-f85d-4902-b883-5c3176ca2d30
	else
	  search --no-floppy --fs-uuid --set=root b6adab9a-f85d-4902-b883-5c3176ca2d30
	fi
	linux16	/boot/memtest86+.bin
}
menuentry "Memory test (memtest86+, serial console 115200)" {
	insmod part_msdos
	insmod part_msdos
	insmod diskfilter
	insmod mdraid1x
	insmod ext2
	set root='mduuid/fd5ebc37a023aa8fbdb1c261978f8969'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint='mduuid/fd5ebc37a023aa8fbdb1c261978f8969'  b6adab9a-f85d-4902-b883-5c3176ca2d30
	else
	  search --no-floppy --fs-uuid --set=root b6adab9a-f85d-4902-b883-5c3176ca2d30
	fi
	linux16	/boot/memtest86+.bin console=ttyS0,115200n8
}
menuentry "Memory test (memtest86+, experimental multiboot)" {
	insmod part_msdos
	insmod part_msdos
	insmod diskfilter
	insmod mdraid1x
	insmod ext2
	set root='mduuid/fd5ebc37a023aa8fbdb1c261978f8969'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint='mduuid/fd5ebc37a023aa8fbdb1c261978f8969'  b6adab9a-f85d-4902-b883-5c3176ca2d30
	else
	  search --no-floppy --fs-uuid --set=root b6adab9a-f85d-4902-b883-5c3176ca2d30
	fi
	multiboot	/boot/memtest86+_multiboot.bin
}
menuentry "Memory test (memtest86+, serial console 115200, experimental multiboot)" {
	insmod part_msdos
	insmod part_msdos
	insmod diskfilter
	insmod mdraid1x
	insmod ext2
	set root='mduuid/fd5ebc37a023aa8fbdb1c261978f8969'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint='mduuid/fd5ebc37a023aa8fbdb1c261978f8969'  b6adab9a-f85d-4902-b883-5c3176ca2d30
	else
	  search --no-floppy --fs-uuid --set=root b6adab9a-f85d-4902-b883-5c3176ca2d30
	fi
	multiboot	/boot/memtest86+_multiboot.bin console=ttyS0,115200n8
}
### END /etc/grub.d/20_memtest86+ ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/30_uefi-firmware ###
### END /etc/grub.d/30_uefi-firmware ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###

brak44 · 05-20-2015, 10:30 PM

created a RAID 1 partition for / and one for swap.

What about the boot partition?
The boot partition will have a the boot flag set.

frznchckn · 05-20-2015, 10:40 PM

boot is not in a separate partition, it's just under / when the system boots. Since the system will boot with both hard drives attached I don't think it's a boot flag issue?

Also, I removed a hard drive, booted, and got dumped into the initramfs "shell". I was able to get the RAID 1 array (/dev/md1) started but now I'm still in initramfs and unsure how to force the system to try and start again.... I'm looking for that in the documentation right now.

frznchckn · 05-20-2015, 10:50 PM

Apparently typing "exit" closes the initramfs shell and it tries to boot again.

I had to wait something to timeout for the second hard drive I think, but then the system was able to boot!

So I *think* I'm in a good place now.

I appreciate the responses and ideas. It certainly focused the debugging.

syg00 · 05-20-2015, 11:23 PM

Yes, exit is correct. Be aware that you'll need to resync the array when you add the disk back. Looks like --re-add might be the best option.

frznchckn · 05-21-2015, 06:41 AM

Yes, thanks. In hindsight it makes sense that the array will continue to run in degraded fashion

Code:

mdadm --manage /dev/md1 -add /dev/sdb1

Added the "failed" disk back and it rebuilt quite quickly.

Very much in a happy place now.

Thanks!