Raid1 array won't start on boot

brew1brew · 01-02-2011, 09:02 PM

I created a raid1 disk with Disk Utilities on with Karmic, then upgraded my system to Lucid then Maverick. This Raid disk is just a data store, I'm not booting off of it.

when I reboot the raid1 disk does not start. I have to go into Disk Utilities, stop the array and then start it. Then it comes up and I can mount it. I ran dpkg-reconfigure mdadm and it created a valid entry in mdadm.conf. but the array still does not start on boot. I want to have it auto mount with fstab but need to make sure the array starts first.

I searched but didn't find the answer on here or on Ubuntu forums. thanks in advance for your help.

Les

crts · 01-03-2011, 04:21 AM

Quote:

Originally Posted by brew1brew

when I reboot the raid1 disk does not start. I have to go into Disk Utilities, stop the array and then start it.

Hi,

I am a bit confused by this statement because it contradicts itself a bit. If you have to stop it first then it was started. Anyway, please post your mdadm.conf and your fstab here. Also run blkid on your raid-devices:

Code:

sudo blkid /dev/sdXX
sudo blkid /dev/sdXY

They should all have the same UUID.
Is it a 'pure' raid or do you have a more complex setup, e.g. encryption, lvm ?

brew1brew · 01-03-2011, 07:54 AM

Quote:

Originally Posted by crts

Hi,

I am a bit confused by this statement because it contradicts itself a bit. If you have to stop it first then it was started. Anyway, please post your mdadm.conf and your fstab here. Also run blkid on your raid-devices:

Code:

sudo blkid /dev/sdXX
sudo blkid /dev/sdXY

They should all have the same UUID.
Is it a 'pure' raid or do you have a more complex setup, e.g. encryption, lvm ?

Good question, when I bring up Disk Utility it shows the DataArray as "not started" but the only button available is Stop RAID Array. I click stop then it becomes "start RAID Array" and I click it and it starts fine.

Here's the output you asked for. Thanks for looking.

Oh, last question, just a pure RAID setup, nothing else special.

Code:

les@les-desktop:~$ sudo blkid /dev/sdc1
/dev/sdc1: UUID="539414cc-25fe-7412-8697-1df6b4d8489c" LABEL=":DataMirror" TYPE="linux_raid_member" 
les@les-desktop:~$ sudo blkid /dev/sdd1
/dev/sdd1: UUID="539414cc-25fe-7412-8697-1df6b4d8489c" LABEL=":DataMirror" TYPE="linux_raid_member" 
les@les-desktop:~$ cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md/DataMirror level=raid1 metadata=1.2 num-devices=2 UUID=539414cc:25fe7412:86971df6:b4d8489c name=:DataMirror

# This file was auto-generated on Sun, 02 Jan 2011 20:07:56 -0600
# by mkconf $Id$
les@les-desktop:~$

crts · 01-03-2011, 08:24 AM

This sounds strange. And you say that it doesn't help if you put it in fstab, like
/dev/md/DataMirror /mountpoint ....

Before you restart with the disk utility is it present in /dev? What is the output of
ls -l /dev/md/DataMirror

Are you sure the array is clean? Can you post the output of
sudo mdadm --detail /dev/md/DataMirror

Since you said that you upgraded, maybe the array is resyncing. Depending on the array size this can take up to a couple of hours. If you manually stop and restart it you might interrupt that process and it restarts next time you boot. Just a thought.

brew1brew · 01-03-2011, 10:20 AM

Quote:

Originally Posted by crts

This sounds strange. And you say that it doesn't help if you put it in fstab, like
/dev/md/DataMirror /mountpoint ....

I haven't put it in fstab yet. I wanted to make sure it was running first.

Quote:

Originally Posted by crts

Before you restart with the disk utility is it present in /dev? What is the output of ls -l /dev/md/DataMirror

I'll have to reboot and look, when I get home from work tonight.

Quote:

Originally Posted by crts

Are you sure the array is clean? Can you post the output of
sudo mdadm --detail /dev/md/DataMirror

Yes, I did an array check/repair in Disk Utility, and yes it took quite a long time to do, it's 1.5TB and I moved over 100 meg onto it.

Quote:

Originally Posted by crts

Since you said that you upgraded, maybe the array is resyncing. Depending on the array size this can take up to a couple of hours. If you manually stop and restart it you might interrupt that process and it restarts next time you boot. Just a thought.

Disk Utility doesn't indicate that it's resyncing, it just says "not started" and I don't have an option to mount it in disk utility until I stop and start it. Is there a way to see if it's syncing?

I appreciate your comments/ideas, This is my first time playing with RAID arrays. I decided that I should have a mirrored data store for my photo's, I'm a hobby photog.

Les

crts · 01-03-2011, 01:47 PM

Quote:

Originally Posted by brew1brew

Is there a way to see if it's syncing?

You could post the output of the commands I posted above. I am not familiar with disk utility so I cannot say whether you can see if it is doing some maintenance/consistency work in the background.Can you mount the array via CLI before you restart it?
sudo mount /dev/md/DataMirror /mountpoint

In addition you can post the output of
cat /proc/mdstat

brew1brew · 01-03-2011, 10:05 PM

ok, here we go.

Quote:

Originally Posted by crts

This sounds strange. And you say that it doesn't help if you put it in fstab, like
/dev/md/DataMirror /mountpoint ....

So, until we get an active array I don't think this would do anything. nothing to mount if it's not active.

Quote:

Originally Posted by crts

Before you restart with the disk utility is it present in /dev? What is the output of
ls -l /dev/md/DataMirror

ok, here's the first one.

Code:

les@les-desktop:~$ sudo ls -l /dev/md/DataMirror
brw------- 1 root root 254, 8128 2011-01-03 21:04 /dev/md/DataMirror
les@les-desktop:~$

Also was curious what was in /dev/md/ so here it is, I thought it was sort of strange that there are 5 of them.

Code:

les@les-desktop:~$ sudo ls -l /dev/md/
total 0
brw------- 1 root root 254, 8128 2011-01-03 21:04 DataMirror
brw------- 1 root root 254, 8129 2011-01-03 21:04 DataMirror1
brw------- 1 root root 254, 8130 2011-01-03 21:04 DataMirror2
brw------- 1 root root 254, 8131 2011-01-03 21:04 DataMirror3
brw------- 1 root root 254, 8132 2011-01-03 21:04 DataMirror4
les@les-desktop:~$

Quote:

Originally Posted by crts

Are you sure the array is clean? Can you post the output of
sudo mdadm --detail /dev/md/DataMirror

Code:

les@les-desktop:~$ sudo mdadm --detail /dev/md/DataMirror
mdadm: md device /dev/md/DataMirror does not appear to be active.
les@les-desktop:~$

As I said before, not active.

Quote:

Originally Posted by crts

Since you said that you upgraded, maybe the array is resyncing. Depending on the array size this can take up to a couple of hours. If you manually stop and restart it you might interrupt that process and it restarts next time you boot. Just a thought.

There doesn't seem to be anything in the output that indicates syncing.

Now, here are some photos of Disk Utility
1 & 2 are physical hard disks
http://anthisphoto.smugmug.com/Photo...75_USMNn-M.png
http://anthisphoto.smugmug.com/Photo...68_ehe6R-L.png
3 is the RAID1 array after boot, before I click "stop" and "Start"
http://anthisphoto.smugmug.com/Photo...64_i9ar8-L.png
4 is after I stopped and started
http://anthisphoto.smugmug.com/Photo...70_Yct3Z-L.png

also here is the output of "sudo mdadm --detail /dev/md0" I used "/dev/md0" because that's what it was listed as in Disk Utility screen after it was started.

Code:

les@les-desktop:~$ sudo mdadm --detail /dev/md0
/dev/md0:
        Version : 01.02
  Creation Time : Fri Dec 31 13:01:20 2010
     Raid Level : raid1
     Array Size : 1465135865 (1397.26 GiB 1500.30 GB)
  Used Dev Size : 2930271730 (2794.52 GiB 3000.60 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Mon Jan  3 21:04:22 2011
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : :DataMirror
           UUID : 539414cc:25fe7412:86971df6:b4d8489c
         Events : 273

    Number   Major   Minor   RaidDevice State
       0       8       49        0      active sync   /dev/sdd1
       1       8       33        1      active sync   /dev/sdc1
les@les-desktop:~$

Of coarse I can mount it after it's started.

crts · 01-04-2011, 05:13 AM

Ok,

after looking at the last output I noticed that /dev/md/DataMirrorX is only read/writeable by root. Here are the permissions of my array:

Code:

$ ls -l /dev/md0 
brw-rw---- 1 root disk 9, 0 2011-01-04 11:48 /dev/md0

As you can see, the group it is also r/w for group disk. Maybe this has something to do with it. Can you try changing the permissions/group to the whatever your /dev/md0 has after you restart it? The permissions of the folder /dev/md/ might also be improtant.

Code:

ls -ld /dev/md/
ls -l /dev/md0

After you correct the permissions/group I would advise you to stop ALL arrays by using the CLI. That is how I manage my arrays. I do not like the disk utility.

Code:

sudo mdadm -S /dev/md0
sudo mdadm -S /dev/md/DataArray
sudo mdadm -S /dev/md/DataArray1
sudo mdadm -S /dev/md/DataArray2
sudo mdadm -S /dev/md/DataArray3
sudo mdadm -S /dev/md/DataArray4
...

Afterwards reassemble it with

Code:

sudo mdadm --assemble /dev/md/DataArray -u 539414cc:25fe7412:86971df6:b4d8489c

Reboot and see if it stays persistent.

If this does not work you might have to compromise and edit you mdadm.conf to name your array /dev/md0.

buckminster · 01-07-2011, 09:13 PM

brew1brew, I had an identical problem to yours... took me a little while to resolve it but the answer was quite simple.
Change your mdadm.conf file as follows and it should work.

Change
ARRAY /dev/md/DataMirror level=raid1 metadata=1.2 num-devices=2 UUID=539414cc:25fe7412:86971df6:b4d8489c name=

ataMirror

to

ARRAY /dev/md0 level=raid1 metadata=1.2 num-devices=2 UUID=539414cc:25fe7412:86971df6:b4d8489c name=

ataMirror

Cheers

brew1brew · 01-09-2011, 09:58 PM

So, I noticed that my kernel was only loading 1 CPU, I have an AMD Phenom II 3x, and read up and found that maybe it was a bios issue. So I updated my bios, and now when I boot the RAID1 array seems to come up. and the kernel loads all 3 cores.

I say "seems" to because after I booted this is what I get

Code:

les@les-desktop:~$ sudo mdadm --detail --scan
[sudo] password for les: 
ARRAY /dev/md/DataMirror level=raid1 num-devices=2 metadata=01.02 name=:DataMirror UUID=539414cc:25fe7412:86971df6:b4d8489c

and this is my current mdadm.conf

Code:

les@les-desktop:~$ cat /etc/md/mdadm.conf
cat: /etc/md/mdadm.conf: No such file or directory
les@les-desktop:~$ cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md/DataMirror level=raid1 num-devices=2 metadata=1.2 name=:DataMirror UUID=539414cc:25fe7412:86971df6:b4d8489c

# This file was auto-generated on Sun, 02 Jan 2011 20:07:56 -0600
# by mkconf $Id$
les@les-desktop:~$

So I put this in my fstab file

Code:

/dev/md/DataMirror /home/les/DataMirror ext4 user,nodev,nosuid,relatime 0 0

now when I boot, I get an error that says the mount point is not ready and /home/les/DataMirror did not mount, wait, S to skip, or M to manually fix.

So I skip it and check and it's there, I do mount -a and it mounts fine. So new problem now.

but hey, I'm closer

crts · 01-09-2011, 10:15 PM

Quote:

Originally Posted by brew1brew

So I put this in my fstab file

Code:

/dev/md/DataMirror /home/les/DataMirror ext4 user,nodev,nosuid,relatime 0 0

now when I boot, I get an error that says the mount point is not ready and /home/les/DataMirror did not mount, wait, S to skip, or M to manually fix.

So I skip it and check and it's there, I do mount -a and it mounts fine. So new problem now.

but hey, I'm closer

Do you have a separate /home partition? Maybe it can't be mounted because /home has not been mounted, yet. So maybe you should post your complete /etc/fstab to investigate further.

brew1brew · 01-10-2011, 07:59 AM

Quote:

Originally Posted by crts

Do you have a separate /home partition? Maybe it can't be mounted because /home has not been mounted, yet. So maybe you should post your complete /etc/fstab to investigate further.

No, just have a root partition.

Here's the fstab

Code:

les@les-desktop:~$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'vol_id --uuid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>                                                                                                                             
proc            /proc           proc    defaults        0       0                                                                                                                                  
# / was on /dev/sda1 during installation                                                                                                                                                           
UUID=62442335-f15a-499f-b4b7-2d4f02ec681f /               ext3    relatime,errors=remount-ro 0       1                                                                                             
# swap was on /dev/sda5 during installation
UUID=556eb689-a1e5-48bb-957f-100469cc0fc9 none            swap    sw              0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto,exec,utf8 0       0
# none /proc/bus/usb usbfs devgid=1001,devmode=664 0 0
/dev/md/DataMirror /home/les/DataMirror ext4 user,nodev,nosuid,relatime 0 0

#
les@les-desktop:~$

crts · 01-10-2011, 08:30 AM

Ok,

have you also tried to mount it with its UUID in fstab?
In any case, when you reboot and it hangs again then press 'M'. This will give you a root shell and you can further investigate. Issue these commands in the 'rescue' shell:

Code:

# what has been mounted so far
mount

# are there any permission issues?
ls -ld /home
ls -l /home
ls -l /home/les
ls -l /home/les/DataMirror

# is the array OK?
mdadm --detail --scan
mdadm

Since those commands will probably give a lot of output you might want to redirect both stderr and stdout to a file, e.g. /root/mount_output, ls_ld_home_output etc., so you can paste it here.

PS:
Just in case you are not comfortable with the CLI, here is a small example how you can redirect stderr as well as stdout:

Code:

mdadm --detail --scan &> /root/mdadm_detail_scan_output