Linux - GeneralThis Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
The problem is that when I reboot my drive assignments change; what was /dev/sda becomes /dev/sdc and so on. GRUB is able to boot, and mdadm is able to start the array... BUT since I have a spare, mdadm sometimes assembles the array thinking that the spare is a degraded member drive, and immediately begins to rebuild on the spare, effectively swapping out the correct drive as the spare.
A better description of the scenario:
(I will use numbers to designate the actual drive IDs, letters to describe their device assignments at boot).
The system worked perfectly with Ubuntu Dapper, but this problem has persisted through upgrades to Edgy, Feisty and now Gutsy.
When I set the machine up:
Code:
DRIVE1: sda,boot drive
DRIVE2-DRIVE6: sd[b-f] are the RAID5 drives (4+1 parity)
DRIVE7: sdg, RAID spare
$cat /etc/mdadm/mdadm.conf
DEVICE partitions
ARRAY /dev/md0 level=raid5 num-devices=5 spares=1 UUID=e7356e2b:71e53a26:94b87bc7:e6a9e6b2
$mdadm --detail /dev/md0
/dev/md0:
Version : 00.90.03
Creation Time : Sat Apr 7 23:32:58 2007
Raid Level : raid5
Array Size : 48869120 (46.61 GiB 50.04 GB)
Used Dev Size : 12217280 (11.65 GiB 12.51 GB)
Raid Devices : 5
Total Devices : 6
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Mon Nov 26 15:05:07 2007
State : clean
Active Devices : 5
Working Devices : 6
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 64K
UUID : e7356e2b:71e53a26:94b87bc7:e6a9e6b2
Events : 0.2601904
Number Major Minor RaidDevice State
0 8 65 0 active sync /dev/sdb1
1 8 81 1 active sync /dev/sdc1
2 8 17 2 active sync /dev/sdd1
3 8 1 3 active sync /dev/sde1
4 8 33 4 active sync /dev/sdf1
5 8 97 - spare /dev/sdg1
So all is well. Then I reboot and GRUB (or whatever is in charge of that) seeming arbitrarily reassigns my drives like this:
Code:
DRIVE1: sdd,boot drive
DRIVE2-DRIVE6: sda,b,e,f,g are the RAID5 drives (4+1 parity)
DRIVE7: sdc, RAID spare
(it looks like the 2nd drive controller was assigned before the first drive controller).
The problem is that mdadm "sees" DRIVE7 (the spare) before some of the other drives in the array, and includes it when it assembles. DRIVE4
ends up as the spare. But since DRIVE7 was the spare, mdadm decides this is a degraded array and immediately begins to rebuild on DRIVE7.
Note, if I were not to have a spare drive, things would work fine, even though the drive assignments are changed.
I can see two cures for this:
1) Prevent the drives from being assigned differently each time. Is there any way to do this?
2) Fix mdadm to recognize UUIDs.
Looks like you will need to write some udev rules to force the drives to come up as the same device each time. if the drives are all identical the rules will have to reference the drive serial numbers..
Doesn't mdadm look at superblocks for info? Isn't the problem here that mdadm encounters the spare drive and assumes it's degraded BEFORE it encounters the actual member drive?
*OR* does mdadm scan through the devices in order /dev/sda, /dev/sdb... and so on?
My mdadm.conf file doesn't mention devices, just UUIDs.
I am not going to try this. Looking at the logs, md_mod is a kernel module and is loaded before udev runs. It appears that the arrays are assembled well before udev rules are run. It seems to me that md kicks in as soon as the drives are assigned.
From the dmesg excerpt below (clutter removed and line-breaks added), it looks to me like my system is inspecting my 'secondary' drive controller first (three drives), attempting to start the RAID5 on those drives unsuccessfully, then inspecting the main drive controller (MB) and then staring the arrays.
Here's the questions I have...
-Why, if GRUB sees my boot partition as (hd0,0) does the system then inspect the OTHER controller first and end up assigning that hd0 drive to sdd?
-Why does md attempt to start after inspecting the 2nd controller, first binding to sdc and then again to sd[abc]?
-When md attempts to bind again after it has inpected all 7 drives, note that it binds in this order: sd[fbacge]1 and assembles: sd[ecabf] for my first partition (which turns out to be the last-boot configuration), but for the second partition binds in order of sd[fbgcae]2 and assembles sd[ecgbf]2? (Which causes a rebuild of md1)
-Is there any way to run a udev rule as the drive gets detected? That is, at the point in the log where it posts "[XX] sd 0:0:0:0: [sda] 488397168 512-byte hardware sectors (250059 MB)"
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
