Drives change device ids while on-line!

BloodBlight · 12-04-2010, 10:59 AM

Ok, I'm not sure what to make of this. I have setup an Ubuntu 10.10 server with two software raids.

md0 is a four disk raid5 - 3TB
md1 is a two disk mirror - 300GB

I think I have a drive failing (and am going to replace it regardless, but I have to take an outage), what appears to happen is it comes on-line with one id (/dev/sda) then something happens AFTER the rebuild completes and the drive changes to another id (in this case /dev/sdh) and puts the array in a failed state.

Is this some sort of protection mechanism to prevent degradation to the array? When setting this up, presumably before the disk started to fail, Ids seemed to jump from reboot to reboot and caused me all kinds of issues.

Also, neither device appears to return info after the change.

Code:

bwoods@MediaServer:~$ cat /proc/mdstat 

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md_d0 : inactive sdh1[0](S)
      976759936 blocks
       
md1 : active raid1 sdg1[1] sdf1[0]
      293033024 blocks [2/2] [UU]
      
md0 : active raid5 sda1[4](F) sdd1[3] sdc1[2] sdb1[1]
      2930279808 blocks level 5, 4k chunk, algorithm 2 [4/3] [_UUU]
      
unused devices: <none>

bwoods@MediaServer:~$ sudo hdparm -i /dev/sda

/dev/sda: No such file or directory

bwoods@MediaServer:~$ sudo hdparm -i /dev/sdh

/dev/sdh: No such file or directory

BloodBlight · 12-05-2010, 12:34 AM

More info.

It appears the drive that jumped in this last instance is actually one of my brand new drives (I have some new WDs and old SpinPoints) and not one of the aging SpinPoints.

Also, though the mdstat lists a disk at sdh, the drive actually jumped to sdi...

This is really odd.

Can anyone give some incite on how Linux assigns device paths?

gd2shoe · 12-07-2010, 12:57 AM

This is truly bizarre behavior. I'm not sure what's going on here. My guess is hardware failure (RAM, MB, PSU, loose cables, etc). There might be a problem with udevd (which manages /dev) or with /etc/udev.

Check:

Code:

dmesg | grep -E 'udev|sd|hd|md'

for clues. Also examine /var/log/messages.

Code:

udevadm monitor

might provide clues in realtime.

Drives should receive the same identifier each time they're detected. I used to know where udev stored this... When a new device is detected in the system, udevd checks a convoluted set of rules in /etc/udev. It uses this information to create the /dev file.

It may also create several sym links to the device. Because of this, drives can often be accessed by more than one name. This highly depends on how udev is setup by your distribution. On my system:

Code:

ls -l /dev/disk/by-id

gives me a list of all drives by serial number. I believe these references can be used with md, should you need to rebuild. THAT should put an end to this funny business (assuming you don't have a deeper problem, which worries me).