mdadm: active, degraded ... what do I do?

mesosphere · 06-03-2008, 12:25 PM

One of my drives in my raid 5 array seems not to be working. Here is my mdadm -D...

Quote:

aim:~ # mdadm -D /dev/md0
/dev/md0:
Version : 01.00.03
Creation Time : Sat May 17 13:00:26 2008
Raid Level : raid5
Array Size : 5858679552 (5587.27 GiB 5999.29 GB)
Used Dev Size : 1952893184 (1862.42 GiB 1999.76 GB)
Raid Devices : 7
Total Devices : 8
Preferred Minor : 0
Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Tue Jun 3 12:35:12 2008
State : active, degraded
Active Devices : 6
Working Devices : 8
Failed Devices : 0
Spare Devices : 2

Layout : left-symmetric
Chunk Size : 128K

Name : 0
UUID : 1c281031:8b24cbc4:91966683:27fa2fbd
Events : 902218

Number Major Minor RaidDevice State
0 8 34 0 active sync /dev/sdc2
1 8 50 1 active sync /dev/sdd2
2 8 66 2 active sync /dev/sde2
3 8 82 3 active sync /dev/sdf2
4 8 98 4 active sync /dev/sdg2
5 8 114 5 active sync /dev/sdh2
7 8 18 6 spare rebuilding /dev/sdb2

8 8 2 - spare /dev/sda2

The /dev/sda2 partition was recently added, and the problem existed before it was added.

I tried to grow my array with the sda2 partition, but I got the following error.

Quote:

aim:~ # mdadm --grow /dev/md0 --raid-devices=8
mdadm: Need to backup 5376K of critical section..
mdadm: /dev/md0: failed to find device 6. Array might be degraded.
--grow aborted

What is wrong with my array, and how can I fix it?

One thing that was different about the sdb2 partition is at the time it was the only drive in my array plugged into one of the two pata ports on my mobo. The others were all on the sata ports. The new drive is on the second pata port. Other than that they are all identical 1TB drives.

Thanks for any help. Let me know if there is any additional information I could provide to help troubleshoot.

-Justin

Pearlseattle · 06-04-2008, 06:22 AM

Hi
Sorry, I didn't understand if your raid in the end comes online & you're able to mount it or not.
I have no clue if this might help, but eventually, this is a note I took a couple of weeks ago when one of the HDD-controllers I had broke, bringing down 2 HDDs of my 4-HDDs raid5:

=============
If in a RAID array two HDDs or more fail / get an unhealthy status at the same time, there is the possibility that something might have gone wrong with the drivers of the controller or the controller itself. Fix the problem with the controller & try to restart the array with all the HDDs (e.g. a RAID 5 with 4 HDDs)...

Code:

mdadm --assemble --force /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1

...and check in "cat /proc/mdstat" if both the HDDs are still marked as "failed". If not, try to re-add the faulty HDD with...

Code:

mdadm --manage /dev/md0 --add /dev/sdd1

...or whichever HDD failed and see if the RAID rebuilds correctly and works again afterwards.
=============
This worked for me and I got back the whole raid5 with the data I had in there.
Greetings.

mesosphere · 06-04-2008, 08:03 AM

Yea, the array mounts fine and on the surface all seems fine. When I had only 7 drives I had the expected 5.4 TB of space available, but drive 6 seems not to be in use, so I don't think I have any failure protection.

I don't think any of my drives are marked as failed, but here is my mdstat...

Quote:

aim:~ # cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid0] [raid1]
md0 : active raid5 sda2[8](S) sdc2[0] sdb2[7] sdh2[5] sdg2[4] sdf2[3] sde2[2] sdd2[1]
5858679552 blocks super 1.0 level 5, 128k chunk, algorithm 2 [7/6] [UUUUUU_]
bitmap: 22/466 pages [88KB], 1024KB chunk

unused devices: <none>

mesosphere · 06-04-2008, 10:54 AM

I've set sdb2 as failed then rebooted the system. It's now rebuilding with sda2. Progress!!!

Quote:

aim:~ # mdadm -D /dev/md0
/dev/md0:
Version : 01.00.03
Creation Time : Sat May 17 13:00:26 2008
Raid Level : raid5
Array Size : 5858679552 (5587.27 GiB 5999.29 GB)
Used Dev Size : 1952893184 (1862.42 GiB 1999.76 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Wed Jun 4 11:51:02 2008
State : active, degraded, recovering
Active Devices : 6
Working Devices : 7
Failed Devices : 0
Spare Devices : 1

Layout : left-symmetric
Chunk Size : 128K

Rebuild Status : 18% complete

Name : 0
UUID : 1c281031:8b24cbc4:91966683:27fa2fbd
Events : 972560

Number Major Minor RaidDevice State
0 8 34 0 active sync /dev/sdc2
1 8 50 1 active sync /dev/sdd2
2 8 66 2 active sync /dev/sde2
3 8 82 3 active sync /dev/sdf2
4 8 98 4 active sync /dev/sdg2
5 8 114 5 active sync /dev/sdh2
8 8 2 6 spare rebuilding /dev/sda2

Once its done, I'll try re-adding sdb2. It was in the "spare rebuilding" state as sda2 is except it never actually started rebuilding.

Pearlseattle · 06-05-2008, 05:35 AM

Sorry, now I have a question: I was thinking about setting up a raid6 instead of raid5, but before doing that I would like to know how it looks like with performance & CPU usage. Did you ever do any experiments with raid6?
Thank you

mesosphere · 06-05-2008, 08:55 AM

No, I never tried raid6. I didn't want to lose more space. Perhaps I've been lucky, but I've never had a drive fail out of the 20-30 I've owned/used, so I'm not to worried about 2 simultaneous drive failures.

But raid6 is (I think) just double the parity calculations, so I would think the CPU usage wouldn't be more than double raid5. Probably not even that much since those calculations aren't all the CPU is being used for. This is just my guess though.

Pearlseattle · 06-06-2008, 01:00 PM

ok, thank you!

mesosphere · 06-09-2008, 03:18 PM

This may require a new thread as its a completely separate problem.

After recovering with the alternate drive I grew the array by placing the 8 th drive back in the array and using the --grow option. All seemed fine after this was completed. mdadm -D /dev/md0 reported that it successfully grew the array and that the size of the array grew by the appropriate amount.

The additional space was not available however since i had not resized the file system, but the array ran without problems all weekend. Now this morning I attempted to resize, but could not since the filesystem was mounted. The raid array includes my '/' partition, so to resize the file system i booted to the ubuntu live CD. After installing mdadm, the array was recognized without a problem. And the --detail reported no problems, so I ran the file system check and resized the array. All seemed to work with no problems reported.

But after rebooting it was unable to assemble the array. I'm not sure if it was damaged, or what. Every time i try to assemble it it gives the error...

mdadm: /dev/md0 assembled from 1 drive - not enough to start the array.

when I run in verbose mode, all the correct partitions are identified as members of /dev/md0, but they are all identified as being slot 0. Any idea what could cause this and/or if its fixable? Can I manually set the slot numbers some how?