LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 11-06-2009, 10:05 AM   #1
ivanmacx
Member
 
Registered: Apr 2006
Location: Cambridge, UK
Distribution: Ubuntu Jaunty
Posts: 45

Rep: Reputation: 16
Fixing degraded RAID5 array


I have a problem with a degraded RAID5 array on my server, which contains 3 x 500gb drives to give just under 1Tb total space. It is /dev/md0 on my system and should be made up of /dev/sda1 /dev/sdb1 and /dev/sdc1. The RAID array is just for data, the system is on a separate drive and sdd1 is the root partition.

The drive is functioning fine, and will mount and the data is intact. However, one of the drives has either dropped off the array, or was not properly inserted in the first place.

The array is showing that it's comprised of just two discs, dm-1 and sdc1. I don't know what dm-1 is. From my research it seems to be a device mapper device and possibly something to do with LVM but to the best of my knowledge I'm not using LVM. I assume it maps to either sda1 or sdb2.

I have no evidence (that I can see) of any disc failure, so I want to re-insert the third drive back into the array.

In case it's relevant, this array was originally created as a 2 disc RAID5 array, and was then grown to 3. I know the third was added and synced because I monitored it and it took the best part of a day.

I'm using Ubuntu and I have just upgraded from jaunty to karmic, but I don't know whether this problem existed before the upgrade or came about as a result of it.

Another thing that confuses me is that although fdisk shows me that sda1 and sdb1 exist as Linux RAID Autodetect partitions (type fd), I have no /dev/sda1 or /dev/sdb2 files in my /dev directory. For some reason /dev/sdc1 is showing up as a Linux type partition (type 83) even though I'm sure I set it to type fd.

So my questions are:

What is dm-1 and how do I find out what it points to?

Can I get rid of dm-1 somehow and revert to using the partitions directly?

How do I add the third drive back into the array without losing data?

Some relevant info below. Thanks in advance if someone can help me with this because I'm having trouble getting my head around this.

Code:
cat /proc/mdstat

Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] 
md0 : active raid5 dm-1[1] sdc1[2]
      976767872 blocks level 5, 64k chunk, algorithm 2 [3/2] [_UU]
      
unused devices: <none>
Code:
sudo mdadm -D /dev/md0

      Version : 00.90
  Creation Time : Sat Aug 22 15:50:05 2009
     Raid Level : raid5
     Array Size : 976767872 (931.52 GiB 1000.21 GB)
  Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
   Raid Devices : 3
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Fri Nov  6 15:09:04 2009
          State : clean, degraded
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 7e6e178d:35bd18bf:77ce5359:408cfd9e
         Events : 0.324778

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1     252        1        1      active sync   /dev/block/252:1
       2       8       33        2      active sync   /dev/sdc1
Code:
sudo fdisk -l

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x10f635e2

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1       60801   488384001   fd  Linux RAID autodetect

Disk /dev/sdb: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000b5116

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1       60801   488384001   fd  Linux RAID autodetect

Disk /dev/sdc: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x0003e19f

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1       60801   488384001   83  Linux

Disk /dev/sdd: 40.0 GB, 40020664320 bytes
255 heads, 63 sectors/track, 4865 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x0b0ac1bf

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1        4343    34885116   83  Linux
/dev/sdd2            4344        4865     4192965    5  Extended
/dev/sdd5            4344        4865     4192933+  82  Linux swap / Solaris

Disk /dev/md0: 1000.2 GB, 1000210300928 bytes
2 heads, 4 sectors/track, 244191968 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table
 
Old 11-06-2009, 11:21 AM   #2
Morphine Drip
LQ Newbie
 
Registered: Nov 2009
Posts: 1

Rep: Reputation: 0
i cant speak to what the dm-1 is. but if it were me I would try the following...

mdadm -E /dev/sda1
mdadm -E /dev/sdb1
mdadm -E /dev/sdc1

and make sure the UUID's are a match

if a and b are (and c is not) i would (because i don't know how dm-1 was renamed - but clearly your box reflects sdb1 from the fdisk output) stop the array

mdadm --stop /dev/md0

then re-assemble (it wont mess with existing data)

mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1

then change the partition type on sdc1 back to what you had (fd)...and add that back to the array (it will then do a re-sync (based on sda and sdb).

mdadm -a /dev/md0 /dev/sdc1

--------------------------------------

Or...

if all 3 drives (sda,sdb,sdc) all are seen to have matching UUID's from the mdadm -E output...i would...do one of two things.

if you have NOT written any data to the array SINCE sdc has been missing i would stop the whole array

mdadm --stop /dev/md0

and

mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1

to get it back (might need to use --force (but read the mdadm man page first - when using force))

if you HAVE written data to the drive(s) since...i would take the first approach. stop the array, reassemble drives a/b....then add in c - so it can resync it.

good luck.
 
Old 11-06-2009, 03:39 PM   #3
ivanmacx
Member
 
Registered: Apr 2006
Location: Cambridge, UK
Distribution: Ubuntu Jaunty
Posts: 45

Original Poster
Rep: Reputation: 16
Thanks for the reply.

My problem is that I can't examine sda1 and sdb1 because /dev/sda1 and /dev/sdb1 don't exist, even though fdisk is telling me that the partitions exist. Hence I get:

Code:
sudo mdadm -E /dev/sda1

mdadm: cannot open /dev/sda1: No such file or directory
Any idea how to get around that, or what might lead to it?

Last edited by ivanmacx; 11-06-2009 at 03:40 PM. Reason: error in code tags
 
Old 11-06-2009, 06:42 PM   #4
ivanmacx
Member
 
Registered: Apr 2006
Location: Cambridge, UK
Distribution: Ubuntu Jaunty
Posts: 45

Original Poster
Rep: Reputation: 16
So I had another thought and I think I've now fixed it.

Firstly I amended /boot/grub/menu.lst and added the 'nodmraid' as a kernel option, to turn off the device mapper. That meant it didn't grab the two drives on startup.

Then I did:

Code:
sudo mdadm --stop /dev/md0
sudo mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1
To try and re-assemble all three drives into the array. However, this got me to the same place as before ie. 2 out of 3 drives assembled (sda and sdc). So then I did:

Code:
sudo mdadm /dev/md0 --add /dev/sdb1
This worked fine and the array is currently syncing itself onto the 3rd drive. I'm watching with

Code:
watch cat /proc/mdstat
and it's 3.3% in and should be done by morning. I'll report back when it's done in the hope that this might be useful to someone else with the same problem.

When I can get into the Bios I may turn on the pseudo hardware Raid if a bit of Googling tells me that's faster, but for now it seems like the software Raid array is fixed.

Last edited by ivanmacx; 11-07-2009 at 02:33 AM.
 
Old 11-07-2009, 03:59 AM   #5
ivanmacx
Member
 
Registered: Apr 2006
Location: Cambridge, UK
Distribution: Ubuntu Jaunty
Posts: 45

Original Poster
Rep: Reputation: 16
I have since found a bug report for this issue:

https://bugs.launchpad.net/ubuntu/+s...id/+bug/442735

The outcome of my situation is that after re-adding the third drive and syncing it seemed I had got back the Raid array, but lost all my data. However I recovered it with a simple:

Code:
sudo fsck /dev/md0
So now I'm back where I was with a degraded RAID5 array running on 2 out of 3 discs. I'm going to back up all the data, then wipe the partitions and recreate the array from scratch. Going forward I will either use the motherboard fakeraid if there's any advantage to that, or I'll just boot with the 'nodmraid' option going forward to avoid a repeat of this issue.

Last edited by ivanmacx; 11-07-2009 at 04:42 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
dirty degraded md raid array edgjerp Linux - Hardware 1 01-07-2009 01:51 PM
Degraded Array on Software Raid pcinfo-az Linux - Hardware 8 07-03-2008 10:43 AM
RAID help restoring a degraded array partition jahlewis Linux - Hardware 2 10-17-2006 07:55 PM
Trouble booting a degraded RAID-1 array aluchko Linux - Software 3 09-09-2006 10:26 PM
RAID 1 Degraded Array gsoft Debian 2 08-18-2006 02:17 PM


All times are GMT -5. The time now is 11:17 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration