LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 04-16-2007, 03:36 PM   #1
hazmatt20
Member
 
Registered: Jan 2006
Distribution: FC5, Ubuntu
Posts: 126

Rep: Reputation: 15
RAID5 Array Recovery after OS upgrade


I need to know if I can fix this or if I should bite the bullet and start reloading my dvd backups. Again. I have 90-95% backed up, but it's about 1.6 TB of data on hundreds of DVDs, so you know how painful reloading is.

I have 6 400 GB SATA drives on a RAID5 array mounted to /home. System is Ubuntu 6.06 Server kernel 2.6.15 with mdadm (don't know the version). I recently installed a new motherboard with more on-board SATA connectors as I was also planning to start adding more drives. The plan was to add a 3 bay enclosure that can hold 5 drives and setup 3 500 GB drives in another RAID5 array now and expand to 5 later. There were two issues with this, both of which I realize now could probably have been resolved if I had simply taken the time to learn how to compile a new kernel. The network drivers weren't loaded when it booted (two on the board), so I had to use a card, and I remember reading that I would need a newer kernel than was available in the apt repository for ubuntu 6.06 to expand an array.

So, instead of compiling a new kernel, I decided to do a fresh install of 6.10 server. During install, there was a some problem with DHCP, and it took me back to the menu. I got it sorted out but didn't realize until it finished that it managed to skip several sections of the installation, including user setup. With no login, I ran the recovery. When it finished, it looked ok. mdadm showed the device as /dev/md0(what it had been), and it mounted fine. If everything else had been fine, it would have been what I wanted, however, the install had missed other stuff besides user accounts, for example, not only were there no apt sources configured, there were no man pages installed.

Another reinstall, all the way through this time. mdadm didn't set it up correctly this time. Here is the current status.

Quote:
# mdadm --detail /dev/md0
/dev/md0:
Version : 00.90.03
Creation Time : Sun Apr 15 19:41:18 2007
Raid Level : raid5
Array Size : 1953556480 (1863.06 GiB 2000.44 GB)
Device Size : 390711296 (372.61 GiB 400.09 GB)
Raid Devices : 6
Total Devices : 6
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Mon Apr 16 02:17:39 2007
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

UUID : 9db5f426:b7ce1681:eb04cbd7:2b95de32
Events : 0.2

Number Major Minor RaidDevice State
0 8 64 0 active sync /dev/sde
1 8 96 1 active sync /dev/sdg
2 8 80 2 active sync /dev/sdf
3 8 48 3 active sync /dev/sdd
4 8 128 4 active sync /dev/sdi
5 8 112 5 active sync /dev/sdh
Quote:
Disk /dev/md0: 2000.4 GB, 2000441835520 bytes
255 heads, 63 sectors/track, 243206 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/md0p1 1 48641 390708801 fd Linux raid autodetect

The only other thing is that last night, it showed it as degraded and resyncing one drive, and it finished the resync. What should my next step be?

Last edited by hazmatt20; 04-16-2007 at 03:44 PM.
 
Old 04-16-2007, 03:57 PM   #2
Quakeboy02
Senior Member
 
Registered: Nov 2006
Distribution: Debian Linux 11 (Bullseye)
Posts: 3,407

Rep: Reputation: 141Reputation: 141
Quote:
What should my next step be?
I think I would rule out reinstalling the kernel again.

Quote:
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
What's the problem? The array is clean and there are no failed devices. Granted I'm not an expert on mdadm, but...
 
Old 04-16-2007, 04:05 PM   #3
hazmatt20
Member
 
Registered: Jan 2006
Distribution: FC5, Ubuntu
Posts: 126

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by Quakeboy02
What's the problem? The array is clean and there are no failed devices. Granted I'm not an expert on mdadm, but...
Heh, sorry. The problem is that I can't mount /dev/md0. fdisk /dev/md0 shows a 400 GB /dev/md0p1 (I dunno).

Quote:
#mount /dev/md0 /mnt/md0
mount: you must specify the filesystem type

#mount /dev/md0p1 /mnt/md0
mount: special device /dev/md0p1 does not exist
 
Old 04-16-2007, 04:11 PM   #4
Quakeboy02
Senior Member
 
Registered: Nov 2006
Distribution: Debian Linux 11 (Bullseye)
Posts: 3,407

Rep: Reputation: 141Reputation: 141
Quote:
Heh, sorry. The problem is that I can't mount /dev/md0. fdisk /dev/md0 shows a 400 GB /dev/md0p1 (I dunno).
Take a look at this thread. dgar is pretty sharp on this stuff, and he haunts the raid posts, too, so maybe he'll chime in and fix you up.

http://www.linuxquestions.org/questi...d.php?t=544557
 
Old 04-16-2007, 06:08 PM   #5
hazmatt20
Member
 
Registered: Jan 2006
Distribution: FC5, Ubuntu
Posts: 126

Original Poster
Rep: Reputation: 15
I looked at it, but it doesn't solve this problem since I want to keep my data. Does help a little with learning about mdadm, though.
 
Old 04-16-2007, 07:10 PM   #6
Quakeboy02
Senior Member
 
Registered: Nov 2006
Distribution: Debian Linux 11 (Bullseye)
Posts: 3,407

Rep: Reputation: 141Reputation: 141
Did you build a new mdadm.conf as a result of this, or did you keep the one that had been on it? You didn't run mdadm -create did you? Where did the current mdadm.conf come from (auto gen or did you make it?) and what are its contents?
 
Old 04-16-2007, 07:49 PM   #7
rtspitz
Member
 
Registered: Jan 2005
Location: germany
Distribution: suse, opensuse, debian, others for testing
Posts: 307

Rep: Reputation: 33
just a guess...

I've read a german thread about the very same error.
someone had built a raid array with /dev/sda1, /dev/sdb1 and so on.
after a kernel update his raid seemed ok, but mdadm showed /dev/sda, /dev/sdb and so on as members. fdisk -l would still show /dev/sda1, /dev/sdb1 were there.

there was also the discrepancy between device/array size.
his solution was to zero the superblocks on the false mebers /dev/sda, /dev/sdb .... + reboot

Code:
mdadm --zero-superblock /dev/sd[a-e]
https://lists.uni-koeln.de/pipermail...er/011313.html

in case you want to give it a shot, I can translate it in detail

Last edited by rtspitz; 04-16-2007 at 07:52 PM.
 
Old 04-16-2007, 10:41 PM   #8
hazmatt20
Member
 
Registered: Jan 2006
Distribution: FC5, Ubuntu
Posts: 126

Original Poster
Rep: Reputation: 15
I don't think I ran mdadm --create. The mdadm.conf was auto-generated.

Quote:
DEVICE partitions
ARRAY /dev/md0 level=raid5 num-devices=6 UUID=9db5f426:b7ce1681:eb04cbd7:2b95de32
That German thread is exactly what mine is doing. If you don't mind translating the solution, I'd be grateful. I tried

Quote:
# mdadm --zero-superblock /dev/sd[d-i]
mdadm: Couldn't open /dev/sdd for write - not zeroing
mdadm: Couldn't open /dev/sde for write - not zeroing
mdadm: Couldn't open /dev/sdf for write - not zeroing
mdadm: Couldn't open /dev/sdg for write - not zeroing
mdadm: Couldn't open /dev/sdh for write - not zeroing
mdadm: Couldn't open /dev/sdi for write - not zeroing
(sd[a-c] are the 3 500 GB drives)
 
Old 04-16-2007, 11:15 PM   #9
Quakeboy02
Senior Member
 
Registered: Nov 2006
Distribution: Debian Linux 11 (Bullseye)
Posts: 3,407

Rep: Reputation: 141Reputation: 141
Quote:
The mdadm.conf was auto-generated.
When I was messing with mdadm, it once created several arrays out of just random drives on my system just after installing it. How reassembling it by hand. Have you tried that?

Code:
mdadm --stop /dev/md0
mdadm --assemble /dev/md0 --level=5 /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi
If this works, then you need to create a new mdadm.conf file for it.
 
Old 04-16-2007, 11:52 PM   #10
hazmatt20
Member
 
Registered: Jan 2006
Distribution: FC5, Ubuntu
Posts: 126

Original Poster
Rep: Reputation: 15
Um, so it may be worse now. I stopped the array and assembled it, but it came up the same as before with /dev/md0p1 in fdisk. Stopped it again and tried the zero-superblock because last time I had the command out of order and didn't stop it. It worked. Tried to assemble, and it said,

Quote:
# mdadm --assemble /dev/md0 /dev/sd[e-i]
mdadm: no recogniseable superblock
Rebooted, and it did the same thing. I'll point out that when it booted, it made md0 out of sd[a-c] (the 500's) and md1 with 4 of the 6 400's. After stopping both, assemble gave me the no recogniseable superblock error for both arrays even though I didn't run zero-superblock on the sd[a-c] array.
 
Old 04-16-2007, 11:58 PM   #11
Quakeboy02
Senior Member
 
Registered: Nov 2006
Distribution: Debian Linux 11 (Bullseye)
Posts: 3,407

Rep: Reputation: 141Reputation: 141
Code:
mdadm --assemble /dev/md0 /dev/sd[e-i]
I don't know the consequences of not specifying which type of raid during an assemble operation. Also, is that a typo, or did you really use sd[e-i]? Or did you type something completely different but you're reporting this?
 
Old 04-16-2007, 11:58 PM   #12
hazmatt20
Member
 
Registered: Jan 2006
Distribution: FC5, Ubuntu
Posts: 126

Original Poster
Rep: Reputation: 15
This thread mentions mdadm --assemble --force. Would it be a bad idea to try it?

http://www.issociate.de/board/post/2...lyaborted.html
 
Old 04-16-2007, 11:59 PM   #13
Quakeboy02
Senior Member
 
Registered: Nov 2006
Distribution: Debian Linux 11 (Bullseye)
Posts: 3,407

Rep: Reputation: 141Reputation: 141
First, I'd like to know exactly what you typed.
 
Old 04-17-2007, 06:29 AM   #14
rtspitz
Member
 
Registered: Jan 2005
Location: germany
Distribution: suse, opensuse, debian, others for testing
Posts: 307

Rep: Reputation: 33
this one will be interesting as well:

http://kev.coolcavemen.com/2007/03/h...d-superblocks/

basically what is discussed here is recovery of a raid5 after zero-ing all the superblocks of the included partitions, seems to work.

I've tested it with vmware and a raid1, killed all superblocks, mdadm would not assemble/start the array. then tried above mentioned --create and lo and behold, I could mount it, no data was lost. mdadm complained about a preexisting filesystem, but I forced it to do its magic and it worked.

Last edited by rtspitz; 04-17-2007 at 06:31 AM.
 
Old 04-17-2007, 09:28 AM   #15
hazmatt20
Member
 
Registered: Jan 2006
Distribution: FC5, Ubuntu
Posts: 126

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by Quakeboy02
First, I'd like to know exactly what you typed.
Sorry. We updated at the same time.

Quote:
mdadm: option --level not valid in assemble mode
I tried both of these.

Quote:
# mdadm --assemble /dev/md0 /dev/sd[d-i]
mdadm: no recogniseable superblock
mdadm: /dev/sdd has no superblock - assembly aborted
# mdadm --assemble /dev/md0 /dev/sd[d-i]1
mdadm: cannot open device /dev/sdd1: Device or resource busy
mdadm: /dev/sdd1 has no superblock - assembly aborted

Last edited by hazmatt20; 04-17-2007 at 09:46 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
cannot rebuild RAID5 array after two disks went offline waimea Linux - General 2 03-22-2007 03:40 AM
RAID 1 Array recovery ? Andrew_OC Linux - Server 6 01-30-2007 02:49 PM
Secure Deletion with RAID5 array neilschelly Linux - Security 2 12-05-2004 06:25 PM
problem in replacing hardisk raid5 array slack66 Linux - Newbie 0 08-13-2003 04:13 AM
Resizing ext3 partitions on RAID5 array greenhornet Linux - General 2 04-09-2002 08:22 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 11:55 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration