LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Server (http://www.linuxquestions.org/questions/linux-server-73/)
-   -   mdadm clean install of four drives overwritting any previous raid5 disk array (http://www.linuxquestions.org/questions/linux-server-73/mdadm-clean-install-of-four-drives-overwritting-any-previous-raid5-disk-array-671362/)

javaholic 09-21-2008 07:20 AM

mdadm clean install of four drives overwritting any previous raid5 disk array
 
Having given up trying to add a fourth drive to a what was three disk raid5 array. i was wondering was can be done to clean up all the disks ready for a clean install of a four disk raid5 array. if you can link me to a good tutorial on the matter that would be grand.

mostlyharmless 09-24-2008 03:47 PM

Depends on why you want to wipe the disks I suppose. If it's security, then I'd wipe the disks first and possibly over write with random data using the dd command prior to re-using them.

http://howto.wikia.com/wiki/Howto_wi...clean_in_Linux

If it isn't a security concern, I'd just repartition with fdisk (if needed) and use mdadm to create the new array(s). How you set it up depends on your distro, in Ubuntu, for example:

http://ubuntuforums.org/showthread.php?t=408461

racracracrac 09-26-2008 07:32 AM

I would recommend blowing away the first blocks on the drive. This is where the info mdadm uses is stored.


dd count=1 bs=128k if=/dev/zero of=/dev/sda1

javaholic 10-02-2008 07:14 AM

oh okay i still can't get array to create properly having cleaned the state of the disks.

how do i narrow down SATA controller card problems?

How can i make sure i know which drive is referring to the correct drive from the innards of the machine?

Just in case i did buy two new drives just in case. NOt entirely sure took out the correct drive(s). How can i be sure?

so the resulting cat /proc/mdstat after about a day of building is as follows:

Quote:

Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[4](S) sdc1[5](F) sdb1[1] sda1[0]
2930279808 blocks level 5, 64k chunk, algorithm 2 [4/2] [UU__]

unused devices: <none>
So it tells me one drive failed and another is a spare i know that much. If the numbers in the square brackets indicates the channel on the SATA card that these drives are attached to that would be wrong as there are only 4 ports and they are listed as 0,1,2,3.

Could this be my problem? If so how can i rectify this?

i should also point out that the contents of the above "cat /proc/mdstat" command was and is to be best of my recollection identical with both times i tried to build/create array.

javaholic 10-03-2008 01:57 AM

Array seemed to build with --assume-clean option.

But failed after about the mkfs.ext3 /dev/md0

/proc/mdstat stated one spare drive and one failed drive then after boot only only two of the drives showed up i assumed the ones that werenn't spare and failed.

Is there a chance i replaced the wrong two drives the first time round?

mostlyharmless 10-03-2008 11:30 AM

It'd probably be helpful to have more details of what you did, what distro you're using, what hardware you're using etc. The particulars can be important.

There are any number of ways of telling drives apart. You have to be methodical about what you're doing. Getting in the machine and noting the drive models and numbers and which port they're attached to is helpful. You can find a lot of information with dmesg as well. Be methodical here as well.

As far as SATA issues and RAID, it depends a lot on the distro, as I mentioned. Slackware has excellent documentation included in the release. I mentioned one Ubuntu resource that seems straightforward above. Additional links that might be helpful follow:

OpenSUSE
http://www.howtoforge.com/forums/showthread.php?t=1664
Ubuntu:
http://ubuntu-in.info/wiki/index.php/SATA_RAID_Howto

FAQ
http://linux-ata.org/faq-sata-raid.html

You might want to post in your distro's forum for help.

javaholic 10-03-2008 06:08 PM

Linux distro is Debian.

Last time i checked the forums.debian.net wasn't as active i sight as you would hope to get a reasonably sharp response. I guess it might have changed.

sata driver: sata_sil

4 port Silicon 31** Image Card.

I knew when i bought it that it wasn't a hardware sata card. At the time of purchase i didn't think i would every have enough to buy a hardware. perhaps you guys could provide the names of good quality brands for one. That would be very helpful.


Following is the content of dmesg with that details my drives are referenced:

Quote:

sata_sil 0000:00:0b.0: version 2.0
ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10
PCI: setting IRQ 10 as level-triggered
ACPI: PCI Interrupt 0000:00:0b.0[A] -> Link [LNKB] -> GSI 10 (level, low) -> IRQ 10
sata_sil 0000:00:0b.0: Applying R_ERR on DMA activate FIS errata fix
8139cp: 10/100 PCI Ethernet driver v1.2 (Mar 22, 2004)
ata1: SATA max UDMA/100 cmd 0xF8820080 ctl 0xF882008A bmdma 0xF8820000 irq 10
ata2: SATA max UDMA/100 cmd 0xF88200C0 ctl 0xF88200CA bmdma 0xF8820008 irq 10
ata3: SATA max UDMA/100 cmd 0xF8820280 ctl 0xF882028A bmdma 0xF8820200 irq 10
ata4: SATA max UDMA/100 cmd 0xF88202C0 ctl 0xF88202CA bmdma 0xF8820208 irq 10
scsi0 : sata_sil
usb 1-1: new full speed USB device using ohci_hcd and address 2
usb 1-1: configuration #1 chosen from 1 choice
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata1.00: ATA-7, max UDMA/133, 1953525168 sectors: LBA48 NCQ (depth 0/32)
ata1.00: ata1: dev 0 multi count 16
ata1.00: configured for UDMA/100
scsi1 : sata_sil
usb 1-2: new low speed USB device using ohci_hcd and address 3
usb 1-2: configuration #1 chosen from 1 choice
usbcore: registered new driver hiddev
input: USBPS2 as /class/input/input0
input: USB HID v1.00 Keyboard [USBPS2] on usb-0000:00:02.0-2
input: USBPS2 as /class/input/input1
input: USB HID v1.00 Mouse [USBPS2] on usb-0000:00:02.0-2
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata2.00: ATA-7, max UDMA/133, 1953525168 sectors: LBA48 NCQ (depth 0/32)
ata2.00: ata2: dev 0 multi count 16
ata2.00: configured for UDMA/100
scsi2 : sata_sil
ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata3.00: ATA-7, max UDMA/133, 1953525168 sectors: LBA48 NCQ (depth 0/32)
ata3.00: ata3: dev 0 multi count 16
ata3.00: configured for UDMA/100
scsi3 : sata_sil
ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata4.00: ATA-8, max UDMA/133, 1953525168 sectors: LBA48 NCQ (depth 0/32)
ata4.00: ata4: dev 0 multi count 16
ata4.00: configured for UDMA/100
Vendor: ATA Model: WDC WD10EACS-00Z Rev: 01.0
Type: Direct-Access ANSI SCSI revision: 05
Vendor: ATA Model: WDC WD10EACS-32Z Rev: 01.0
Type: Direct-Access ANSI SCSI revision: 05
Vendor: ATA Model: WDC WD10EACS-00Z Rev: 01.0
Type: Direct-Access ANSI SCSI revision: 05
Vendor: ATA Model: WDC WD10EACS-00Z Rev: 01.0
Type: Direct-Access ANSI SCSI revision: 05
SCSI device sda: 1953525168 512-byte hdwr sectors (1000205 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: drive cache: write back
SCSI device sda: 1953525168 512-byte hdwr sectors (1000205 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: drive cache: write back
sda: sda1
sd 0:0:0:0: Attached scsi disk sda
SCSI device sdb: 1953525168 512-byte hdwr sectors (1000205 MB)
sdb: Write Protect is off
sdb: Mode Sense: 00 3a 00 00
SCSI device sdb: drive cache: write back
SCSI device sdb: 1953525168 512-byte hdwr sectors (1000205 MB)
sdb: Write Protect is off
sdb: Mode Sense: 00 3a 00 00
SCSI device sdb: drive cache: write back
sdb: sdb1
sd 1:0:0:0: Attached scsi disk sdb
SCSI device sdc: 1953525168 512-byte hdwr sectors (1000205 MB)
sdc: Write Protect is off
sdc: Mode Sense: 00 3a 00 00
SCSI device sdc: drive cache: write back
SCSI device sdc: 1953525168 512-byte hdwr sectors (1000205 MB)
sdc: Write Protect is off
sdc: Mode Sense: 00 3a 00 00
SCSI device sdc: drive cache: write back
sdc: sdc1
sd 2:0:0:0: Attached scsi disk sdc
SCSI device sdd: 1953525168 512-byte hdwr sectors (1000205 MB)
sdd: Write Protect is off
sdd: Mode Sense: 00 3a 00 00
SCSI device sdd: drive cache: write back
SCSI device sdd: 1953525168 512-byte hdwr sectors (1000205 MB)
sdd: Write Protect is off
sdd: Mode Sense: 00 3a 00 00
SCSI device sdd: drive cache: write back
sdd: sdd1
sd 3:0:0:0: Attached scsi disk sdd
raid5: automatically using best checksumming function: pIII_sse
pIII_sse : 4010.000 MB/sec
raid5: using function: pIII_sse (4010.000 MB/sec)
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
raid6: int32x1 583 MB/s
raid6: int32x2 708 MB/s
raid6: int32x4 554 MB/s
raid6: int32x8 401 MB/s
raid6: mmxx1 1216 MB/s
raid6: mmxx2 1931 MB/s
raid6: sse1x1 1120 MB/s
raid6: sse1x2 1910 MB/s
raid6: using algorithm sse1x2 (1910 MB/s)
md: raid6 personality registered for level 6
md: raid5 personality registered for level 5
md: raid4 personality registered for level 4
md: md0 stopped.
md: bind<sda1>
md: bind<sdc1>
md: bind<sdd1>
md: bind<sdb1>
md: kicking non-fresh sdc1 from array!
md: unbind<sdc1>
md: export_rdev(sdc1)
md: kicking non-fresh sda1 from array!
md: unbind<sda1>
md: export_rdev(sda1)
raid5: device sdb1 operational as raid disk 1
raid5: device sdd1 operational as raid disk 3
raid5: not enough operational devices for md0 (2/4 failed)
RAID5 conf printout:
--- rd:4 wd:2 fd:2
disk 1, o:1, dev:sdb1
disk 3, o:1, dev:sdd1
raid5: failed to run raid set md0

mostlyharmless 10-04-2008 01:16 PM

Quote:

Last time i checked the forums.debian.net wasn't as active i sight as you would hope to get a reasonably sharp response. I guess it might have changed.
I guess I was thinking THIS forum :) maybe it can be moved

http://www.linuxquestions.org/questions/debian-26/

But, anyway, looking at your information, it looks like sda and c are still not clean, but otherwise are recognized normally. I doubt therefore that it's a controller or drive issue.

Did you wipe them completely with dd and repartition them with fdisk?
What does fdisk -l show?

javaholic 10-04-2008 02:05 PM

Well the response from fdisk -l is a bit confusing cause i get the following.

Quote:

Disk /dev/hda: 12.9 GB, 12997361664 bytes
255 heads, 63 sectors/track, 1580 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/hda1 * 1 1507 12104946 83 Linux
/dev/hda2 1508 1580 586372+ 5 Extended
/dev/hda5 1508 1580 586341 82 Linux swap / Solaris

Disk /dev/hdc: 200.0 GB, 200049647616 bytes
255 heads, 63 sectors/track, 24321 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/hdc1 1 24321 195358401 83 Linux

Disk /dev/hdd: 203.9 GB, 203928109056 bytes
255 heads, 63 sectors/track, 24792 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/hdd1 1 24793 199148512+ 83 Linux

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 121601 976760001 fd Linux raid autodetect
Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 121601 976760001 fd Linux raid autodetect
Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdc1 1 121601 976760001 fd Linux raid autodetect
Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdd1 1 121601 976760001 fd Linux raid autodetect
but say with a command like cfdisk /dev/sd{whichever drive i'm trying to look at i get unformatted response for the contents of the two drives in question}

I would most certainly bank of neither drive being faulty due to their age more than anything.

My most underlining thought tells me that the kernel is interferring with drive channel assignment. Meaning that drive at /dev/sda1 would be assigned channel 0 and then assigned to channel 3 at next boot but i thought this only happened at boot.

javaholic 10-05-2008 08:37 AM

So if i can create and format the result of a three disk array totalling 1.7TB then i can be pretty certain debian doesn't come with GPT support and will need to find a reference material to help instruct me how to add it to teh kernel.

I'll let you know in due course.

javaholic 10-06-2008 02:39 AM

failed.

drives were low level formatted using dd.

fdisk and so i'm pretty sure cfdisk as well use parted 1.7 with the debian system so i'm going to try and update that.

mostlyharmless 10-06-2008 12:04 PM

fdisk says there's a partition table and cfdisk doesn't? That's odd. Are you using an MBR or GPT?

GPT support is a kernel option; if it's not compiled in , you'll have to recompile the kernel. But I don't think you need that until you're over 2 TB.

javaholic 10-08-2008 10:25 AM

it downed on me last night that i was missing one piece of very important data in my quest to find out what is wrong with this array.

The result of the mkfs.ext3 /dev/md0 command. It is as follows:



Quote:

mke2fs 1.40-WIP (14-Nov-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
366297088 inodes, 732569952 blocks
36628497 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
22357 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544

Writing inode tables: ^[[A^[[Adone
ext2fs_mkdir: Attempt to read block from filesystem resulted in short read while creating root dir
I haven't recompiled the kernel at any point so one has to assume that all it is setup up with the defaults of the kernel hashed by the debian team which i don't think is set up with GPT in mind.

I would have to be shown how to compile kernel with GPT support if that were possible.

mostlyharmless 10-08-2008 12:58 PM

regrouping for a moment
 
Well, I don't think you need to recompile with GPT enabled (it's a kernel config option); I still think it's a mdadm problem. You dmesg seemed to indicate it wasn't a drive/controller/driver problem.

I just want to recap the situation so that I understand it.

Regrouping: you wiped with dd:

dd if=/dev/zero of=/dev/sda bs=1M
dd if=/dev/zero of=/dev/sdc bs=1M

You partitioned with fdisk:

fdisk /dev/sda
...made whatever partitions...
fdisk /dev/sdc
...made whatever partitions...
and made them type "fd" if you want, though it's not absolutely necessary usually..

You made the arrays with mdadm:

mdadm --create /dev/md0 --level=5 --raid devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1

and mdadm said:
[fill in the blank]


see for example
http://www.linuxhomenetworking.com/w..._Software_RAID

javaholic 10-12-2008 04:29 PM

before system reboot it would claim sdb1 was spare and sdd1 was faulty

after that only two drives existed in the array sda1 and sdc1 neither sb1 or sdd1 were shown to be part of the array.

Sorry for the delay in response.


All times are GMT -5. The time now is 09:41 PM.