Software RAID (5) issues

BlackRabbit · 03-28-2008, 02:04 PM

Hi all,

decided to go for Linux Software RAID 5 on my HighPoint 1820A controller.

Followed some guides around the net, summarized I did this:

Code:

> mdadm --create /dev/md0 -l5 --chunk=128 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sdd

As 1 can see: I build a RAID5 array from 4 drives (each of 400GB).

This is what my mdadm.conf looks like:

Code:

DEVICE /dev/sda /dev/sdb /dev/sdc /dev/sdd
ARRAY /dev/md0 devices=/dev/sda,/dev/sdb,/dev/sdc,/dev/sdd

However, upon boot the array wasn't assebled for some reason (can anyone explain why?).

So I assembled manually:

Code:

> mdadm --assemble --scan

Check 'cat /proc/mdstat' showed that the array was resyncing.
Is it normal that it needs to resync each time it is assembled?

I was able to format it:

Code:

> mkreiserfs /dev/md0

So far so good, I was able to write file to the array (at fair speeds), so I was a happy man.

However, troubles started when I rebooted the system. As the array was still resyncing, the system hung upon shutdown (before reboot). Last line mentioned was something like "saving resync checkpoints". As the system did not react anymore, I did a hard-reset. (how long does it take to save those checkpoints? How bad is it to reboot during resync?)

As I try to assemble the array now, I get the following message:

Code:

root@sun:~# mdadm --assemble /dev/md0
mdadm: /dev/md0 assembled from 1 drive - not enough to start the array.

I'm not sure what to do next.. Any advise and some answers on the questions above would be much appreciated!

PS: as there is no real data (just some test files) on the array yet, I could create a new array. However, it would be great if that wouldn't be necessary, as it takes a long time to create this array). Also, it doesn't give a very safe feeling

Some more info:

Code:

> mdadm --examine --scan
ARRAY /dev/md0 level=raid5 num-devices=4 UUID=fdf567f0:4a5cdbf9:209911b2:ab5853da

Right after reboot:

Code:

root@sun:~# dmesg |grep md
ata1: SATA max UDMA/133 cmd 0x00000000 ctl 0xe0ba2120 bmdma 0x00000000 irq 10
ata2: SATA max UDMA/133 cmd 0x00000000 ctl 0xe0ba4120 bmdma 0x00000000 irq 10
ata3: SATA max UDMA/133 cmd 0x00000000 ctl 0xe0ba6120 bmdma 0x00000000 irq 10
ata4: SATA max UDMA/133 cmd 0x00000000 ctl 0xe0ba8120 bmdma 0x00000000 irq 10
ata5: SATA max UDMA/133 cmd 0x00000000 ctl 0xe0bb2120 bmdma 0x00000000 irq 10
ata6: SATA max UDMA/133 cmd 0x00000000 ctl 0xe0bb4120 bmdma 0x00000000 irq 10
ata7: SATA max UDMA/133 cmd 0x00000000 ctl 0xe0bb6120 bmdma 0x00000000 irq 10
ata8: SATA max UDMA/133 cmd 0x00000000 ctl 0xe0bb8120 bmdma 0x00000000 irq 10
md: linear personality registered for level -1
md: raid0 personality registered for level 0
md: raid1 personality registered for level 1
md: raid10 personality registered for level 10
md: raid6 personality registered for level 6
md: raid5 personality registered for level 5
md: raid4 personality registered for level 4
md: multipath personality registered for level -4
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.

After trying to assemble:

Code:

root@sun:~# mdadm --assemble --scan
mdadm: /dev/md0 assembled from 1 drive - not enough to start the array.
root@sun:~# dmesg |grep md
... ... ...
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
md: md0 stopped.
md: bind<sdb>
md: bind<sdc>
md: bind<sdd>
md: bind<sda>

BlackRabbit · 03-28-2008, 05:12 PM

This is really bugging me..

Tried again:
* created a 40GB partition on each of the 4 disks (sd[abcd]1)
* created a new array:
> mdadm --create /dev/md0 -l5 -c128 --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
(Finished without issues.)
* Formatted /dev/md0:
> mkreiserfs /dev/md0
* Rebooted (just 'to be sure')
* Although config file looks ok, array wasn't created upon boot, so I assemble manually:
> mdadm --assemble /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
* Mounted it:
> mount /dev/md0 /mnt/md0
Suddenly the machine crashed (no warnings/errors shown), system just hangs)

Hard-reset, tried to assemble again:

Code:

root@sun:~# mdadm --assemble /dev/sda /dev/sdb /dev/sdc /dev/sdd
mdadm: no recogniseable superblock on /dev/sdb
mdadm: /dev/sdb has no superblock - assembly aborted

//SO I TRY THIS
root@sun:~# mdadm --zero-superblock /dev/sda1
root@sun:~# mdadm --zero-superblock /dev/sdb1
root@sun:~# mdadm --zero-superblock /dev/sdc1
root@sun:~# mdadm --zero-superblock /dev/sdd1
root@sun:~# mdadm --assemble /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
mdadm: no recogniseable superblock on /dev/sdb1
mdadm: /dev/sdb1 has no superblock - assembly aborted

Could it be Reiser causing these problems?

BlackRabbit · 03-28-2008, 06:05 PM

This is just too weird..

Redid the whole thing, formatted with ext3:

Code:

root@sun:/etc# mkfs.ext3 /dev/md0
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
14663680 inodes, 29296416 blocks
1464820 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=29360128
895 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
root@sun:/etc#

Everything looks ok:

Code:

root@sun:/etc# mdadm -Q -D /dev/md0
/dev/md0:
        Version : 00.90.03
  Creation Time : Fri Mar 28 23:38:42 2008
     Raid Level : raid5
     Array Size : 117185664 (111.76 GiB 120.00 GB)
  Used Dev Size : 39061888 (37.25 GiB 40.00 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Sat Mar 29 00:03:56 2008
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 128K

           UUID : eb55d4f6:3fa2025e:dc065d20:a79325b0
         Events : 0.4

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
       2       8       33        2      active sync   /dev/sdc1
       3       8       49        3      active sync   /dev/sdd1

Was able to 'touch' some files and mkdir some directories, chmod & chown etc: all no problem. Fired up proftpd and started transferring data to the array -> system crash

BlackRabbit · 03-29-2008, 04:16 AM

BTW: after the system crash I did a (hard)reset, and once again I was no longer able to assemble /dev/md0..

BlackRabbit · 03-29-2008, 10:52 AM

*sigh*, anybody?

Tried once again (you never know..) but system failed again:
- was able to create the array
- was able to assemble it after reboot (without it starting to resync btw)
- was able to format /dev/md0 in ext3 FS
BUT THEN:
- did not succeed to mount /dev/md0:
> mount /dev/md0 /mnt/md0
=> system just hangs, no response whatsoever

**UPDATE**
As I wanted to be sure that it wasn't just mount taking a lot of time to actually mount the device, I waited for about 30min. STill, the system didn't respond.
Did a hard reset. After reboot I was able to assemble /dev/md0 without errors. It is now resyncing though, so it is clear that something went wrong. Did a "ef2fsck /dev/md0", came back clean.

Will now wait for resync to finish and try to mount again.

Some more questions:
- do I need to partition/format the drives before I start creating the array?
- can I use reiserfs/ext3 as FS on a md device?
- can I (may I) mount /dev/md0 while it is resyncing?

BlackRabbit · 03-29-2008, 02:46 PM

Could it be the drivers of my controller causing these issues?

It's a HighPoint 1820A, I'm using SlackWare built-in drivers.

In what log might I be able to find some useful information?

BlackRabbit · 03-29-2008, 03:28 PM

I don't get it: why does mount make my system crash?

I've spent the last few days trying & retrying all kinds of guides & howto's, googled for any possible sollutions to similar problems but it just doesn't want to work

BlackRabbit · 03-30-2008, 08:55 AM

So..

Installed Slack 10.1 (kernel 2.4), installed highpoint drivers (from their site) and used the distro-provided mdadm -> no issues at all.

Reinstalled Slack 12.0, highpoint drivers from hpsite, mdadm of distro. As soon as I try to access the drives, the controller starts to beep (see this thread.

So:
- I suppose it's the highpoint drivers + my kernel (2.6)?
- Am I doomed to run Slack 10.1 (until new drivers from highpoint are released)?

BlackRabbit · 03-30-2008, 12:39 PM

Last attempt: during boot the kernel-built-in drivers are loaded, so maybe they cause trouble with the hpt drivers?

Code:

root@sun:~# dmesg|grep 3xx
RocketRAID 3xxx SATA Controller driver v1.0 (060426)

How can I force those drivers not to be loaded? (the hptmv don't have to be loaded upon boot)

BlackRabbit · 03-31-2008, 04:48 AM

Final update:
- installed Slack 12.0 again
- recompiled the kernel without support for any HighPoint controllers
- installed opensource drivers from highpoint

Although it took a bit more time before the controller started beeping, it still did when I tried to access a device on the controller (eg cfdisk /dev/sda).

I've sent a mail to Highpoint with my findings, hopefully a fix will be published soon.

Until then, I'm back to Slack 10.1

BlackRabbit · 04-27-2008, 05:18 PM

FYI: I was able to get everything to work without problems under Ubuntu Hardy Heron Server Edition (8.04). Note that I don't use the HighPoint drivers (the 1820A works without problems with the default kernel) as I don't have time to experiment (+ I can't backup my 'old' software raid5 arrays

).

Maybe I'll try Slack again when newer versions are released.