-   Linux - Software (
-   -   Software RAID (5) issues (

BlackRabbit 03-28-2008 02:04 PM

Software RAID (5) issues
Hi all,

decided to go for Linux Software RAID 5 on my HighPoint 1820A controller.

Followed some guides around the net, summarized I did this:

> mdadm --create /dev/md0 -l5 --chunk=128 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sdd
As 1 can see: I build a RAID5 array from 4 drives (each of 400GB).

This is what my mdadm.conf looks like:

DEVICE /dev/sda /dev/sdb /dev/sdc /dev/sdd
ARRAY /dev/md0 devices=/dev/sda,/dev/sdb,/dev/sdc,/dev/sdd

However, upon boot the array wasn't assebled for some reason (can anyone explain why?).

So I assembled manually:

> mdadm --assemble --scan
Check 'cat /proc/mdstat' showed that the array was resyncing.
Is it normal that it needs to resync each time it is assembled?

I was able to format it:

> mkreiserfs /dev/md0
So far so good, I was able to write file to the array (at fair speeds), so I was a happy man.

However, troubles started when I rebooted the system. As the array was still resyncing, the system hung upon shutdown (before reboot). Last line mentioned was something like "saving resync checkpoints". As the system did not react anymore, I did a hard-reset. (how long does it take to save those checkpoints? How bad is it to reboot during resync?)

As I try to assemble the array now, I get the following message:

root@sun:~# mdadm --assemble /dev/md0
mdadm: /dev/md0 assembled from 1 drive - not enough to start the array.

I'm not sure what to do next.. Any advise and some answers on the questions above would be much appreciated!

PS: as there is no real data (just some test files) on the array yet, I could create a new array. However, it would be great if that wouldn't be necessary, as it takes a long time to create this array). Also, it doesn't give a very safe feeling ;)

Some more info:

> mdadm --examine --scan
ARRAY /dev/md0 level=raid5 num-devices=4 UUID=fdf567f0:4a5cdbf9:209911b2:ab5853da

Right after reboot:

root@sun:~# dmesg |grep md
ata1: SATA max UDMA/133 cmd 0x00000000 ctl 0xe0ba2120 bmdma 0x00000000 irq 10
ata2: SATA max UDMA/133 cmd 0x00000000 ctl 0xe0ba4120 bmdma 0x00000000 irq 10
ata3: SATA max UDMA/133 cmd 0x00000000 ctl 0xe0ba6120 bmdma 0x00000000 irq 10
ata4: SATA max UDMA/133 cmd 0x00000000 ctl 0xe0ba8120 bmdma 0x00000000 irq 10
ata5: SATA max UDMA/133 cmd 0x00000000 ctl 0xe0bb2120 bmdma 0x00000000 irq 10
ata6: SATA max UDMA/133 cmd 0x00000000 ctl 0xe0bb4120 bmdma 0x00000000 irq 10
ata7: SATA max UDMA/133 cmd 0x00000000 ctl 0xe0bb6120 bmdma 0x00000000 irq 10
ata8: SATA max UDMA/133 cmd 0x00000000 ctl 0xe0bb8120 bmdma 0x00000000 irq 10
md: linear personality registered for level -1
md: raid0 personality registered for level 0
md: raid1 personality registered for level 1
md: raid10 personality registered for level 10
md: raid6 personality registered for level 6
md: raid5 personality registered for level 5
md: raid4 personality registered for level 4
md: multipath personality registered for level -4
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.

After trying to assemble:

root@sun:~# mdadm --assemble --scan
mdadm: /dev/md0 assembled from 1 drive - not enough to start the array.
root@sun:~# dmesg |grep md
... ... ...
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
md: md0 stopped.
md: bind<sdb>
md: bind<sdc>
md: bind<sdd>
md: bind<sda>

BlackRabbit 03-28-2008 05:12 PM

This is really bugging me..

Tried again:
* created a 40GB partition on each of the 4 disks (sd[abcd]1)
* created a new array:
> mdadm --create /dev/md0 -l5 -c128 --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
(Finished without issues.)
* Formatted /dev/md0:
> mkreiserfs /dev/md0
* Rebooted (just 'to be sure')
* Although config file looks ok, array wasn't created upon boot, so I assemble manually:
> mdadm --assemble /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
* Mounted it:
> mount /dev/md0 /mnt/md0
Suddenly the machine crashed (no warnings/errors shown), system just hangs)

Hard-reset, tried to assemble again:

root@sun:~# mdadm --assemble /dev/sda /dev/sdb /dev/sdc /dev/sdd
mdadm: no recogniseable superblock on /dev/sdb
mdadm: /dev/sdb has no superblock - assembly aborted

root@sun:~# mdadm --zero-superblock /dev/sda1
root@sun:~# mdadm --zero-superblock /dev/sdb1
root@sun:~# mdadm --zero-superblock /dev/sdc1
root@sun:~# mdadm --zero-superblock /dev/sdd1
root@sun:~# mdadm --assemble /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
mdadm: no recogniseable superblock on /dev/sdb1
mdadm: /dev/sdb1 has no superblock - assembly aborted

Could it be Reiser causing these problems?

BlackRabbit 03-28-2008 06:05 PM

This is just too weird..

Redid the whole thing, formatted with ext3:

root@sun:/etc# mkfs.ext3 /dev/md0
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
14663680 inodes, 29296416 blocks
1464820 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=29360128
895 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Everything looks ok:

root@sun:/etc# mdadm -Q -D /dev/md0
        Version : 00.90.03
  Creation Time : Fri Mar 28 23:38:42 2008
    Raid Level : raid5
    Array Size : 117185664 (111.76 GiB 120.00 GB)
  Used Dev Size : 39061888 (37.25 GiB 40.00 GB)
  Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Sat Mar 29 00:03:56 2008
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

        Layout : left-symmetric
    Chunk Size : 128K

          UUID : eb55d4f6:3fa2025e:dc065d20:a79325b0
        Events : 0.4

    Number  Major  Minor  RaidDevice State
      0      8        1        0      active sync  /dev/sda1
      1      8      17        1      active sync  /dev/sdb1
      2      8      33        2      active sync  /dev/sdc1
      3      8      49        3      active sync  /dev/sdd1

Was able to 'touch' some files and mkdir some directories, chmod & chown etc: all no problem. Fired up proftpd and started transferring data to the array -> system crash

BlackRabbit 03-29-2008 04:16 AM

BTW: after the system crash I did a (hard)reset, and once again I was no longer able to assemble /dev/md0..

BlackRabbit 03-29-2008 10:52 AM

*sigh*, anybody?

Tried once again (you never know..) but system failed again:
- was able to create the array
- was able to assemble it after reboot (without it starting to resync btw)
- was able to format /dev/md0 in ext3 FS
- did not succeed to mount /dev/md0:
> mount /dev/md0 /mnt/md0
=> system just hangs, no response whatsoever

As I wanted to be sure that it wasn't just mount taking a lot of time to actually mount the device, I waited for about 30min. STill, the system didn't respond.
Did a hard reset. After reboot I was able to assemble /dev/md0 without errors. It is now resyncing though, so it is clear that something went wrong. Did a "ef2fsck /dev/md0", came back clean.

Will now wait for resync to finish and try to mount again.

Some more questions:
- do I need to partition/format the drives before I start creating the array?
- can I use reiserfs/ext3 as FS on a md device?
- can I (may I) mount /dev/md0 while it is resyncing?

BlackRabbit 03-29-2008 02:46 PM

Could it be the drivers of my controller causing these issues?

It's a HighPoint 1820A, I'm using SlackWare built-in drivers.

In what log might I be able to find some useful information?

BlackRabbit 03-29-2008 03:28 PM

I don't get it: why does mount make my system crash?

I've spent the last few days trying & retrying all kinds of guides & howto's, googled for any possible sollutions to similar problems but it just doesn't want to work :(

BlackRabbit 03-30-2008 08:55 AM


Installed Slack 10.1 (kernel 2.4), installed highpoint drivers (from their site) and used the distro-provided mdadm -> no issues at all.

Reinstalled Slack 12.0, highpoint drivers from hpsite, mdadm of distro. As soon as I try to access the drives, the controller starts to beep (see this thread.

- I suppose it's the highpoint drivers + my kernel (2.6)?
- Am I doomed to run Slack 10.1 (until new drivers from highpoint are released)?

BlackRabbit 03-30-2008 12:39 PM

Last attempt: during boot the kernel-built-in drivers are loaded, so maybe they cause trouble with the hpt drivers?

root@sun:~# dmesg|grep 3xx
RocketRAID 3xxx SATA Controller driver v1.0 (060426)

How can I force those drivers not to be loaded? (the hptmv don't have to be loaded upon boot)

BlackRabbit 03-31-2008 04:48 AM

Final update:
- installed Slack 12.0 again
- recompiled the kernel without support for any HighPoint controllers
- installed opensource drivers from highpoint

Although it took a bit more time before the controller started beeping, it still did when I tried to access a device on the controller (eg cfdisk /dev/sda).

I've sent a mail to Highpoint with my findings, hopefully a fix will be published soon.

Until then, I'm back to Slack 10.1

BlackRabbit 04-27-2008 05:18 PM

FYI: I was able to get everything to work without problems under Ubuntu Hardy Heron Server Edition (8.04). Note that I don't use the HighPoint drivers (the 1820A works without problems with the default kernel) as I don't have time to experiment (+ I can't backup my 'old' software raid5 arrays :)).

Maybe I'll try Slack again when newer versions are released.

All times are GMT -5. The time now is 11:40 PM.