[SOLVED] PIKE 2308 Hardware RAID seems okay but I'm not sure . . .

vitronix · 05-01-2017, 11:03 AM

Hi all of you out there,

A while ago I asked a question about my ASUS PIKE 2308 RAID controller.
I wanted to check if CentOS 7 was supporting the card.
I wanted to install Nethserver 7.
Well Nethserver is a bit disappointing.
So I thought well the only things I really need are Samba DNS an LAMP.
That shouldn't be to hard to install myself.
I choose Ubuntu 16.04 LTS.
Everything looks okay I just wanted to check what the experts have to say about it.
I analyzed the boot output of what I think is coming from the kernel.
Here are the interesting bits:

Code:

    3.419667] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 35
[    3.419668] mpt2sas0-msix1: PCI-MSI-X enabled: IRQ 36
[    3.419668] mpt2sas0-msix2: PCI-MSI-X enabled: IRQ 37
[    3.419668] mpt2sas0-msix3: PCI-MSI-X enabled: IRQ 38
[    3.419669] mpt2sas0-msix4: PCI-MSI-X enabled: IRQ 39
[    3.419669] mpt2sas0-msix5: PCI-MSI-X enabled: IRQ 40
[    3.419670] mpt2sas0-msix6: PCI-MSI-X enabled: IRQ 41
[    3.419670] mpt2sas0-msix7: PCI-MSI-X enabled: IRQ 42
[    3.419671] mpt2sas_cm0: iomem(0x00000000db240000), mapped(0xffffc90001ba0000), size(65536)
[    3.419672] mpt2sas_cm0: ioport(0x000000000000d000), size(256)
[    3.470972] mpt2sas_cm0: sending diag reset !!
[    4.561035] mpt2sas_cm0: LSISAS2308: FWVersion(15.00.00.00), ChipRevision(0x05), BiosVersion(07.29.00.00)
[    4.571057] mpt2sas_cm0: Protocol=(
[    4.614212] scsi host0: Fusion MPT SAS Host
[    4.618768] mpt2sas_cm0: sending port enable !!
[   12.367291] mpt2sas_cm0: port enable: SUCCESS
[   12.371918] scsi 0:1:0:0: Direct-Access     LSI      Logical Volume   3000 PQ: 0 ANSI: 6
[   12.380346] scsi 0:1:0:0: RAID1: handle(0x0522), wwid(0x00830984cffcf63e), pd_count(2), type(SATA)
[   12.390557] scsi 0:0:0:0: Direct-Access     ATA      WDC WD20EFRX-68E 0A82 PQ: 0 ANSI: 6
[   12.398874] scsi 0:0:0:0: SATA: handle(0x0009), sas_addr(0x4433221100000000), phy(0), device_name(0x50014ee262b3c110)
[   12.409500] scsi 0:0:0:0: SATA: enclosure_logical_id(0x5000ea64400c9a00), slot(0)
[   12.417051] scsi 0:0:0:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[   12.428054] scsi 0:0:1:0: Direct-Access     ATA      WDC WD20EFRX-68E 0A82 PQ: 0 ANSI: 6
[   12.436381] scsi 0:0:1:0: SATA: handle(0x000a), sas_addr(0x4433221101000000), phy(1), device_name(0x50014ee260ae8136)
[   12.447028] scsi 0:0:1:0: SATA: enclosure_logical_id(0x5000ea64400c9a00), slot(1)
[   12.454585] scsi 0:0:1:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[   12.464531] sd 0:1:0:0: Attached scsi generic sg1 type 0
[   12.464588] sd 0:1:0:0: [sda] 3904294912 512-byte logical blocks: (2.00 TB/1.82 TiB)
[   12.464589] sd 0:1:0:0: [sda] 4096-byte physical blocks
[   12.464677] sd 0:1:0:0: [sda] Write Protect is off
[   12.464698] sd 0:1:0:0: [sda] No Caching mode page found
[   12.464698] sd 0:1:0:0: [sda] Assuming drive cache: write through
[   12.498093]  sda: sda1 sda2 < sda5 sda6 sda7 >
[   12.498769] sd 0:1:0:0: [sda] Attached SCSI disk
[   12.508437] scsi 0:0:0:0: Attached scsi generic sg2 type 0
[   12.514132] scsi 0:0:1:0: Attached scsi generic sg3 type 0
[   12.520888] sr 7:0:0:0: [sr1] scsi-1 drive
[   12.525174] sr 7:0:0:0: Attached scsi generic sg4 type 5
[   12.530841] sd 8:0:0:0: Attached scsi generic sg5 type 0
[   12.531545] sd 8:0:0:0: [sdb] Attached SCSI removable disk
[   12.541888] sd 9:0:0:0: Attached scsi generic sg6 type 0
[   12.542962] sd 9:0:0:0: [sdc] Attached SCSI removable disk
Begin: Loading essential drivers[   12.984864] md: linear personality registered for level -1
 ... [   12.993083] md: multipath personality registered for level -4
[   13.000276] md: raid0 personality registered for level 0
[   13.007365] md: raid1 personality registered for level 1
[   13.079298] raid6: sse2x1   gen() 11598 MB/s
[   13.151299] raid6: sse2x1   xor()  9098 MB/s
[   13.223301] raid6: sse2x2   gen() 14811 MB/s
[   13.295303] raid6: sse2x2   xor()  9955 MB/s
[   13.367308] raid6: sse2x4   gen() 17075 MB/s
[   13.439306] raid6: sse2x4   xor() 11895 MB/s
[   13.511309] raid6: avx2x1   gen() 22666 MB/s
[   13.583312] raid6: avx2x2   gen() 26359 MB/s
[   13.655315] raid6: avx2x4   gen() 30394 MB/s
[   13.659591] raid6: using algorithm avx2x4 gen() 30394 MB/s
[   13.665069] raid6: using avx2x2 recovery algorithm
[   13.670577] xor: automatically using best checksumming function:
[   13.715316]    avx       : 35994.000 MB/sec
[   13.720384] async_tx: api initialized (async)
[   13.729350] md: raid6 personality registered for level 6
[   13.734685] md: raid5 personality registered for level 5
[   13.740015] md: raid4 personality registered for level 4
[   13.748655] md: raid10 personality registered for level 10

What's all this about RAID 6? I configured a RAID 1.

Then the relevant contents of my /dev directory:

Code:

sda1    
sda2
sda5
sda6
sda7
sdb
sdc

Why doesn't sdb have 5 partitions too?

And fstab

Code:

 /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>            <dump>   <pass>
# / was on /dev/sdc6 during installation
UUID=f65aa074-9b7e-4d22-af5a-db13ba799704 /                      ext4     errors=remount-ro 0       1
# /boot was on /dev/sdc1 during installation
UUID=3389966b-750d-4af4-a483-8bfde29dd109 /boot           ext4       defaults        0       2
# /data was on /dev/sdc7 during installation
UUID=204b17ca-0f9a-4d6c-896b-d689db060fe5 /data            ext4       defaults        0       2
# swap was on /dev/sdc5 during installation
UUID=cec753fc-75ba-4f76-9cc3-507b0ce5625b none                swap    sw              0       0

The only thing that I found to be strange is that mdadm was installed. mdadm is for Fake RAID so why is it installed?
Can I safely remove the mdadm package?

Thanks in advance for your time and checking my findings ;-)

Vitronix

Ser Olmy · 05-02-2017, 04:00 AM

Quote:

Originally Posted by vitronix

Code:

Begin: Loading essential drivers[   12.984864] md: linear personality registered for level -1
 ... [   12.993083] md: multipath personality registered for level -4
[   13.000276] md: raid0 personality registered for level 0
[   13.007365] md: raid1 personality registered for level 1
[   13.079298] raid6: sse2x1   gen() 11598 MB/s
[   13.151299] raid6: sse2x1   xor()  9098 MB/s
[   13.223301] raid6: sse2x2   gen() 14811 MB/s
[   13.295303] raid6: sse2x2   xor()  9955 MB/s
[   13.367308] raid6: sse2x4   gen() 17075 MB/s
[   13.439306] raid6: sse2x4   xor() 11895 MB/s
[   13.511309] raid6: avx2x1   gen() 22666 MB/s
[   13.583312] raid6: avx2x2   gen() 26359 MB/s
[   13.655315] raid6: avx2x4   gen() 30394 MB/s
[   13.659591] raid6: using algorithm avx2x4 gen() 30394 MB/s
[   13.665069] raid6: using avx2x2 recovery algorithm
[   13.670577] xor: automatically using best checksumming function:
[   13.715316]    avx       : 35994.000 MB/sec
[   13.720384] async_tx: api initialized (async)
[   13.729350] md: raid6 personality registered for level 6
[   13.734685] md: raid5 personality registered for level 5
[   13.740015] md: raid4 personality registered for level 4
[   13.748655] md: raid10 personality registered for level 10

What's all this about RAID 6? I configured a RAID 1.

That's just the software RAID subsystem benchmarking a number of different algorithms to figure out which is the fastest. This happens because you have software RAID support compiled into your kernel. Just ignore it.

These, however, are the relevant log entries for your RAID controller:

Code:

[    3.419667] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 35
[    3.419668] mpt2sas0-msix1: PCI-MSI-X enabled: IRQ 36
[    3.419668] mpt2sas0-msix2: PCI-MSI-X enabled: IRQ 37
[    3.419668] mpt2sas0-msix3: PCI-MSI-X enabled: IRQ 38
[    3.419669] mpt2sas0-msix4: PCI-MSI-X enabled: IRQ 39
[    3.419669] mpt2sas0-msix5: PCI-MSI-X enabled: IRQ 40
[    3.419670] mpt2sas0-msix6: PCI-MSI-X enabled: IRQ 41
[    3.419670] mpt2sas0-msix7: PCI-MSI-X enabled: IRQ 42
[    3.419671] mpt2sas_cm0: iomem(0x00000000db240000), mapped(0xffffc90001ba0000), size(65536)
[    3.419672] mpt2sas_cm0: ioport(0x000000000000d000), size(256)
[    3.470972] mpt2sas_cm0: sending diag reset !!
[    4.561035] mpt2sas_cm0: LSISAS2308: FWVersion(15.00.00.00), ChipRevision(0x05), BiosVersion(07.29.00.00)
[    4.571057] mpt2sas_cm0: Protocol=(
[    4.614212] scsi host0: Fusion MPT SAS Host
[    4.618768] mpt2sas_cm0: sending port enable !!
[   12.367291] mpt2sas_cm0: port enable: SUCCESS
[   12.371918] scsi 0:1:0:0: Direct-Access     LSI      Logical Volume   3000 PQ: 0 ANSI: 6
[   12.380346] scsi 0:1:0:0: RAID1: handle(0x0522), wwid(0x00830984cffcf63e), pd_count(2), type(SATA)
[   12.390557] scsi 0:0:0:0: Direct-Access     ATA      WDC WD20EFRX-68E 0A82 PQ: 0 ANSI: 6
[   12.398874] scsi 0:0:0:0: SATA: handle(0x0009), sas_addr(0x4433221100000000), phy(0), device_name(0x50014ee262b3c110)
[   12.409500] scsi 0:0:0:0: SATA: enclosure_logical_id(0x5000ea64400c9a00), slot(0)
[   12.417051] scsi 0:0:0:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[   12.428054] scsi 0:0:1:0: Direct-Access     ATA      WDC WD20EFRX-68E 0A82 PQ: 0 ANSI: 6
[   12.436381] scsi 0:0:1:0: SATA: handle(0x000a), sas_addr(0x4433221101000000), phy(1), device_name(0x50014ee260ae8136)
[   12.447028] scsi 0:0:1:0: SATA: enclosure_logical_id(0x5000ea64400c9a00), slot(1)
[   12.454585] scsi 0:0:1:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[   12.464531] sd 0:1:0:0: Attached scsi generic sg1 type 0
[   12.464588] sd 0:1:0:0: [sda] 3904294912 512-byte logical blocks: (2.00 TB/1.82 TiB)
[   12.464589] sd 0:1:0:0: [sda] 4096-byte physical blocks
[   12.464677] sd 0:1:0:0: [sda] Write Protect is off
[   12.464698] sd 0:1:0:0: [sda] No Caching mode page found
[   12.464698] sd 0:1:0:0: [sda] Assuming drive cache: write through
[   12.498093]  sda: sda1 sda2 < sda5 sda6 sda7 >
[   12.498769] sd 0:1:0:0: [sda] Attached SCSI disk

As you can see, your RAID 1 volume is initialized and working properly.

Quote:

Originally Posted by vitronix

Then the relevant contents of my /dev directory:

Code:

sda1    
sda2
sda5
sda6
sda7
sdb
sdc

Why doesn't sdb have 5 partitions too?

Why would it have? It's not related to your hardware RAID. (Looks like it might be a memory card reader of some sort.)

The whole point of hardware RAID is that the RAID controller handles all the work. All the OS sees, is one or more logical RAID volumes appearing as if they were regular block devices.

The individual physical drives in a hardware RAID set are not exposed to the OS at all. In order to manage a hardware RAID set (check status, rebuild, verify/scrub, expand, convert etc.) you'll have to use software designed to communicate directly with the controller (through the driver).

Quote:

Originally Posted by vitronix

The only thing that I found to be strange is that mdadm was installed. mdadm is for Fake RAID so why is it installed?
Can I safely remove the mdadm package?

mdadm is included in most distributions. If you don't use it, you can safely remove it.

You'll still see md-related entries in the log, though, unless you also replace the kernel with one without software RAID support. I wouldn't bother with that unless I was running Linux on an embedded system with very little RAM.

vitronix · 05-02-2017, 09:54 AM

Well Sir Olmy,

I thought why not test the whole thing?
So I removed a SATA cable from one HDD and booted.
The LSI BIOS detected the malfunctioning of the hard disk.
But the system booted as normal.
However when I disconnected the other HDD the system didn't boot anymore.
So what am I to do? clearly I did something wrong . . .

I am eager for an answer.
I think I should have five partitions on BOTH disks but how do I install that?

Vitronix

AwesomeMachine · 05-02-2017, 07:51 PM

If you lose the other drive you might have to connect the good one to the same controller port that worked in your test. Then you can replace the drive and rebuild the array.

Ser Olmy · 05-04-2017, 05:36 AM

Quote:

Originally Posted by vitronix

So I removed a SATA cable from one HDD and booted.
The LSI BIOS detected the malfunctioning of the hard disk.
But the system booted as normal.

Yes, the controller will see the RAID metadata on the remaining disk, and will mark the RAID as "degraded" before continuing booting.

Quote:

Originally Posted by vitronix

However when I disconnected the other HDD the system didn't boot anymore.
So what am I to do? clearly I did something wrong . . .

If you briefly reconnected both disks and then disconnected the first, this is to be expected.

When a missing drive comes back online, the metadata will show it to be out of sync. The controller will then mark the new drive as out-of-sync and typically initiate a rebuild, and until that process has completed, the RAID remains degraded. Disconnecting the first drive before the rebuild is complete will leave you with one drive without valid data.

Quote:

Originally Posted by vitronix

I think I should have five partitions on BOTH disks but how do I install that?

You don't. /dev/sdb is NOT the second drive in your array.

/dev/sda is the logical RAID volume, and neither physical disk is directly visible to the OS. That's how hardware RAID works.

vitronix · 05-05-2017, 01:38 AM

Hello Sir Olmy,

What the operating system sees, as is clear in the fstab file is sdc
sda and sdb are the drives used for building the array.

My expectations of RAID 1 where too high.

As mentioned when you disconnect sdb the RAID controller will start to sync drives.
You have to wait until everything is rebuilt and then you can boot again.

But everything is clear now to me.

So thank you all for your time!

Best regards,

Vitronix