PCIe SATA controller problem in re hard drive capacity

Brukenet · 06-08-2023, 06:45 AM

Hi all,

I have tried two different expansion cards in a machine running Debian 11 -

1. An older PCIe SATA controller with:
Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller (rev 11)

2. A newer PCIe SATA controller with:
ASMedia Technology Inc. Device 1164 (rev 02)

That data is copy/pasted directly from the output of lspci

I currently have both installed and have connected a 4 TB drive to each of them. In this configuration, everything works. I can mount the disks and set them up in fstab to auto-mount on boot.

My problem is that I don't want to use 4 TB hard drives. I have several drives that range from 10 TB to 18 TB (all tested and known to be good) but none of them seem to work with these two PCIe cards.

I am at a total loss. I've been reading documentation for over a week and can't find any mention of a maximum capacity for these.

I have confirmed that the larger drives are all 6 GB/s transfer speeds, which the PCIe cards claim to support. I have confirmed that when they are connected and the machine is powered on, I can touch them and feel that they are spinning.

Any suggestions are very welcome.

smallpond · 06-08-2023, 04:05 PM

"doesn't work" is not a useful problem description. Does it not see the disk at all? Did you give a command that gives an error? Does it spray ink all over your desk, then scuttle away like an octopus?

With the disk connected, what is the output of lsblk?

Brukenet · 06-08-2023, 11:48 PM

Quote:

Originally Posted by smallpond

"doesn't work" is not a useful problem description. Does it not see the disk at all? Did you give a command that gives an error? Does it spray ink all over your desk, then scuttle away like an octopus?

With the disk connected, what is the output of lsblk?

As I said, they don't work. They don't show up in lsblk when connected (sorry if that wasn't obvious from my first post). They have power, I can touch them and feel them spinning, but they don't show up in /dev/ nor lsblk. I've used dmesg and I don't see any obvious errors but I'm less familiar with dmesg output so I could be missing something.

I want to be very clear - both PCIe SATA cards work fine with smaller hard drives, up to about 4 TB. For those smaller drives they are automatically detected, I can set the drives up in fstab to mount on boot, etc. My search history is littered with results that are about getting the system to recognize the cards and that is NOT the issue.

The issue only appears when I connect larger drives, 10 TB to 18 TB.

Here's a copy paste of my lsblk output (note that all these drives are accounted for and none are the new 10 TB and 18 TB ones I'm trying to connect):

Code:

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    1   3.6T  0 disk 
└─sda1   8:1    1   3.6T  0 part /media/archive-8
sdb      8:16   0  12.7T  0 disk 
└─sdb1   8:17   0  12.7T  0 part /media/archive-3
sdc      8:32   0  12.7T  0 disk 
└─sdc1   8:33   0  12.7T  0 part /media/archive-1
sdd      8:48   0 931.5G  0 disk 
├─sdd1   8:49   0   512M  0 part /boot/efi
├─sdd2   8:50   0 930.1G  0 part /
└─sdd3   8:51   0   977M  0 part [SWAP]
sde      8:64   0  12.7T  0 disk 
└─sde1   8:65   0  12.7T  0 part /media/archive-2
sdf      8:80   0  12.7T  0 disk 
└─sdf1   8:81   0  12.7T  0 part /media/archive-4
sdg      8:96   0  12.7T  0 disk 
└─sdg1   8:97   0  12.7T  0 part /media/archive-5
sdh      8:112  0  12.7T  0 disk 
└─sdh1   8:113  0  12.7T  0 part /media/archive-6
sdi      8:128  0  12.7T  0 disk 
└─sdi1   8:129  0  12.7T  0 part /media/archive-7

The drives listed above include a 1 TB Samsung SSD that I use for a boot drive, i.e. sdd, plus a 4 TB drive connected to one of the PCIe SATA cards (which is a proof of concept that the cards are installed correctly and working), i.e. sda, plus the seven 14 TB drives connected to the motherboard.

In case it helps, here's a relevant section of the lspci output (note that I do see both cards post prior to BIOS when I boot):

Code:

03:00.0 SATA controller: ASMedia Technology Inc. Device 1164 (rev 02)
04:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller (rev 11)
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 26)
06:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
06:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
06:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
07:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
08:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

Here's a portion of the dmesg output that seems relevant (but as I said, I'm not very familiar with dmesg output and don't know what's normal):

Code:

[    1.401687] ahci 0000:04:00.0: AHCI 0001.0000 32 slots 4 ports 6 Gbps 0xf impl SATA mode
[    1.401689] ahci 0000:04:00.0: flags: 64bit ncq sntf led only pmp fbs pio slum part sxs 
[    1.401961] scsi host24: ahci
[    1.402026] scsi host25: ahci
[    1.402090] scsi host26: ahci
[    1.402148] scsi host27: ahci
[    1.402172] ata25: SATA max UDMA/133 abar m2048@0xfc740000 port 0xfc740100 irq 63
[    1.402174] ata26: SATA max UDMA/133 abar m2048@0xfc740000 port 0xfc740180 irq 63
[    1.402177] ata27: SATA max UDMA/133 abar m2048@0xfc740000 port 0xfc740200 irq 63
[    1.402179] ata28: SATA max UDMA/133 abar m2048@0xfc740000 port 0xfc740280 irq 63
[    1.402516] ahci 0000:07:00.0: AHCI 0001.0301 32 slots 4 ports 6 Gbps 0xf impl SATA mode
[    1.402517] ahci 0000:07:00.0: flags: 64bit ncq sntf ilck pm led clo only pmp fbs pio slum part 
[    1.402749] scsi host28: ahci
[    1.402790] scsi host29: ahci
[    1.402829] scsi host30: ahci
[    1.402867] scsi host31: ahci
[    1.402886] ata29: SATA max UDMA/133 abar m2048@0xfc500000 port 0xfc500100 irq 64
[    1.402888] ata30: SATA max UDMA/133 abar m2048@0xfc500000 port 0xfc500180 irq 65
[    1.402890] ata31: SATA max UDMA/133 abar m2048@0xfc500000 port 0xfc500200 irq 66
[    1.402892] ata32: SATA max UDMA/133 abar m2048@0xfc500000 port 0xfc500280 irq 67
[    1.403168] ahci 0000:08:00.0: AHCI 0001.0301 32 slots 4 ports 6 Gbps 0x33 impl SATA mode
[    1.403169] ahci 0000:08:00.0: flags: 64bit ncq sntf ilck pm led clo only pmp fbs pio slum part 
[    1.403455] scsi host32: ahci
[    1.403497] scsi host33: ahci
[    1.403544] scsi host34: ahci
[    1.403631] scsi host35: ahci
[    1.403668] scsi host36: ahci
[    1.403712] scsi host37: ahci
[    1.403734] ata33: SATA max UDMA/133 abar m2048@0xfc400000 port 0xfc400100 irq 80
[    1.403736] ata34: SATA max UDMA/133 abar m2048@0xfc400000 port 0xfc400180 irq 81
[    1.403736] ata35: DUMMY
[    1.403736] ata36: DUMMY
[    1.403738] ata37: SATA max UDMA/133 abar m2048@0xfc400000 port 0xfc400300 irq 84
[    1.403749] ata38: SATA max UDMA/133 abar m2048@0xfc400000 port 0xfc400380 irq 85
[    1.635439] usb 3-6: new full-speed USB device number 2 using xhci_hcd
[    1.636798] usb 1-2: new high-speed USB device number 2 using xhci_hcd
[    1.704193] ata1: SATA link down (SStatus 0 SControl 300)
[    1.716512] ata27: SATA link down (SStatus 0 SControl 300)
[    1.716976] ata28: SATA link down (SStatus 0 SControl 300)
[    1.718450] ata25: SATA link down (SStatus 0 SControl 300)
[    1.798566] usb 3-6: config 1 has an invalid interface number: 2 but max is 1
[    1.798567] usb 3-6: config 1 has no interface number 1
[    1.806551] usb 1-2: New USB device found, idVendor=0557, idProduct=5411, bcdDevice= 1.19
[    1.806552] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    1.806553] usb 1-2: Product: 4-Port USB 2.0 Hub
[    1.806553] usb 1-2: Manufacturer: Generic
[    1.810566] usb 3-6: New USB device found, idVendor=0b05, idProduct=18f3, bcdDevice= 1.00
[    1.810567] usb 3-6: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    1.810568] usb 3-6: Product: AURA LED Controller
[    1.810569] usb 3-6: Manufacturer: AsusTek Computer Inc.
[    1.810569] usb 3-6: SerialNumber: 9876543210
[    1.823735] hid: raw HID events driver (C) Jiri Kosina
[    1.832589] usbcore: registered new interface driver usbhid
[    1.832590] usbhid: USB HID core driver
[    1.833486] hid-generic 0003:0B05:18F3.0001: hiddev0,hidraw0: USB HID v1.11 Device [AsusTek Computer Inc. AURA LED Controller] on usb-0000:06:00.3-6/input2
[    1.859047] hub 1-2:1.0: USB hub found
[    1.860276] hub 1-2:1.0: 4 ports detected
[    1.873043] ata31: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    1.873448] ata31.00: supports DRM functions and may not be fully accessible
[    1.876839] ata30: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    1.876920] ata26: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    1.877082] ata30.00: supports DRM functions and may not be fully accessible
[    1.877540] ata30.00: ATA-11: Samsung SSD 870 EVO 1TB, SVT01B6Q, max UDMA/133
[    1.877541] ata30.00: 1953525168 sectors, multi 1: LBA48 NCQ (depth 32), AA
[    1.877746] ata26.00: ATA-9: ST4000DM000-1F2168, CC54, max UDMA/133
[    1.877747] ata26.00: 7814037168 sectors, multi 0: LBA48 NCQ (depth 32), AA
[    1.878568] ata26.00: configured for UDMA/133
[    1.879304] ata30.00: supports DRM functions and may not be fully accessible
[    1.880457] ata31.00: ATA-9: WDC WD140EDFZ-11A0VA0, 81.00A81, max UDMA/133
[    1.880458] ata31.00: 27344764928 sectors, multi 16: LBA48 NCQ (depth 32), AA
[    1.880863] ata37: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    1.880864] ata32: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    1.880887] ata33: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    1.881165] ata30.00: configured for UDMA/133
[    1.881264] ata37.00: supports DRM functions and may not be fully accessible
[    1.881274] ata32.00: supports DRM functions and may not be fully accessible
[    1.881297] ata33.00: supports DRM functions and may not be fully accessible
[    1.884826] ata38: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    1.884849] ata34: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    1.885237] ata38.00: supports DRM functions and may not be fully accessible
[    1.885251] ata34.00: supports DRM functions and may not be fully accessible
[    1.888199] ata37.00: ATA-9: WDC WD140EDGZ-11B1PA0, 85.00A85, max UDMA/133
[    1.888200] ata37.00: 27344764928 sectors, multi 16: LBA48 NCQ (depth 32), AA
[    1.888219] ata32.00: ATA-9: WDC WD140EDFZ-11A0VA0, 81.00A81, max UDMA/133
[    1.888221] ata32.00: 27344764928 sectors, multi 16: LBA48 NCQ (depth 32), AA
[    1.888246] ata33.00: ATA-9: WDC WD140EDFZ-11A0VA0, 81.00A81, max UDMA/133
[    1.888248] ata33.00: 27344764928 sectors, multi 16: LBA48 NCQ (depth 32), AA
[    1.891757] ata31.00: supports DRM functions and may not be fully accessible
[    1.892253] ata38.00: ATA-9: WDC WD140EDGZ-11B1PA0, 85.00A85, max UDMA/133
[    1.892254] ata38.00: 27344764928 sectors, multi 16: LBA48 NCQ (depth 32), AA
[    1.892269] ata34.00: ATA-9: WDC WD140EDGZ-11B1PA0, 85.00A85, max UDMA/133
[    1.892270] ata34.00: 27344764928 sectors, multi 16: LBA48 NCQ (depth 32), AA
[    1.899368] ata37.00: supports DRM functions and may not be fully accessible
[    1.899392] ata32.00: supports DRM functions and may not be fully accessible
[    1.899423] ata33.00: supports DRM functions and may not be fully accessible
[    1.900243] ata31.00: configured for UDMA/133
[    1.903416] ata38.00: supports DRM functions and may not be fully accessible
[    1.903549] ata34.00: supports DRM functions and may not be fully accessible
[    1.907798] ata37.00: configured for UDMA/133
[    1.907857] ata33.00: configured for UDMA/133
[    1.907867] ata32.00: configured for UDMA/133
[    1.911860] ata38.00: configured for UDMA/133
[    1.912045] ata34.00: configured for UDMA/133
[    1.923048] usb 2-2: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[    1.940837] ata29: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    1.954288] usb 2-2: New USB device found, idVendor=0557, idProduct=2410, bcdDevice= 1.19
[    1.954289] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    1.954290] usb 2-2: Product: 4-Port USB 3.0 Hub
[    1.954291] usb 2-2: Manufacturer: Generic
[    1.990211] hub 2-2:1.0: USB hub found
[    1.991554] hub 2-2:1.0: 4 ports detected
[    2.008982] ata29.00: supports DRM functions and may not be fully accessible
[    2.010705] ata29.00: ATA-11: WDC WD140EDGZ-11B2DA2, 85.00A85, max UDMA/133
[    2.010707] ata29.00: 27344764928 sectors, multi 16: LBA48 NCQ (depth 32), AA
[    2.012392] ata29.00: supports DRM functions and may not be fully accessible
[    2.014901] ata29.00: configured for UDMA/133
[    2.019113] ata2: SATA link down (SStatus 0 SControl 300)

Does anyone see anything in that output that helps explain what I'm seeing?

Thanks.

kilgoretrout · 06-09-2023, 07:43 AM

What distro are you running? Also, post your kernel version in use:

Code:

$ uname -a

With the problem drives connected, check your bios setup and confirm that the drives are detected there. The cards may have there own firmware apart from the bios setup, however. I'm looking at the product page for the Asmedia card:

https://www.asmedia.com.tw/product/1...dYQ8bxZ4UR9wG5

The page references PCIe Gen3 which first came out in 2010 and was replaced by Gen4 in 2017. Accordingly, the card could be using over 10 year old technology. At that time, drives of 10TB or greater were not in common use and the card's firmware may not be able to handle drives of that size. If you are not seeing any reference to the problem drives in your dmesg right after boot, that would be an indication of a hardware/firmware problem. If it's a driver problem, that would be governed by the kernel and a more recent kernel may address the issue.

Arnulf · 06-09-2023, 09:03 AM

Use a more simple setup for testing, please:

Unplug all drives from onboard SATA controllers.
Insert only one of the two PCIe SATA controller cards.
Connect only one drive to it.
Boot a live Linux with a recent kernel from USB stick.
Post output of lspci -v and dmesg.

Both PCIe SATA controller cards may be too old for 10+ TB drives:

Check for firmware updates for both PCIe SATA controller cards. If firmware updates are available:
Connect only one of your 4 TB drive to PCIe SATA controller card.
Install new firmware.
Try it with a 10+ TB drive and simple setup described above again.

Brukenet · 06-10-2023, 03:00 AM

Quote:

Originally Posted by kilgoretrout

What distro are you running? Also, post your kernel version in use:

Code:

$ uname -a

With the problem drives connected, check your bios setup and confirm that the drives are detected there. The cards may have there own firmware apart from the bios setup, however. I'm looking at the product page for the Asmedia card:

https://www.asmedia.com.tw/product/1...dYQ8bxZ4UR9wG5

The page references PCIe Gen3 which first came out in 2010 and was replaced by Gen4 in 2017. Accordingly, the card could be using over 10 year old technology. At that time, drives of 10TB or greater were not in common use and the card's firmware may not be able to handle drives of that size. If you are not seeing any reference to the problem drives in your dmesg right after boot, that would be an indication of a hardware/firmware problem. If it's a driver problem, that would be governed by the kernel and a more recent kernel may address the issue.

As I mentioned initially, it's Debian 11.

The kernal version in use is:
5.10.0-8-amd64 #1 SMP Debian 5.10.46-5 (2021-09-23) x86_64 GNU/Linux

I spent about two hours in a live chat with a StarTech support person and they assured me that it should work and that there's no firmware updates available. They were completely stumped and the live chat ended without a solution.

The smaller drives do show up in BIOS but the larger drives do not.

I agree that a PCIe Gen4 card would be nice, but I can't find one anywhere. If you know of one, please let me know. I picked this Gen3 one because someone on another forum claimed that the ASM1164 can handle larger drives (I was asking then about another card I tried that used a Marvell chipset).

One thought that I have is, could it be caused by the fact that I'm putting it in the second of two PCIe x16 ports? I have a graphics card in the first x16 port. The ASM1164 card requires PCIe x2 or larger and won't fit in the x1 ports so I plugged it into the remaining, i.e., second, PCIe x16 port. Could a bios setting that handles these two ports be causing the issue? I feel like I'm grasping at straws but no other ideas at this point.

Thanks.

Brukenet · 06-10-2023, 03:04 AM

Quote:

Originally Posted by Arnulf

Use a more simple setup for testing, please:

Unplug all drives from onboard SATA controllers.
Insert only one of the two PCIe SATA controller cards.
Connect only one drive to it.
Boot a live Linux with a recent kernel from USB stick.
Post output of lspci -v and dmesg.

Both PCIe SATA controller cards may be too old for 10+ TB drives:

Check for firmware updates for both PCIe SATA controller cards. If firmware updates are available:
Connect only one of your 4 TB drive to PCIe SATA controller card.
Install new firmware.
Try it with a 10+ TB drive and simple setup described above again.

I spoke to a StarTech support person in live chat and they indicated that the card I have does not have any available firmware updates. They also claimed that I should have no problems with drives of up to 20 TB.

I have a busy few days but I will pull out all the other drives, set up a new live Linux distro on a USB, and try your suggestion. Please be patient with me but I will post the results here as soon as I get a chance.

Thanks.

Brukenet · 06-10-2023, 03:19 AM

Quote:

Originally Posted by Arnulf

Use a more simple setup for testing, please:

Unplug all drives from onboard SATA controllers.
Insert only one of the two PCIe SATA controller cards.
Connect only one drive to it.
Boot a live Linux with a recent kernel from USB stick.
Post output of lspci -v and dmesg.

Both PCIe SATA controller cards may be too old for 10+ TB drives:

Check for firmware updates for both PCIe SATA controller cards. If firmware updates are available:
Connect only one of your 4 TB drive to PCIe SATA controller card.
Install new firmware.
Try it with a 10+ TB drive and simple setup described above again.

Hey, I haven't had a chance to pull out all the other drives but I did try a

Code:

lspci -v

instead of just

Code:

lspci

and got this:

Code:

03:00.0 SATA controller: ASMedia Technology Inc. Device 1164 (rev 02) (prog-if 01 [AHCI 1.0])
	Subsystem: ZyDAS Technology Corp. Device 2116
	Flags: bus master, fast devsel, latency 0, IRQ 39, IOMMU group 19
	Memory at fc782000 (32-bit, non-prefetchable) [size=8K]
	Memory at fc780000 (32-bit, non-prefetchable) [size=8K]
	Expansion ROM at fc700000 [disabled] [size=512K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [80] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [130] Secondary PCI Express
	Kernel driver in use: ahci
	Kernel modules: ahci

I noticed that it's referencing 32-bit memory - is that relevant? If so, that would be a smoking gun that the issue is the card itself, not Linux.

Anyway, unless you reply back that the lspci -v output is enough to solve the issue, I'll plan to still go ahead and yank out all the drives later tonight and run the tests you suggested.

lvm_ · 06-10-2023, 05:37 AM

Do larger disks by any chance use 4k sectors? Not all controllers support them.

rknichols · 06-10-2023, 08:05 AM

Never mind. Misread something.

Arnulf · 06-10-2023, 02:37 PM

"32-bit memory" is a compatibility remnant from "32-bit PCI-bus aera" and means that card's memory range is exposed below 4 GiB. Therefore both 32-bit OSs and 64-bit OSs can handle this card. Nowadays this compatibility remnant is more or less useless. "32-bit memory" should not be a problem in modern computers running a 64-bit OS.

Output of lspci -v looks good. The kernel detects this card and loads the ahci module for it. Therefore it might be a hardware problem e.g. an incompatibility of this card with your 10+ TB HDDs.

kilgoretrout · 06-10-2023, 09:56 PM

Quote:

The smaller drives do show up in BIOS but the larger drives do not.

That pretty much cinches it as a hardware problem IMHO.

lvm_ · 06-11-2023, 01:41 AM

Quote:

Originally Posted by kilgoretrout

That pretty much cinches it as a hardware problem IMHO.

It indicates it, but old farts should remember how older BIOSes with no LBA support couldn't see large disks, and yet they could be accessed by LBA-compliant OSes.

Arnulf · 06-11-2023, 04:55 AM

Quote:

Originally Posted by lvm_

older BIOSes with no LBA support couldn't see large disks, and yet they could be accessed by LBA-compliant OSes.

That's right but one hardware limit occurs in this case too: This LBA access by OS is limited to LBA28 drives because these ancient PATA-controllers doesn't support LBA48. This LBA28/48-break comes to stay for commonly available PATA-HDDs between capacities 120 GB (LBA28) and 160 GB (LBA48). Running a PATA-HDD that requires LBA48-support on a PATA-controller that doesn't support LBA48 results in massive data loss independing of "detected" and shown PATA-HDD capacity e.g. by a Linux kernel.

Brukenet · 06-12-2023, 12:10 AM

Quote:

Originally Posted by kilgoretrout

That pretty much cinches it as a hardware problem IMHO.

That makes sense.

Now this question becomes, does anyone know a PCIe SATA card that does support larger hard drives?

Or, do you mean the issue is with the hard drives themselves? I've tested two different ones at 10 TB and one at 18 TB; they all work if plugged directly into the motherboard's SATA connectors; when they're plugged into the motherboard directly they show up in lsblk and can be added to fstab to automatically mount on boot.