Throughput problem with MPT Fusion SAS

herr_tichy · 11-14-2010, 04:05 PM

Hello Forum,

I've got the following setup:

LSI SAS1068E PCI-Express Fusion-MPT SAS
Hitachi Deskstar 7K1000.C (8x)
4-core Xeon
some Intel server board
kernel 2.6.34 and 2.6.35 (tried both)

Code:

Fusion MPT base driver 3.04.15
Fusion MPT SAS Host driver 3.04.15

I've also tried the current driver from LSI themselves (v4.2.4, iirc).

Anyway, I'm not using any of the RAID capabilities of the controller but I am accessing the drives directly. I have a ZFS RAID-Z2 (dedup=on, compression=on) across all eight disks which shows a rather bad performance with small writes. And by bad, I mean this bad (measured using bonnie++):

http://imgur.com/bWHoK.png

Anyway, it's the same without dedup and compression and even the same when writing directly to the devices:

Code:

ltm-memorix ~ # dd if=/dev/zero of=/dev/sdb bs=1kB count=500000
500000+0 Datensätze ein
500000+0 Datensätze aus
500000000 Bytes (500 MB) kopiert, 26,0034 s, 19,2 MB/s

Code:

ltm-memorix ~ # dd if=/dev/zero of=/dev/sdb bs=5kB count=100000
100000+0 Datensätze ein
100000+0 Datensätze aus
500000000 Bytes (500 MB) kopiert, 21,2242 s, 23,6 MB/s

Code:

ltm-memorix ~ # dd if=/dev/zero of=/dev/sdb bs=10kB count=50000
50000+0 Datensätze ein
50000+0 Datensätze aus
500000000 Bytes (500 MB) kopiert, 12,7889 s, 39,1 MB/s

Code:

ltm-memorix ~ # dd if=/dev/zero of=/dev/sdb bs=20kB count=25000
25000+0 Datensätze ein
25000+0 Datensätze aus
500000000 Bytes (500 MB) kopiert, 8,44659 s, 59,2 MB/s

Code:

ltm-memorix ~ # dd if=/dev/zero of=/dev/sdb bs=1000kB count=500
500+0 Datensätze ein
500+0 Datensätze aus
500000000 Bytes (500 MB) kopiert, 6,91047 s, 72,4 MB/s

As you can see, write performance gets a lot better with bigger block sizes. What am I doing wrong here? On my desktop machine with its cheap SiL controller I'm getting WAY better throughput.

stress_junkie · 11-14-2010, 04:26 PM

Here's an idea from http://forums.gentoo.org/viewtopic.php?t=512696

Quote:

Posted: Thu Nov 09, 2006 8:50 pm Post subject: Reply with quote
While googling around I came across this thread for a similar model xw9300

http://www.gatago.com/linux/kernel/18503357.html

I turned ACPI Segmentation off in the bios and the sucker came right up. I haven't tried the pci=noacpi with ACPI Segmentation turned on but I image it would work if I needed ACPI Segmentation on. Thanks to bgregorcy for trying to help me as he was the only one. Hope this thread helps others with similar hardware.

May or may not be related. The above post was about the devices not showing up at all but still maybe playing with ACPI on the motherboard might help.

There is more food for thought at https://www.centos.org/modules/newbb...25692&forum=37

Quote:

so at the end I did what is recommended in the article provided by pschaff, I disabled the RAID on the motherboard (jumper setting) and moved the cables from the SAS ports to the normal SATA ports.