Extremely poor disk performance on SCSI RAID

Yalla-One · 03-27-2007, 01:01 AM

All,

I'm getting extremely low disk performance on my new server, and suspect I have some incorrect kernel options or hardware configurations. Troughput as follows:

Code:

[189] SUPERUSER@spirou:/tmp#dd if=/dev/sda7 of=/dev/null bs=64k count=100
100+0 records in
100+0 records out
6553600 bytes (6.6 MB) copied, 1.50139 s, 4.4 MB/s
[190] SUPERUSER@spirou:/tmp#dd if=/dev/sda7 of=/dev/null bs=64k count=1000
1000+0 records in
1000+0 records out
65536000 bytes (66 MB) copied, 18.0209 s, 3.6 MB/s
[191] SUPERUSER@spirou:/tmp#dd if=/dev/sda7 of=/dev/null bs=64k count=10000
10000+0 records in
10000+0 records out
655360000 bytes (655 MB) copied, 179.74 s, 3.6 MB/s

As you can see, 3.6MB/s is appallingly low...

Some more system info:

Code:

00:00.0 Host bridge: Broadcom CMIC-WS Host Bridge (GC-LE chipset) (rev 13)
00:00.1 Host bridge: Broadcom CMIC-WS Host Bridge (GC-LE chipset)
00:00.2 Host bridge: Broadcom CMIC-LE
00:01.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
00:0f.0 Host bridge: Broadcom CSB5 South Bridge (rev 93)
00:0f.1 IDE interface: Broadcom CSB5 IDE Controller (rev 93)
00:0f.2 USB Controller: Broadcom OSB4/CSB5 OHCI USB Controller (rev 05)
00:0f.3 ISA bridge: Broadcom CSB5 LPC bridge
00:11.0 Host bridge: Broadcom CIOB-X2 PCI-X I/O Bridge (rev 03)
00:11.2 Host bridge: Broadcom CIOB-X2 PCI-X I/O Bridge (rev 03)
01:01.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)
02:01.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5703X Gigabit Ethernet (rev 02)
02:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5703X Gigabit Ethernet (rev 02)

Kernel is 2.6.20.4 with SMP - full kernel config can be found here:
http://pastebin.ca/411667

dmesg output can be found here:
http://pastebin.ca/411670

The server in question is an IBM x335 with an LSI Logic 53c1030 SCSI-RAID controller which has two Ultra320 SCSI disks in RAID-1 set up in hardware. Processor is Dual-XEON 2.4Ghz which gives 4 cores in 2.6 Linux kernels. Memory is 2Gb.

Since these are 2x 10kRPM 36GB Ultra320 SCSI disks, something is clearly wrong given the ultra-low read and write data.

Any insight on this would be more than greatly appreciated!

-Y1

syg00 · 03-27-2007, 08:52 PM

Try copying disk to disk - i.e. don't use /dev pseudo device, and see what your numbers look like then.

Edit: Ooops, missed the underlined bit above. I've seen similar numbers from using /dev/urandom to create test files. Copying them disk to disk gets "real" numbers.

Yalla-One · 03-29-2007, 10:10 AM

Hello, and thank you for responding.

I now did as you recommended and got even worse throughput:

Code:

[495] root@spirou:/root#dd if=testfile.out of=/tmp/test.ttt bs=128k count=1000
1000+0 records in
1000+0 records out
131072000 bytes (131 MB) copied, 86.5747 s, 1.5 MB/s

[496] root@spirou:/root#dd if=testfile.out of=/tmp/test.yalla bs=64k count=1000
1000+0 records in
1000+0 records out
65536000 bytes (66 MB) copied, 12.5143 s, 5.2 MB/s

However, doing the exact same copy once over gives 210MB/s, obviously since the controller (or OS) works in a cache rather than disk. Changing the filename brings me back to the cold, hard reality though...

Any further suggestions would be warmly welcome!

-Y1

syg00 · 03-29-2007, 08:12 PM

I have a quad zeon with hardware raid5, using Adaptec AIC7899 Ultra160 SCSI. Reboot shows my (initial) copy of 80-odd Meg at 8k blocksize getting (variously) 25-30 MB/s. Bumping it to 64k made no discernable difference.

Yalla-One · 03-29-2007, 08:14 PM

Could the problem perhaps be that I have compiled multiple SCSI-controllers into the kernel, so that they are competing for attention?

Have you compiled only the Adaptec-specific RAID driver, or have you also included the md/raid section?

syg00 · 03-29-2007, 09:04 PM

Couldn't say. In my case, this is an Arch build. i686 optimised, and it loads modules for the hardware it finds - guess it must have the AIC7999 or somesuch in the initrd as I don't recall having to rebuild it.
As it happens I have a test kernel I'm using for cpusets and resource container patch testing. It's all compiled in and as lean as I could sensibly make it.
Testing that shows similar numbers.