hard disk drive performance

SaxiDawg · 05-31-2006, 09:28 PM

I put a 8006-2LP 3Ware card in a DL140 and attached two raptor 74G drives with Raid 1. Did a hdparm -t /dev/sda and got around 54MB/s. When I benched the same setup using windows (different controller, but same specs) I get around 65MB/s.

I noticed I can't tweak the performance at all with hdparm and didn't understand what I could do with sdparm. Is there any way to squeeze that extra 11MB/s out of this array with linux?

Also had another configuration with the newer version of the DL140 (G2) that I could only get 2.87MB/s with the ICH5 on-board with a Seagate 7200RPM. I could tweak it with HDPARM to about 5MB/s but I could never enable DMA. I tried the i2c option to enable the intel PIIX as well as PIIX support in the drivers section and I made sure the Intel Sata was enabled under scsi low level drivers. Anyone know what other kernel setting would stop me from enabling DMA and getting good performance with a ICH5? I have DMA enabled in the kernel.

WhatsHisName · 06-01-2006, 01:15 AM

The problem is more likely with hdparm than with the controller or drive setup.

Try a more convention bulk read:

Code:

# time dd if=/dev/sda of=/dev/null bs=4K count=2M
2097152+0 records in
2097152+0 records out

real    1m1.124s
user    0m0.531s
sys     0m9.113s

where “bs” times “count” (4KiB * 2MiB = 8GiB in the example) should be at least twice your installed memory. When using extremely large read sizes, be sure that the file, partition, disk, etc. is at least the desired size.

In this example, 8GiB * 1024MiB/GiB / 61.1sec = 134MiB/s

Bruce Hill · 06-01-2006, 04:17 PM

WhatsHisName,

I'm just curious as to what the significant difference is between "dd" writing to the disk, and hdparm's disk reads?

Code:

root@silas:~# time dd if=/dev/sda of=/dev/null bs=4K count=2M
2097152+0 records in
2097152+0 records out
8589934592 bytes (8.6 GB) copied, 127.862 seconds, 67.2 MB/s

real    2m7.897s
user    0m0.666s
sys     0m12.667s
root@silas:~# hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   1656 MB in  2.01 seconds = 825.63 MB/sec
 Timing buffered disk reads:  206 MB in  3.02 seconds =  68.12 MB/sec

Don't know why "dd" didn't do the calculations for you...

SaxiDawg,
Can you post your kernel .config file somewhere for us to look at?
And also /sbin/lspci?
The kernel is going to be over 1,500 lines, or I'd just say post it here.
Also post the manufacturer and model of you motherboard and hard drive.

Do you have a device for your RAID array such as /dev/md0 ?
That's what Pete M used for his software RAID0 in the long hdparm thread.

Electro · 06-01-2006, 07:21 PM

The utility hdparm gives you an estimate reading. It should never be use for comparison between two drives. Just use it to tweak the hard drive.

Seagate drives are known to have DMA problems in Linux. Same goes for Maxtor hard drives. Use Western Digital or Hitachi drives instead.

Bruce Hill · 06-02-2006, 02:47 AM

Electro posts FUD again. I have all four brands you mentioned, and all have DMA working fine in Slackware Linux. Perhaps your kernels/configurations are giving you the problems electro.

Electro · 06-02-2006, 04:07 AM

Quote:

Originally Posted by Chinaman

Electro posts FUD again. I have all four brands you mentioned, and all have DMA working fine in Slackware Linux. Perhaps your kernels/configurations are giving you the problems electro.

My kernel is running excellent. Really, Seagate and Maxtor SATA hard drives will fail to run in DMA mode. If you are stating that you are using only IDE drives not SATA, then yes Seagate and Maxtor hard drives are ok. I still stand by my recommended hard drive brands for the reliability with both PATA and SATA controllers.

You have posted FUD too. dd is a sector by sector copy. It will not calculate time and megabytes per second unless you rewrite the code. The kernel has nothing to do with how well the driver IDE or SATA driver works. It is up to the maintainer for the controller. If the maintainer did all what he or she could do to optimize the code, then the kernel have to be rewritten. However, if the kernel is set to preemptive, the performance will be slightly lower than being in server mode. My kernel is set to server mode and clock is set to 1000 Hz. This provides very good performance and the processor spends more time on programs such as mplayer, tvtime, xine, transcode, ffmpeg, and other processor intense programs.

WhatsHisName · 06-02-2006, 10:39 AM

Chinaman: Redhat didn’t implement the transfer calculation in dd (example output was from CentOS4.3), but it would be nice to have. I always enjoy the feature in Debian.

SaxiDawg · 06-02-2006, 04:56 PM

Quote:

Originally Posted by Chinaman

WhatsHisName,

I'm just curious as to what the significant difference is between "dd" writing to the disk, and hdparm's disk reads?

Code:

root@silas:~# time dd if=/dev/sda of=/dev/null bs=4K count=2M
2097152+0 records in
2097152+0 records out
8589934592 bytes (8.6 GB) copied, 127.862 seconds, 67.2 MB/s

real    2m7.897s
user    0m0.666s
sys     0m12.667s
root@silas:~# hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   1656 MB in  2.01 seconds = 825.63 MB/sec
 Timing buffered disk reads:  206 MB in  3.02 seconds =  68.12 MB/sec

Don't know why "dd" didn't do the calculations for you...

SaxiDawg,
Can you post your kernel .config file somewhere for us to look at?
And also /sbin/lspci?
The kernel is going to be over 1,500 lines, or I'd just say post it here.
Also post the manufacturer and model of you motherboard and hard drive.

Do you have a device for your RAID array such as /dev/md0 ?
That's what Pete M used for his software RAID0 in the long hdparm thread.

I am using the default kernel at this point now.

lspci:

0000:00:00.0 Host bridge: ServerWorks GCNB-LE Host Bridge (rev 32)
0000:00:00.1 Host bridge: ServerWorks GCNB-LE Host Bridge
0000:00:03.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
0000:00:0f.0 ISA bridge: ServerWorks CSB6 South Bridge (rev a0)
0000:00:0f.1 IDE interface: ServerWorks CSB6 RAID/IDE Controller (rev a0)
0000:00:0f.2 USB Controller: ServerWorks CSB6 OHCI USB Controller (rev 05)
0000:00:0f.3 Host bridge: ServerWorks GCLE-2 Host Bridge
0000:00:10.0 Host bridge: ServerWorks CIOB-E I/O Bridge with Gigabit Ethernet (rev 12)
0000:00:10.2 Host bridge: ServerWorks CIOB-E I/O Bridge with Gigabit Ethernet (rev 12)
0000:01:06.0 RAID bus controller: 3ware Inc 3ware 7000-series ATA-RAID (rev 01)
0000:02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 02)
0000:02:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 02)

Western Digital Raptor 74G & 3Ware 8006-2LP.

The machine I was having problems with 2.87MB was a Maxtor 7200RPM with a DL140 G2 using a ICH5 controller. I don't have the .config handy as I since blew away the machine.

Bruce Hill · 06-02-2006, 10:35 PM

Quote:

Originally Posted by Electro

My kernel is running excellent. Really, Seagate and Maxtor SATA hard drives will fail to run in DMA mode.

Perhaps this evidence will help:

Code:

root@silas:~# hdparm -I /dev/sda

/dev/sda:

ATA device, with non-removable media
        Model Number:       WDC WD740GD-00FLC0 < -- that's a Western Digial Raptor SATA 
        Serial Number:      WD-WMAKE2225466
        Firmware Revision:  33.08F33
Standards:
        Supported: 6 5 4 3
        Likely used: 6
Configuration:
        Logical         max     current
        cylinders       16383   16383
        heads           16      16
        sectors/track   63      63
        --
        CHS current addressable sectors:   16514064
        LBA    user addressable sectors:  145226112
        LBA48  user addressable sectors:  145226112
        device size with M = 1024*1024:       70911 MBytes
        device size with M = 1000*1000:       74355 MBytes (74 GB)
Capabilities:
        LBA, IORDY(can be disabled)
        bytes avail on r/w long: 74     Queue depth: 32
        Standby timer values: spec'd by Standard, with device specific minimum
        R/W multiple sector transfer: Max = 16  Current = 16
        Recommended acoustic management value: 128, current value: 254
        DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6 < -- DMA 
             Cycle time: min=120ns recommended=120ns
        PIO: pio0 pio1 pio2 pio3 pio4
             Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
        Enabled Supported:
           *    NOP cmd
           *    READ BUFFER cmd
           *    WRITE BUFFER cmd
           *    Host Protected Area feature set
           *    Release interrupt
           *    Look-ahead
           *    Write cache
           *    Power Management feature set
                Security Mode feature set
           *    SMART feature set
           *    FLUSH CACHE EXT command
           *    Mandatory FLUSH CACHE command
           *    Device Configuration Overlay feature set
           *    48-bit Address feature set
                Automatic Acoustic Management feature set
                SET MAX security extension
           *    SET FEATURES subcommand required to spinup after power up
                Power-Up In Standby feature set
           *    READ/WRITE DMA QUEUED
           *    DOWNLOAD MICROCODE cmd
           *    SMART self-test
           *    SMART error logging
Security:
                supported
        not     enabled
        not     locked
        not     frozen
        not     expired: security count
        not     supported: enhanced erase
Checksum: correct
root@silas:~# hdparm -I /dev/sdb

/dev/sdb:

ATA device, with non-removable media
        Model Number:       ST3160827AS < -- that's a Seagate Barracuda SATA 
        Serial Number:      4MT1CEQC
        Firmware Revision:  3.42
Standards:
        Used: ATA/ATAPI-6 T13 1410D revision 2
        Supported: 6 5 4 3
Configuration:
        Logical         max     current
        cylinders       16383   16383
        heads           16      16
        sectors/track   63      63
        --
        CHS current addressable sectors:   16514064
        LBA    user addressable sectors:  268435455
        LBA48  user addressable sectors:  312581808
        device size with M = 1024*1024:      152627 MBytes
        device size with M = 1000*1000:      160041 MBytes (160 GB)
Capabilities:
        LBA, IORDY(can be disabled)
        bytes avail on r/w long: 4      Queue depth: 32
        Standby timer values: spec'd by Standard, no device specific minimum
        R/W multiple sector transfer: Max = 16  Current = 16
        Recommended acoustic management value: 254, current value: 0
        DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6 < -- DMA 
             Cycle time: min=120ns recommended=120ns
        PIO: pio0 pio1 pio2 pio3 pio4
             Cycle time: no flow control=240ns  IORDY flow control=120ns
Commands/features:
        Enabled Supported:
           *    READ BUFFER cmd
           *    WRITE BUFFER cmd
           *    Host Protected Area feature set
           *    Look-ahead
           *    Write cache
           *    Power Management feature set
                Security Mode feature set
           *    SMART feature set
           *    FLUSH CACHE EXT command
           *    Mandatory FLUSH CACHE command
           *    Device Configuration Overlay feature set
           *    48-bit Address feature set
                SET MAX security extension
           *    DOWNLOAD MICROCODE cmd
           *    SMART self-test
           *    SMART error logging
Security:
        Master password revision code = 65534
                supported
        not     enabled
        not     locked
        not     frozen
        not     expired: security count
        not     supported: enhanced erase
Checksum: correct

root@silas:~# hdparm /dev/sda
/dev/sda:
 IO_support   =  0 (default 16-bit)
 readonly     =  0 (off)
 readahead    = 256 (on)
 geometry     = 9039/255/63, sectors = 145226112, start = 0

root@silas:~# hdparm /dev/sdb
/dev/sdb:
 IO_support   =  0 (default 16-bit)
 readonly     =  0 (off)
 readahead    = 256 (on)
 geometry     = 19457/255/63, sectors = 312581808, start = 0

root@silas:~# hdparm -tT /dev/sda
/dev/sda:
 Timing cached reads:   1576 MB in  2.00 seconds = 788.37 MB/sec
 Timing buffered disk reads:  204 MB in  3.00 seconds =  67.94 MB/sec

root@silas:~# hdparm -tT /dev/sdb
/dev/sdb:
 Timing cached reads:   1288 MB in  2.00 seconds = 644.21 MB/sec
 Timing buffered disk reads:  170 MB in  3.02 seconds =  56.21 MB/sec

Not the greatest times, but you don't get that without DMA.

Quote:

Originally Posted by Electro

You have posted FUD too. dd is a sector by sector copy. It will not calculate time and megabytes per second unless you rewrite the code.

Code:

root@silas:~# dd --version
dd (coreutils) 5.94
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.

Written by Paul Rubin, David MacKenzie, and Stuart Kemp.

and from "man dd"
       status=noxfer
              suppress transfer statistics

So unless Pat has 'rewritten the code' ...

Quote:

Originally Posted by Electro

The kernel has nothing to do with how well the driver IDE or SATA driver works. It is up to the maintainer for the controller.

You may find it interesting that the controller uses the driver in the kernel.

Quote:

Originally Posted by Electro

However, if the kernel is set to preemptive, the performance will be slightly lower than being in server mode.

So you've chosen:
No Forced Preemption (Server) (PREEMPT_NONE)

This is the traditional Linux preemption model, geared towards
throughput. It will still provide good latencies most of the
time, but there are no guarantees and occasional longer delays
are possible.

Select this option if you are building a kernel for a server or
scientific/computation system, or if you want to maximize the
raw processing power of the kernel, irrespective of scheduling
latencies.

I've been using voluntary kernel preemption for my desktops.
Perhaps I should try with different preemption models...

Bruce Hill · 06-02-2006, 10:58 PM

SaxiDawg,

I'm trying to help, while some people only care to post where others are wrong --- with a history of not always adding anything helpful for the OP.

Can you now post some information about your 3ware 7000-series ATA-RAID controller? You say "I am using the default kernel at this point now." Which one is that? Which distro and which kernel?

For instance, for Slackware-10.2 you have the following choices:

Quote:

For IDE: bare.i, bareacpi.i, ataraid.i, lowmem.i, old_cd.i, pportide.i, sata.i
Then, the SCSI kernels (these also support IDE):adaptec.s, ibmmca.s, jfs.s, raid.s, scsi.s, scsi2.s, scsi3.s, speakup.s, xfs.s
There is also a 2.6.13 install kernel that might be useful if your machine has new hardware such as a Serial-ATA controller that isn't working with the 2.4 kernel: test26.s

It would help to know which kernel, or maybe even see your present .config file. There are several different variables to set.

Jeff Garzik, the maintainer responsible for developing the Linux SATA drivers, says:

Quote:

Broadcom/ServerWorks/Apple
Summary: Huge per-device queues, full SATA control including hotplug and PM for the "Frodo4" and "Frodo8" boards. Apple K2 SATA, which also uses this chipset, has all the feature of Frodo4/8 save the host DMA queueing feature ("QDMA"). QDMA supports legacy TCQ, but not NCQ.

Update: Newer cards support NCQ via QDMA.

libata driver status: Production, but no QDMA support yet.

and

Quote:

3ware SATA RAID
Not suited for libata architecture.

Separate open source SCSI driver for 3-ware hardware exists.

But your "lspci" output says "3ware 7000-series ATA-RAID".

Is that NOT for SATA, or just the way the pci.ids file outputs it?

It may help to read more of Jeff's pages, including -> http://linux-ata.org/software-status...recent_updates

Electro · 06-04-2006, 05:27 PM

Chinaman, that is untrue I give very good information while you beat around the bush. I gave that hdparm is utility to optimize the hard drive but it is not a good utility for benchmarking. The hard drive and the filesystem have to be benchmark all together. It seems people here skim posts to only pick what they think they need. Since the 3ware card is working, I suggest that you do not fix it. The PCI ID that states it as a 7000 series is nothing to worry about. The card is similar to the 8000 series. The problem with the Maxtor or Seagate SATA hard drive and the SATA Intel controller is probably the software is blacklisting the drive so it will not enable DMA because there is a high possibility of data corruption. I recommend SaxiDawg to upgrade his or her kernel to the latest stable version or use the latest unstable mm or git kernels.

I search this forum and you only have a VIA SATA controller. I have looked through the kernel source code. There is no lines of code that includes the quirk.c like the Intel SATA controller. You stating that DMA is working perfectly is ok, but SaxiDawg is using an Intel SATA controller which is whole different setup. Each manufacture has their own ways to handle hard drives. It seems that Intel is using black art to provide high performance which the kernel maintainers do not know about.

I suggest forcing a certain DMA mode because of some flaw in the Intel SATA controller.

SaxiDawg, start with `hdparm -d 1 -X 65 -k 1 -K 1 /dev/sda` and increase the -X number, but do not go above 68 which is UDMA-4. You may need to include the -c 3 option which is 32-bit with sync. If DMA gets enabled but you are experiencing data corruption, you have to go back to PIO settings or buy a more reliable SATA controller.