turning on dma and write back cache on SATA?

niverson · 11-21-2005, 11:02 PM

I have read most of the post about turning on DMA on IDE using /sbin/hdparm -d 1 /dev/hdX.
I have a LSI megaRAID SATA 150-4 with 4 Seagate SATA drives in RAID 5. I know that losing power to the system can cause lost data when write back is enabled. The RAID is mostly used for read operations with the only writes being recording of tv shows by mythtv. I have a separate IDE drive for the OS.

Does anyone know how to turn on writeback cache and DMA on a LSI megaRAID sata 150-4 in Linux(fedora core 4)?

[root@mythtv scsi]# /sbin/hdparm -d 1 /dev/sda
/dev/sda:
setting using_dma to 1 (on)
HDIO_SET_DMA failed: Invalid argument

[root@mythtv mythtv]# /sbin/hdparm /dev/sda
/dev/sda:
readonly = 0 (off)
readahead = 256 (on)
geometry = 58370/255/63, sectors = 480110444544, start = 0

From /var/log/dmesg
scsi[0]: scanning scsi channel 1 [virtual] for logical drives
Vendor: MegaRAID Model: LD 0 RAID5 457G Rev: 713N
Type: Direct-Access ANSI SCSI revision: 02
SCSI device sda: 937715712 512-byte hdwr sectors (480110 MB)
sda: asking for cache data failed
sda: assuming drive cache: write through
SCSI device sda: 937715712 512-byte hdwr sectors (480110 MB)
sda: asking for cache data failed
sda: assuming drive cache: write through
sda: sda1
Attached scsi disk sda at scsi0, channel 1, id 0, lun 0

[root@mythtv scsi]# uname -r
2.6.13-1.1526_FC4smp

Thanks for your time,
Nate

Bruce Hill · 11-22-2005, 10:13 AM

Nate,

I'm in the same boat, so I'll post some of my stats and the little I can add.

Code:

bruce@silas:~$ /usr/sbin/hdparm /dev/sda /dev/sdb

/dev/sda:
 IO_support   =  0 (default 16-bit)
 readonly     =  0 (off)
 readahead    = 256 (on)
 geometry     = 19457/255/63, sectors = 312581808, start = 0

/dev/sdb:
 IO_support   =256 (???)
 readonly     =  0 (off)
 readahead    = 256 (on)
 geometry     = 30401/255/63, sectors = 488397168, start = 0

Code:

bruce@silas:~$ /usr/sbin/hdparm -iI /dev/sda /dev/sdb

/dev/sda:
 HDIO_GET_IDENTITY failed: Inappropriate ioctl for device
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device

/dev/sdb:
 HDIO_GET_IDENTITY failed: Inappropriate ioctl for device
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device

Code:

bruce@silas:~$ /usr/sbin/hdparm -d 1 /dev/sda /dev/sdb

/dev/sda:
 setting using_dma to 1 (on)
 HDIO_SET_DMA failed: Inappropriate ioctl for device

/dev/sdb:
 setting using_dma to 1 (on)
 HDIO_SET_DMA failed: Inappropriate ioctl for device

Code:

root@silas:~# dmesg | less
libata version 1.12 loaded.
sata_via version 1.1
ACPI: PCI Interrupt 0000:00:0f.0[B] -> GSI 20 (level, low) -> IRQ 16
PCI: Via IRQ fixup for 0000:00:0f.0, from 10 to 0
sata_via(0000:00:0f.0): routed to hard irq line 0
ata1: SATA max UDMA/133 cmd 0xE800 ctl 0xE402 bmdma 0xD400 irq 16
ata2: SATA max UDMA/133 cmd 0xE000 ctl 0xD802 bmdma 0xD408 irq 16
ata1: dev 0 cfg 49:2f00 82:346b 83:7d01 84:4003 85:3469 86:3c01 87:4003 88:407f
ata1: dev 0 ATA, max UDMA/133, 312581808 sectors: lba48
ata1: dev 0 configured for UDMA/133
scsi0 : sata_via
ata2: dev 0 cfg 49:2f00 82:74eb 83:7fea 84:4023 85:74e8 86:3c02 87:4023 88:203f
ata2: dev 0 ATA, max UDMA/100, 488397168 sectors: lba48
ata2: dev 0 configured for UDMA/100
scsi1 : sata_via
  Vendor: ATA       Model: ST3160827AS       Rev: 3.42
  Type:   Direct-Access                      ANSI SCSI revision: 05
  Vendor: ATA       Model: HDS722525VLSA80   Rev: V36O
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sda: 312581808 512-byte hdwr sectors (160042 MB)
SCSI device sda: drive cache: write back
SCSI device sda: 312581808 512-byte hdwr sectors (160042 MB)
SCSI device sda: drive cache: write back
 sda: sda1 sda2 sda3
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
SCSI device sdb: 488397168 512-byte hdwr sectors (250059 MB)
SCSI device sdb: drive cache: write back
SCSI device sdb: 488397168 512-byte hdwr sectors (250059 MB)
SCSI device sdb: drive cache: write back
 sdb: sdb1
Attached scsi disk sdb at scsi1, channel 0, id 0, lun 0
Attached scsi generic sg0 at scsi0, channel 0, id 0, lun 0,  type 0
Attached scsi generic sg1 at scsi1, channel 0, id 0, lun 0,  type 0

According to the hdparm manual page:

Quote:

DESCRIPTION
hdparm provides a command line interface to various hard disk ioctls supported by
the stock Linux ATA/IDE device driver subsystem. Some options may work correctly
only with the latest kernels. For best results, compile hdparm with the include
files from the latest kernel source code

The tool for SCSI disks, and some Serial ATA disks, might be sdparm.
I haven't tried it yet, though it's on my list of things to do...

According to that sdparm link:

Quote:

SATA
SATA disks and S-ATAPI devices (e.g. a DVD drive) may appear as either IDE (e.g. /dev/hda) or SCSI (e.g. /dev/scd0) devices in linux. In some cases this may depend on which hardware drivers have been chosen in the kernel configuration. For SATA disks that appear as IDE devices in linux, the hdparm utility is the appropriate tool. S-ATAPI devices, regardless of how the appear in linux, may find sdparm useful because there is a high probability that they use a SCSI command set (e.g. MMC for DVD drives).

SATA disks that appear in linux as SCSI devices (e.g. /dev/sda) will have a SCSI ATA Translation layer (SATL) between the disk and the host computer. SCSI ATA Translation (SAT) is an emerging standard at t10.org . The SATL may be within the linux kernel (e.g. libata) or in an external enclosure (e.g. Fibre channel has FC to SATA translation chips that include a SATL). [One reason for doing this is that the SCSI protocols scale better in multi-initiator, multi-port environments (e.g. SATA NCQ assumes one host).] Recent versions of hdparm may work on device attributes; since it tries to use the ATA PASS THROUGH SCSI commands defined by SAT. Those commands have been added to libata in lk 2.6.15-rc1 and thus should be available when lk 2.6.15 is released.

It's my understanding that at the present time, there are only two motherboard
manufacturers with an AHCI board with full NCQ support, full SATA control.

What does your performance look like? These are two drives in this PC. I
have a couple more in China, one of which doesn't perform well at all.

Code:

root@silas:~# /usr/sbin/hdparm -tT /dev/sda /dev/sdb

/dev/sda:
 Timing cached reads:   1508 MB in  2.01 seconds = 751.86 MB/sec
HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device
 Timing buffered disk reads:  170 MB in  3.01 seconds =  56.47 MB/sec
HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device

/dev/sdb:
 Timing cached reads:   1712 MB in  2.00 seconds = 855.70 MB/sec
HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device
 Timing buffered disk reads:  176 MB in  3.03 seconds =  58.08 MB/sec
HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device

deepclutch · 11-22-2005, 03:45 PM

buddys,got same problem with seagate SATA will try sdparm and post results..

niverson · 11-22-2005, 07:50 PM

I don't think that sdparm is going to help us. Sata isn't quite SCSI and this mainly looks like it is used to changes the SCSI mode pages. I also have a request into LSI tech support on how to enable writeback and DMA --- I will post anything I hear from LSI.

[root@mythtv tmp]# sdparm -v -a /dev/sda
/dev/sda: MegaRAID LD 0 RAID5 457G 713N
>> Read write error recovery mode page [0x1] not supported
>> Format (SBC) mode page [0x3] not supported
>> Rigid disk (SBC) mode page [0x4] not supported
>> Verify error recovery (SBC) mode page [0x7] not supported
>> Caching (SBC) mode page [0x8] not supported
>> Control mode page [0xa] not supported
>> Control extension mode subpage [0xa,0x1] not supported
>> XOR control (SBC) mode page [0x10] not supported
>> Protocol specific logical unit mode page [0x18] not supported
>> Protocol specific port mode page [0x19] not supported
>> Power condition mode page [0x1a] not supported

Bruce Hill · 11-22-2005, 07:55 PM

I don't exactly know how Fedora Core works. If you aren't confident compiling
your own kernel, can you get a 2.6.15-rc1 built for Fedora that you can install?

I'll try one on this box, but it might be next week.

niverson · 11-22-2005, 07:58 PM

[root@mythtv tmp]# /sbin/hdparm -tT /dev/sda

/dev/sda:
Timing cached reads: 808 MB in 2.00 seconds = 403.17 MB/sec
Timing buffered disk reads: 68 MB in 3.00 seconds = 22.67 MB/sec
[root@mythtv tmp]#

I also have a query into LSI about using FLEXraid to tune the storage(change striping and other parameter) without corrupting data. I'm not sure it is possible, but it can't hurt to ask.

niverson · 11-22-2005, 08:04 PM

What does the new kernel give us? I'm very comfortable with compiling kernels, but I have mythtv on my system and it requires many other driver and applications dependency that could be very painful to manually solve.

Bruce Hill · 11-22-2005, 08:39 PM

Look what I quoted from the sdparm link in post #2.

Bruce Hill · 11-22-2005, 08:54 PM

Quote:

Originally posted by niverson
[root@mythtv tmp]# /sbin/hdparm -tT /dev/sda

/dev/sda:
Timing cached reads: 808 MB in 2.00 seconds = 403.17 MB/sec
Timing buffered disk reads: 68 MB in 3.00 seconds = 22.67 MB/sec
[root@mythtv tmp]#

I also have a query into LSI about using FLEXraid to tune the storage(change striping and other parameter) without corrupting data. I'm not sure it is possible, but it can't hurt to ask.

That's really poor performance. All of my ATA drives outperform that.
Which controller does your motherboard use? There have been reports
of problems with Sil controllers and Seagate drives.

This board uses a VIA VT6420 SATA RAID Controller.

You might want to do some thorough Google searches, and also
some LKML (Linux Kernel Mailing List) mailing list searches.

Here's one thread > http://www.ussg.iu.edu/hypermail/lin...12.0/0120.html

Also read the latest SATA / SATA RAID information from Jeff Garzik,
who is the kernel developer of libata.

niverson · 11-23-2005, 07:58 AM

I think I had backgroud IO going in the previous read performance posting. I got a response back from LSI.

There are 2 ways to enable write back cache. You can do it in the
controller's bios (Control M) or in Megamgr. The process is exactly the
same since megamgr mimics control M.
1. Go into control M or Megamgr.
2. Go to objects menu
3. Go to logical drive menu.
4. Select view/update parameters.
5. Scroll down to write cache option and press enter to change the setting.
Escape and exit.

Used these settings from the LSI knowledge base:
Configure the array for 64KB Stripe Size, Write Back, Normal Read, and Direct IO.

[root@mythtv mythtv]# /sbin/hdparm /dev/sda

/dev/sda:
readonly = 0 (off)
readahead = 256 (on)
geometry = 58370/255/63, sectors = 480110444544, start = 0
[root@mythtv mythtv]# /sbin/hdparm -tT /dev/sda

/dev/sda:
Timing cached reads: 800 MB in 2.00 seconds = 399.98 MB/sec
Timing buffered disk reads: 186 MB in 3.02 seconds = 61.59 MB/sec
[root@mythtv mythtv]#

Kinobi · 12-07-2005, 08:27 PM

I have the LSI megaraid SATA 150-6 running on ubuntu breezy, and these are my settings:
Write Policy= WRTHRU
READ Policy = NORMAL
Cache Policy = DirectIO

If I change any of those, after long sustained writes I will get I/O Errors and have to reboot before the drive responds again. With my current settings long sustained writes will go for a while then eventually stop for short periods, then continue... Very strange for directIO etc. The peformance is disapointing to say the least.

Any Ideas? Oh on boot I get things like :
asking for cache data failed ...

thanks.

niverson · 12-20-2005, 10:13 AM

At boot, I also see the request for cache policy failed message. I'm wondering if write-back caching is only available if the backup battery is connected. As for the crashes, I think you should connect LSI tech support through their web site.

Nate

niverson · 02-12-2006, 11:32 PM

After messing with the megaRAID SATA150-4 long enough I finally found why the write performance was so bad. I had write-back enabled in the BIOS and the cache flush interval was set to 2 seconds. These setting were not the problem.

In the BIOS go to
OBJECTS->Physical Drive->Drive Information and enable the write cache on each drive.

I have 3 400GB WD raid edition drives(use to have 4 160GB seagate drives) set up in RAID 5, only one of the physical drives had the write cache enabled. Once the write cache was enabled on all drives, write throughput almost doubled.

./dt of=/mnt/data/text bs=64k capacity=1g

Write Statistics:
Total records processed: 16384 @ 65536 bytes/record (64.000 Kbytes)
Total bytes transferred: 1073741824 (1048576.000 Kbytes, 1024.000 Mbytes)
Average transfer rates: 36398028 bytes/sec, 35544.949 Kbytes/sec
Number I/O's per second: 555.390
Total passes completed: 0/1
Total errors detected: 0/1
Total elapsed time: 00m29.50s
Total system time: 00m04.93s
Total user time: 00m06.26s

Read Statistics:
Total records processed: 16384 @ 65536 bytes/record (64.000 Kbytes)
Total bytes transferred: 1073741824 (1048576.000 Kbytes, 1024.000 Mbytes)
Average transfer rates: 56811737 bytes/sec, 55480.212 Kbytes/sec
Number I/O's per second: 866.878
Total passes completed: 1/1
Total errors detected: 0/1
Total elapsed time: 00m18.90s
Total system time: 00m02.69s
Total user time: 00m15.75s

Total Statistics:
Output device/file name: /mnt/data/text (device type=regular)
Type of I/O's performed: sequential (forward)
Data pattern read/written: 0x39c39c39
Total records processed: 32768 @ 65536 bytes/record (64.000 Kbytes)
Total bytes transferred: 2147483648 (2097152.000 Kbytes, 2048.000 Mbytes)
Average transfer rates: 44369497 bytes/sec, 43329.587 Kbytes/sec
Number I/O's per second: 677.025
Total passes completed: 1/1
Total errors detected: 0/1
Total elapsed time: 00m48.40s
Total system time: 00m07.62s
Total user time: 00m22.01s
Starting time: Sun Feb 12 22:24:29 2006
Ending time: Sun Feb 12 22:25:18 2006

The LSI tech support never even suggested this might be an issue. Now I want NCQ on this card. Does anyone know if NCQ is done in firmware or the driver?

Kinobi · 02-14-2006, 11:10 AM

I don't understand, when I go into the bios, OBJECTS->Physical Drive->Drive Information has no editable fields, it just says press any key to continue.

nevelis · 02-14-2006, 01:36 PM

Hey guys,

The only thing I can think of is that your kernel wasn't compiled with a driver for your IDE/SCSI chipset. This means that you won't be able to use DMA on them.

Have you tried compiling your own/replacing your kernel?

Cheers,
Aaron