WD sata slow speed
 

Hello I have a 320 gb SATA 2 [wd3200aaks] on P4 2.66 with Mercury P4VM800M7 mobo
WHen I copy a movie in windows xp fat32 to fat 32, a programme called burst copy it shows speed of about 58 mbps [one partition to another]
when I copy same movie in pclinuxos 2007 (or debian4.0) i get speed of roughly 35 mbps, [XFS one partition to another XFS partition] why such a great difference in performance? and how can I speed up my harddisk under linux?
I have 2 gb ram
2 gb swap on same sata
no other harddisks
I have tested this number of times (and dont run any other software in between the copy process just to be sure of speed on both OS)

P.S does chosing syncronouse writting [all i/o/ synchronously] in XFS speed up the performance ? (because I think I had this feature unchecked (also if i check it will i get any performance penelty? and what its exact purpose btw, googled but didnt get something clearly )

If you are using Konqueror or Midnight Commander to measure the writing throughput, the bandwidth will worsen. I suggest use time with cp or mv to clock how fast or how long it will take. Then divide the type to the amount of bytes of the file. Any file system will have a bandwidth penalty when moving or copying data from one partition and then to the next partition. I suggest do the test with two hard drives on a different IDE channel or even better on a separate storage controller of the same model and brand.

On my notebook computer I get increase write and read performance using XFS compared to NTFS. Thanks to XFS buffering, I get more than 5 MB/sec compared to NTFS. My notebook computer has a Intel T7300 with 2 GB of DDR2.

My other computers also uses XFS and their throughput is high too. Again copying or moving data from partition to another partition cost performance.

I just play around with the format options.

The option async is faster than sync. The option sync will be slow because it tells the kernel and hard drive to flush cache to the medium. The async option will only flush the cache when the data has reached to a certain limit.

Try 'hdparm -Tt' to see what speed you get there.

Also look at 'hdparm -I' to see if something is fishy with the parameters for the drive.

hdparm -Tt :-

/dev/sda:
Timing cached reads: 826 MB in 2.00 seconds = 412.09 MB/sec
Timing buffered disk reads: 230 MB in 3.03 seconds = 76.00 MB/sec

--------------------------------------------------------------------
hdparm -I

/dev/sda:

ATA device, with non-removable media
Model Number: WDC WD3200AAKS-00VYA0
Serial Number: WD-WCARW1935518
Firmware Revision: 12.01B02
Transport: Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5
Standards:
Supported: 8 7 6 5
Likely used: 8
Configuration:
Logical max current
cylinders 16383 16383
heads 16 16
sectors/track 63 63
--
CHS current addressable sectors: 16514064
LBA user addressable sectors: 268435455
LBA48 user addressable sectors: 625142448
device size with M = 1024*1024: 305245 MBytes
device size with M = 1000*1000: 320072 MBytes (320 GB)
Capabilities:
LBA, IORDY(can be disabled)
Queue depth: 32
Standby timer values: spec'd by Standard, with device specific minimum
R/W multiple sector transfer: Max = 16 Current = 16
Recommended acoustic management value: 128, current value: 254
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
Enabled Supported:
* SMART feature set
Security Mode feature set
* Power Management feature set
* Write cache
* Look-ahead
* Host Protected Area feature set
* WRITE_BUFFER command
* READ_BUFFER command
* NOP cmd
* DOWNLOAD_MICROCODE
Power-Up In Standby feature set
* SET_FEATURES required to spinup after power up
SET_MAX security extension
Automatic Acoustic Management feature set
* 48-bit Address feature set
* Device Configuration Overlay feature set
* Mandatory FLUSH_CACHE
* FLUSH_CACHE_EXT
* SMART error logging
* SMART self-test
* General Purpose Logging feature set
* 64-bit World wide name
* Segmented DOWNLOAD_MICROCODE
* SATA-I signaling speed (1.5Gb/s)
* SATA-II signaling speed (3.0Gb/s)
* Native Command Queueing (NCQ)
* Host-initiated interface power management
* Phy event counters
DMA Setup Auto-Activate optimization
* Software settings preservation
* SMART Command Transport (SCT) feature set
* SCT Long Sector Access (AC1)
* SCT LBA Segment Access (AC2)
* SCT Error Recovery Control (AC3)
* SCT Features Control (AC4)
* SCT Data Tables (AC5)
unknown 206[12]
unknown 206[13]
Security:
Master password revision code = 65534
supported
not enabled
not locked
not frozen
not expired: security count
supported: enhanced erase
84min for SECURITY ERASE UNIT. 84min for ENHANCED SECURITY ERASE UNIT.
Checksum: correct

I tried using time, i get roughly same speed.
I am aware that If i copy data from one partition to another on same harddisk i face some performance penelty however I am tellling from my experience that in windows I was getting about 60 m/s from One partition partition to another on same harddisk ! how could it be while on linux i am getting virtuallly half of it

76 mb/s doesn't look too bad - I'd look into the acoustic management and the xfs options and into this.

Pls. don't get started on file systems and benchmarks there are enough threads flaming away already about it - just interesting benchmarks.

I am aware of performance penelties, but difference b/w speed of copying in windows (both partitions on same drive) is far higher (double) than xfs

hdparm -M /dev/sda

/dev/sda:
acoustic = 254 (128=quiet ... 254=fast)

i cant find any such helpful option for xfs

In copying between partitions on the same drive, the two big factors are:

1) How large are the chunks in which copying actually occurs. The OS may be buffering or caching to make the actual chunks larger than the chunks the copy program uses. In Linux, the dd program is available, which can copy in whatever chunk size you specify (subject to available memory).

2) How far apart are the partitions physically. Seek time is not linear with distance, but it does go up with distance. If I understood you correctly, four different partitions are involved: two for Windows and two for Linux. Maybe fdisk -l output would help us understand that aspect.

Sorry - got that all wrong.Thats what happens if you don't read the original post well.

First of all burstcopy uses RAM as buffer - wether that is a big factor depends on a lot of things.

Copy of oblivion.cab (1gb) ntfs->ntfs 42 sec with burstcopy
ntfs-> jfs 22 sec with Linux

In both cases that went from on disc to another with defragged M$ drives.

If you use partitions on the same disk or the drive is fragmented as hell I see how the buffer could speed up things a lot because it doesn't jump back and forth that much with an additional 64 mb or so buffer from ram.

Also disks are a lot slower the further the partition is to the inside of the platter because the relative speed is slower.

Sorry. I never heard of "burst copy" before reading the first post of this thread. I assume from its name it tries to do something intelligent regarding the buffer size. I have no idea what. But:

Any program copying from disk to disk uses ram as a buffer. How much ram it uses is usually a big factor.

But when going from one disk to another, the amount of ram isn't as big a factor as it would be from one partition to another on the same disk.

From one disk to another, it matters a lot whether something (OS or program) enables the overlap to start reading a second chunk before you're done reading the first. Given that overlap, even with unlimited ram, you wouldn't want the buffer to be a large fraction of the size of a large file.

With one disk and two partitions, you can't really start reading the second chunk before you're done writing the first. So with unlimited RAM, the fastest copy method would buffer the entire file in one chunk.

When copying a big file between partitions on one disk on a system with 2gig of ram, I hope the software is smart enough to use a lot more than 64mb for buffering. (I don't know it is smart enough. I just hope).

I didn't know that. Do you happen to have a URL for hard facts on that effect? (I almost never have a URL for the facts I mention in threads like this, but if you happen to, I'd appreciate it).

Long ago, drives used lower physical density on outer tracks so the temporal density wouldn't change very much. Obviously, electronics have improved faster than disk surface quality has improved, so consistent physical density now should be more important than consistent temporal density. So the effect you describe should exist. I just wonder how big it is.

I didn't know that. Do you happen to have a URL for hard facts on that effect? (I almost never have a URL for the facts I mention in threads like this, but if you happen to, I'd appreciate it).

Have a look here

That any program uses RAM as a disk I/O buffer is new to me.Thats what they put buffers on the disks for.

Softwarebuffering generally isn't needed except with bad dvd burners that run out of data to burn and produce coasters.

What gets read ahead and back and forth and so on depends to large part on the interface (scsi,sata,whatever).

The density is the same all over the platter - htat it gets slower is just simple physics.
You got always the same rpm's - on the outside it is faster than on the inside.It is a disk after all.

Disk /dev/sda: 320.0 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 685 5502231 b W95 FAT32
/dev/sda2 686 33168 260919697+ 5 Extended
/dev/sda5 686 3279 20836273+ b W95 FAT32
/dev/sda6 3280 5876 20860371 b W95 FAT32
/dev/sda7 5877 12268 51343708+ b W95 FAT32
/dev/sda8 12269 12536 2152678+ 82 Linux swap / Solaris
/dev/sda9 12537 14392 14908288+ 83 Linux
/dev/sda10 14393 15501 8908011 83 Linux
/dev/sda11 15502 15757 2056288+ 83 Linux
/dev/sda12 15758 33168 139853826 83 Linux


For windows I was copying from SDA1 to SDA7
For linux I was copying from SDA11 to SDA12

oh yeah and
hdparm -M /dev/sda

/dev/sda:
acoustic = 254 (128=quiet ... 254=fast)

in these tests , the pci ide bus mastering was enabled from bios.
I dual boot to windows/linux using GRUB (linux is pclinuxos 2007)

I have just noticed that If is Disable pci bus mastering it takes grub like 3 seconds FASTER to load in constast to when pci bus mastering is Enabled.Though the copying speed difference wasent noticeable in linux (wether bus mastering is enabled or disabled)
I thought it might help telling you this

hdparm -Q /dev/sda
-Q: bad/missing queue-depth value (0..32)


and what does this mean? why isint it showing the DMA status for the device?

I think crashmeister has explained the whole issue.

I downloaded the (windows based) disk test program he suggested and found the various disks I tested had nearly two to one transfer rate difference between the low address (outer edge) and high address (inner edge).

You compared a transfer between two locations near the outer edge for Windows against a transfer between two locations near the inner edge for Linux, so you would expect a difference slightly less than the full outer to inner difference, which in turn is slightly less than two to one.

So the difference you found might be entirely explained by the difference in raw transfer rate at outer vs. inner edges of the disk.

An average random seek betwen SDA1 and SDA7 would be 8730 "cylinders", while between SDA11 and SDA12 it would be 8833. But actual cylinders are not equal to the reported "cylinders" and vary for the same reason as the transfer rate between outer and inner. So a random seek between SDA11 and SDA12 is twice as far as a random seek between SDA1 and SDA7. So that MIGHT also be a factor.

But we have no idea how many actual seeks are involved nor how the actual seeks compare to the random seek (in both cases the destination size is large enough that the uncertainty in the seek distance is most of the total seek distance).

Comparing a simple file system like FAT to a Linux file system such as XFS which is complex and then basing the throughput between these two completely different file systems. Windows handles data IO differently than Linux. I suggest compared the results using both EXT2/3 and FAT. There is EXT IFS to provide EXT2 support in Windows 2000/XP.

EXT2/3, XFS, ReiserFS, have one thing in common. They can defrag while writing data to them. JFS does not defrag, so it gets fragmented over time. To stop the defraging, you can include noalign, notail, and a few others. Also using noatime can help. Before using these options, check if it is supported by the file system during mounting.

Disk benchmark programs in Windows does not include the file system during the test, so it goes by raw performance. Raw performance versus disk + file system performance are two different things. Real world testing is disk + file system performance.

Again basing write performance on transferring data from partition and then to the next partition just going to create more confusion. I suggest create a garbage file using /dev/urandom that is at least two times bigger than the capacity of RAM installed. Then include time with cat, dd, cpio the file to /dev/null. This will give you an estimate reading throughput. Sure you can use bonnie or bonnie++ to benchmark a drive, but that program is old and I think it is not accurate.

The utility hdparm is meant to be use with IDE/ATAPI devices. I suggest use sdparm for SCSI and SATA devices.

All the worries of throughput does not matter at all in Linux because programs barely comes close to the size of Windows programs.

Linux has a lot of reduntancies in place to provide

It is new to you. Have you been living in a cave far too long. Programs since the DOS ages uses RAM for buffering. DOS had simple buffers for software and data. When smartdrive was invented, it sped up the performance a lot. It helped high latency drives such as hard drives and optical drives to find data fast and transfer them as fast as memory. One problem, it uses RAM for this performance improvement. If I ran my 80386DX-40 that has 8 MB of RAM with out smartdrive, it ran slower.

The memory on drives are cache. The algorithm in today's hard drives and optical drives are very, very efficient, so they can find data very fast compared to their ancestor models that has similar hardware.

Linux, Windows, and Mac uses buffering and cache to increase performance of these high latency drives.

Device Boot Start End Blocks Id System
/dev/sda1 * 1 685 5502231 b W95 FAT32
/dev/sda2 686 33168 260919697+ 5 Extended
/dev/sda5 686 3279 20836273+ 83 Linux
/dev/sda6 3280 5876 20860371 83 Linux
/dev/sda7 5877 12268 51343708+ b W95 FAT32
/dev/sda8 12269 12536 2152678+ 82 Linux swap / Solaris
/dev/sda9 12537 14392 14908288+ 83 Linux
/dev/sda10 14393 15501 8908011 83 Linux
/dev/sda11 15502 15757 2056288+ 83 Linux
/dev/sda12 15758 33168 139853826 83 Linux

i formatted sda5 & sda6 to xfs but still the speed that I got was around 35mbps [johnsfine said it might be because of the distance in seeking...]
:(
i dont see any way of optimising my hd via sdparm/hdparm .

Maybe I should have asked the size of the test file.

You seem to be saying the performance is the same (with the same software) for a copy sda5 to sda6 and for sda11 to sda12. That doesn't make sense if the actual transfer or seek times matter.

If the file being copied is small enough, then other factors dominate and the actual transfer and seek times wouldn't have much impact on the measured data rate.

I don't know what "small enough" is. There are too many caching and buffering layers involved to make a reasonable estimate.

I should have mentioned the size too, the file size is 1023 mb (kida 1 GB , its a .vob , movie file).I copied a 644 mb (Debian 4.0 cd iso 1) and it took roughly 35 seconds from sda5 to sda6 (xfs partitions) [cant say benchmark windows since I dont have it on my pC]
I also 'tested' by copying a 1 1023 mb video file from sda11 to sda 5 and got roughly speed of 31 mbps
while I got a speed of roughly 35 mbps when copying from sda5 to sda 6 so[All partitions mentioned above are xfs]
It does prove there is tangible difference due to the cylinder seek issue but doesnt answer the massive difference between windows and linux's speed difference as now I have tested by converting those windows partitions to linux.

I would just like to ask [because I am feeling that may be the bci busmastering... thing i mentioned in my last post of page 1 Might have a hand in it]
Is there a way to check if linux is using/taking advantage of pci bus mastering option thats enabled from my BIOS?
[The real reason I got into this issue was because a friend,windows user asked me how fast my sata's copying speed is in windows , [we have satas of same model but my system only supports 1.5 gbps while his is advanced and supports 3 gbps SATA interface, so he was also surprised to see the copying speed of around 60 mbps in windows on my pc because thats what his system was providing with new interface and this HD supports new interface i.e sata 2... that copying was done with bus mastering enabled on my system]

People still do not understand the mechanics of hard drives. If the hard drive is rated for 150 MB per second (so called 1.5 Gb per second spec), it does not mean it will transfer data at that rate. Same goes with 300 MB per second (so called 3 Gb per second spec). Also the bus limits the throughput, so a SATA rated at 1.5 Gb per second or 150 MB per second will only get 133 MB per second is on a PCI bus. Same goes true for a SATA 3 Gb per second or 300 MB per second on a PCIe X1 which is only rated at 250 MB per second.

The explaining about throughput of a hard drive does not matter in Linux or in other operating systems because RAID can improve it. RAID can not improve latency, so find a hard drive or storage medium that has the lowest accessing time.

Please use the units below.

M = mega
b = bits
G = giga
B = bytes

One thing that can hurt xfs performance is file deletion. This can be greatly improved by increasing log buffer size. Create the xfs filesystem from the command line with
mkfs.xfs -l size=64M /dev/???
and mount with the options noatime, nodiratime, logbufs=8.

I was aware that 1.5 Gbps wont give me 1.5 giga BYTEs / second speed (because data rate is basically given in BITS).That wasent the issue , the issue is I am experienceing far slower speed in linux than windows.
I had some time so I formatted both sda5 and sda6 to the following filesystems and following are the speeds that I got :-
both ext3 =31 mbps
both ext2 = 34 mbps
both reiser [prolly 3, reiser 3, as mkfs.reiserfs -V shows :-
mkfs.reiserfs 3.6.19 (2003 www.namesys.com)
= speed on both reiser 30.9 mbps
both jfs = 34 mbps

OS= pclinuxos Minime
uname -a = Linux localhost 2.6.22.15.tex2 #1 SMP Mon Dec 17 23:18:44 CST 2007 i686 Intel(R) Pentium(R) 4 CPU 2.66GHz GNU/Linux

No reason to get personal.

Thank god we got you around to keep me in line.

cp and mv have their own sw buffering now?

Stupid me thought the kernel handled the buffer cache.

I looked at a number of benchmarks of WD 160GB drives. 30-35MB/s appears to be the normal transfer speed for data located at 110GB and above (while data on the outer edge can reach about 55MB/s).

Western digital makes atleast 3 different types of harddisk
within the range of PATA or SATA.
they are with 2mb cache, 8mb cache, and 16 mb cache (model no have aabs , aajs and aaks respectively in their names)the cache might influence the speed too.but ok
I will keep searching for the solution

It is 1.5 gigabits (Gb) not 1.5 gigabytes (GB).

Thanks bro..