LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 04-25-2010, 11:42 PM   #1
koooee
LQ Newbie
 
Registered: Apr 2010
Distribution: slackware 13.0
Posts: 7

Rep: Reputation: 1
data transfer rate decreases with dd command?


anyone know why this would happen? I ran the following dd command:

Code:
dd if=/dev/sdb1 of=hard/old_dell_desk_part1.img
a USB external drive was mounted to the 'hard' directory with the standard mount command. it is a ntfs-3g filesystem. it is a mechanical drive.

same behavior with solid state memory (flash, and solid state hard drives)


3693492+0 records in
3693492+0 records out
1891067904 bytes (1.9 GB) copied, 274.562 s, 6.9 MB/s
7083789+0 records in
7083789+0 records out
3626899968 bytes (3.6 GB) copied, 552.678 s, 6.6 MB/s
7349092+0 records in
7349092+0 records out
3762735104 bytes (3.8 GB) copied, 576.682 s, 6.5 MB/s
9762943+0 records in
9762943+0 records out
4998626816 bytes (5.0 GB) copied, 782.539 s, 6.4 MB/s
10257790+0 records in
10257790+0 records out
5251988480 bytes (5.3 GB) copied, 826.804 s, 6.4 MB/s
17000321+0 records in
17000321+0 records out
8704164352 bytes (8.7 GB) copied, 1494.76 s, 5.8 MB/s
37069401+0 records in
37069401+0 records out
18979533312 bytes (19 GB) copied, 3982.31 s, 4.8 MB/s
79391274+0 records in
79391274+0 records out
40648332288 bytes (41 GB) copied, 14068.8 s, 2.9 MB/s
144385881+0 records in
144385881+0 records out
73925571072 bytes (74 GB) copied, 47609.8 s, 1.6 MB/s
194700873+0 records in
194700873+0 records out
99686846976 bytes (100 GB) copied, 94929.3 s, 1.1 MB/s
201734145+0 records in
201734145+0 records out
103287882240 bytes (103 GB) copied, 103429 s, 999 kB/s
202499967+0 records in
202499967+0 records out
103679983104 bytes (104 GB) copied, 104408 s, 993 kB/s
204796557+0 records in
204796557+0 records out
104855837184 bytes (105 GB) copied, 107298 s, 977 kB/s
 
Old 04-26-2010, 12:31 AM   #2
pixellany
LQ Veteran
 
Registered: Nov 2005
Location: Annapolis, MD
Distribution: Arch/XFCE
Posts: 17,802

Rep: Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728
All that in response to one dd command??!!

What is the size of the partition being copied?

How much free space is on the target drive?

Quote:
same behavior with solid state memory (flash, and solid state hard drives)
If the source partition is ~100GB as implied by the command output, then what kind of flash drive were you copying to? More generally, are you saying that the behavior is identical for different target drives?
 
Old 04-26-2010, 06:48 AM   #3
koooee
LQ Newbie
 
Registered: Apr 2010
Distribution: slackware 13.0
Posts: 7

Original Poster
Rep: Reputation: 1
Quote:
Originally Posted by pixellany View Post
All that in response to one dd command??!!
Not sure what this is suppose to mean....but yes, so you can see the transfer rate decline in a fairly rapid linear fashion.

Quote:
Originally Posted by pixellany View Post
What is the size of the partition being copied?
105GB
Quote:
Originally Posted by pixellany View Post
How much free space is on the target drive?
300GB

Quote:
Originally Posted by pixellany View Post
If the source partition is ~100GB as implied by the command output, then what kind of flash drive were you copying to? More generally, are you saying that the behavior is identical for different target drives?
Yes, this was meant in a more general sense, I tried various types of storage devices (as source and target) and had a similar linear decline. At this point the size, and type of device is irrelevant since the behavior is the same for all.
 
Old 04-26-2010, 08:56 AM   #4
pixellany
LQ Veteran
 
Registered: Nov 2005
Location: Annapolis, MD
Distribution: Arch/XFCE
Posts: 17,802

Rep: Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728
Quote:
Quote:
Originally Posted by pixellany View Post
All that in response to one dd command??!!
Not sure what this is suppose to mean....but yes, so you can see the transfer rate decline in a fairly rapid linear fashion.
I simply asked if dd did all that by itself....I have never seen such a thing. Anytime I ever used dd, it printed the statistics once at the end of the job.

For the moment, I can't think of what might be causing this. Have you tried different block sizes?
 
Old 04-26-2010, 10:23 AM   #5
japa-fi
LQ Newbie
 
Registered: Feb 2010
Location: Finland
Distribution: Fedora 11
Posts: 5

Rep: Reputation: 0
Quote:
Originally Posted by koooee View Post
anyone know why this would happen? I ran the following dd command:
The destination hard drive has a cache (they all have cache, some bigger, some smaller). At the start, notable part of data has been written to cache, which is fast. As more and more data is copied, then the effect of the cache gets smaller and the write speed you are seeing is getting closer to actual write speed of the magnetic media on HD.

by default, dd uses 512 byte writes, which has really poor performance on some drives. if the amount of data to be copied is multiple of 2kb (2048 bytes), try using bs=2k parameter with dd. That will increase speed noticeably. Even better if you can use 4k (multiple of 4096) or 8k (8192)
 
1 members found this post helpful.
Old 04-26-2010, 11:16 AM   #6
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,524
Blog Entries: 27

Rep: Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175
Quote:
Originally Posted by japa-fi View Post
The destination hard drive has a cache (they all have cache, some bigger, some smaller). At the start, notable part of data has been written to cache, which is fast. As more and more data is copied, then the effect of the cache gets smaller and the write speed you are seeing is getting closer to actual write speed of the magnetic media on HD.
That's correct but HDD drive caches are a few tens of MB so the phenomenon you describe would only be effective for the first part of the reported transfer, and would be invisible for the data quantities reported.

AFAIK HDDs read/write faster at the outer edge of the drive because the spin speed is constant and so the linear distance travelled per second is greater when the head is further out. This could be causing (some of?) the effect but doesn't make sense when writing only ~30% of the HDD capacity, assuming writing starts at the outside and moves inward.

It may be instructive to explore what happens when using dd's seek option to start recording, say, 80 GB from the start.

Another possibility is that the limiting factor is not the output device but the input device, especially if this unusual behaviour is seen on multiple output devices. This could be tested by using /dev/null as the output device.

Was there any other activity on the input and output devices when the tests were run?

Last edited by catkin; 04-26-2010 at 11:21 AM. Reason: typo-anarchy
 
Old 04-26-2010, 01:23 PM   #7
koooee
LQ Newbie
 
Registered: Apr 2010
Distribution: slackware 13.0
Posts: 7

Original Poster
Rep: Reputation: 1
Quote:
Originally Posted by catkin View Post
Was there any other activity on the input and output devices when the tests were run?
Nope, I made sure to do nothing else on either IO device...

I am aware of drive caches, but my only reason for not exploring that as a solution was because we should have seen the data transfer rate level off at some point, as catkin implied, but it continues to decline in a linear fashion??

Quote:
Originally Posted by pixellany;
I have never seen such a thing. Anytime I ever used dd, it printed the statistics once at the end of the job
if you send the currently running dd process the USR1 signal it will print the current statistics
Code:
kill -s USR1 PID_of_dd
 
1 members found this post helpful.
Old 04-26-2010, 02:01 PM   #8
GazL
Senior Member
 
Registered: May 2008
Posts: 3,312

Rep: Reputation: Disabled
Because linux has a filesystem cache/buffers in system memory in addition to the much smaller write buffers on the device itself it's very likely that the first few GB or so (depending on the amount of free ram your system has) is going to get an exceptionally high transfer rate, but once those buffers are saturated the actual rate will drop to the sustained rate of the device. However, what you need to remember is that the transfer rate that dd reports is the averaged transfer rate from the start of the dd operation which will be artificially high due to the head-start it got from the buffering and will gradually trend towards the actual sustained transfer rate. This may explain what you're seeing.

If you want to be certain, then instead of watching the averaged rate reported by dd, try running something like iostat that can report the actual rate over real time intervals.


Anyway, that's my guess at what's happening. I could be wrong.
 
Old 04-26-2010, 02:19 PM   #9
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,524
Blog Entries: 27

Rep: Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175Reputation: 1175
Quote:
Originally Posted by GazL View Post
Because linux has a filesystem cache/buffers in system memory in addition to the much smaller write buffers on the device itself it's very likely that the first few GB or so (depending on the amount of free ram your system has) is going to get an exceptionally high transfer rate, but once those buffers are saturated the actual rate will drop to the sustained rate of the device. However, what you need to remember is that the transfer rate that dd reports is the averaged transfer rate from the start of the dd operation which will be artificially high due to the head-start it got from the buffering and will gradually trend towards the actual sustained transfer rate. This may explain what you're seeing.

If you want to be certain, then instead of watching the averaged rate reported by dd, try running something like iostat that can report the actual rate over real time intervals.


Anyway, that's my guess at what's happening. I could be wrong.
Are the filesystem cache/buffers likely to hold the data that dd reads, though? I'm no expert but I assume they only hold already-read data plus maybe some read-ahead. Even if the dd input had already been done, wouldn't the cache/buffers hold the last data read, up to their capacity? Unless the system has ~100 GB memory I'm dubious that the cache/buffers are significant.

Regards dd printing an averaged rate that's useful to know but it means the real rate slowed down even more than the figures reported.

The initial rate looks like a reasonable maximum for current storage devices.

I wonder if some device in the chain may be overheating and de-loading itself but AFAIK only CPUs do that, not HDDs or bus controllers, and the reported transfer rate shouldn't make a current model CPU break sweat unless its cooling system is defective.
 
Old 04-26-2010, 03:40 PM   #10
GazL
Senior Member
 
Registered: May 2008
Posts: 3,312

Rep: Reputation: Disabled
@catkin,

I'm not really sure about the internals of the vfs cache/buffer, I was thinking more of the effect of write buffering rather than any effect the read cache may have had on the overall average, but looking a little closer at the raw data between 5.0GB and 74GB we get:

4998626816 bytes (5.0 GB) copied, 782.539 s, 6.4 MB/s
73925571072 bytes (74 GB) copied, 47609.8 s, 1.6 MB/s

So the average rate between the two observations:

73925571072 - 4998626816 =
68926944256 (bytes)
47609 - 782 =
46827 (seconds)
68926944256 / 46827
1471948 (bytes/sec) = 1471KB/sec


Similarly between the 41GB and 100GB points, it's:

99686846976 - 40648332288
59038514688
94929 - 14068
80861
59038514688 / 80861
730123 = 730KB/sec

So, it's certainly not a consistent rate and any buffering should have been well out of the way after that initial 5GB, so that shouldn't factor either.

Really not sure what's going on now.
 
Old 04-26-2010, 05:33 PM   #11
koooee
LQ Newbie
 
Registered: Apr 2010
Distribution: slackware 13.0
Posts: 7

Original Poster
Rep: Reputation: 1
I'm not sure if the cache would have anything to do with it at this point?? and here is why:

there are various cache algorithms out there. Some common ones are MRU, and LRU, (Most/Least Recently Used) google search of "cache algorithms" should get you there fine. I have implemented a MRU cache, basically, any new data that hasn't been seen before will overwrite the piece of data with the lowest frequency/visits if you will. In this case where you are making an exact copy of a hard drive there really are no addresses that are visited more than once, effectively ruling out the vary purpose of the cache. Again, I am unsure, this is my best guess....I am confused.

Perhaps the hard drive gets tired, maybe I should give it some water?
 
Old 04-26-2010, 06:07 PM   #12
pixellany
LQ Veteran
 
Registered: Nov 2005
Location: Annapolis, MD
Distribution: Arch/XFCE
Posts: 17,802

Rep: Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728Reputation: 728
I'm still stuck on the question of how to get dd to print out multiple progress reports----did I miss it?

I have NEVER seen dd give anything except the total time at the end.
 
Old 04-26-2010, 06:46 PM   #13
minrich
Member
 
Registered: Aug 2003
Location: Isles of Man & Wight
Distribution: See signature
Posts: 548

Rep: Reputation: 40
Koooee, what hardware software (distro) and type of computer and usb port type and file system types are you using?

On a 2GHz AMD64 debian lenny laptop, bought in 2004, with 2GB ram and a 100GB 5400rpm hdd and usb2, I regularly (one a week) get dd transfer rates of between 22MB/s and 17MB/s transferring (backing-up) partitions of between 11GB to 23GB to a (2008 vintage) Western Digital Passport 320GB 5400rpm usb2 powered external drive. This is approximately the same speed that I get when copying between partitions on my internal harddrive.

So I am surprised that you are getting such slow speed. However, my command using 'dd' adds a few things that you are missing: dd if=/dev/hda6 of/dev/sdb9/deb.image bs=2048 conv=notrunc,noerror

My thoughts are that without the notrunc(ate) and noerror parameters you are suffering from severe error checking or that your external hdd is suffering from bad sectors are some other ailment.

The foregoing is based on the assumption that the if= and of= file systems are formatted with the same filesystems, that you are not using an old usb port (rather than usb2) on both ends of your connection, and that your wire is no longer than 3 feet (mine is only about 12 inches long). I also assume that you are reformatting the external hdd between running the dd commands in your first post - otherwise you are trying to overwrite existing data, which while not usually a disaster, does mean that the process will be slower than writing to a clean partition.

finally, I would add that I am careful to ensure that my external hdd partition is *unmounted* before issuing the dd command. try this it may be faster, but you will have to know which partition on the external hdd you want to dd to. It is partitioned, isn't it?
 
1 members found this post helpful.
Old 04-26-2010, 07:20 PM   #14
jschiwal
Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654
Writes aligned to the erase buffer size of the SSD may provide the best performance. You may be performing a read/update/erase/write cycle on your SSD device for each 512 byte write at some point. For normal writing, the file system partition alignment may need to match some the erase buffer size as well to prevent the device from writing twice.

This blog entry covers these items: http://thunk.org/tytso/blog/2009/02/...se-block-size/

To maximize throughput, you may need to make the sector size of the dd command match the size the erase buffer and also align the beginning of the partition to start on the start of the a cylinder for the ssd drive. Otherwise it will need to erase two buffer sized areas before updating them for each one buffer sized write.
 
Old 04-26-2010, 08:19 PM   #15
koooee
LQ Newbie
 
Registered: Apr 2010
Distribution: slackware 13.0
Posts: 7

Original Poster
Rep: Reputation: 1
Quote:
Originally Posted by minrich View Post
Koooee, what hardware software (distro) and type of computer and usb port type and file system types are you using?
Slackware 13.0 64bit
SMP_Kernal (default)
2 Intel Xeon Quad-Core's @ 2.3GHz
12GB Memory
source= 160GB SeaGate hdd SATA (fairly old so I'm guessing 5400rpm, specs aren't on the drive)
target= 400GB SeaGate external 5400rpm USB 2.0, external power

Quote:
Originally Posted by minrich View Post
So I am surprised that you are getting such slow speed. However, my command using 'dd' adds a few things that you are missing: dd if=/dev/hda6 of/dev/sdb9/deb.image bs=2048 conv=notrunc,noerror

My thoughts are that without the notrunc(ate) and noerror parameters you are suffering from severe error checking or that your external hdd is suffering from bad sectors are some other ailment.
Sector size for each drive is 512 bytes, I guess I never tried to go higher than that because I was afraid of alignment issues? Perhaps I don't have a deep enough understanding of this? Could you help?
Quote:
Originally Posted by minrich View Post
The foregoing is based on the assumption that the if= and of= file systems are formatted with the same filesystems, that you are not using an old usb port (rather than usb2) on both ends of your connection, and that your wire is no longer than 3 feet (mine is only about 12 inches long). I also assume that you are reformatting the external hdd between running the dd commands in your first post - otherwise you are trying to overwrite existing data, which while not usually a disaster, does mean that the process will be slower than writing to a clean partition.
Your assumptions are correct.
Quote:
Originally Posted by minrich View Post
finally, I would add that I am careful to ensure that my external hdd partition is *unmounted* before issuing the dd command. try this it may be faster, but you will have to know which partition on the external hdd you want to dd to. It is partitioned, isn't it?
the drive in which it received the 6mb/s transfer rate had one massive partition, slow speed on those two could very likely be due to bad sectors and/or old hardware. But, I am most concerned with the consistent linear decline in the transfer rate, regardless of the hardware I am using for the source and target??

@jschiwal --
Thanks for the informative article, very useful information. But, the article was only for SSD's, the effect I am interested about is a superset of SSD's which contains SS Memory, and Mechanical Memory.

@pixellany --
I have seen this command in different forms
Code:
kill -SIGUSR1 PID_of_dd
kill -USR1 PID_of_dd
kill -s USR1 PID_of_dd
I can't condone for the first two, as the last one is what I use. The first two don't work for me, nor are they specified in the grammar of my kill command, I would check out 'man kill' for grammar specs.
 
  


Reply

Tags
dd


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
NIC Data Transfer Rate questions SHENGTON Linux - Networking 3 08-07-2009 09:02 PM
How to increase Data transfer Rate of ftp arunabh_biswas Linux - Networking 1 11-18-2006 02:44 PM
estimating the data transfer rate between hard disk sharathksin Programming 1 02-27-2006 09:25 PM
samba transfer rate deflin39 Linux - Software 4 01-12-2004 12:15 PM
Nic transfer rate ? wizade Linux - Networking 3 01-07-2004 05:24 PM


All times are GMT -5. The time now is 12:17 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration