I used

to copy a tree of 22GB in a vfat32 partition in external hard drive #1
to another vfat32 partition in external hard drive #2.
These two drives are connected to my linux box with usb 2.0.

The command of course is issued from CentOS 5.2.

Now, the seemingly easy task turns to be excruciating painful because
the it took 6 hours 45 mins to transfer. I worked the maths, the transfer rate is 0.9 MB/sec !!

My friends told me that I should get at least 30 MB/sec transfer rate.

I wonder the mere 0.9 MB has to do with the fact that data are on vfat32 partitions ?

I suspect the driver to handle vfat is not optimal ? I don't know if I have asked the right question.

I am so upset by the slow rate.

I believe I achieve a higher rate when I transferred files between
2 ext3 partitions. Can anyone confirm ?

The speed sounds like USB 1.1, not USB 2.0. Check /var/log/messages or dmesg when you plug it in, to see if it's being recognized as a "high speed" device. If it says "full speed", it's running slow.

30 MB sounds too fast for usb. That is closer to the rate you will get from a firewire drive. It does sound like one of the ports is running on USB 1.0 speeds.

If both ports work of the same hub, then you can cut the rate in half. Also, usb 2.0 obtains it's high rate by compressing data before it goes down the wire. If what you are copying is already compressed, the speed will be less.
If both ports are from the same hub controller, you can cut the speed in half.

Some ports on your computer may be only USB 1.0. If you have a USB 1.0 device using the same parent hub, that may also reduce the speed.

Plugging into different ports and using "sudo /sbin/hdparm -t <dev>" might allow you to find the better usb ports to use.

Using "dd" to create a 50M file on an external (esata) drive, I get 5.2 MB/s. Using hdparm, I get 80 MB/sec. Using dd to copy the file I created, I get 100 MB/sec reads and 89 MB/sec writes.

You could use "dd" to test reads and writes for each device. E.G. "dd if=~/mytestfile of=/mnt/drive1/mytestfile". It will print out the statistics. Try it with each drive and see if one particular drive or usb port is causing the problem.
----
For copying from one partition to another, I would use tar.
tar -C /mnt/drive1 -cf - . | tar -C /mnt/drive2 -xf -

You could pipe through dd to get an on demand speed. You might try different sizes for block size with the dd command. It could effect the rate.
tar -c /mnt/drive1 -cf - . | dd if=/dev/stdin of=/dev/stdout bs=65536 | tar -C /mnt/drive2 -xf -

Running "kill -USR1 <pid>" in another console will cause dd to print out it's statistics so far on stderr.
Using the above (without the bs option) I get 34 MB/sec going from one directory to another on the same esata device.

---
By the way, you could use "ssh user@host tar -c /mnt/drive2 -xf -" on the right to copy from one mounted drive to a directory on another computer on the network or even on the internet.

If I were to transfer the 22GB file.tar to the disk #2 first and then run tar -xvf will I get maximum speed since there won't be excessive use of USB cable? Strange enough I didn't feel the slowness when I move the 22 GB file around the external disks. Either tar or vfat is at fault?

I can't remember the details --- another tar file file2.tar of 18GB either on disk1 or disk2, I untar file2.tar with ext3 destination partition and it took something about 2 hours. Right now I am so confused. I may have to repeat the experiment when I can just do the experiment in the background.

Another thing I wrote a script that run df -h every 5 sec and will this slow down everything?

Possibly because the head needs to move back and forth to another part of the disk.
If you move a file to another location on the same filesystem, you just need to change the directory entries. The file itself stays put. So it will be a lot faster.

One thing is sure: I get 2.6 MB/sec untar rate if the tar file and and
the destination is another partition on the same physical disk.
This is done using the standard tar -xvf a.tar command.

2.6 MB/sec might seem not much better than 0.9 MB/sec, but it is a much BETTER value. Imagine the time is shorted by 3 times. This is really
important for me.


The command I issued in post #1 is probably too sophisticated (1) it involves two physical disks. (2) it seems two tar command are running at the same time. Synchronization has to be achieved.

I will do more tests and if I see any interesting results I will report back.
The next time my hard disk crashes, I have a better strategy to restore data to the new disk !

If the filesystem is corrupt, cloning it will replicate the errors.
You could try cloning it and then run fsck on the image file.

There is a command "ddrescue" that is similar to dd but will retry in event of a read error. There is a dd option to ignore errors and continue with the copy.

I didn't realize you were copying files from a damaged drive.

If you use dd to fill up the drive with a file containing zeroes and then delete the file, the blank space on the drive will then compress to about the size of file usage. For a partially used drive, this can be significant.

This can be handy if the destination is on another computer via ssh, or you want to save an emergency restore image. gunzip the compressed image and pipe it into the dd command.

My disk is not corrupted. It is just that the mini usb connection is
rather unstable lately. It caused the mounted partitions to disappear and
reappear as if a new disk is inserted. This upset me that why I bought and new disk and try to copy everthing over using as little time as possible. If I tighten the connection for the old disk, the dismount/mount problem can be delayed so I have a small window to copy
the data between the disks.

anyway, I did a few tests on my systems (both Dell desktop and
Fujitsu laptop, both installed with the same 64-bit CentOS 5.2).

I untar (using the standard tar -xvf ) the 22GB tar file on the the same physical disk. And I am quite amassed that I get 3.25 MB/sec, 3.7 MB/sec, and 2.6 MB/sec rates on different machines (I didn't do the full extraction: I stopped after 1 to 2 hours and stop the process, but it is good enough to let me figure out how much data is copied/created). In short, the number varies, but it is definitely
better than miserable 0.9 MB/sec I have gotten in post #1.

Armed with this, I tried one more time of

This test involves data on two physical disks, i.e., tar cf in disk 1 and tar xf in another disk.
During the copying process, I use du -ak to find out the disk space,
and divided by the elapsed time in secs, and I get a seemingly progressively slower rate of 1.6, 1.4, 1.3 MB/sec (statistics collected
after at least one hour in 3 occasions, since I have the initial time
in ~/timeinfo, I can work out the time difference at any time during
the copyign process) I discontinued the
test since I need to do some work on the physical drive.

I have no theory why (1) the speed drop by quite a lot when 2 disks are
involved (from more than 2.6MB/sec to 1.6 MB/sec at best), and (2) the rate seems to get smaller and smaller (i.e., 1.6, 1.4, 1.3 MB/sec). I am feeling rather defeated by all sorts of numbers without a concrete theory. But the only conclusion is that I should (1) tar everything and put the a.tar file in the same physical disk if possible, (2) use cp to copy a.tar to the new disk, and (3) tar -xvf a.tar to get a higher rate.

There are many issues involved here : I am not sure the read/write rate for vfat32 is the same as the read/write rate for ext3. There are just too many issues involved, but since I guess my job of copying is already done as mentioned in post 1 with 6 hours 45 mins, I think I shall leave this issue behind and move on.

I felt stupid of not using scp -p -r for copying.

Now, an interesting finding is revealed:
When copying the file on the new hard drive to another two other old drives, I achieved a fantastic 28.3 MB/sec and 15 MB/sec. This is very
statistic.

Then, I reversed the role, I copied the file from the older drive
to the new drive, and I get a small 4.6 MB/sec speed! So I can conclude that
the new drive is very slow in writing (or the old drives is very slow
in reading).

I check the new HDD, it is Toshiba brand made in year 2009, with HDDR320E04X.
I am extremely puzzled by its slow speed in writing.

Did anyone else do similar tests ?

Run "lsusb" to see if the devices are using the same bus. That could account for a 50% reduction in speed.
Files such as videos will take longer to copy using a USB drive because they are already compressed. USB compresses the data before it sends the data and then decompresses it on the recipient end. An already compressed file may take twice as long as average.

Also determine whether the device is detected as a usb 1 device or a usb 2 device, by reading the kernel messages.
Sharing a parent hub with a 1.0 device may reduce the speed as well.

How to interpret this ?

I found a very convenient way of figuring out the actual speed. scp

I did `scp -r -p myusername@localmachinename:/media/VFDATAD1 .'
keyed the passwd, I can see the MB/sec. It looks like my new disk
is not too slow after all, and the rate of transfer fluctuates depending on the size, usually larger size file will get higher rate.

I have "new low speed" and "new high speed" in the /var/log/messages, but I don't understand them.