I have been trying to copy some files (about 3 million) from a linux machine to a USB hard drive over the past couple of weeks. The performance copying files has been in the 3.5MB/sec range. I had tested the speed of the drives involved using hdparm and they were above 50MB/sec. I have tried different programs, cp, cpio, tar, star, rsync and the performance remains in this range both for copying from the IDE drive to a USB drive and the IDE drive to another IDE drive.

I have been copying tiff files that range in size from 32K to 10MB.

Just to see if there would be any difference I just copied one 1GB file from the IDE drive to the USB drive and it took 54 seconds. So that transfer happened at 18.5MB/sec.

Is there any software or scheme that can get the transfer of multiple files up to the level of the single file? I know there would be a little overhead in the opening of the individual files, but it seems like there should be a way to get closer to the 18.5MB/sec range.

Use hot swapping

or

compress - copy - uncompress

Or you can try using database applications