Best utility to copy massive amounts of data
 

Just asking out of curiosity...

Sometimes I need to copy a huge directory to another directory (local filesystem), and usually I will use the "cp" or "rsync" commands. These commands are good, but depending on the size of the data being copied, the copy is painfully slow.

I realize we are limited because of the hardware we have with it's limitations, ie, I/O speed, and the filesystem (which is usually ext3).

Are there any other utilities that maybe not well known, but can handle copying large amounts of data? (mostly in the TB range)

My best to you,

~~Rick

You could try a private backend network between the servers so you could rsync much faster. Just an opinion, not sure of the physical locations of your servers.

If copy or rsync is slow then almost any other copy will be slow. I'd think about your OS and hardware. Might be something there causing it or maybe some tweaks to speed it up. Block size may be a choice as well as how ram is used in this or setting priorities. The only advantage of rsync would be if only part of the data changed otherwise it is a copy. Rsync may work if you can allow it to be working all the time so that when you do need to officially copy it could only be a matter of the last few hours.

I'd try to keep copy operations like that between drives and not between partitions.