-   Linux - Hardware (
-   -   Best utility to copy massive amounts of data (

king0770 07-16-2010 12:01 PM

Best utility to copy massive amounts of data
Just asking out of curiosity...

Sometimes I need to copy a huge directory to another directory (local filesystem), and usually I will use the "cp" or "rsync" commands. These commands are good, but depending on the size of the data being copied, the copy is painfully slow.

I realize we are limited because of the hardware we have with it's limitations, ie, I/O speed, and the filesystem (which is usually ext3).

Are there any other utilities that maybe not well known, but can handle copying large amounts of data? (mostly in the TB range)

My best to you,


nkitmitto 07-16-2010 01:08 PM

You could try a private backend network between the servers so you could rsync much faster. Just an opinion, not sure of the physical locations of your servers.

jefro 07-16-2010 02:45 PM

If copy or rsync is slow then almost any other copy will be slow. I'd think about your OS and hardware. Might be something there causing it or maybe some tweaks to speed it up. Block size may be a choice as well as how ram is used in this or setting priorities. The only advantage of rsync would be if only part of the data changed otherwise it is a copy. Rsync may work if you can allow it to be working all the time so that when you do need to officially copy it could only be a matter of the last few hours.

I'd try to keep copy operations like that between drives and not between partitions.

All times are GMT -5. The time now is 07:08 AM.