Share your knowledge at the LQ Wiki.
Go Back > Forums > Linux Forums > Linux - Hardware
User Name
Linux - Hardware This forum is for Hardware issues.
Having trouble installing a piece of hardware? Want to know if that peripheral is compatible with Linux?


  Search this Thread
Old 07-16-2010, 11:01 AM   #1
LQ Newbie
Registered: May 2006
Posts: 4

Rep: Reputation: 0
Best utility to copy massive amounts of data

Just asking out of curiosity...

Sometimes I need to copy a huge directory to another directory (local filesystem), and usually I will use the "cp" or "rsync" commands. These commands are good, but depending on the size of the data being copied, the copy is painfully slow.

I realize we are limited because of the hardware we have with it's limitations, ie, I/O speed, and the filesystem (which is usually ext3).

Are there any other utilities that maybe not well known, but can handle copying large amounts of data? (mostly in the TB range)

My best to you,

Old 07-16-2010, 12:08 PM   #2
LQ Newbie
Registered: Jul 2010
Posts: 16

Rep: Reputation: 0
You could try a private backend network between the servers so you could rsync much faster. Just an opinion, not sure of the physical locations of your servers.
Old 07-16-2010, 01:45 PM   #3
Registered: Mar 2008
Posts: 17,000

Rep: Reputation: 2513Reputation: 2513Reputation: 2513Reputation: 2513Reputation: 2513Reputation: 2513Reputation: 2513Reputation: 2513Reputation: 2513Reputation: 2513Reputation: 2513
If copy or rsync is slow then almost any other copy will be slow. I'd think about your OS and hardware. Might be something there causing it or maybe some tweaks to speed it up. Block size may be a choice as well as how ram is used in this or setting priorities. The only advantage of rsync would be if only part of the data changed otherwise it is a copy. Rsync may work if you can allow it to be working all the time so that when you do need to officially copy it could only be a matter of the last few hours.

I'd try to keep copy operations like that between drives and not between partitions.


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Searching through massive amounts of Data sxa Linux - Software 5 02-27-2009 09:42 PM
Using wget to copy large amounts of data via ftp. AndrewCAtWayofthebit Linux - General 1 05-11-2006 11:55 AM
DISCUSSION: Network Attached Storage – An Alternative To Tape Drives In Managing Massive Amounts of Data LQ Articles Discussion 0 04-02-2006 04:48 PM
rm command is choking on large amounts of data? Jello Linux - General 18 02-28-2003 07:11 PM > Forums > Linux Forums > Linux - Hardware

All times are GMT -5. The time now is 01:27 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration