LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Hardware
User Name
Password
Linux - Hardware This forum is for Hardware issues.
Having trouble installing a piece of hardware? Want to know if that peripheral is compatible with Linux?

Notices



Reply
 
Search this Thread
Old 07-16-2010, 12:01 PM   #1
king0770
LQ Newbie
 
Registered: May 2006
Posts: 4

Rep: Reputation: 0
Best utility to copy massive amounts of data


Just asking out of curiosity...

Sometimes I need to copy a huge directory to another directory (local filesystem), and usually I will use the "cp" or "rsync" commands. These commands are good, but depending on the size of the data being copied, the copy is painfully slow.

I realize we are limited because of the hardware we have with it's limitations, ie, I/O speed, and the filesystem (which is usually ext3).

Are there any other utilities that maybe not well known, but can handle copying large amounts of data? (mostly in the TB range)

My best to you,

~~Rick
 
Old 07-16-2010, 01:08 PM   #2
nkitmitto
LQ Newbie
 
Registered: Jul 2010
Posts: 16

Rep: Reputation: 0
You could try a private backend network between the servers so you could rsync much faster. Just an opinion, not sure of the physical locations of your servers.
 
Old 07-16-2010, 02:45 PM   #3
jefro
Guru
 
Registered: Mar 2008
Posts: 12,087

Rep: Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521Reputation: 1521
If copy or rsync is slow then almost any other copy will be slow. I'd think about your OS and hardware. Might be something there causing it or maybe some tweaks to speed it up. Block size may be a choice as well as how ram is used in this or setting priorities. The only advantage of rsync would be if only part of the data changed otherwise it is a copy. Rsync may work if you can allow it to be working all the time so that when you do need to officially copy it could only be a matter of the last few hours.

I'd try to keep copy operations like that between drives and not between partitions.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Searching through massive amounts of Data sxa Linux - Software 5 02-27-2009 10:42 PM
Using wget to copy large amounts of data via ftp. AndrewCAtWayofthebit Linux - General 1 05-11-2006 12:55 PM
DISCUSSION: Network Attached Storage – An Alternative To Tape Drives In Managing Massive Amounts of Data primearray.com LQ Articles Discussion 0 04-02-2006 05:48 PM
rm command is choking on large amounts of data? Jello Linux - General 18 02-28-2003 08:11 PM


All times are GMT -5. The time now is 12:14 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration