Visit Jeremy's Blog.
Go Back > Forums > Linux Forums > Linux - Software
User Name
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.


  Search this Thread
Old 06-22-2008, 03:24 PM   #1
Registered: Feb 2007
Posts: 386

Rep: Reputation: 30
Can rsync work with 10 terabyte data?

I am transfering video & audio data from one server to the others for backup.
The total data size is about 10 Terabytes. If I used rsync from the root directory, rsync seems to take a long time before start the actual copy.
I guess it was doing some indexing. So I stopped them since I was afraid it might crash the system because of the size of my transfering data.

So I choosed to rysnc directory by directory instead. I created a shell script with each line having a rsync cmd to copy each directory. It seems to work fine.

I would like some expert opinion if I were to rysnc from the root, will it cause any problem for the data of this size?

Once my shell finish copy all the data , I was planing to excute another rsync from the root to make sure that all the data are copied.

Last edited by ufmale; 06-22-2008 at 03:25 PM.
Old 06-22-2008, 03:43 PM   #2
LQ Guru
Registered: Feb 2003
Location: Virginia, USA
Distribution: Debian 12
Posts: 8,347

Rep: Reputation: 552Reputation: 552Reputation: 552Reputation: 552Reputation: 552Reputation: 552
I don't think that you will crash the system. You could experience the problem that the system runs very slow for a very long time while you are moving that much data.

After you run rsync for the first time it will go a lot faster on subsequent runs because after the first run rsync only copies data that has changed. The files that are still the same as the first rsync are not copied again.

Steve Stites
Old 06-22-2008, 05:35 PM   #3
Registered: Feb 2007
Posts: 386

Original Poster
Rep: Reputation: 30
Does anyone know what rsync was doing before it actually copy the data? It seems to stay still for a long time. Will it get better in the subsequencial run?
Old 06-22-2008, 06:09 PM   #4
Senior Member
Registered: Jan 2006
Posts: 4,363

Rep: Reputation: 172Reputation: 172
It has to compare the current files with the backedup files, that takes time. How much time depends on how much data. What you can do is split it up into multiple jobs, with each job handling a different subdirectory. It will probably take the same amount of time(total) but you will "see" more progress as each of the separate jobs finishes.


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
DISCUSSION: Using rsync to mirror data between servers david_ross LinuxAnswers Discussion 8 03-29-2012 01:37 AM
copy terabyte of data from one machine to another ufmale Linux - Newbie 3 05-16-2008 12:17 AM
Problem with data backup using rsync and samba coolamit78 Linux - Networking 0 10-31-2004 09:01 AM
How can I stop rsync from deleting data if network link is dropped? skyfly Linux - Networking 2 06-30-2004 10:35 AM
Rsync to mirror data for backup - having difficulties tapanga Linux - Newbie 1 05-17-2004 05:45 PM > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 06:57 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration