LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 02-16-2012, 03:15 AM   #1
Lumify
Member
 
Registered: Jan 2012
Distribution: Debian Stable
Posts: 52

Rep: Reputation: Disabled
Rsync local drives transfers even existing files


Whenever I try rsyncing two local drives, rsync transfers everything instead of skipping the already-transferred files. I use rsync -rvhS --progress --stats sourcedir destdir, and using the -u option doesn't help either. When I transfer from one server to another, it only transfers changed files.

I'm trying to transfer data from one local 3TB drive to another larger local drive, and I can't finish the transfer because if I have to restart after 3 days, everything starts over again. Is this an intended feature of rsync, or am I doing it wrong?
 
Old 02-16-2012, 03:31 AM   #2
deep27ak
Senior Member
 
Registered: Aug 2011
Location: Bangalore, India
Distribution: rhel 5x,6.0,6.2, centOS 5x,6.0,6.2
Posts: 1,194
Blog Entries: 4

Rep: Reputation: 221Reputation: 221Reputation: 221
Quote:
Originally Posted by Lumify View Post
When I transfer from one server to another, it only transfers changed files.
So this is what you exactly want to do isn't it?
Update the existing directory with new and updated files.
Rsync will copy only those files which does not exist in your current or destination directory wrt source directory.

Quote:
Originally Posted by Lumify View Post
I'm trying to transfer data from one local 3TB drive to another larger local drive, and I can't finish the transfer because if I have to restart after 3 days, everything starts over again.
So you can try running simultaneous rsync process distributing the data which are trying to copy. It would utilize greater amount of Memory but would increase the rate of data transfer but don't run more than 2 process at a time.
 
1 members found this post helpful.
Old 02-16-2012, 04:47 AM   #3
hallucinative
LQ Newbie
 
Registered: Feb 2012
Location: Western Australia
Distribution: Ubuntu Server
Posts: 18

Rep: Reputation: 5
I won't pretend to understand why your copying hard links but if you keep it simple... do you have the same issue? and rsync handles sparce files very well with its on the fly hash checking anyway so unless you've got very large sparce files I would drop it..

give this a try..
Code:
rsync -rP <Source> <Destination>
see if you have the same issue..


Storage use:
Also I found hash checking files in my storage drive while rsyncing didn't really benefit me, if it needed to be synced there would normally just be a size difference.. the chance you've edited a file and its the same size would be extremely unlikely... (--size-only)

Server use:
If this is for your Minecraft server or system backup then hash checking would still be worth doing..

Or this might even have to do with some strange time issue where its copying to the new drive and its time format has changed... Apparently ext4 has 2 more digits of accuracy... but I'm speculating, you could give --ignore-times a try


Good luck!

-Donny
 
1 members found this post helpful.
Old 02-17-2012, 06:58 AM   #4
Lumify
Member
 
Registered: Jan 2012
Distribution: Debian Stable
Posts: 52

Original Poster
Rep: Reputation: Disabled
I don't know why this would cause rsync to retransfer everything when I restart it, but the --size-only option fixed the issue! I also did notice that rsync is rather slow; running it two times at once speeds it up a lot. Thank you both for the advice.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync not delete non existing files on remote server proNick Linux - Software 2 09-10-2008 03:47 AM
[rsync] get the differences between the source files and the existing files djgerbavore Linux - Networking 2 06-04-2008 12:05 PM
resume broken transfers at file-level using rsync prvnrk Linux - Server 1 08-25-2007 12:04 AM
performance issue: rsync sends existing files microtim Linux - Networking 1 12-02-2004 02:27 AM
rsync always transfers brentos Linux - Software 2 04-29-2004 06:53 PM


All times are GMT -5. The time now is 08:03 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration