LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices



Reply
 
Search this Thread
Old 01-01-2008, 11:31 AM   #1
Unclesmiff
LQ Newbie
 
Registered: Feb 2007
Posts: 17

Rep: Reputation: 0
Question rsync - inefficient behaviour?


Hi,

I'm using rsync to sync a large file (NT bkf file 21GB) from one server to another.

The --write-batch option produces a difference file of 880MB between the local and remote file.

If I rsync the file directly, it takes days.

When I use the --only-write-batch option, sftp the batch (difference) file, then use --read-batch to apply the differences to the remote file, I find that it does it in a few hours.

What is going on???

It's like the default rsync is massivley inefficient.
 
Old 01-01-2008, 12:55 PM   #2
David1357
Senior Member
 
Registered: Aug 2007
Location: South Carolina, U.S.A.
Distribution: Ubuntu, Fedora Core, Red Hat, SUSE, Gentoo, DSL, coLinux, uClinux
Posts: 1,302
Blog Entries: 1

Rep: Reputation: 107Reputation: 107
Quote:
Originally Posted by Unclesmiff View Post
If I rsync the file directly, it takes days.
I'm no rsync expert, in fact I have never used it. But I am pretty sure it is not optimized for synching large binaries.

Why don't you just scp the file?
 
Old 01-01-2008, 07:18 PM   #3
neomanyon
LQ Newbie
 
Registered: Jan 2008
Location: Australia
Distribution: Debian, Mandriva
Posts: 1

Rep: Reputation: 0
I have at times thought that rsync is not that efficient but have not worried about it as i only use over a small LAN.

Just looking at the man page...

-W, --whole-file
With this option the incremental rsync algorithm is not used and the whole file is sent as-is instead.
The transfer may be faster if this option is used when the bandwidth between the source and destination machines is higher than the bandwidth to disk (especially when the “disk” is actually a networked filesystem). This is the default when both the source and destination are specified as local paths.


Are you connecting to the server using a local path?
 
Old 01-02-2008, 11:57 AM   #4
Unclesmiff
LQ Newbie
 
Registered: Feb 2007
Posts: 17

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by David1357 View Post
I'm no rsync expert, in fact I have never used it. But I am pretty sure it is not optimized for synching large binaries.

Why don't you just scp the file?
scp a 21GB file - that would be *much* slower than even the inefficient rsync method.
 
Old 01-02-2008, 12:13 PM   #5
Unclesmiff
LQ Newbie
 
Registered: Feb 2007
Posts: 17

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by neomanyon View Post
I have at times thought that rsync is not that efficient but have not worried about it as i only use over a small LAN.

Just looking at the man page...

-W, --whole-file
With this option the incremental rsync algorithm is not used and the whole file is sent as-is instead.
The transfer may be faster if this option is used when the bandwidth between the source and destination machines is higher than the bandwidth to disk (especially when the “disk” is actually a networked filesystem). This is the default when both the source and destination are specified as local paths.

Are you connecting to the server using a local path?
It is not a local path.

The data is to be sent over the internet during the night, and since the upload speeds are not great on ADSL --whole-file is not an option, and so neither is scp as this would send the whole file.

I was just wondering why if I ask rsync to create a batch file, this batch file can be ftp’d in hours, then use rsync on the receiver to build the final file, rather than letting rsync do it all in days.

Is there a bug in rsync?

Surely rsync is meant for large binary files to send just the changes as small files can be sent as whole.

Last edited by Unclesmiff; 01-02-2008 at 02:29 PM.
 
  


Reply

Tags
rsync


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
is this normal rsync behaviour? nass Slackware 5 05-03-2007 02:40 PM
Rsync server vs rsync over ssh humbletech99 Linux - Networking 1 10-18-2006 01:10 PM
vi Behaviour! dlublink Suse/Novell 5 09-20-2006 10:14 AM
Strange behaviour Anmol Suse/Novell 2 10-28-2005 12:05 AM
Windows Rsync Upload to Linux Rsync - permissions inspleak Linux - Software 0 10-12-2004 03:49 PM


All times are GMT -5. The time now is 01:20 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration