rsync - inefficient behaviour?
 

Hi,

I'm using rsync to sync a large file (NT bkf file 21GB) from one server to another.

The --write-batch option produces a difference file of 880MB between the local and remote file.

If I rsync the file directly, it takes days.

When I use the --only-write-batch option, sftp the batch (difference) file, then use --read-batch to apply the differences to the remote file, I find that it does it in a few hours.

What is going on???

It's like the default rsync is massivley inefficient.

I'm no rsync expert, in fact I have never used it. But I am pretty sure it is not optimized for synching large binaries.

Why don't you just scp the file?

I have at times thought that rsync is not that efficient but have not worried about it as i only use over a small LAN.

Just looking at the man page...

-W, --whole-file
With this option the incremental rsync algorithm is not used and the whole file is sent as-is instead.
The transfer may be faster if this option is used when the bandwidth between the source and destination machines is higher than the bandwidth to disk (especially when the “disk” is actually a networked filesystem). This is the default when both the source and destination are specified as local paths.


Are you connecting to the server using a local path?

scp a 21GB file - that would be *much* slower than even the inefficient rsync method.

It is not a local path.

The data is to be sent over the internet during the night, and since the upload speeds are not great on ADSL --whole-file is not an option, and so neither is scp as this would send the whole file.

I was just wondering why if I ask rsync to create a batch file, this batch file can be ftp’d in hours, then use rsync on the receiver to build the final file, rather than letting rsync do it all in days.

Is there a bug in rsync?

Surely rsync is meant for large binary files to send just the changes as small files can be sent as whole.