[SOLVED] rsync -z vs. scp ... which one is faster over network?

someshpr · 12-03-2010, 09:34 AM

Hi,
I often have to transfer huge data over our LAN from one computer to another. The size of the files varies and can be somewhere from 2 GB to 50GB or more! The only accepted connection protocol between machines are ssh; and rsync and scp are the only options available for copying over network (unison is not installed).
I usually use rsync with "-z" option to copy over network.

Does anyone have any idea if "rsync -z" is faster than "scp" for data transfer?

TIA,

AlucardZero · 12-03-2010, 10:20 AM

It probably is, especially if your connection sometimes gets interrupted and you have to resume (and double especially if you use rsync's --partial, and triple especially if you transfer the same files with minor changes to the same destination).

Have you benchmarked it? rsync has a "-h --stats" and scp also gives stats.

someshpr · 12-07-2010, 04:26 PM

Quote:

Originally Posted by AlucardZero

It probably is, especially if your connection sometimes gets interrupted and you have to resume (and double especially if you use rsync's --partial, and triple especially if you transfer the same files with minor changes to the same destination).

Have you benchmarked it? rsync has a "-h --stats" and scp also gives stats.

Thanks AlucardZero! I haven't benchmarked it using "-h --stats", but using "/usr/bin/time" I have seen rsync -z to be faster. Here is one set of results for same source and no prior existence of destination:

Code:

scp -r: RunTime: System: 1.34; User: 2.42; Elapsed:15:39.39
rsync -az: RunTime: System: 0.66; User: 2.69; Elapsed:3:32.51

chrism01 · 12-08-2010, 12:02 AM

As AlucardZero pointed out, rsync is better for broken txfrs & by default it only sends diffrences, so if you sent the same file twice it would only send the diffs. You're also using -z = compress.
scp encrypts everything, so its likely what's sent is the same size as what's on the disk. Also, encrypted stuff doesn't compress very well as a rule.
You'd notice a slowdown if you told rsync to use ssh encryption. (-e ssh ...)

AlucardZero · 12-08-2010, 06:05 AM

I don't understand, chrism01. rsync goes over SSH just like SCP. Also, it's possible to compress, THEN encrypt.

someshpr · 12-08-2010, 06:06 PM

I did some more timing studies, and here is what I found:
For a directory predominantly full of music (mp3/ogg)

Code:

rsync -aze 'ssh' ... RunTime: System: 22.37; User: 99.39; Elapsed:3:08.79
rsync -ae 'ssh' ... RunTime: System: 21.52; User: 80.57; Elapsed:3:06.14
scp -r ... RunTime: System: 19.48; User: 69.78; Elapsed:3:23.12
rsync -az ... RunTime: System: 22.29; User: 99.86; Elapsed:3:10.09

For a directory full of various data and text and binary files

Code:

rsync -aze 'ssh' ... RunTime: System: 3.90; User: 15.54; Elapsed:0:48.96
rsync -ae 'ssh' ... RunTime: System: 12.61; User: 47.56; Elapsed:1:49.80
scp -r ... RunTime: System: 11.69; User: 41.92; Elapsed:2:03.16
rsync -az ... RunTime: System: 4.02; User: 15.52; Elapsed:0:47.11
rsync -a ... RunTime: System: 12.69; User: 47.53; Elapsed:1:55.53

I believe mp3/ogg files cannot be compressed much and hence doesn't really matter whether we give "-z" option, or not. However for other cases "-z" option seem to make a big difference! Also timing for rsync with and without "-e ssh" are pretty much the same.