LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 12-03-2010, 09:34 AM   #1
someshpr
Member
 
Registered: Jul 2009
Location: CA, USA
Distribution: Debian, RHEL5.4, CentOS 5.4, 6.2, Ubuntu 11.04,12.04
Posts: 107

Rep: Reputation: 26
rsync -z vs. scp ... which one is faster over network?


Hi,
I often have to transfer huge data over our LAN from one computer to another. The size of the files varies and can be somewhere from 2 GB to 50GB or more! The only accepted connection protocol between machines are ssh; and rsync and scp are the only options available for copying over network (unison is not installed).
I usually use rsync with "-z" option to copy over network.

Does anyone have any idea if "rsync -z" is faster than "scp" for data transfer?

TIA,
 
Old 12-03-2010, 10:20 AM   #2
AlucardZero
Senior Member
 
Registered: May 2006
Location: USA
Distribution: Debian
Posts: 4,647

Rep: Reputation: 524Reputation: 524Reputation: 524Reputation: 524Reputation: 524Reputation: 524
It probably is, especially if your connection sometimes gets interrupted and you have to resume (and double especially if you use rsync's --partial, and triple especially if you transfer the same files with minor changes to the same destination).

Have you benchmarked it? rsync has a "-h --stats" and scp also gives stats.
 
Old 12-07-2010, 04:26 PM   #3
someshpr
Member
 
Registered: Jul 2009
Location: CA, USA
Distribution: Debian, RHEL5.4, CentOS 5.4, 6.2, Ubuntu 11.04,12.04
Posts: 107

Original Poster
Rep: Reputation: 26
Quote:
Originally Posted by AlucardZero View Post
It probably is, especially if your connection sometimes gets interrupted and you have to resume (and double especially if you use rsync's --partial, and triple especially if you transfer the same files with minor changes to the same destination).

Have you benchmarked it? rsync has a "-h --stats" and scp also gives stats.
Thanks AlucardZero! I haven't benchmarked it using "-h --stats", but using "/usr/bin/time" I have seen rsync -z to be faster. Here is one set of results for same source and no prior existence of destination:

Code:
scp -r: RunTime: System: 1.34; User: 2.42; Elapsed:15:39.39
rsync -az: RunTime: System: 0.66; User: 2.69; Elapsed:3:32.51
 
Old 12-08-2010, 12:02 AM   #4
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,280

Rep: Reputation: 2032Reputation: 2032Reputation: 2032Reputation: 2032Reputation: 2032Reputation: 2032Reputation: 2032Reputation: 2032Reputation: 2032Reputation: 2032Reputation: 2032
As AlucardZero pointed out, rsync is better for broken txfrs & by default it only sends diffrences, so if you sent the same file twice it would only send the diffs. You're also using -z = compress.
scp encrypts everything, so its likely what's sent is the same size as what's on the disk. Also, encrypted stuff doesn't compress very well as a rule.
You'd notice a slowdown if you told rsync to use ssh encryption. (-e ssh ...)
 
Old 12-08-2010, 06:05 AM   #5
AlucardZero
Senior Member
 
Registered: May 2006
Location: USA
Distribution: Debian
Posts: 4,647

Rep: Reputation: 524Reputation: 524Reputation: 524Reputation: 524Reputation: 524Reputation: 524
I don't understand, chrism01. rsync goes over SSH just like SCP. Also, it's possible to compress, THEN encrypt.
 
Old 12-08-2010, 06:06 PM   #6
someshpr
Member
 
Registered: Jul 2009
Location: CA, USA
Distribution: Debian, RHEL5.4, CentOS 5.4, 6.2, Ubuntu 11.04,12.04
Posts: 107

Original Poster
Rep: Reputation: 26
compression is the key!

I did some more timing studies, and here is what I found:
For a directory predominantly full of music (mp3/ogg)
Code:
rsync -aze 'ssh' ... RunTime: System: 22.37; User: 99.39; Elapsed:3:08.79
rsync -ae 'ssh' ... RunTime: System: 21.52; User: 80.57; Elapsed:3:06.14
scp -r ... RunTime: System: 19.48; User: 69.78; Elapsed:3:23.12
rsync -az ... RunTime: System: 22.29; User: 99.86; Elapsed:3:10.09
For a directory full of various data and text and binary files
Code:
rsync -aze 'ssh' ... RunTime: System: 3.90; User: 15.54; Elapsed:0:48.96
rsync -ae 'ssh' ... RunTime: System: 12.61; User: 47.56; Elapsed:1:49.80
scp -r ... RunTime: System: 11.69; User: 41.92; Elapsed:2:03.16
rsync -az ... RunTime: System: 4.02; User: 15.52; Elapsed:0:47.11
rsync -a ... RunTime: System: 12.69; User: 47.53; Elapsed:1:55.53
I believe mp3/ogg files cannot be compressed much and hence doesn't really matter whether we give "-z" option, or not. However for other cases "-z" option seem to make a big difference! Also timing for rsync with and without "-e ssh" are pretty much the same.
 
  


Reply

Tags
network, rsync, scp


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Linux scp or rsync without password sanjee Linux - Server 15 09-13-2012 02:46 PM
I think I need rsync or scp -- something to copy from http://... to /somewhere/else nithra Linux - Server 2 01-18-2011 06:09 AM
Auto syncing folder from Linux to Windows, SCP/Rsync? batfastad Linux - Software 2 11-26-2010 02:08 AM
Home network is fine, until i try to rsync/scp lagartoflojo Linux - Networking 4 01-15-2006 09:09 PM
scp problem related to network architecture? rdorte Linux - Networking 2 07-22-2004 09:17 AM


All times are GMT -5. The time now is 01:59 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration