LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 12-26-2012, 08:24 AM   #1
rjo98
Senior Member
 
Registered: Jun 2009
Location: US
Distribution: RHEL, CentOS
Posts: 1,685

Rep: Reputation: 48
to -z or not to -z during an rsync


Hi guys. Looking at a bunch of rsync cron jobs we have, they all have -z in them to compress the files before transfer. I'm wondering if that's even really needed or not. Does that add overhead to the server running the rsync? Is it actually trying to zip each individual file before sending it or something?
 
Old 12-26-2012, 09:08 AM   #2
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978Reputation: 1978
It strongly depends on the bandwidth you have at disposal. For fast connections the compression can actually add overhead to the whole process, whereas for slow connections or if most of the available bandwidth is required by other critical processes the compression is strongly advised.

On the other hand there are some tricks you can adopt in order to limit the bandwith usage and/or speed up the syncronization. First of all, take in mind that rsync has a --bwlimit option by which you can control the bandwidth usage. Then you can instruct ssh to spare some time, by using a quicker decipher algorithm, disabling x11 forwarding or disabling ssh compression (actually the last two are the default behaviour). For example:
Code:
rsync -az -e "ssh -x -c Arcfour -o Compression=no" source user@host:destination
Moreover you can determine the compression level in rsync by means of the --compress-level option. Which combination of these conditions is the best for you, can be told only by testing. I'd say that -z is generally a good option, but you must always take in account the balance between the usage of the local resources and the bandwidth usage in order to not interfere with other critical processes running on your server. Just my
 
1 members found this post helpful.
Old 12-26-2012, 09:42 AM   #3
rjo98
Senior Member
 
Registered: Jun 2009
Location: US
Distribution: RHEL, CentOS
Posts: 1,685

Original Poster
Rep: Reputation: 48
Thanks colucix. Really I only run my rsync's during off hours, and it's all on the same network between servers. Like one server i was looking at is like 10s of thousands of jpg's, but has the z enabled. I'd imagine the overhead on it trying to compress every jpg individually outweighs any gains that might be doing.
 
Old 12-26-2012, 12:26 PM   #4
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Arch
Posts: 3,131

Rep: Reputation: 1317Reputation: 1317Reputation: 1317Reputation: 1317Reputation: 1317Reputation: 1317Reputation: 1317Reputation: 1317Reputation: 1317Reputation: 1317
Quote:
Originally Posted by rjo98 View Post
Like one server i was looking at is like 10s of thousands of jpg's, but has the z enabled.
The kind of files matters too; jpg's generally can't be compressed any further. In fact, rsync won't attempt to compress jpg's even if you pass -z, by default:

Quote:
man rsync
...
The default list of suffixes that will not be compressed is this (several
of these are newly added for 3.0.0):
gz/zip/z/rpm/deb/iso/bz2/t[gb]z/7z/mp[34]/mov/avi/ogg/jpg/jpeg
 
2 members found this post helpful.
Old 12-26-2012, 12:29 PM   #5
rjo98
Senior Member
 
Registered: Jun 2009
Location: US
Distribution: RHEL, CentOS
Posts: 1,685

Original Poster
Rep: Reputation: 48
Ah, that's good to know. I didn't even notice that little note in the man when I looked at it. I only looked at the text along with the z option itself.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync solaris + ld.so.1: rsync: fatal: libiconv.so.2: open failed: xxx_anuj_xxx Solaris / OpenSolaris 25 02-23-2012 03:23 AM
[SOLVED] rsync execution issue with crontab - Have given full path to rsync too!! Prabagaran Linux - Server 6 04-15-2011 01:39 AM
[SOLVED] rsync fails in cron - ssh key prob for rsync? jonathansfl Linux - Server 6 12-09-2010 09:48 AM
Could I run rsync to download files from a server without rsync daemon? Richard.Yang Linux - Software 1 09-18-2009 04:08 AM
Windows Rsync Upload to Linux Rsync - permissions inspleak Linux - Software 0 10-12-2004 02:49 PM


All times are GMT -5. The time now is 06:33 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration