to -z or not to -z during an rsync
 

Hi guys. Looking at a bunch of rsync cron jobs we have, they all have -z in them to compress the files before transfer. I'm wondering if that's even really needed or not. Does that add overhead to the server running the rsync? Is it actually trying to zip each individual file before sending it or something?

It strongly depends on the bandwidth you have at disposal. For fast connections the compression can actually add overhead to the whole process, whereas for slow connections or if most of the available bandwidth is required by other critical processes the compression is strongly advised.

On the other hand there are some tricks you can adopt in order to limit the bandwith usage and/or speed up the syncronization. First of all, take in mind that rsync has a --bwlimit option by which you can control the bandwidth usage. Then you can instruct ssh to spare some time, by using a quicker decipher algorithm, disabling x11 forwarding or disabling ssh compression (actually the last two are the default behaviour). For example:
Moreover you can determine the compression level in rsync by means of the --compress-level option. Which combination of these conditions is the best for you, can be told only by testing. I'd say that -z is generally a good option, but you must always take in account the balance between the usage of the local resources and the bandwidth usage in order to not interfere with other critical processes running on your server. Just my :twocents:

Thanks colucix. Really I only run my rsync's during off hours, and it's all on the same network between servers. Like one server i was looking at is like 10s of thousands of jpg's, but has the z enabled. I'd imagine the overhead on it trying to compress every jpg individually outweighs any gains that might be doing.

The kind of files matters too; jpg's generally can't be compressed any further. In fact, rsync won't attempt to compress jpg's even if you pass -z, by default:

Ah, that's good to know. I didn't even notice that little note in the man when I looked at it. I only looked at the text along with the z option itself.