large tar nightly + rsync to compare question
 

I have a few sites that need to be backed up and moved off.

The current solution was a HUGE tar nightly, gzipped, then scp'd over. There is 2 much data to keep things unzipped, as I wanted to just rsync the data.

But, rather than zipping 50+GB nightly, is there a way to tar things, rsync compare them to the current tar, and just rsync the changes inside the tar file?

I'm sure there is a better way to google/ask the question, but I am posting while reading and trying to figure the best way to ask/approach. I simply don't want to copy all that data nightly if 95% is the same.

Thanks.

Hello,

If you only want to synchronize the changes you could use rsync, but I've had some bad experiences with that. When looking for another solution I came across Unison File Synchronizer. I'm sure it'll be of use.

Kind regards,

Eric

Thanks, but I didn't want to compare the unzipped storage due to storage limitations. The remote server can't store ALL the data from ALL servers uncompressed. So nightly they backup via tar and copy it over. So rather than taking 50gb, tarring and gzipping nightly to copy over, I wanted to try and tar on the host server, then via rsync or something compare the tarred file and only update the changed file (still inside the tar) but I am not sure if it's possible.

I will chk the provided link, but the key is both the remote backup machine will only have a tar / gzipped copy of the data so the compare has to go into that file to compare and I am not sure if it's possible.

Thanks,

Hi again,

How about 'tarring' your directory(ies)/file(s) directly on the destination server over an ssh tunnel and using the --update or --incremental/--listed-incremental option of tar? Have a look at the following sites, they clarify it and I believe that with a little trial and error you'll be able to avoid having to tar and copy over a big file and instead tar directly on the destination server.

Tar through SSH tunnel
Incremental dumps using tar
Update tar archive

Kind regards,

Eric

Looks like a great solution, reading now.

Thanks for the help and quick replies.

Hi,

You're welcome! Let us know your findings when you got it up and running.

Kind regards,

Eric