[SOLVED] large tar nightly + rsync to compare question
Linux - ServerThis forum is for the discussion of Linux Software used in a server related context.
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
I have a few sites that need to be backed up and moved off.
The current solution was a HUGE tar nightly, gzipped, then scp'd over. There is 2 much data to keep things unzipped, as I wanted to just rsync the data.
But, rather than zipping 50+GB nightly, is there a way to tar things, rsync compare them to the current tar, and just rsync the changes inside the tar file?
I'm sure there is a better way to google/ask the question, but I am posting while reading and trying to figure the best way to ask/approach. I simply don't want to copy all that data nightly if 95% is the same.
If you only want to synchronize the changes you could use rsync, but I've had some bad experiences with that. When looking for another solution I came across Unison File Synchronizer. I'm sure it'll be of use.
Thanks, but I didn't want to compare the unzipped storage due to storage limitations. The remote server can't store ALL the data from ALL servers uncompressed. So nightly they backup via tar and copy it over. So rather than taking 50gb, tarring and gzipping nightly to copy over, I wanted to try and tar on the host server, then via rsync or something compare the tarred file and only update the changed file (still inside the tar) but I am not sure if it's possible.
I will chk the provided link, but the key is both the remote backup machine will only have a tar / gzipped copy of the data so the compare has to go into that file to compare and I am not sure if it's possible.
How about 'tarring' your directory(ies)/file(s) directly on the destination server over an ssh tunnel and using the --update or --incremental/--listed-incremental option of tar? Have a look at the following sites, they clarify it and I believe that with a little trial and error you'll be able to avoid having to tar and copy over a big file and instead tar directly on the destination server.