LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 01-05-2011, 01:03 PM   #1
sir-lancealot
Member
 
Registered: Aug 2007
Posts: 336

Rep: Reputation: 31
large tar nightly + rsync to compare question


I have a few sites that need to be backed up and moved off.

The current solution was a HUGE tar nightly, gzipped, then scp'd over. There is 2 much data to keep things unzipped, as I wanted to just rsync the data.

But, rather than zipping 50+GB nightly, is there a way to tar things, rsync compare them to the current tar, and just rsync the changes inside the tar file?

I'm sure there is a better way to google/ask the question, but I am posting while reading and trying to figure the best way to ask/approach. I simply don't want to copy all that data nightly if 95% is the same.

Thanks.
 
Old 01-05-2011, 01:07 PM   #2
EricTRA
Guru
 
Registered: May 2009
Location: Gibraltar, Gibraltar
Distribution: Fedora 20 with Awesome WM
Posts: 6,805
Blog Entries: 1

Rep: Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290
Hello,

If you only want to synchronize the changes you could use rsync, but I've had some bad experiences with that. When looking for another solution I came across Unison File Synchronizer. I'm sure it'll be of use.

Kind regards,

Eric
 
Old 01-05-2011, 01:24 PM   #3
sir-lancealot
Member
 
Registered: Aug 2007
Posts: 336

Original Poster
Rep: Reputation: 31
Thanks, but I didn't want to compare the unzipped storage due to storage limitations. The remote server can't store ALL the data from ALL servers uncompressed. So nightly they backup via tar and copy it over. So rather than taking 50gb, tarring and gzipping nightly to copy over, I wanted to try and tar on the host server, then via rsync or something compare the tarred file and only update the changed file (still inside the tar) but I am not sure if it's possible.

I will chk the provided link, but the key is both the remote backup machine will only have a tar / gzipped copy of the data so the compare has to go into that file to compare and I am not sure if it's possible.

Thanks,
 
Old 01-05-2011, 01:40 PM   #4
EricTRA
Guru
 
Registered: May 2009
Location: Gibraltar, Gibraltar
Distribution: Fedora 20 with Awesome WM
Posts: 6,805
Blog Entries: 1

Rep: Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290
Hi again,

How about 'tarring' your directory(ies)/file(s) directly on the destination server over an ssh tunnel and using the --update or --incremental/--listed-incremental option of tar? Have a look at the following sites, they clarify it and I believe that with a little trial and error you'll be able to avoid having to tar and copy over a big file and instead tar directly on the destination server.

Tar through SSH tunnel
Incremental dumps using tar
Update tar archive

Kind regards,

Eric
 
1 members found this post helpful.
Old 01-05-2011, 02:23 PM   #5
sir-lancealot
Member
 
Registered: Aug 2007
Posts: 336

Original Poster
Rep: Reputation: 31
Looks like a great solution, reading now.

Thanks for the help and quick replies.
 
Old 01-05-2011, 02:28 PM   #6
EricTRA
Guru
 
Registered: May 2009
Location: Gibraltar, Gibraltar
Distribution: Fedora 20 with Awesome WM
Posts: 6,805
Blog Entries: 1

Rep: Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290Reputation: 1290
Hi,

You're welcome! Let us know your findings when you got it up and running.

Kind regards,

Eric
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync can't handle large files??? deweirdt Linux - Server 19 02-28-2012 02:35 PM
Problem using rsync to back up large files to external harddrive donnaaparis Linux - Software 7 05-20-2009 03:05 PM
Verifying Rsync Backups of Large Volumes of Files mcgirvanmedia Linux - Server 2 06-03-2008 11:30 PM
tar --compare ygloo Linux - General 2 02-23-2007 04:21 PM
using rsync and tar disorderly Linux - Software 4 11-19-2005 06:40 PM


All times are GMT -5. The time now is 11:17 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration