[SOLVED] Confused by TAR incremental feature

digity · 02-04-2010, 02:45 PM

I'm following the guide on using tar to perform incremental backups: http://www.gnu.org/software/automake...tal-Dumps.html

I'm confused on how the snapshot file actually works. I want to do a daily backup system, performing full backups only on the 1st of the month and incrementals during the rest of the month. So I make the 1st of the month/full/level 0 backup like so:

Code:

tar -zvcf /backups/users/users.20100201.tar.gz --listed-incremental=/backups/users/users.20100201.snar /users

The confusion starts here for me - for daily/incrementals/level 1 backups do I simply reference that same snapshot file each and every time like so?:

Code:

# Today is 02/02/2010
tar -zvcf /backups/users/users.20100202.tar.gz --listed-incremental=/backups/users/users.20100201.snar /users

.....

# Today is 02/03/2010
tar -zvcf /backups/users/users.20100203.tar.gz --listed-incremental=/backups/users/users.20100201.snar /users

.....

# Today is 02/04/2010
tar -zvcf /backups/users/users.20100204.tar.gz --listed-incremental=/backups/users/users.20100201.snar /users

Or do I have to make a copy of the snapshot for each day for the next day like so:

Code:

# Today is 02/02/2010
tar -zvcf /backups/users/users.20100202.tar.gz --listed-incremental=/backups/users/users.20100201.snar /users

cp /backups/users/users.20100201.snar /backups/users/users.20100202.snar

.....

# Today is 02/03/2010
tar -zvcf /backups/users/users.20100203.tar.gz --listed-incremental=/backups/users/users.20100202.snar /users

cp /backups/users/users.20100202.snar /backups/users/users.20100203.snar

.....

# Today is 02/04/2010
tar -zvcf /backups/users/users.20100204.tar.gz --listed-incremental=/backups/users/users.20100203.snar /users

cp /backups/users/users.20100203.snar /backups/users/users.20100204.snar

Or do I have to do something different?

TIA

choogendyk · 02-04-2010, 08:20 PM

Yup. It's a bit confusing. Read it over slowly another time or two, and you'll get it.

For abbreviation and ease of reference, let's say on day 0 you do tar0 and snar0. That's now a level 0 backup.

Day 1. Suppose you want to do a level 1 based on that level 0. Make a new copy of the snar0. Then make a tar1 referencing that new copy, which we will now call snar1. tar1 is a level 1 incremental backup. It's reference point is snar1, and it is an incremental from snar0.

Day 2. You decide to continue incrementally. Make a new copy of snar1, call it snar2. Now make a new tar2 using snar2. It will be an incremental level 2 and snar2 is its reference point. It is based on the incremental 1 referenced in snar1.

Day 3. Here you decide you don't want to have to go back through all the levels of incremental. So you decide to go back to an incremental level 1. Make a new copy of snar0 and call it snar1-2 (the second level 1 backup of the cycle). Make a new tar1-2 pointing to this. It will update snar1-2 to be the new referenece point for tar1-2.

See the pattern? You can come up with your own labeling scheme. Maybe call it users_0.20100201.tar and snar to indicate that it is a level 0 with that date stamp. If on the 7th of the month you want to go back to a level 1, you would copy the level 0 snar from the 1st of the month, call it users_1.20100207.snar, and then run a new tar users_1.20100207.tar pointing at that. The snar will get updated along with the tar so that it will contain the information for the level 1 incremental.

HTH

choogendyk · 02-04-2010, 08:24 PM

As an alternative, you could adopt Amanda and let it manage all that incremental stuff for you.

It would still use gnutar if you configured it that way. And you could do the backups to disk if you like or to tapes or out to the "cloud" (as in, say, Amazon S3).

digity · 02-04-2010, 09:26 PM

Thanks for the replies, but I'm still a bit confused. The language and technique used to explain the process is what I'm having trouble with. In addition I have an idea of how I want and think it should work and since the suggested method doesn't seem identical to what's in my head and my usually smart brain is shutting down. I imagine a pass the baton/pick up where the previous day left off type of simple incremental system - just back up the new/modified stuff made after the last backup.

Can you put it in laymen terms and provide examples like I did and leave out the level 0 stuff (the level stuff is throwing me off)??

sohail0399 · 02-05-2010, 12:16 AM

i dont know, if am i correct to understand?

first copy with fresh date, and then tar with new copy.

choogendyk · 02-05-2010, 07:19 AM

OK, I'm going to do this in two posts. Brief script answer in second post, but you have to understand the level stuff to really understand backups. It's fundamental. Language like incremental can vary from one backup application to another. See http://wiki.zmanda.com/index.php/FAQ...da_use_them%3F for an explanation and comparison of different terminology.

Here's one reason it is important to understand levels. Suppose you do a full (level 0) at the beginning of the month, and then do "incremental" backups every day through the rest of the month (which is your original request), where, by incremental you mean only the things that have changed that day. Now, in order to recover your system to its condition on the 15th of the month, you have to recover the original full followed by recovering every daily backup from then until the 15th. Thinking in levels, you see that you have backup levels 0,1,2,3,4,5,6, etc. up to the end of the month. If you had to recover for the 28th of the month, then, yeah, . . . what a pain.

Someone else might be thinking of using what they call "differential" backups. For that you do your full at the beginning of the month. Then you do a backup every day of the month that backs up everything that has changed since the original full (not just what changed that day). This makes your recoveries easy. You only have to recover the full backup and the one backup for the day you want to get to. But your daily backups get larger and larger as the month goes on. So, you haven't achieved the space savings expected from doing incremental backups. Thinking in levels, you see that you have backup levels 0,1,1,1,1,1, etc. to the end of the month.

And that is why people get into more complex plans. You might do a level 0 at the beginning of the month (that's a full). Then you do increasing levels ("incremental") for the first week, and drop back to a level 1 at the beginning of the second week, increasing again. That is a compromise that keeps the size of the backups smaller while not requiring you to go through any more than a week of backups to do a recovery. Thinking in levels, you see that you have 0,1,2,3,4,5,6, 1,2,3,4, etc. to the end of the month. If you skip weekends, it would be 0,1,2,3,4, 1,2,3,4,5, 1,2 etc. to the end of the month.

You can do a man page on dump and get an explanation of levels with a reference to the Towers of Hanoi strategy, which is an attempt to optimize the trade offs between backup size and number of backups you have to recover to restore a system. This is one reason why people end up using some kind of backup software, such as Amanda, to plan and organize their backups.

I should note that things like the Towers of Hanoi were more a topic of interest when backups were almost always to tape and sysadmins had to manually change each individual tape. Going through 28 tapes to do a recovery would be almost unthinkably tedious. Now, with inexpensive TB sized disks it is easier to script strategies, and you just have to let the script run and do its thing. But it still takes scripting or software. If you try to do it by hand, one command at a time, it is still tedious and error prone.

OK, enough for now. Commands to do what you asked coming up.

choogendyk · 02-05-2010, 07:45 AM

Commands to do what you asked.

If you want to do your full at the beginning of the month and incremental backups ever subsequent day catching only what changed that day, then ...

Either of your command sequences will work just as they are. If you aren't going to refer back to earlier snapshots at later dates in the month, then you don't need to copy them, but it doesn't hurt either. It works either way.

You only need the copies if you want to refer back to them later in the month, and that only comes into play if you choose a more complex strategy. In other words if you choose any of the alternate strategies discussed in my previous post.

If you are going to do the copies, I would do them as the first step on the day it is going to be used. That makes more logical sense and is more flexible for more complex strategies where you would be copying different snapshots rather than just the immediately previous one.

I hope you don't feel cheated.

After looking at your commands, it didn't seem necessary to repeat them here. For what you want, they work.

digity · 02-08-2010, 03:42 PM

my gawd, you deserve more than a thanks on that one, you deserve a lap dance (from a professional exotic dancer of course, not me)!

Thanks, that was extremely helpful and extremely clear.

From the looks of my backups, I'm actually doing differential backups (all the file sizes are the same) - epic failure! But thanks to your help I know how to fix it

Thanks again!