Differential and incremental backups
 

I've come across a somewhat confusing situation while trying to put together a new backup scheme.

The Mandriva Drakxtools Guide says that the difference between incremental and differential backups is as follows:

However, two Linux books that I own have these the other way round so that an incremental backup saves changes since the last full backup while a differential backup saves only the day to day changes.

I wondered which set of definitions is correct and whether Mandriva will follow its own convention regardless. :scratch:

In both cases you start with a full backup. This contains every file on the disk.

Every differential backup contains all of the files that have changed since the full backup.

Incremental backups contain only the files that have changed since the last backup whether the last backup was full or incremental.

This is how the backup software implements this:
When you perform a full backup all of the files are marked as having been backed up. When a file is created or modified it is marked as not being backed up. The differential backup copies all of the files that are marked as not backed up and does nothing else. The incremental backup copies all of the files that are marked as not backed up then it marks all of those files as having been backed up. This way the incremental backup will not make a second copy of a file that only changed since the last full backup.

Differential backups tend to be larger than incremental backups for this reason.

The most important fact about differential versus incremental backups is this: almost every business uses incremental backups, not differential backups. When you see a business using differential backups they probably have an inexperienced system administrator.

As for myself, I try to always make full backups every day on my clients' machines that run UNIX or Linux or VMS if time permits. (Yes, VMS.) I only resort to incremental backups when a full backup takes more time than is available such as if a full backup takes twenty hours. Then incremental backups are required in order to finish before the beginning of business the next day. In these cases full backups are performed on Friday night or any time Saturday and incremental backups are made the other six days of the week.

My clients' Windows machines are backed up according to what their Windows based backup software wants to do. That is typically a weekly full and daily incrementals.

In my home and in my home office I always perform full backups. My personal machines get backed up about once a month. My business machines get backed up after I've spent several hours entering data, so that works out to about three per week.

Full = Backs up all files specified in backup set.
Differential = Performs a backup of all files that changed from previous Full backup.
Incremental = Backs up all changed files from previous backup, whether it's a Full, Differential or Incremental.

If you read it differently elsewhere, the person or thing who wrote such things is smoking crack.

Thanks for a very clear explanation. :)

I'm backing up the my home desktop, but as the multimedia files occupy a lot of disk space but rarely change, only get added to, I think incremental backups would be a good way to go.

Thanks again,

Mikie

Remember, media files might not change but if you play or open them, it modifies the access time so the backups will believe they changed due to that, so they get added to the next round of backups.

Probably a good way to back these up is to omit the backup program from backing them up if only the access time has changed, if given such option.

Incremental backups can be tedious to manage by hand, such as if you are just using tar to make your archives. I use incrementals on Windows machines because the backup software (Norton Ghost or Brightstore Arcserve for example) do all of the tedious archive file management and present a nice GUI when you want to restore a file.

Exactly. Amanda and Bacula are probably the most popular open source that will do all the tedious work for you. Just select the files and press ok, come back when it's done restoring. ;)

As the the description of differential backups implies, to restore, you restore from the full backup and then the last differential backup. Using incremental backups, you restore from each in succession. In some situations, a full restore as quickly as possible may be desirable.

One thing to keep in mind is whether acls and attributes are backed up as well. A gui backup program may use a program like tar or star to create the actual archives. The star program is similar to tar but also backs up security attributes and acls. However a recent version of tar may do the same. You might want to create some test files with attributes & acls and see if you can restore them with these attributes preserved.

If you only have one or two machines to backup, your Mandriva wizard may be fine. There is a program that I like called "dar" and it's gui front end "kdar". You can set up a full backup job and an incremental backup job in kdar and then export the jobs as bash commands. These commands can then be dropped into /etc/cron.daily/, /etc/cron.weekly/, etc. to perform automated backups. This would work best if you are backing up to an external drive or a network share.
The slices of the backup can than be burned to cd or dvd at a convenient time.

Good Luck!

@trickykid: I have wanted to learn about Amanda and Backula but other things get prioritized ahead of that. For example, right now I'm struggling with IPSec. There's always something to learn.

@jschiwal: I hadn't heard about star. Thanks.

I played around with it a bit after posting. I found that you need the options "-H=exustar -acl -xattr -xfflags" to back up ACLs, attributes and file attributes.

It took a while to figure out that -xfflags are what provides a backup of the attributes (such as "i" & "s" set with chattr).

Regular tar does backup file attributes but not file ACLs. If you wanted to use tar or another backup program that doesn't save file ACLs, then you could use the getfacl options "-R --skip-base" to produce a list of files with ACLs and what they are.

One way of managing stress and time is by getting out from under manually managed backups. Over a number of years I had developed some pretty sophisticated scripts with config files and everything to do my backups. But it still left a fair bit of work keeping tabs on backups, tweaking configs, tracking tapes, etc. I finally made the switch to running Amanda nearly 2 years ago. It took a bit of concerted effort to get it set up and get all my servers configured and have it connected to the tape library and so on. But, it totally freed me from the regular drudge of backups. I can take them out of my "personal cron" and focus on other projects.

You still have to be responsible for backups, but it becomes more a matter of skimming email reports for errors or red flags and checking in on things periodically to reassure yourself that they are running properly (with a periodic test restore to be really sure).

From the Linux man pages for tar - the GNU version:

--acls
this option causes tar to store each file's ACLs in the archive.

http://linux.die.net/man/1/tar

maybe check `tar --version`, since many installations may actually have GNU tar as their primary tar.

Your tar is better than my tar!

If amanda backs up all of the attributes and acls that you need backed up, that would be great. However, a tool like amanda may use tar or star to produce the archive files. It would be a good idea find if ACLs and X_ATTRs are backed up as well.

File attributes probably are, or the program wouldn't be usable if you use se_linux. It's possible that the program you use may need to be rebuilt with these enabled. Testing and reading the README are important.

Bacula has a good tip on how to backup ACL's just in case you want to test to make sure Bacula is backing ACL's up correctly and or if you want to back them up separately, in order to restore if necessary. Currently only for ext3 or xfs though, but a start.

http://www.bacula.org/fr/dev-manual/...00000000000000

Since Amanda uses native tools to do the backups, you just need to confirm that your tools work the way you want. If your distribution had an inadequate tar that didn't do acls, then you could install a better tar that did acls. That's easy. Confirm that it works. Then you know that Amanda will work. The standard recommendation is gnu tar.

Of course, no matter what you use for backups, it is always the case that you should test recovers to make sure you can really get your files back with the attributes you require. Even if 25 people tell you it works, test it yourself. There just might be something unique about your system or something you haven't installed quite right.