LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 07-04-2008, 03:42 PM   #1
mikieboy
Member
 
Registered: Apr 2004
Location: Warrington, Cheshire, UK
Distribution: Linux Mint 12 LXDE
Posts: 555

Rep: Reputation: 33
Differential and incremental backups


I've come across a somewhat confusing situation while trying to put together a new backup scheme.

The Mandriva Drakxtools Guide says that the difference between incremental and differential backups is as follows:

Quote:
Choosing Use Incremental Backups will only save files which have been changed or added since the last backup operation. Choosing Use Differential Backups will only save files which have been changed or added since the first backup operation (also known as the “base” backup). This last option takes more space than the first one, but allows you to more easily restore the system “as it was” at any given point in time for which a differential backup was made.
However, two Linux books that I own have these the other way round so that an incremental backup saves changes since the last full backup while a differential backup saves only the day to day changes.

I wondered which set of definitions is correct and whether Mandriva will follow its own convention regardless.
 
Old 07-04-2008, 04:40 PM   #2
stress_junkie
Senior Member
 
Registered: Dec 2005
Location: Massachusetts, USA
Distribution: Ubuntu 10.04 and CentOS 5.5
Posts: 3,873

Rep: Reputation: 331Reputation: 331Reputation: 331Reputation: 331
In both cases you start with a full backup. This contains every file on the disk.

Every differential backup contains all of the files that have changed since the full backup.

Incremental backups contain only the files that have changed since the last backup whether the last backup was full or incremental.

This is how the backup software implements this:
When you perform a full backup all of the files are marked as having been backed up. When a file is created or modified it is marked as not being backed up. The differential backup copies all of the files that are marked as not backed up and does nothing else. The incremental backup copies all of the files that are marked as not backed up then it marks all of those files as having been backed up. This way the incremental backup will not make a second copy of a file that only changed since the last full backup.

Differential backups tend to be larger than incremental backups for this reason.

The most important fact about differential versus incremental backups is this: almost every business uses incremental backups, not differential backups. When you see a business using differential backups they probably have an inexperienced system administrator.

As for myself, I try to always make full backups every day on my clients' machines that run UNIX or Linux or VMS if time permits. (Yes, VMS.) I only resort to incremental backups when a full backup takes more time than is available such as if a full backup takes twenty hours. Then incremental backups are required in order to finish before the beginning of business the next day. In these cases full backups are performed on Friday night or any time Saturday and incremental backups are made the other six days of the week.

My clients' Windows machines are backed up according to what their Windows based backup software wants to do. That is typically a weekly full and daily incrementals.

In my home and in my home office I always perform full backups. My personal machines get backed up about once a month. My business machines get backed up after I've spent several hours entering data, so that works out to about three per week.

Last edited by stress_junkie; 07-04-2008 at 04:58 PM.
 
Old 07-04-2008, 05:12 PM   #3
trickykid
Guru
 
Registered: Jan 2001
Posts: 24,133

Rep: Reputation: 197Reputation: 197
Full = Backs up all files specified in backup set.
Differential = Performs a backup of all files that changed from previous Full backup.
Incremental = Backs up all changed files from previous backup, whether it's a Full, Differential or Incremental.

If you read it differently elsewhere, the person or thing who wrote such things is smoking crack.
 
Old 07-04-2008, 05:21 PM   #4
mikieboy
Member
 
Registered: Apr 2004
Location: Warrington, Cheshire, UK
Distribution: Linux Mint 12 LXDE
Posts: 555

Original Poster
Rep: Reputation: 33
Thanks for a very clear explanation.

I'm backing up the my home desktop, but as the multimedia files occupy a lot of disk space but rarely change, only get added to, I think incremental backups would be a good way to go.

Thanks again,

Mikie
 
Old 07-04-2008, 05:30 PM   #5
trickykid
Guru
 
Registered: Jan 2001
Posts: 24,133

Rep: Reputation: 197Reputation: 197
Quote:
Originally Posted by mikieboy View Post
Thanks for a very clear explanation.

I'm backing up the my home desktop, but as the multimedia files occupy a lot of disk space but rarely change, only get added to, I think incremental backups would be a good way to go.

Thanks again,

Mikie
Remember, media files might not change but if you play or open them, it modifies the access time so the backups will believe they changed due to that, so they get added to the next round of backups.

Probably a good way to back these up is to omit the backup program from backing them up if only the access time has changed, if given such option.
 
Old 07-04-2008, 05:46 PM   #6
stress_junkie
Senior Member
 
Registered: Dec 2005
Location: Massachusetts, USA
Distribution: Ubuntu 10.04 and CentOS 5.5
Posts: 3,873

Rep: Reputation: 331Reputation: 331Reputation: 331Reputation: 331
Incremental backups can be tedious to manage by hand, such as if you are just using tar to make your archives. I use incrementals on Windows machines because the backup software (Norton Ghost or Brightstore Arcserve for example) do all of the tedious archive file management and present a nice GUI when you want to restore a file.

Last edited by stress_junkie; 07-04-2008 at 05:47 PM.
 
Old 07-04-2008, 05:55 PM   #7
trickykid
Guru
 
Registered: Jan 2001
Posts: 24,133

Rep: Reputation: 197Reputation: 197
Quote:
Originally Posted by stress_junkie View Post
Incremental backups can be tedious to manage by hand, such as if you are just using tar to make your archives. I use incrementals on Windows machines because the backup software (Norton Ghost or Brightstore Arcserve for example) do all of the tedious archive file management and present a nice GUI when you want to restore a file.
Exactly. Amanda and Bacula are probably the most popular open source that will do all the tedious work for you. Just select the files and press ok, come back when it's done restoring.
 
Old 07-04-2008, 06:54 PM   #8
jschiwal
Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654
As the the description of differential backups implies, to restore, you restore from the full backup and then the last differential backup. Using incremental backups, you restore from each in succession. In some situations, a full restore as quickly as possible may be desirable.

One thing to keep in mind is whether acls and attributes are backed up as well. A gui backup program may use a program like tar or star to create the actual archives. The star program is similar to tar but also backs up security attributes and acls. However a recent version of tar may do the same. You might want to create some test files with attributes & acls and see if you can restore them with these attributes preserved.

If you only have one or two machines to backup, your Mandriva wizard may be fine. There is a program that I like called "dar" and it's gui front end "kdar". You can set up a full backup job and an incremental backup job in kdar and then export the jobs as bash commands. These commands can then be dropped into /etc/cron.daily/, /etc/cron.weekly/, etc. to perform automated backups. This would work best if you are backing up to an external drive or a network share.
The slices of the backup can than be burned to cd or dvd at a convenient time.

Good Luck!

Last edited by jschiwal; 07-04-2008 at 07:13 PM.
 
Old 07-04-2008, 07:31 PM   #9
stress_junkie
Senior Member
 
Registered: Dec 2005
Location: Massachusetts, USA
Distribution: Ubuntu 10.04 and CentOS 5.5
Posts: 3,873

Rep: Reputation: 331Reputation: 331Reputation: 331Reputation: 331
@trickykid: I have wanted to learn about Amanda and Backula but other things get prioritized ahead of that. For example, right now I'm struggling with IPSec. There's always something to learn.

@jschiwal: I hadn't heard about star. Thanks.
 
Old 07-05-2008, 07:11 AM   #10
jschiwal
Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654
I played around with it a bit after posting. I found that you need the options "-H=exustar -acl -xattr -xfflags" to back up ACLs, attributes and file attributes.

It took a while to figure out that -xfflags are what provides a backup of the attributes (such as "i" & "s" set with chattr).

Regular tar does backup file attributes but not file ACLs. If you wanted to use tar or another backup program that doesn't save file ACLs, then you could use the getfacl options "-R --skip-base" to produce a list of files with ACLs and what they are.
 
Old 07-05-2008, 05:31 PM   #11
choogendyk
Senior Member
 
Registered: Aug 2007
Location: Massachusetts, USA
Distribution: Solaris 9 & 10, Mac OS X, Ubuntu Server
Posts: 1,189

Rep: Reputation: 105Reputation: 105
Quote:
Originally Posted by stress_junkie View Post
@trickykid: I have wanted to learn about Amanda and Backula but other things get prioritized ahead of that. For example, right now I'm struggling with IPSec. There's always something to learn.
One way of managing stress and time is by getting out from under manually managed backups. Over a number of years I had developed some pretty sophisticated scripts with config files and everything to do my backups. But it still left a fair bit of work keeping tabs on backups, tweaking configs, tracking tapes, etc. I finally made the switch to running Amanda nearly 2 years ago. It took a bit of concerted effort to get it set up and get all my servers configured and have it connected to the tape library and so on. But, it totally freed me from the regular drudge of backups. I can take them out of my "personal cron" and focus on other projects.

You still have to be responsible for backups, but it becomes more a matter of skimming email reports for errors or red flags and checking in on things periodically to reassure yourself that they are running properly (with a periodic test restore to be really sure).
 
Old 07-05-2008, 05:43 PM   #12
choogendyk
Senior Member
 
Registered: Aug 2007
Location: Massachusetts, USA
Distribution: Solaris 9 & 10, Mac OS X, Ubuntu Server
Posts: 1,189

Rep: Reputation: 105Reputation: 105
Quote:
Originally Posted by jschiwal View Post
Regular tar does backup file attributes but not file ACLs. If you wanted to use tar or another backup program that doesn't save file ACLs, then you could use the getfacl options "-R --skip-base" to produce a list of files with ACLs and what they are.
From the Linux man pages for tar - the GNU version:

--acls
this option causes tar to store each file's ACLs in the archive.

http://linux.die.net/man/1/tar

maybe check `tar --version`, since many installations may actually have GNU tar as their primary tar.
 
Old 07-06-2008, 04:18 PM   #13
jschiwal
Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654
Your tar is better than my tar!

If amanda backs up all of the attributes and acls that you need backed up, that would be great. However, a tool like amanda may use tar or star to produce the archive files. It would be a good idea find if ACLs and X_ATTRs are backed up as well.

File attributes probably are, or the program wouldn't be usable if you use se_linux. It's possible that the program you use may need to be rebuilt with these enabled. Testing and reading the README are important.
 
Old 07-06-2008, 09:15 PM   #14
trickykid
Guru
 
Registered: Jan 2001
Posts: 24,133

Rep: Reputation: 197Reputation: 197
Quote:
Originally Posted by jschiwal View Post
Your tar is better than my tar!

If amanda backs up all of the attributes and acls that you need backed up, that would be great. However, a tool like amanda may use tar or star to produce the archive files. It would be a good idea find if ACLs and X_ATTRs are backed up as well.

File attributes probably are, or the program wouldn't be usable if you use se_linux. It's possible that the program you use may need to be rebuilt with these enabled. Testing and reading the README are important.
Bacula has a good tip on how to backup ACL's just in case you want to test to make sure Bacula is backing ACL's up correctly and or if you want to back them up separately, in order to restore if necessary. Currently only for ext3 or xfs though, but a start.

http://www.bacula.org/fr/dev-manual/...00000000000000
 
Old 07-06-2008, 09:27 PM   #15
choogendyk
Senior Member
 
Registered: Aug 2007
Location: Massachusetts, USA
Distribution: Solaris 9 & 10, Mac OS X, Ubuntu Server
Posts: 1,189

Rep: Reputation: 105Reputation: 105
Since Amanda uses native tools to do the backups, you just need to confirm that your tools work the way you want. If your distribution had an inadequate tar that didn't do acls, then you could install a better tar that did acls. That's easy. Confirm that it works. Then you know that Amanda will work. The standard recommendation is gnu tar.

Of course, no matter what you use for backups, it is always the case that you should test recovers to make sure you can really get your files back with the attributes you require. Even if 25 people tell you it works, test it yourself. There just might be something unique about your system or something you haven't installed quite right.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Theory behind full, differential and incremental backups? mohammednv Linux - Server 5 06-15-2008 08:29 AM
Creating Differential Backups with Tar FeX_05 Linux - Newbie 1 04-29-2008 03:55 AM
Encrypted incremental backups? DaneelGiskard Linux - Security 3 07-25-2007 06:57 AM
incremental & differential backups to CD? SheldonPlankton Linux - General 0 06-30-2004 05:33 PM
Incremental tar backups Pepe Linux - General 5 03-18-2002 03:31 AM


All times are GMT -5. The time now is 11:15 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration