LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 09-24-2007, 01:10 PM   #1
lrt
Member
 
Registered: Sep 2007
Location: Chattanooga, TN
Distribution: Ubuntu
Posts: 84

Rep: Reputation: 15
using tar --listed-incremental as backup


hi,

i'm using tar's --listed-incremental option to perform incremental backups. i have made this work using test directories on my linux machine. however, the backup jobs are backing up three mounted windows cifs shares and are placing the .tar files in a network attached storage. this works fine for a full backup using the --listed-incremental option to create a snapshot file. to get an incremental backup i simply have to run the tar command again specifying the same snapshot file, and tar will only archive the modified files. like i said, this works fine when i run it on test directories in my linux machine, but when i run this on the smb shares it wants to do a full backup every time. and the only explanation i can come up with is that my snapshot file has been modified, but i never changed the contents of the snapshot file, only its name. i also changed the mounted shares to a different directory, but then again, i did not change contents of any file. my users may have changed some of the files' contents (which is fine) but i know for sure they have not changed all of them for tar to think that it needs to backup every single file as in a full backup. i am hesitant to run a full backup again to create a fresh snapshot file (i think i can do this, however, without actually creating the archive) because it will take a long time.

so what's my questions??? does renaming or moving the snapshot file or the mounted shares that need to be archived cause tar to think the files have changed and therefore the snapshot file becomes useless if it was created before?

any ideas on this is very welcome! thanks.

lrt
 
Old 09-25-2007, 06:33 AM   #2
merijnv
LQ Newbie
 
Registered: May 2007
Location: The Netherlands
Distribution: ubuntu-desktop 8.10 and ubuntu-server 8.10
Posts: 21

Rep: Reputation: 15
I do not know your exact use of tar, however, I have a suggestion for backups: maybe you'd try to use another approach to snapshotting backups by using something like rdup.

--
Merijn Vogel
 
Old 09-25-2007, 08:34 AM   #3
choogendyk
Senior Member
 
Registered: Aug 2007
Location: Massachusetts, USA
Distribution: Solaris 9 & 10, Mac OS X, Ubuntu Server
Posts: 1,189

Rep: Reputation: 105Reputation: 105
I presume you are using gnutar. It ought to be able to do it. That's actually the way amanda does backups of smb shares, including incrementals. You might want to check out how they do it. I haven't dug into that section of the program, so I'm not sure of the details, but it is open source.

You could also just adopt amanda and let it manage your gnutar backups of the smb shares.

http://www.linuxquestions.org/bookmarks/tags/amanda
 
Old 09-25-2007, 01:57 PM   #4
lrt
Member
 
Registered: Sep 2007
Location: Chattanooga, TN
Distribution: Ubuntu
Posts: 84

Original Poster
Rep: Reputation: 15
i've looked at amanda. i'm a little lazy when it comes to learning new software. i like to write my own (although i have to admit i'm not very good at it). i am looking at rdup right now (thank you merijnv). i like the idea and its very low level. stay tuned for updates, if you have any other suggestions please feel free.

lrt
 
Old 09-25-2007, 08:13 PM   #5
choogendyk
Senior Member
 
Registered: Aug 2007
Location: Massachusetts, USA
Distribution: Solaris 9 & 10, Mac OS X, Ubuntu Server
Posts: 1,189

Rep: Reputation: 105Reputation: 105
I can understand that. Software is supposed to save you time and effort, or do something for you that you couldn't do otherwise.

I used my own scripts for backups for years; and, if I may say so myself, they were fairly sophisticated. Eventually, I decided I needed to find something that wasn't so labor intensive. I settled on Amanda and a tape library.

There is always an initial overhead of effort to get into a new application, and often it is proportional to the complexity of what it does, though it's also related to how well the software is designed. The initial overhead for Amanda wasn't bad; and, for me, the payoff has been huge. It just runs, and I hardly ever have to do anything. Several months ago, I had a tape drive breakdown. Amanda just kept running. It saw the drive gone, dropped back to incrementals, and saved them to my holding disk. Then when the drive was back up a couple of days later, it flushed everything out to tape. I never had to intervene at all. It just worked.

Anyway, that's the solution I chose. Someone else would need to evaluate their alternatives and choose the appropriate solution for their own situation and needs.
 
Old 09-26-2007, 07:19 AM   #6
lrt
Member
 
Registered: Sep 2007
Location: Chattanooga, TN
Distribution: Ubuntu
Posts: 84

Original Poster
Rep: Reputation: 15
thank you for that insightful comment choogendyk!
 
Old 09-26-2007, 09:21 AM   #7
lrt
Member
 
Registered: Sep 2007
Location: Chattanooga, TN
Distribution: Ubuntu
Posts: 84

Original Poster
Rep: Reputation: 15
merijnv,

have you used rdup before? i'm having trouble figuring out how to use this program. the man pages a little vague. can u provide example commands and explain to me what they do? thanks.
 
Old 09-26-2007, 11:16 AM   #8
lrt
Member
 
Registered: Sep 2007
Location: Chattanooga, TN
Distribution: Ubuntu
Posts: 84

Original Poster
Rep: Reputation: 15
The GNU tar manual has a warning about using --after-date and --newer-mtime for incremental backup, but it doesn't explain why. Could anyone enlighten me?

The warning is at the bottom: http://www.gnu.org/software/tar/manu...ode/after.html

"Please Note: --after-date and --newer-mtime should not be used for incremental backups. See Incremental Dumps, for proper way of creating incremental backups."

lrt
 
Old 09-26-2007, 08:33 PM   #9
choogendyk
Senior Member
 
Registered: Aug 2007
Location: Massachusetts, USA
Distribution: Solaris 9 & 10, Mac OS X, Ubuntu Server
Posts: 1,189

Rep: Reputation: 105Reputation: 105
I guess there are two issues here. One is, "follow the link." It explains how to do proper incrementals using gnutar.

But it sounds like you are just wondering why there is a problem with using what would seem like a reasonable method (--after-date and --newer-mtime). For answers to this, you just have to watch the discussion lists of backup software packages that try to write their own backup procedures (as opposed to using native tools like Amanda does). They end up getting into extended discussions of the situations that fail for incremental backups, possible workarounds, and the interactions of the various time stamps.

For a couple of simple examples, try deleting a few files from a directory and then doing an incremental followed by a full recovery in which you recover the full and then recover the incrementals. Use the wrong procedures and the deleted files will still be there. For another example, try moving some files from one directory to another. Then see how incremental backup and recovery works. Use the wrong methods and the recovered system will not properly reflect the move.

Using mtime won't catch changes to permissions. That would be ctime. Dump bases it's incrementals on ctime. However, common lore (Linus) says that dump is broken in Linux (for other reasons). Ufsdump in Solaris uses additional information. I've tried experiments with it, and it actually catches deleted files and files moved from one directory to another properly.

If you google "mtime atime ctime", you'll find an interesting link or two:

Definitions: http://www.brandonhutchinson.com/ctime_atime_mtime.html

Backup issues: http://seer.support.veritas.com/docs/200644.htm
 
Old 09-27-2007, 03:02 AM   #10
merijnv
LQ Newbie
 
Registered: May 2007
Location: The Netherlands
Distribution: ubuntu-desktop 8.10 and ubuntu-server 8.10
Posts: 21

Rep: Reputation: 15
The author of rdup has a website with examples. I must admit that I haven't used rdup myself, I used the previous tool by this author named hdup.
More info about rdup may be found on: http://www.miek.nl/projects/rdup/index.html

Good luck backing up (I should raise the priority on implementing a backup-strategy myself).

--
Merijn Vogel
 
Old 09-27-2007, 09:14 AM   #11
lrt
Member
 
Registered: Sep 2007
Location: Chattanooga, TN
Distribution: Ubuntu
Posts: 84

Original Poster
Rep: Reputation: 15
here's another hint: make sure all system clocks are synchronized between machines backing to and from. this must be done if your backups will rely on checking a, c, m times. i don't have much else to say. thanks for everyones suggestions. i think i'm going to stick with tar and run it with a bash script or eventually upgrade it to a perl script, with web interface for easier administration :-)

lrt
 
Old 06-18-2009, 12:51 PM   #12
orange.toaster
LQ Newbie
 
Registered: Jun 2009
Posts: 9

Rep: Reputation: 0
I solved a very similar problem and got incremental backups working over a cifs share. http://www.linuxquestions.org/questi...8/#post3578596
 
Old 06-19-2009, 07:55 AM   #13
lrt
Member
 
Registered: Sep 2007
Location: Chattanooga, TN
Distribution: Ubuntu
Posts: 84

Original Poster
Rep: Reputation: 15
i solved this problem by using BackupPC for all our backups. this software is still being actively developed and it works beautifully.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
incremental backup using tar eaglegst Linux - Software 9 02-01-2009 05:00 PM
tar --listed-incremental & filellist alexandernewald Linux - Software 0 10-31-2005 06:05 AM
Incremental Backup (tar command) maginotjr Slackware 4 07-21-2005 01:04 AM
Incremental backup script - Tar problem joshheald Linux - Software 1 09-28-2004 10:02 AM
Getting date format right in tar incremental backup jonr Linux - Software 3 08-19-2004 09:53 AM


All times are GMT -5. The time now is 03:01 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration