LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 07-13-2010, 09:51 PM   #1
indubitableness
Member
 
Registered: Feb 2009
Distribution: Slackware64 14.0 multilib
Posts: 62

Rep: Reputation: 15
Three stage backup with rsync?


I've been trying to make a three stage backup with stage 0 being a full monthly back up, stage 1 being a weekly backup, and stage 2 being a daily backup. I've been trying very hard to use rsync for this but sorting files by date is proving to be problematic. Sometimes it seems to work from the command line directly, but the same command causes errors and warnings from a script while entirely failing to sort the correct files.

The common example I see for this involves commands like this:

Code:
rsync -Rav `find /home/ -ctime -7 -print` /path/to/home_backup
The problem seems to be that since the user directories in /home contain files that have been altered within the time frame specified the whole directory is matched first which means that the whole directory is recursively archived as opposed to just the changed files.

I've also seen examples using the --files-from tag using the same find parameters and this one seems to ALMOST work but gives me strange warnings and fails to run at all when launched from a script.

Many of the things I've googled about using rsync to backup stuff by the date modified involves a rather snarky "You're missing the point of rsync!" to which I respond by yelling at my computer monitor followed by "JUST TELL ME WHAT I NEED TO KNOW!" I understand that rsync is meant to take care of incremental backups on it's own, that's why I want to use it specifically for a traditional 3 stage backup scheme.

What am I doing wrong?

Last edited by XavierP; 07-19-2010 at 06:09 PM. Reason: Language tidied up
 
Old 07-13-2010, 10:34 PM   #2
bryanl
Member
 
Registered: Dec 2003
Posts: 84

Rep: Reputation: 33
- looks like maybe copying recursively with just one file might make things interesting ...

For what you want to do, you might look at Brice Burgess' idea to create a series of time stamped backup destinations then use the '--link-dest' option so unchanged files are only links. see at linux.com.

I changed to cp -al step to the rsync --link-dest to modernize the script a bit. I also tested for running as root and a few other things. I have done some thinking about adding a three stage snapshot for monthly, weekly, and daily by using a runtime parameter and a prefix on the backup directory so I could still use the most recent for the latest snapshot but maintain an appropriate number of snapshots for each level. That would only require the rotations loop to look for target directories with the appropriate prefix.

At any rate, the link provided should give you some ideas, I hope.
 
1 members found this post helpful.
Old 07-13-2010, 11:21 PM   #3
Valery Reznic
ELF Statifier author
 
Registered: Oct 2007
Posts: 666

Rep: Reputation: 132Reputation: 132
Quote:
Originally Posted by indubitableness View Post
I've been trying to make a three stage backup with stage 0 being a full monthly back up, stage 1 being a weekly backup, and stage 2 being a daily backup. I've been trying very hard to use rsync for this but sorting files by date is proving to be problematic. Sometimes it seems to work from the command line directly, but the same command causes errors and warnings from a script while entirely failing to sort the correct files.

The common example I see for this involves commands like this:

Code:
rsync -Rav `find /home/ -ctime -7 -print` /path/to/home_backup
The problem seems to be that since the user directories in /home contain files that have been altered within the time frame specified the whole directory is matched first which means that the whole directory is recursively archived as opposed to just the changed files.

I've also seen examples using the --files-from tag using the same find parameters and this one seems to ALMOST work but gives me strange warnings and fails to run at all when launched from a script.

Many of the things I've googled about using rsync to backup stuff by the date modified involves a rather snarky "You're missing the point of rsync!" to which I respond by yelling at my computer monitor followed by "JUST TELL ME WHAT I NEED TO KNOW!" I understand that rsync is meant to take care of incremental backups on it's own, that's why I want to use it specifically for a traditional 3 stage backup scheme.

What am I doing wrong?
adding '-type f' for find parameters may help

Last edited by XavierP; 07-19-2010 at 06:09 PM. Reason: Language tidied up
 
1 members found this post helpful.
Old 07-14-2010, 01:27 AM   #4
indubitableness
Member
 
Registered: Feb 2009
Distribution: Slackware64 14.0 multilib
Posts: 62

Original Poster
Rep: Reputation: 15
I think I have it figured out, but I'm still seeing some strangeness.

The final command I've come up with looks like this:

Code:
rsync -Rav $OtherOptions `find /home/ -type f -ctime 7 -print0 | xargs -0` /path/to/home_BCK
Where $OtherOptions are things like --exclude-from=file, --delete-excluded, and --delete.

It seems to be working correctly. It copies recursively and it seems to be copying ALL the directories, but it only populates those directories with files that have been modified in the last 7 days, which is exactly the behavior I want.

the -print0 | xargs -0 part of the command is meant to handle filenames with spaces in them and it took me an hour and a half of reading to figure that bit of it out.

I'm still seeing some strange behavior though. Even though the command seems to be behaving, for the most part, I'm seeing errors that look like this:

Code:
rsync: -: unknown option
rsync error: syntax or usage error (code 1) at main.c(1425) [client=3.0.7]
rsync: -: unknown option
rsync error: syntax or usage error (code 1) at main.c(1425) [client=3.0.7]
sending incremental file list
rsync: link_stat "/home/indubitableness/.wine/wineprefixes/Psychonauts/drive_c/Program" failed: No such file or directory (2)
rsync: link_stat "/root/Files/Double" failed: No such file or directory (2)
rsync: link_stat "/root/Fine" failed: No such file or directory (2)
rsync: link_stat "/root/Productions/Psychonauts/Profiles/Profile" failed: No such file or directory (2)
rsync: link_stat "/root/1/SavedGame2" failed: No such file or directory (2)
rsync: link_stat "/home/indubitableness/.wine/wineprefixes/Psychonauts/drive_c/Program" failed: No such file or directory (2)
rsync: link_stat "/root/Files/Double" failed: No such file or directory (2)
rsync: link_stat "/root/Fine" failed: No such file or directory (2)
rsync: link_stat "/root/Productions/Psychonauts/Profiles/Profile" failed: No such file or directory (2)
rsync: link_stat "/root/1/Profile" failed: No such file or directory (2)
rsync: link_stat "/root/1-" failed: No such file or directory (2)
rsync: link_stat "/root/Raz.ini" failed: No such file or directory (2)
rsync: link_stat "/home/indubitableness/.wine/wineprefixes/Psychonauts/drive_c/Program" failed: No such file or directory (2)
rsync: link_stat "/root/Files/Double" failed: No such file or directory (2)
rsync: link_stat "/root/Fine" failed: No such file or directory (2)
rsync: link_stat "/root/Productions/Psychonauts/Profiles/Profile" failed: No such file or directory (2)
rsync: link_stat "/root/1/SavedGame3" failed: No such file or directory (2)
rsync: link_stat "/home/indubitableness/.wine/wineprefixes/Psychonauts/drive_c/Program" failed: No such file or directory (2)
rsync: link_stat "/root/Files/Double" failed: No such file or directory (2)
rsync: link_stat "/root/Fine" failed: No such file or directory (2)
rsync: link_stat "/root/Productions/Psychonauts/Profiles/Profile" failed: No such file or directory (2)
rsync: link_stat "/root/1/SavedGame1" failed: No such file or directory (2)
rsync: link_stat "/home/indubitableness/.wine/wineprefixes/Psychonauts/drive_c/Program" failed: No such file or directory (2)
rsync: link_stat "/root/Files/Double" failed: No such file or directory (2)
rsync: link_stat "/root/Fine" failed: No such file or directory (2)
rsync: link_stat "/root/Productions/Psychonauts/Profiles/Profile" failed: No such file or directory (2)
rsync: link_stat "/root/1/SavedGame0" failed: No such file or directory (2)
rsync: link_stat "/home/indubitableness/.wine/wineprefixes/Psychonauts/drive_c/Program" failed: No such file or directory (2)
rsync: link_stat "/root/Files/Double" failed: No such file or directory (2)
rsync: link_stat "/root/Fine" failed: No such file or directory (2)
rsync: link_stat "/root/Productions/Psychonauts/Profiles/Profile" failed: No such file or directory (2)
rsync: link_stat "/root/1/Profile" failed: No such file or directory (2)
rsync: link_stat "/root/1-" failed: No such file or directory (2)
rsync: link_stat "/root/Raz" failed: No such file or directory (2)
rsync: link_stat "/home/indubitableness/.wine/wineprefixes/Psychonauts/drive_c/Program" failed: No such file or directory (2)
rsync: link_stat "/root/Files/Double" failed: No such file or directory (2)
rsync: link_stat "/root/Fine" failed: No such file or directory (2)
rsync: link_stat "/root/Productions/Psychonauts/AudioSettings.ini" failed: No such file or directory (2)
rsync: link_stat "/home/indubitableness/Torrent/[isoHunt]" failed: No such file or directory (2)
rsync: link_stat "/root/The" failed: No such file or directory (2)
rsync: link_stat "/root/Metal" failed: No such file or directory (2)
rsync: link_stat "/root/Gear" failed: No such file or directory (2)
rsync: link_stat "/root/Solid" failed: No such file or directory (2)
rsync: link_stat "/root/PSX" failed: No such file or directory (2)
rsync: link_stat "/root/Collection" failed: No such file or directory (2)
rsync: link_stat "/root/(NTSC-US).torrent" failed: No such file or directory (2)
rsync: link_stat "/home/indubitableness/.cache/vlc/art/artistalbum/Cake/Prolonging" failed: No such file or directory (2)
rsync: link_stat "/root/the" failed: No such file or directory (2)
rsync: link_stat "/root/Magic/art" failed: No such file or directory (2)
rsync: link_stat "/home/indubitableness/.cache/vlc/art/artistalbum/Cake/Fashion" failed: No such file or directory (2)
rsync: link_stat "/root/Nugget/art" failed: No such file or directory (2)
What is this "link_stat" error and why is rsync looking in my /root directory in the first place?

These all seem to be errors caused by the lack of escaped spaces, a problem I fixed with the "find ... -print0 | xargs -0" commands I mentioned before. The files in /home/indubitableness that have spaced in them are getting backed up with this command despite the error messages.

Furthermore I can't seem to figure out how to make rsync copy the contents of the source directory to the destination directory using this find command as the input for rsync. I know when using rsync normally doing /home/ as opposed to /home is supposed to mean "copy the contents of home" but I keep getting a "home" directory in the destination. Is this something I'm just going to have to live with or can I force the behavior I'm looking for?


Edit:

This is the script I've been working on.

http://lfs.pastebin.com/x8LaH1G5

Note: The line '"${OUTPUT_DIR[$dirCount]}"' on line 86 includes the quotes because I was desperately casting around for a solution to the errors I'm seeing. It didn't help to put them in quotes.

The command doesn't seem to above posted errors from the command line. I also have a test script intended to simulate the variable substitution used in the above script in an attempt to replicate the errors. It looks like this:

Code:
#!/bin/sh

SOURCE[1]=/home/
DEST[1]=/media/Sanctuary/DAILY_BACKUP_INCREMENT/home_rsync/vroomfondel/ 
src_destNum=1

rsync --dry-run -Rav --delete --exclude-from=EXCLUDE_SYNC --delete-excluded `find ${SOURCE[$src_destNum]} -type f -ctime -1 -print0 | xargs -0` ${DEST[$src_destNum]}
The errors existed in the original script before I relegated all of rsync's options to variables and so I haven't yet put them in variables in the test script. The test script appears to run fine and I only seem to be getting the "rsync: -: unknown option" followed by the litany of "link_stat" errors when it moves into copying the /home directory. I am not seeing any of these errors using the test script.

HALP!

Last edited by indubitableness; 07-14-2010 at 02:23 AM.
 
Old 07-14-2010, 02:44 AM   #5
indubitableness
Member
 
Registered: Feb 2009
Distribution: Slackware64 14.0 multilib
Posts: 62

Original Poster
Rep: Reputation: 15
Okay upon closer inspection using sh -x I THIIIINK I've narrowed it down to this.

"find ... -print0 | xargs -0" is not working as I intentended. It SEEMS to be getting files with spaces but at this point I'm too confused to be certain. It doesn't seem to be escaping those spaces in the output anymore, which it was doing earlier, so I don't know what I'm doing wrong.

I know for certain it's not escaping the character "-" when it's in a filename, hence the "rsync: -: unknown option" error. When it hits filenames with that it thinks it's an option for rsync and chokes on itself.

What I need is some certain way for the find command to output filenames exactly, escaping all spaces and other special characters particularly dash (-).

For the love of god someone tell me how to do that!

edit: Fuck it. I'll use sed. Seems like that shit should be built in. the -ls option does it with find but it also includes all the ownership, permission, and timestamp information in the output just to piss me off.

Last edited by indubitableness; 07-14-2010 at 02:49 AM.
 
Old 07-14-2010, 03:53 AM   #6
indubitableness
Member
 
Registered: Feb 2009
Distribution: Slackware64 14.0 multilib
Posts: 62

Original Poster
Rep: Reputation: 15
For goodness sake!!! Now sed is inserting the backslashes to escape characters but it's separating shit into it's own files.

So the output of the "find ... | sed ..." command is like '/media/Media_Share/Some\' 'Movie.avi'

Instead of '/media/Media_Share/Some\ Movie.avi'

This at least explains why the script keeps looking for files in /root. When it sees a line like that it tries to check the current working directory for it.

This stuff works flawlessly on the command line and then chokes once it's in the script.

How can I make the find command escape spaces and special characters for the love of GOD how do I do this?

Last edited by XavierP; 07-19-2010 at 06:11 PM. Reason: Language tidied up
 
  


Reply

Tags
rsync, sort, timestamp


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
backup using rsync mkrems Linux - Software 6 04-04-2009 02:07 PM
Using Rsync to Backup FredGSanford Slackware 5 08-30-2008 10:09 PM
rsync backup songeek Solaris / OpenSolaris 1 07-05-2007 07:01 AM
Using RSync to backup a secondary off-site backup server pezdspencer Linux - Software 4 06-29-2007 03:40 PM
Using rsync to backup data. Best way to backup folders? Micro420 Linux - General 2 11-23-2006 01:13 AM


All times are GMT -5. The time now is 07:13 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration