Three stage backup with rsync?

indubitableness · 07-13-2010, 09:51 PM

I've been trying to make a three stage backup with stage 0 being a full monthly back up, stage 1 being a weekly backup, and stage 2 being a daily backup. I've been trying very hard to use rsync for this but sorting files by date is proving to be problematic. Sometimes it seems to work from the command line directly, but the same command causes errors and warnings from a script while entirely failing to sort the correct files.

The common example I see for this involves commands like this:

Code:

rsync -Rav `find /home/ -ctime -7 -print` /path/to/home_backup

The problem seems to be that since the user directories in /home contain files that have been altered within the time frame specified the whole directory is matched first which means that the whole directory is recursively archived as opposed to just the changed files.

I've also seen examples using the --files-from tag using the same find parameters and this one seems to ALMOST work but gives me strange warnings and fails to run at all when launched from a script.

Many of the things I've googled about using rsync to backup stuff by the date modified involves a rather snarky "You're missing the point of rsync!" to which I respond by yelling at my computer monitor followed by "JUST TELL ME WHAT I NEED TO KNOW!" I understand that rsync is meant to take care of incremental backups on it's own, that's why I want to use it specifically for a traditional 3 stage backup scheme.

What am I doing wrong?

bryanl · 07-13-2010, 10:34 PM

- looks like maybe copying recursively with just one file might make things interesting ...

For what you want to do, you might look at Brice Burgess' idea to create a series of time stamped backup destinations then use the '--link-dest' option so unchanged files are only links. see at linux.com.

I changed to cp -al step to the rsync --link-dest to modernize the script a bit. I also tested for running as root and a few other things. I have done some thinking about adding a three stage snapshot for monthly, weekly, and daily by using a runtime parameter and a prefix on the backup directory so I could still use the most recent for the latest snapshot but maintain an appropriate number of snapshots for each level. That would only require the rotations loop to look for target directories with the appropriate prefix.

At any rate, the link provided should give you some ideas, I hope.

Valery Reznic · 07-13-2010, 11:21 PM

Quote:

Originally Posted by indubitableness

I've been trying to make a three stage backup with stage 0 being a full monthly back up, stage 1 being a weekly backup, and stage 2 being a daily backup. I've been trying very hard to use rsync for this but sorting files by date is proving to be problematic. Sometimes it seems to work from the command line directly, but the same command causes errors and warnings from a script while entirely failing to sort the correct files.

The common example I see for this involves commands like this:

Code:

rsync -Rav `find /home/ -ctime -7 -print` /path/to/home_backup

The problem seems to be that since the user directories in /home contain files that have been altered within the time frame specified the whole directory is matched first which means that the whole directory is recursively archived as opposed to just the changed files.

I've also seen examples using the --files-from tag using the same find parameters and this one seems to ALMOST work but gives me strange warnings and fails to run at all when launched from a script.

Many of the things I've googled about using rsync to backup stuff by the date modified involves a rather snarky "You're missing the point of rsync!" to which I respond by yelling at my computer monitor followed by "JUST TELL ME WHAT I NEED TO KNOW!" I understand that rsync is meant to take care of incremental backups on it's own, that's why I want to use it specifically for a traditional 3 stage backup scheme.

What am I doing wrong?

adding '-type f' for find parameters may help

indubitableness · 07-14-2010, 01:27 AM

I think I have it figured out, but I'm still seeing some strangeness.

The final command I've come up with looks like this:

Code:

rsync -Rav $OtherOptions `find /home/ -type f -ctime 7 -print0 | xargs -0` /path/to/home_BCK

Where $OtherOptions are things like --exclude-from=file, --delete-excluded, and --delete.

It seems to be working correctly. It copies recursively and it seems to be copying ALL the directories, but it only populates those directories with files that have been modified in the last 7 days, which is exactly the behavior I want.

the -print0 | xargs -0 part of the command is meant to handle filenames with spaces in them and it took me an hour and a half of reading to figure that bit of it out.

I'm still seeing some strange behavior though. Even though the command seems to be behaving, for the most part, I'm seeing errors that look like this:

Code:

rsync: -: unknown option
rsync error: syntax or usage error (code 1) at main.c(1425) [client=3.0.7]
rsync: -: unknown option
rsync error: syntax or usage error (code 1) at main.c(1425) [client=3.0.7]
sending incremental file list
rsync: link_stat "/home/indubitableness/.wine/wineprefixes/Psychonauts/drive_c/Program" failed: No such file or directory (2)
rsync: link_stat "/root/Files/Double" failed: No such file or directory (2)
rsync: link_stat "/root/Fine" failed: No such file or directory (2)
rsync: link_stat "/root/Productions/Psychonauts/Profiles/Profile" failed: No such file or directory (2)
rsync: link_stat "/root/1/SavedGame2" failed: No such file or directory (2)
rsync: link_stat "/home/indubitableness/.wine/wineprefixes/Psychonauts/drive_c/Program" failed: No such file or directory (2)
rsync: link_stat "/root/Files/Double" failed: No such file or directory (2)
rsync: link_stat "/root/Fine" failed: No such file or directory (2)
rsync: link_stat "/root/Productions/Psychonauts/Profiles/Profile" failed: No such file or directory (2)
rsync: link_stat "/root/1/Profile" failed: No such file or directory (2)
rsync: link_stat "/root/1-" failed: No such file or directory (2)
rsync: link_stat "/root/Raz.ini" failed: No such file or directory (2)
rsync: link_stat "/home/indubitableness/.wine/wineprefixes/Psychonauts/drive_c/Program" failed: No such file or directory (2)
rsync: link_stat "/root/Files/Double" failed: No such file or directory (2)
rsync: link_stat "/root/Fine" failed: No such file or directory (2)
rsync: link_stat "/root/Productions/Psychonauts/Profiles/Profile" failed: No such file or directory (2)
rsync: link_stat "/root/1/SavedGame3" failed: No such file or directory (2)
rsync: link_stat "/home/indubitableness/.wine/wineprefixes/Psychonauts/drive_c/Program" failed: No such file or directory (2)
rsync: link_stat "/root/Files/Double" failed: No such file or directory (2)
rsync: link_stat "/root/Fine" failed: No such file or directory (2)
rsync: link_stat "/root/Productions/Psychonauts/Profiles/Profile" failed: No such file or directory (2)
rsync: link_stat "/root/1/SavedGame1" failed: No such file or directory (2)
rsync: link_stat "/home/indubitableness/.wine/wineprefixes/Psychonauts/drive_c/Program" failed: No such file or directory (2)
rsync: link_stat "/root/Files/Double" failed: No such file or directory (2)
rsync: link_stat "/root/Fine" failed: No such file or directory (2)
rsync: link_stat "/root/Productions/Psychonauts/Profiles/Profile" failed: No such file or directory (2)
rsync: link_stat "/root/1/SavedGame0" failed: No such file or directory (2)
rsync: link_stat "/home/indubitableness/.wine/wineprefixes/Psychonauts/drive_c/Program" failed: No such file or directory (2)
rsync: link_stat "/root/Files/Double" failed: No such file or directory (2)
rsync: link_stat "/root/Fine" failed: No such file or directory (2)
rsync: link_stat "/root/Productions/Psychonauts/Profiles/Profile" failed: No such file or directory (2)
rsync: link_stat "/root/1/Profile" failed: No such file or directory (2)
rsync: link_stat "/root/1-" failed: No such file or directory (2)
rsync: link_stat "/root/Raz" failed: No such file or directory (2)
rsync: link_stat "/home/indubitableness/.wine/wineprefixes/Psychonauts/drive_c/Program" failed: No such file or directory (2)
rsync: link_stat "/root/Files/Double" failed: No such file or directory (2)
rsync: link_stat "/root/Fine" failed: No such file or directory (2)
rsync: link_stat "/root/Productions/Psychonauts/AudioSettings.ini" failed: No such file or directory (2)
rsync: link_stat "/home/indubitableness/Torrent/[isoHunt]" failed: No such file or directory (2)
rsync: link_stat "/root/The" failed: No such file or directory (2)
rsync: link_stat "/root/Metal" failed: No such file or directory (2)
rsync: link_stat "/root/Gear" failed: No such file or directory (2)
rsync: link_stat "/root/Solid" failed: No such file or directory (2)
rsync: link_stat "/root/PSX" failed: No such file or directory (2)
rsync: link_stat "/root/Collection" failed: No such file or directory (2)
rsync: link_stat "/root/(NTSC-US).torrent" failed: No such file or directory (2)
rsync: link_stat "/home/indubitableness/.cache/vlc/art/artistalbum/Cake/Prolonging" failed: No such file or directory (2)
rsync: link_stat "/root/the" failed: No such file or directory (2)
rsync: link_stat "/root/Magic/art" failed: No such file or directory (2)
rsync: link_stat "/home/indubitableness/.cache/vlc/art/artistalbum/Cake/Fashion" failed: No such file or directory (2)
rsync: link_stat "/root/Nugget/art" failed: No such file or directory (2)

What is this "link_stat" error and why is rsync looking in my /root directory in the first place?

These all seem to be errors caused by the lack of escaped spaces, a problem I fixed with the "find ... -print0 | xargs -0" commands I mentioned before. The files in /home/indubitableness that have spaced in them are getting backed up with this command despite the error messages.

Furthermore I can't seem to figure out how to make rsync copy the contents of the source directory to the destination directory using this find command as the input for rsync. I know when using rsync normally doing /home/ as opposed to /home is supposed to mean "copy the contents of home" but I keep getting a "home" directory in the destination. Is this something I'm just going to have to live with or can I force the behavior I'm looking for?

Edit:

This is the script I've been working on.

http://lfs.pastebin.com/x8LaH1G5

Note: The line '"${OUTPUT_DIR[$dirCount]}"' on line 86 includes the quotes because I was desperately casting around for a solution to the errors I'm seeing. It didn't help to put them in quotes.

The command doesn't seem to above posted errors from the command line. I also have a test script intended to simulate the variable substitution used in the above script in an attempt to replicate the errors. It looks like this:

Code:

#!/bin/sh

SOURCE[1]=/home/
DEST[1]=/media/Sanctuary/DAILY_BACKUP_INCREMENT/home_rsync/vroomfondel/ 
src_destNum=1

rsync --dry-run -Rav --delete --exclude-from=EXCLUDE_SYNC --delete-excluded `find ${SOURCE[$src_destNum]} -type f -ctime -1 -print0 | xargs -0` ${DEST[$src_destNum]}

The errors existed in the original script before I relegated all of rsync's options to variables and so I haven't yet put them in variables in the test script. The test script appears to run fine and I only seem to be getting the "rsync: -: unknown option" followed by the litany of "link_stat" errors when it moves into copying the /home directory. I am not seeing any of these errors using the test script.

HALP!

indubitableness · 07-14-2010, 02:44 AM

Okay upon closer inspection using sh -x I THIIIINK I've narrowed it down to this.

"find ... -print0 | xargs -0" is not working as I intentended. It SEEMS to be getting files with spaces but at this point I'm too confused to be certain. It doesn't seem to be escaping those spaces in the output anymore, which it was doing earlier, so I don't know what I'm doing wrong.

I know for certain it's not escaping the character "-" when it's in a filename, hence the "rsync: -: unknown option" error. When it hits filenames with that it thinks it's an option for rsync and chokes on itself.

What I need is some certain way for the find command to output filenames exactly, escaping all spaces and other special characters particularly dash (-).

For the love of god someone tell me how to do that!

edit: Fuck it. I'll use sed. Seems like that shit should be built in. the -ls option does it with find but it also includes all the ownership, permission, and timestamp information in the output just to piss me off.

indubitableness · 07-14-2010, 03:53 AM

For goodness sake!!! Now sed is inserting the backslashes to escape characters but it's separating shit into it's own files.

So the output of the "find ... | sed ..." command is like '/media/Media_Share/Some\' 'Movie.avi'

Instead of '/media/Media_Share/Some\ Movie.avi'

This at least explains why the script keeps looking for files in /root. When it sees a line like that it tries to check the current working directory for it.

This stuff works flawlessly on the command line and then chokes once it's in the script.

How can I make the find command escape spaces and special characters for the love of GOD how do I do this?