Rsync + hard links backup strategy questions

TrainingPeaks · 07-24-2012, 12:53 PM

I'm using several of Amazon's EC2 linux instances for a Deploy & Live MongoDB environment, and have been working on my backup strategy for a few days.

I've narrowed it down to using LVM snapshots to obtain a consistent data volume which I rsync to my backup server.

I do this on an hourly basis from the backup server itself, using the following script:

Code:

#!/bin/bash

# ... some config stuff before this ...

cd /backup

#Rotate all existing backups back 1 hour, and entirely delete the oldest one.
rm -fr 24h
for i in {23..1}
do
    mv ${i}h $(($i + 1))h
done

#Make a copy of the current backup into the 1h bucket, using hard links to avoid duplicating data.
cp -la current 1h

# Rsync 
echo "Created on `date`" | tee -a /backup/current/DATETIME
rsync -e "ssh -i ***.pem" --verbose --recursive --times --perms --links --delete ***@***:~/backup/ /backup/current

Then, once a day at midnight, I run the following script:

Code:

#!/bin/bash

mv 6d 7d
mv 5d 6d
mv 4d 5d
mv 3d 4d
mv 2d 3d
mv 1d 2d
mv 24h 1d
rm -fr 24h/*

The idea being to have daily backups for a weekly as well as hourly backups for a day.
Finally, I run the following script once a week on a Saturday morning:

Code:

#!/bin/bash

cd /backup
mv current weekly
rm -fr current/*

The idea being to have a weekly backup, and by deleting my /backup/current folder to force rsync to do a completely fresh copy once a week to ensure consistency.

Any input on this backup strategy?
Any input on my scripts? Are they doing what I expect them to do?
Just looking for general feedback....

Thanks!

pan64 · 07-25-2012, 01:31 AM

I would try the option --link-dest probably you can save a lot of time and space.

TrainingPeaks · 07-25-2012, 09:07 AM

Thanks! But

Code:

cp -la current 1h

My understanding is, using hard-link copy like above achieves the same space savings as link-dest (which is just a hard-link copy command, in my understanding). Further, my doing the hard-link copy myself, I have more control over how to store/keep/rotate the backups.

pan64 · 07-25-2012, 10:59 AM

No, I think you misunderstood. --link-dest will work file by file: if file changed it will be copied if file was not changed since the last backup it will be linked instead (that will save the time of copy and space too). You can give several --link-dest options to use several backups. For example: you will get every time a new backup, but the files not changed since the first backup will be stored only once.
Also you can try to read rsync forums there are a lot of discussion about how to use rsync for incremental backups and similar tasks, you may find usable ideas.