LinuxQuestions.org - Rsync + hard links backup strategy questions

- Linux - Server (https://www.linuxquestions.org/questions/linux-server-73/)

- - Rsync + hard links backup strategy questions (https://www.linuxquestions.org/questions/linux-server-73/rsync-hard-links-backup-strategy-questions-4175418442/)

Rsync + hard links backup strategy questions

I'm using several of Amazon's EC2 linux instances for a Deploy & Live MongoDB environment, and have been working on my backup strategy for a few days.

I've narrowed it down to using LVM snapshots to obtain a consistent data volume which I rsync to my backup server.

I do this on an hourly basis from the backup server itself, using the following script:

Code:

#!/bin/bash



# ... some config stuff before this ...



cd /backup



#Rotate all existing backups back 1 hour, and entirely delete the oldest one.

rm -fr 24h

for i in {23..1}

do

    mv ${i}h $(($i + 1))h

done



#Make a copy of the current backup into the 1h bucket, using hard links to avoid duplicating data.

cp -la current 1h



# Rsync 

echo "Created on `date`" | tee -a /backup/current/DATETIME

rsync -e "ssh -i ***.pem" --verbose --recursive --times --perms --links --delete ***@***:~/backup/ /backup/current

Then, once a day at midnight, I run the following script:

Code:

#!/bin/bash



mv 6d 7d

mv 5d 6d

mv 4d 5d

mv 3d 4d

mv 2d 3d

mv 1d 2d

mv 24h 1d

rm -fr 24h/*

The idea being to have daily backups for a weekly as well as hourly backups for a day.
Finally, I run the following script once a week on a Saturday morning:

Code:

#!/bin/bash



cd /backup

mv current weekly

rm -fr current/*

The idea being to have a weekly backup, and by deleting my /backup/current folder to force rsync to do a completely fresh copy once a week to ensure consistency.

Any input on this backup strategy?
Any input on my scripts? Are they doing what I expect them to do?
Just looking for general feedback....

Thanks!

I would try the option --link-dest probably you can save a lot of time and space.

Thanks! But

Code:

cp -la current 1h

My understanding is, using hard-link copy like above achieves the same space savings as link-dest (which is just a hard-link copy command, in my understanding). Further, my doing the hard-link copy myself, I have more control over how to store/keep/rotate the backups.

No, I think you misunderstood. --link-dest will work file by file: if file changed it will be copied if file was not changed since the last backup it will be linked instead (that will save the time of copy and space too). You can give several --link-dest options to use several backups. For example: you will get every time a new backup, but the files not changed since the first backup will be stored only once.
Also you can try to read rsync forums there are a lot of discussion about how to use rsync for incremental backups and similar tasks, you may find usable ideas.