Backup solution like rsnapshot that's easy on the hard disks...

cabrilo · 04-15-2013, 04:21 AM

Hello everyone!

Currently, I have one server which I use to keep backups from various places. I use rsnapshot and keep backups of 3 remote servers, as well as some local directories. So, there is a lot of files to sync.

The problem is, rsnapshot is very slow, especially when it's removing directories, which can take literally hours (and this is amplified by the fact that I use relatively slow hard disks for storage - both for their price and to keep the server quiet as it's in my office). I'd like to do hourly backups every 4 hours, but it will, every once in a while, take more than 3-4 hours to complete one backup cycle!

Are there any other incremental backup solutions which are as simple as rsnapshot (which is very nice otherwise) but are better suited for a large number of files?

Thanks

lleb · 04-15-2013, 07:11 PM

just create a basic bash script that will use rsync and if you want to have rsync remove defunct files i believe there is a delete option you can flag, see the man page for the exact syntax.

a simple basic script could as easy as:

Code:

#!/bin/bash

### set your variables

DTSTAMP="`date +%Y-%m-%d-%H.%M.%S `"
RUSER=remote_user_name
RHOST=remote_IP_backup_server    ### or DNS host name if you have that configured ###
LDIR=/path/to/local/dir/to/backup
RDIR=/path/on/backup/server/to/store/data
DOW=`date +%A`
LOG=/path/to/log/file/${DTSTAMP}-rsync.log

### call the rsync portion

     rsync -aviS --exclude-from=/path/to/excludes.txt ${LDIR}/ ${RUSER}@${RHOST}:/${DOW}/ >> ${LOG} 2>&1

### clean up your log files

find /path/to/log/files/*.log -mtime +30 -exec rm '{}' \;

exit

This will store 7 days worth of data from each of the servers that are set to run. also keep in mind if you have this set to PUSH from the remote servers into the backup server you CAN run them all at the same time, but keep in mind your I/O and Network performance might suffer if you do this. I might be a good idea to stager the start times of each remote server by 15-20min that way you can still have them all run every hour.

your needs may very, but this is a start and again very basic.

sajansijin@ymail.com · 04-16-2013, 02:09 AM

Hi lleb,

It seems the password would be prompted for when we run this script.

Can this be avoided with an attribute in rsync itself.?

cabrilo · 04-16-2013, 02:47 AM

Quote:

Originally Posted by lleb

just create a basic bash script that will use rsync and if you want to have rsync remove defunct files i believe there is a delete option you can flag, see the man page for the exact syntax.

Thanks for the reply. There is a problem though: If I understand correctly, each of 7 days stores a full copy of everything. Given that I currently keep 6 snapshots a day, 7 daily, 4 weekly, 12 monthly and 4 yearly, that would quite quickly fill up my disks (that would be 30 copies of everything). Rsnapshot solves this by keeping hard links to same files.

chrism01 · 04-16-2013, 08:36 AM

@sajansijin@ymail.com: tell rsync to use ssh and ssh auth-keys

lleb · 10-13-2013, 07:08 PM

Quote:

Originally Posted by sajansijin@ymail.com

Hi lleb,

It seems the password would be prompted for when we run this script.

Can this be avoided with an attribute in rsync itself.?

look at both the links in my signature, and my script again.

as you are using user@remote_location, this is over ssh and with ssh keys you will not be prompted for a p/w if configured properly.