how to create daily incremental backups easily?

crypted · 11-29-2010, 07:57 AM

I've had several HDD crashes on my personal server over the years and it's just gotten to be a real pain in the rear. Crashed again this morning.

Currently, I make monthly tarball backups of the entire filesystem using my script:

Code:

#!/bin/sh

# Removes the tarball from the previous execution.
rm -rf /backup/data/*.tar.gz

# Dates the new tarballs of current builds.
DATE=`date +%m_%d_%Y`

# Directory structure that is being tarballed. 
# Have this be what is found in your "/" root
# excluding the /backup/ partition.

tar -pzcf /backup/data/bin.$DATE.tar.gz /bin/
tar -pzcf /backup/data/boot.$DATE.tar.gz /boot/
tar -pzcf /backup/data/dev.$DATE.tar.gz /dev/
tar -pzcf /backup/data/dist.$DATE.tar.gz /dist/
tar -pzcf /backup/data/etc.$DATE.tar.gz /etc/
tar -pzcf /backup/data/kernel.$DATE.tar.gz /kernel*
tar -pzcf /backup/data/lib.$DATE.tar.gz /lib
tar -pzcf /backup/data/root.$DATE.tar.gz /root/
tar -pzcf /backup/data/sbin.$DATE.tar.gz /sbin/
tar -pzcf /backup/data/stand.$DATE.tar.gz /stand/
tar -pzcf /backup/data/usr.$DATE.tar.gz /usr/
tar -pzcf /backup/data/var.$DATE.tar.gz /var/
tar -pzcf /backup/data/emul.$DATE.tar.gz /emul/
tar -pzcf /backup/data/home.$DATE.tar.gz /home/
tar -pzcf /backup/data/selinux.$DATE.tar.gz /selinux/
tar -pzcf /backup/data/srv.$DATE.tar.gz /srv/
tar -pzcf /backup/data/ssl.$DATE.tar.gz /ssl/
tar -pzcf /backup/data/sys.$DATE.tar.gz /sys/
ls -lah > /backup/data/rootmap
cp /quota* /backup/data/

I'd like to get daily backups of everything, but at least of /var/ since it contains my email, websites, and sql databases.

Any free, simple solutions out there to run such automated backups to my secondary hdd?

fordeck · 11-29-2010, 08:40 AM

You might take a look at rdiff-backup:

http://www.nongnu.org/rdiff-backup/

Regards,

Fordeck

fordeck · 11-29-2010, 08:41 AM

See previous post

catkin · 11-29-2010, 09:25 AM

9 out of 10 cat owners who were asked said their cats prefer rsync. Me, I like Bacula with crunchy fish bits but it's not what any sane puss would call simple.

jlcasado · 11-29-2010, 12:55 PM

rsync...simple and really effective

repo · 11-29-2010, 01:01 PM

+1 for rsync

Quote:

# Removes the tarball from the previous execution.
rm -rf /backup/data/*.tar.gz

BTW, I would remove the old backups, AFTER the new backup is made.

Kind regards

crypted · 11-29-2010, 10:25 PM

any suggestions for the rsync thing?

i'd like to do a backup every 24 hrs on my /backup/ harddrive.

1) /var/www/
2) /var/vmail/
3) /var/lib/mysql

not sure how it'd work out...

fbsduser · 11-29-2010, 11:27 PM

+1 for rsync. Also

Code:

#!/bin/sh

# Dates the new tarballs of current builds.
DATE=`date +%m_%d_%Y`

# Directory structure that is being tarballed. 
# Have this be what is found in your "/" root
# excluding the /backup/ partition.

tar -pzcf /backup/data/bin.$DATE.tar.gz /bin/
tar -pzcf /backup/data/boot.$DATE.tar.gz /boot/
tar -pzcf /backup/data/dev.$DATE.tar.gz /dev/
tar -pzcf /backup/data/dist.$DATE.tar.gz /dist/
tar -pzcf /backup/data/etc.$DATE.tar.gz /etc/
tar -pzcf /backup/data/kernel.$DATE.tar.gz /kernel*
tar -pzcf /backup/data/lib.$DATE.tar.gz /lib
tar -pzcf /backup/data/root.$DATE.tar.gz /root/
tar -pzcf /backup/data/sbin.$DATE.tar.gz /sbin/
tar -pzcf /backup/data/stand.$DATE.tar.gz /stand/
tar -pzcf /backup/data/usr.$DATE.tar.gz /usr/
tar -pzcf /backup/data/var.$DATE.tar.gz /var/
tar -pzcf /backup/data/emul.$DATE.tar.gz /emul/
tar -pzcf /backup/data/home.$DATE.tar.gz /home/
tar -pzcf /backup/data/selinux.$DATE.tar.gz /selinux/
tar -pzcf /backup/data/srv.$DATE.tar.gz /srv/
tar -pzcf /backup/data/ssl.$DATE.tar.gz /ssl/
tar -pzcf /backup/data/sys.$DATE.tar.gz /sys/
ls -lah > /backup/data/rootmap
cp /quota* /backup/data/

# Removes the tarball from the previous execution.
rm -rf /backup/data/*.tar.gz

Works more safely than what you had.

repo · 11-30-2010, 02:18 AM

Quote:

Originally Posted by crypted

any suggestions for the rsync thing?

i'd like to do a backup every 24 hrs on my /backup/ harddrive.

1) /var/www/
2) /var/vmail/
3) /var/lib/mysql

not sure how it'd work out...

Start with reading the documentation for rsync.
Then create a script to backup your directories.
Make sure the script works.
Create a cronjob for the script.
Some info
http://www.thegeekstuff.com/2010/09/...mand-examples/
http://www.cyberciti.biz/tips/tag/rsync-examples

Kind regards

choogendyk · 11-30-2010, 06:54 AM

I like rsync too, and use it regularly. However, the original post did say incremental. That would be rdiff-backup, which uses librsync. See http://www.nongnu.org/rdiff-backup/, or, for the general concept and to roll your own with rsync, see http://www.mikerubel.org/computers/rsync_snapshots/.

I would also note that in the original configuration, rather than deleting all the old tarballs with the *, you could use `find /backup/data/ -name '*.tar.gz' -mtime +7 -exec rm {} \;`. That would remove any tarballs older than 7 days. Of course, make sure you have enough space for the extras, since they are all full backups. You can do incrementals with gnutar, but then you're getting more complicated (scripting which day to run fulls, incrementals on others, removing the older ones, etc.), and you might as well use something off the shelf.

As far as cat owners go, I like Amanda

, and it's not too complicated to get running (there is a quick start with backup to disk on the wiki); but, both Amanda and Bacuala come into their own in networked environments with multiple machines being backed up. For a single computer, there are many simpler solutions.

crypted · 12-11-2010, 04:46 PM

I've tried using the find command above from choogendyk.

It doesn't work.

Code:

my:/backup# find /backup/databases/ -name '*.sql' -mtime +30 -exec rm {} \
>

Just sits there at >

Code:

my:/backup/databases# find /backup/databases/ -name '*.sql' -mtime +30 -exec rm {} 
find: missing argument to `-exec'

How could I get this to work?

It'd be very useful to remove backup files whether tar.gz or sql dumps (not incremental) after so many days...

barriehie · 12-11-2010, 09:02 PM

Try putting a ; on the end.

Code:

my:/backup# find /backup/databases/ -name '*.sql' -mtime +30 -exec rm {} \;

crypted · 12-11-2010, 10:24 PM

Ah yes, correct you are! Thanks for catching that...

barriehie · 12-12-2010, 02:41 AM

Quote:

Originally Posted by crypted

Ah yes, correct you are! Thanks for catching that...

Yep, I'll probably do that a few more times too!