Linux - ServerThis forum is for the discussion of Linux Software used in a server related context.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
what about time stamps in differrntial backup. is it normal? in disaster recovery?
I don't know. What you are doing with the time stamp is very dependent on the naming convention you use for your files and how often files are replaced or rotated.
If you are using the time stamp to try to differentiate between generations of backup then I suggest that you create several directories in /mnt/bck and run a different rsync command each day in rotation to the directories. For example have 7 directories in /mnt/bck:
then have 7 rsync commands to do a differential backup for each day of the week. The limit as to how many generations you can keep is the space available.
I have exactly the same setup with a samba4 domain controller and RAID-1. The problem with rsync is that it makes a copy of the source hierarchy w/o compression on disk. Therefore, you need a lot of space for saving the generations as jailbait pointed out. I use tar. My backup script is shown below (btw I hadn't used --suffix on the rsync command before. I tried it, but it didn't seem to do anything. My suffix didn't appear in any of the target directories).
My script does a full backup every month and differentials every day. And it creates a special backup each quarter eliminating the users' email and redirected folders. Something to be aware of is that tar won't preserve ATTRs and ACLs, so these have to be saved in a separate file (which can be saved with the backup) and restored later. I've been using this backup for a while and have had occasion to restore the complete system twice.
Code:
#!/usr/bin/bash
echo `date "+Y-%m-%d %H:%M Starting"`
# Day-of-month for full backup
fullBUday=01
# Keep differential backups for this many days
keepDays=60
today=`date +%d`
BUDIR=/mnt/backup/MAIL
dte=`date -d yesterday +%Y-%m-%d`
EXCLUDE="--exclude /backup --exclude /mnt --exclude /proc --exclude /sys --exclude /dev --exclude /run"
EXCLUDE="$EXCLUDE --exclude lost+found"
# Kill full backup tagfile if this is the $fullBUdate
if [ $fullBUday == $today ]; then rm -f $HOME/.fullBUOK; fi
x=`df | grep /mnt/backup`
if [ -z "$x" ]
then
mount /mnt/backup
RC=$?
if [ "$RC" != 0 ]
then
echo Mount of /mnt/backup failed, status: $RC
echo `date "+%Y-%m-%d %H:%M Finished"`
echo
echo Mount of /mnt/backup failed, status: $RC | mail -r noreply -s "OHPRS ${HOSTNAME^^} backup failed" sysadmin
exit
fi
else
echo /mnt/backup already mounted
isMounted=true
fi
if [ ! -e $BUDIR ]
then
echo Directory $BUDIR not found, backup aborted
echo `date "+%Y-%m-%d %H:%M Finished"`
echo
echo Directory $BUDIR not found, backup aborted | mail -r noreply -s "OHPRS ${HOSTNAME^^} backup failed" sysadmin
exit
fi
if [ ! -e $BUDIR/logs ]; then mkdir $BUDIR/logs; fi
# Backup ATTRs, extended file attributes
# Restore with cd /; setfattr --restore=backupATTR.txt
getfattr -dR /redirectedFolders /var/lib/samba/sysvol /var/lib/samba/share >/backupATTR.txt 2>/tmp/this
cat /tmp/this
# Backup ACLs. This will put ACL info on tarfile
# Restore with: cd /;setfacl --restore=/backupACL.txt
getfacl -R /redirectedFolders /var/lib/samba/sysvol /var/lib/samba/share > /backupACL.txt 2>/tmp/this
cat /tmp/this
rm /tmp/this
# Check for Quarterly Backup - Exclude email (retention policy)
thisYear=`date +%Y`
thisMonth=`date +%m`
case $thisMonth in
01 | 04 | 07 | 10 )
if [ "$today" == "01" ]
then
if [ "$thisMonth" == "01" ]; then QTR="Q4"
elif [ "$thisMonth" == "04" ]; then QTR="Q1"
elif [ "$thisMonth" == "07" ]; then QTR="Q2"
elif [ "$thisMonth" == "10" ]; then QTR="Q3"; fi
echo Create Quarter Backup for $QTR $thisYear
tar -C / $EXCLUDE --exclude Maildir --exclude /var/spool/mail/allmail\* \
--exclude /var/spool/mail/spam\* --exclude /redirectedFolders/Users --warning=no-file-ignored \
-cvjf $BUDIR/${thisYear}${QTR}.bz2 /
echo `date "+%Y-%m-%d %H:%M Quarterly Backup completed"`
fi
;;
esac
lastBackup=`/usr/bin/ls -t $BUDIR/*MAILfullbackup.tar.bz2 | head -1`
if [ ! -e $HOME/.fullBUOK ]
then
echo Full Backup
FULLBACKUP=true
TARFILE=$BUDIR/${dte}-MAILfullbackup.tar.bz2
LOGFILE=$BUDIR/logs/${dte}-MAILfullbackup.log
else
echo Differential since \
`stat $lastBackup | grep "^Modify" | cut "-d " -f2- | cut -d. -f1`
NEWER="--warning=no-file-unchanged --newer $lastBackup"
TARFILE=$BUDIR/${dte}-MAILdifferential.tar.bz2
LOGFILE=$BUDIR/logs/${dte}-MAILdifferential.log
fi
tar -C / $EXCLUDE $NEWER --warning=no-file-ignored --warning=no-file-changed -cvjf $TARFILE / >$LOGFILE
retCd=$?
if [ $retCd -gt 1 ]
then
echo backup Failed, status: $retCd
echo backup Failed, status $retCd | mail -r noreply -s "OHPRS ${HOSTNAME^^} backup failed" sysadmin
else
echo Clean up differentials older than $keepDays days
x=`ls $BUDIR/*differential.tar.bz2 | wc | awk '{print $1}'`
if [ "$x" -gt 1 ] # don't delete last differential
then
find $BUDIR/*differential.tar.bz2 -mtime +$keepDays -exec rm -v {} \;
find $BUDIR/logs/*differential*.log -mtime +$keepDays -exec rm -v {} \;
fi
echo -e "$(df -h)" "\n\nBackup OK" | mail -r noreply -s "OHPRS ${HOSTNAME^^} backup OK" sysadmin
fi
if [ -n "$FULLBACKUP" ]
then
date > $HOME/.fullBUOK
fi
if [ -z "$isMounted" ]; then umount /mnt/backup; fi
echo `date "+%Y-%m-%d %H:%M Finished"`
echo
So in case of disaster I just can directly restore a data folder from NAS with latest information and if I will need the previous version of
some file I will take it from incremental backup kept in time stamped directories. In my case space is not a problem and file sizes are not
big. For me it is important to recover as soon as possible from disaster and later if will need previous versions of files, manually can be
restored.
What you think?
And want to thank both jailbait and mfoley for scripts and advices
My compressed backups are on the NAS. I had to restore an important 125GB directory hierarchy last week from a 95GB compressed tar. It took 3 hours! If you've got the space, a ready-to-go rsync'd archive is the way to go. As for me, my NAS RAID-6 is 70 full, so not possible.
If you are really looking for actual "disaster recovery", I would suggest also making an offsite backup, or at least an offline backup. A true disaster might take out your backup drive. I have a 3TB, self-powered USB drive on our backup server having complete compressed backups of the important servers, files and database. This is rsync'd nightly with a same-sized USB drive located off-site -- and the offsite server makes a duplicate of that drive as soon as the rsync is finished so there is always at least one such drive that has the most recent complete backup. All these backups are encrypted.
Having at least a local external USB drive gives you something to grab and go if the office catches fire. If an offsite backup is not possible, the local external USB could be swapped weekly taking the alternate drive off-site. That way, if a small thermonuclear device went off in your building (or thieves cleaned out the office), you'd at least have the as-of last week backup.
here is a very simple backup script. add some of this to the error checking provided above and you will have a simple 7 days worth of backups in day of the week subfolder names.
Code:
#!/bin/bash
#
###########################################################
# Created by Ray Brunkow January 11, 2015
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 2 or version 3 of the
# license, at your option.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#############################################################
#
#
#############################################################
#
# Backup program for laptops to Linux server using
# rsync -aviS
#
#############################################################
# Veriables
RUSER=<---insert remote user name here--->
RHOST=<----insert remote host IP/URL here---->
WHO=`whoami`
DOW=`date +A`
RDIR=/path/to/remote/location/without/a/trailing/foo
HOMEDIR="$HOME"
dtstamp="`date +Y-%m-%d-%H:%M `"
log=${HOMEDIR}/logs/${dtstamp}-rsync.log
###############################################################
### Check for a logs directory, if not found create one
###############################################################
[ ! -d "${HOMEDIR}/logs" ] && mkdir -p ${HOMEDIR}/logs >> /dev/null 2>&1
###############################################################
#
# Rsync preserving permissions
#
###############################################################
rsync -aviS --exclude-from=${HOMEDIR}/excludes.txt ${HOMEDIR}/ ${RUSER}@${RHOST}:${RDIR}/${WHO}/${DOW}/ >> ${log} 2>&1
### Cleaning up log files
###############################
find ${HOMEDIR}/logs/*.log -mtime +30 -exec rm '{}' \;
exit
This will also create a simple log file for you to monitor for network issues and what not that will automatically clean up older log files greater then 30 days old.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.