help with modifying a backup script

dfwcomputer · 09-25-2006, 07:15 PM

hi all,

i have a backup script i aquired from somewhere (credit where credit is due).
i would like to modify it a little.What it does is backs up daily (/etc/cron.daily) what ever directories you enter into it.
e.g. BACKUPTHESE="/home /root/"

it backs up to directories i have created.

e.g on a monday it backsup everything inside /home & /root and places it in /media/other/backup/daily.On Tuesday it first moves everything from /dailt into /yesterday an so on. (see the script below)

Code:

#!/bin/bash
# A Simple (Poor Man) disk based backup script
# Will backup all user home directories in seperate archives
# as a single /home backup file can be quite large and unwieldly.
# Does not need to be modified if users are added/deleted

# Step - 1 Create Timestamp and set up variables and functions
# BUDTSTAMP = Backup Date/Time Stamp
BUTDSTAMP=$(date +%Y%m%d)

# variable holding directories containing files to backup eg: BACKUPTHESE="/home /root /etc"
BACKUPTHESE="/home"

#directory containing today's backup
BKUPDIR="/media/other/backup/daily/home"
#directory containing yesterday's backup
YDBKUPDIR="/media/other/backup/yesterday/home"
#directory containing the day before yesterday's backup
DBBKUPDIR="/media/other/backup/daybefore/home"

# log files
LOG="mail.txt"
ERRORLOG="ERROR.txt"

# tag for backup file name
TAG="home-backup"

# function that quits and logs on error
exiterror()
{
# use this function by supplying $LINENO as first arg
echo "Fatal error caused by line ${1} of ${0}" >> $ERRORLOG
mail somebody@somedomain.com -s "Backup Job ERROR" -v < $ERRORLOG
mail somebody@somedomain.com -s "Backup Job Report" -v < $LOG
exit 1
}

# Step - 2 Start Email Message To Be Sent

# Remove mail message from previous backup
# I do this at the beginning of the script instead of the end
# in case the mail does not send for whatever reason or
# I need to debug it

rm $LOG > /dev/null 2>&1
echo "System Backup $BUTDSTAMP" >> $LOG

# The email sends a user friendly note showing the start and
# end time/dates. This is important so you can compare logs
# and see if a backup ran ok.

echo "Backup Began $(date)" >> $LOG

# Step - 3 Rotate Backups
# Simple three backupset rotation, keeps only last three
# Use directory /media/other/backup as an example, make sure you change this path
# to fit your local settings. Have three directories in /media/other/backup named daily,
# yesterday and daybefore.
# Make sure /media/other/backup is on a different disk (preferably a different machine)
# than the files you are backing up

#looks specifically for backup files in case other files are kept in these directories
for bkfile in $DBBKUPDIR/*; do
echo $bkfile | grep $TAG >/dev/null &&
(rm $bkfile || exiterror $LINENO )
done
for bkfile in $YDBKUPDIR/*; do
echo $bkfile | grep $TAG >/dev/null &&
(mv $bkfile $DBBKUPDIR/ || exiterror $LINENO )
done
for bkfile in $BKUPDIR/*; do
echo $bkfile | grep $TAG >/dev/null &&
(mv $bkfile $YDBKUPDIR/ || exiterror $LINENO )
done

# Step - 4 Archive Home Directories
# Creates a seperate tar file for each directory in the directories in $BACKUPTHESE


# change to dir to backup. Check in case cd failed for some reason.
cd $BACKUPTHIS || exiterror $LINENO
#backup all files listed in BACKUPTHESE
for DIRTOBACKUP in $BACKUPTHESE; do
for FOLDERNAME in $DIRTOBACKUP/*
do
# Archives are created in the format someuser-$TAG-datetime.tar.gz
# basename is used here so as not to include absolute paths
# -p preserves permissions
echo -e "-------------------\n>>>taring ${FOLDERNAME}\n\n" >> $LOG
tar -czvpf ${BKUPDIR}/$( basename $FOLDERNAME )-${TAG}-${BUTDSTAMP}.tar.gz ${FOLDERNAME} >> $LOG || exiterror $LINENO
done
done

# Step - 5 Finish Email Report and Send

echo "Backup ended $(date)" >> $LOG

# df -h includes a human readable disk usage report of the media that /media/other/backup
# is mounted on. Good to now if your backup disk is running out of space.
# Of course /dev/hdb1 is the device I use, modify it for your local settings

df -h /dev/hdb1 >> $LOG
mail somebody@somedomain.com -s "Backup Job Report" -v < $LOG

# Step - 6 All Done

The trouble is you end up with a folder full of individual archives, and not knowing where they come from.If you also add /etc to the backups it starts to.....well i think you can see what happens.Is there a way to maybe have it put backups from each folder e.g. /home & /etc into maybe say /media/other/backup/daily/home.

Or any other ideas on a better setup.

Thanks

dfwcomputer · 09-27-2006, 02:50 AM

bump

anyone got any ideas

konsolebox · 09-27-2006, 03:05 AM

hello. what's BACKUPTHIS in 'cd $BACKUPTHIS || exiterror $LINENO'?

dfwcomputer · 09-27-2006, 03:50 AM

i'm not really sure it looks like it just checks if it changed directory alright.
e.g. if you didn't create the directory your changing to, maybe?

The script works great though.

kloss · 09-27-2006, 05:22 AM

Code:

#!/bin/sh

backupdir="/path/to/backup/dir"
backupthese="/foo/bar /baz/qux"

day=`date +%Y%m%d`
dir="$backupdir/$day"; mkdir $dir
for d in $backupthese; do
	tar cfz "$dir/`echo $d | tr / !`.tar.gz" $d
done
rm -rf "$backupdir/`ls -1 $backupdir | head -1`"

This script backups /foo/bar und /baz/qux to /path/to/backup/dir/20060927/!foo!bar.tar.gz and /path/to/backup/dir/20060927/!baz!qux.tar.gz (since we are the 27th of september).

The names of the tar.gz files let you know where they come from.

The script also removes the oldest backup: if you want to give it a try, make sure to create /path/to/backup/dir/20060926 and newer dummy backup directories before running it.

An advice: NEVER run a script if you do not understand each line of it.

Bye!

konsolebox · 09-27-2006, 05:26 AM

Quote:

Originally Posted by dfwcomputer

i'm not really sure it looks like it just checks if it changed directory alright.
e.g. if you didn't create the directory your changing to, maybe?

The script works great though.

it appears that BACKUPTHIS doesn't have any value. it's not BACKUPTHESE right? so cd to where? is it really included in the script?

dfwcomputer · 09-27-2006, 08:33 PM

Quote:

it appears that BACKUPTHIS doesn't have any value. it's not BACKUPTHESE right? so cd to where? is it really included in the script?

Sorry yes your right it is BACKUPTHESE

dfwcomputer · 09-27-2006, 09:26 PM

Quote:

Originally Posted by kloss

Code:

#!/bin/sh

backupdir="/path/to/backup/dir"
backupthese="/foo/bar /baz/qux"

day=`date +%Y%m%d`
dir="$backupdir/$day"; mkdir $dir
for d in $backupthese; do
    tar cfz "$dir/`echo $d | tr / !`.tar.gz" $d
done
rm -rf "$backupdir/`ls -1 $backupdir | head -1`"

This script backups /foo/bar und /baz/qux to /path/to/backup/dir/20060927/!foo!bar.tar.gz and /path/to/backup/dir/20060927/!baz!qux.tar.gz (since we are the 27th of september).

The names of the tar.gz files let you know where they come from.

The script also removes the oldest backup: if you want to give it a try, make sure to create /path/to/backup/dir/20060926 and newer dummy backup directories before running it.

Thanks i will try it out.

Quote:

Originally Posted by kloss

An advice: NEVER run a script if you do not understand each line of it.

the system i run is just one i draged out of the shed to muck around on so i not really concerned even if the pc self destructed.

But i here the warning

konsolebox · 09-27-2006, 10:05 PM

What do you prefer to identify the folders, tags or directories?

dfwcomputer · 09-28-2006, 02:06 AM

not quite sure of the difference, but i would imagine that it would not really bother me either way.

konsolebox · 09-28-2006, 02:31 AM

please find a way to test this mod. if you don't like each pair to be separated in a different directory just tell me.

Code:

#!/bin/bash

# A Simple (Poor Man) disk based backup script
# Will backup all user home directories in seperate archives
# as a single /home backup file can be quite large and unwieldly.
# Does not need to be modified if users are added/deleted

# Step - 1 Create Timestamp and set up variables and functions
# BUDTSTAMP = Backup Date/Time Stamp
BUTDSTAMP=$(date +%Y%m%d)

# variable holding directories containing files to backup with tags
# format: "DIR:TAG ..."
# eg: BACKUPPAIRS="/homehome-backup /root:root-backup /etc:etc-backup"
BACKUPPAIRS="/home:home-backup"

#directory containing today's backup
BKUPDIR="/media/other/backup/daily/home"
#directory containing yesterday's backup
YDBKUPDIR="/media/other/backup/yesterday/home"
#directory containing the day before yesterday's backup
DBBKUPDIR="/media/other/backup/daybefore/home"

# log files
LOG="mail.txt"
ERRORLOG="ERROR.txt"

# function that quits and logs on error
exiterror()
{
	# use this function by supplying $LINENO as first arg
	echo "Fatal error caused by line ${1} of ${0}" >> $ERRORLOG
	mail somebody@somedomain.com -s "Backup Job ERROR" -v < $ERRORLOG
	mail somebody@somedomain.com -s "Backup Job Report" -v < $LOG
	exit 1
}

# Step - 2 Start Email Message To Be Sent

# Remove mail message from previous backup
# I do this at the beginning of the script instead of the end
# in case the mail does not send for whatever reason or
# I need to debug it

rm $LOG > /dev/null 2>&1
echo "System Backup $BUTDSTAMP" >> $LOG

# The email sends a user friendly note showing the start and
# end time/dates. This is important so you can compare logs
# and see if a backup ran ok.

echo "Backup Began $(date)" >> $LOG

# Step - 3 Rotate Backups
# Simple three backupset rotation, keeps only last three
# Use directory /media/other/backup as an example, make sure you change this path
# to fit your local settings. Have three directories in /media/other/backup named daily,
# yesterday and daybefore.
# Make sure /media/other/backup is on a different disk (preferably a different machine)
# than the files you are backing up

#looks specifically for backup files in case other files are kept in these directories
for PAIR in $BACKUPPAIRS; do
	TAG=${PAIR/*:}
	[ -d $DBBKUPDIR/$TAG ] && \
        ( rm -fr $DBBKUPDIR/$TAG || exiterror $LINENO )
	[ -d $YDBKUPDIR/$TAG ] && \
        ( mv $YDBKUPDIR/$TAG $DBBKUPDIR/ || exiterror $LINENO )
	[ -d $BKUPDIR/$TAG ] && \
        ( mv $BKUPDIR/$TAG $YDBKUPDIR/ || exiterror $LINENO )
done

# Step - 4 Archive Home Directories
# Creates a seperate tar file for each directory in the directories in $BACKUPPAIRS

# change to dir to backup. Check in case cd failed for some reason.
# cd $BACKUPPAIRS || exiterror $LINENO
#backup all files listed in BACKUPPAIRS
for PAIR in $BACKUPPAIRS; do
	DIRTOBACKUP=${PAIR/:*}
	TAG=${PAIR/*:}
	[ -d $DIRTOBACKUP ] || exiterror $LINENO
	for FOLDERNAME in $DIRTOBACKUP/*
	do
		# Archives are created in the format someuser-$TAG-datetime.tar.gz
		# basename is used here so as not to include absolute paths
		# -p preserves permissions
		echo -e "-------------------\n>>>taring ${FOLDERNAME}\n\n" >> $LOG
		mkdir -p ${BKUPDIR}/${TAG} || exiterror $LINENO
		tar -czvpf ${BKUPDIR}/${TAG}/$( basename $FOLDERNAME )-${TAG}-${BUTDSTAMP}.tar.gz ${FOLDERNAME} >> $LOG || exiterror $LINENO
	done
done

# Step - 5 Finish Email Report and Send

echo "Backup ended $(date)" >> $LOG

# df -h includes a human readable disk usage report of the media that /media/other/backup
# is mounted on. Good to now if your backup disk is running out of space.
# Of course /dev/hdb1 is the device I use, modify it for your local settings

df -h /dev/hdb1 >> $LOG
mail somebody@somedomain.com -s "Backup Job Report" -v < $LOG

# Step - 6 All Done

edit: enclosed some of the commands

dfwcomputer · 09-28-2006, 09:37 AM

o.k gave it a go but i get errors.Not sure what this means (see below)

Code:

root@server:~# sh backup
: command not found
: command not found
: command not found
: command not found
: command not found
'ackup: line 28: syntax error near unexpected token `
'ackup: line 28: `exiterror()
root@server:~#

any ideas

konsolebox · 09-28-2006, 07:42 PM

Text format error. Sorry about that. I tested the code myself and made new changes. Please try again.

Code:

#!/bin/bash

# A Simple (Poor Man) disk based backup script
# Will backup all user home directories in seperate archives
# as a single /home backup file can be quite large and unwieldly.
# Does not need to be modified if users are added/deleted

# Step - 1 Create Timestamp and set up variables and functions
# BUDTSTAMP = Backup Date/Time Stamp
BUTDSTAMP=$(date +%Y%m%d)

# variable holding directories containing files to backup with tags
# format: "DIR:TAG ..."
# eg: BACKUPPAIRS="/homehome-backup /root:root-backup /etc:etc-backup"
BACKUPPAIRS="/home:home-backup /etc:etc-backup"

#directory containing today's backup
BKUPDIR="/media/other/backup/daily"
#directory containing yesterday's backup
YDBKUPDIR="/media/other/backup/yesterday"
#directory containing the day before yesterday's backup
DBBKUPDIR="/media/other/backup/daybefore"

# log files
LOG="mail.txt"
ERRORLOG="ERROR.txt"

# function that quits and logs on error
exiterror()
{
	# use this function by supplying $LINENO as first arg
	echo "Fatal error caused by line ${1} of ${0}" >> $ERRORLOG
	mail somebody@somedomain.com -s "Backup Job ERROR" -v < $ERRORLOG
	mail somebody@somedomain.com -s "Backup Job Report" -v < $LOG
	exit 1
}

# Step - 2 Start Email Message To Be Sent

# Remove mail message from previous backup
# I do this at the beginning of the script instead of the end
# in case the mail does not send for whatever reason or
# I need to debug it

rm $LOG > /dev/null 2>&1
echo "System Backup $BUTDSTAMP" >> $LOG

# The email sends a user friendly note showing the start and
# end time/dates. This is important so you can compare logs
# and see if a backup ran ok.

echo "Backup Began $(date)" >> $LOG

# Step - 3 Rotate Backups
# Simple three backupset rotation, keeps only last three
# Use directory /media/other/backup as an example, make sure you change this path
# to fit your local settings. Have three directories in /media/other/backup named daily,
# yesterday and daybefore.
# Make sure /media/other/backup is on a different disk (preferably a different machine)
# than the files you are backing up

# create the backup directories if they don't exist
[ ! -d $DBBKUPDIR ] && ( mkdir -p $DBBKUPDIR || exiterror $LINENO )
[ ! -d $YDBKUPDIR ] && ( mkdir -p $YDBKUPDIR || exiterror $LINENO )
[ ! -d $BKUPDIR ] && ( mkdir -p $BKUPDIR || exiterror $LINENO )

#looks specifically for backup files in case other files are kept in these directories
for PAIR in $BACKUPPAIRS; do
	TAG=${PAIR/*:}
	[ -d $DBBKUPDIR/$TAG ] && \
        ( rm -fr $DBBKUPDIR/$TAG || exiterror $LINENO )
	[ -d $YDBKUPDIR/$TAG ] && \
        ( mv $YDBKUPDIR/$TAG $DBBKUPDIR/ || exiterror $LINENO )
	[ -d $BKUPDIR/$TAG ] && \
        ( mv $BKUPDIR/$TAG $YDBKUPDIR/ || exiterror $LINENO )
done

# Step - 4 Archive Home Directories
# Creates a seperate tar file for each directory in the directories in $BACKUPPAIRS

# change to dir to backup. Check in case cd failed for some reason.
# cd $BACKUPPAIRS || exiterror $LINENO
#backup all files listed in BACKUPPAIRS
for PAIR in $BACKUPPAIRS; do
	DIRTOBACKUP=${PAIR/:*}
	TAG=${PAIR/*:}
	echo "Doing backups for $DIRTOBACKUP using TAG $TAG" >> $LOG
	[ -d $DIRTOBACKUP ] || exiterror $LINENO
	# if we don't want to include DIRTOBACKUP from the archive, we should
	# note the annoying 'removing trailing lines' tar messages so i included 2>/dev/null
	# cd $DIRTOBACKUP || exiterror $LINENO
	# for FOLDERNAME in *
	for FOLDERNAME in $DIRTOBACKUP/*
	do
		# Archives are created in the format someuser-$TAG-datetime.tar.gz
		# basename is used here so as not to include absolute paths
		# -p preserves permissions
		echo -e "-------------------\n>>> tarring ${FOLDERNAME}" >> $LOG
		mkdir -p ${BKUPDIR}/${TAG} || exiterror $LINENO
		tar -czvpf ${BKUPDIR}/${TAG}/$( basename $FOLDERNAME )-${TAG}-${BUTDSTAMP}.tar.gz ${FOLDERNAME} >> $LOG 2>/dev/null || exiterror $LINENO
		echo >> $LOG
	done
done

# Step - 5 Finish Email Report and Send

echo "Backup ended $(date)" >> $LOG

# df -h includes a human readable disk usage report of the media that /media/other/backup
# is mounted on. Good to now if your backup disk is running out of space.
# Of course /dev/hdb1 is the device I use, modify it for your local settings

df -h /dev/hdb1 >> $LOG
mail somebody@somedomain.com -s "Backup Job Report" -v < $LOG

# Step - 6 All Done

dfwcomputer · 09-28-2006, 08:23 PM

Works great, thanks for spending the time to do that it's much appreciated

is there a simple thing i can change to make the script verbose.Not a big issue just wondering.

Also any ideas on the best way to restore these backup's if failure did happen to occur.

Once again thanks for the script (much appreciated)

Dean

konsolebox · 09-28-2006, 09:00 PM

added verbosity (script.sh -v)

Code:

#!/bin/bash

# A Simple (Poor Man) disk based backup script
# Will backup all user home directories in seperate archives
# as a single /home backup file can be quite large and unwieldly.
# Does not need to be modified if users are added/deleted

# Some variables
# Set to 1 for verbose mode by default (might not work with cron).
VERBOSE=

# Some messaging functions
log() {
	[ "$VERBOSE" ] && echo -e "$@"
	echo -e "$@" >> $LOG
}

log_error() {
	[ "$VERBOSE" ] && echo -e "$@"
	echo -e "$@" >> $LOG
}

# Check args
for a in $*; do
	case $a in
	-v)
		VERBOSE=1
		;;
	*)
		;;
	esac
done

# Step - 1 Create Timestamp and set up variables and functions
# BUDTSTAMP = Backup Date/Time Stamp
BUTDSTAMP=$(date +%Y%m%d)

# variable holding directories containing files to backup with tags
# format: "DIR:TAG ..."
# eg: BACKUPPAIRS="/homehome-backup /root:root-backup /etc:etc-backup"
BACKUPPAIRS="/home:home-backup /etc:etc-backup"

#directory containing today's backup
BKUPDIR="/media/other/backup/daily"
#directory containing yesterday's backup
YDBKUPDIR="/media/other/backup/yesterday"
#directory containing the day before yesterday's backup
DBBKUPDIR="/media/other/backup/daybefore"

# log files
LOG="mail.txt"
ERRORLOG="ERROR.txt"

# function that quits and logs on error
exiterror()
{
	# use this function by supplying $LINENO as first arg
	log_error "Fatal error caused by line ${1} of ${0}"
	mail somebody@somedomain.com -s "Backup Job ERROR" -v < $ERRORLOG
	mail somebody@somedomain.com -s "Backup Job Report" -v < $LOG
	exit 1
}

# Step - 2 Start Email Message To Be Sent

# Remove mail message from previous backup
# I do this at the beginning of the script instead of the end
# in case the mail does not send for whatever reason or
# I need to debug it

rm $LOG > /dev/null 2>&1
log "System Backup $BUTDSTAMP"

# The email sends a user friendly note showing the start and
# end time/dates. This is important so you can compare logs
# and see if a backup ran ok.

log "Backup Began $(date)"

# Step - 3 Rotate Backups
# Simple three backupset rotation, keeps only last three
# Use directory /media/other/backup as an example, make sure you change this path
# to fit your local settings. Have three directories in /media/other/backup named daily,
# yesterday and daybefore.
# Make sure /media/other/backup is on a different disk (preferably a different machine)
# than the files you are backing up

# create the backup directories if they don't exist
[ ! -d $DBBKUPDIR ] && ( mkdir -p $DBBKUPDIR || exiterror $LINENO )
[ ! -d $YDBKUPDIR ] && ( mkdir -p $YDBKUPDIR || exiterror $LINENO )
[ ! -d $BKUPDIR ] && ( mkdir -p $BKUPDIR || exiterror $LINENO )

#looks specifically for backup files in case other files are kept in these directories
for PAIR in $BACKUPPAIRS; do
	TAG=${PAIR/*:}
	[ -d $DBBKUPDIR/$TAG ] && \
        ( rm -fr $DBBKUPDIR/$TAG || exiterror $LINENO )
	[ -d $YDBKUPDIR/$TAG ] && \
        ( mv $YDBKUPDIR/$TAG $DBBKUPDIR/ || exiterror $LINENO )
	[ -d $BKUPDIR/$TAG ] && \
        ( mv $BKUPDIR/$TAG $YDBKUPDIR/ || exiterror $LINENO )
done

# Step - 4 Archive Home Directories
# Creates a seperate tar file for each directory in the directories in $BACKUPPAIRS

# change to dir to backup. Check in case cd failed for some reason.
# cd $BACKUPPAIRS || exiterror $LINENO
# check verbosity
if [ "$VERBOSE" ]; then
	TAROUTPUT="| tee -a $LOG"
else
	TAROUTPUT=">> $LOG 2>/dev/null"
fi
# backup all files listed in BACKUPPAIRS
for PAIR in $BACKUPPAIRS; do
	DIRTOBACKUP=${PAIR/:*}
	TAG=${PAIR/*:}
	log "Doing backups for $DIRTOBACKUP using TAG $TAG"
	[ -d $DIRTOBACKUP ] || exiterror $LINENO
	# if we don't want to include DIRTOBACKUP from the archive, we should
	# note the annoying 'removing trailing lines' tar messages so i included 2>/dev/null
	# cd $DIRTOBACKUP || exiterror $LINENO
	# for FOLDERNAME in *
	for FOLDERNAME in $DIRTOBACKUP/*
	do
		# Archives are created in the format someuser-$TAG-datetime.tar.gz
		# basename is used here so as not to include absolute paths
		# -p preserves permissions
		log "-------------------\n>>> tarring ${FOLDERNAME}"
		mkdir -p ${BKUPDIR}/${TAG} || exiterror $LINENO
		# tar -czvpf ${BKUPDIR}/${TAG}/$( basename $FOLDERNAME )-${TAG}-${BUTDSTAMP}.tar.gz ${FOLDERNAME} >> $LOG 2>/dev/null || exiterror $LINENO
		eval "tar -czvpf ${BKUPDIR}/${TAG}/$( basename $FOLDERNAME )-${TAG}-${BUTDSTAMP}.tar.gz ${FOLDERNAME} ${TAROUTPUT} || exiterror $LINENO"
		log
	done
done

# Step - 5 Finish Email Report and Send

log "Backup ended $(date)"

# df -h includes a human readable disk usage report of the media that /media/other/backup
# is mounted on. Good to now if your backup disk is running out of space.
# Of course /dev/hdb1 is the device I use, modify it for your local settings

df -h /dev/hdb1 >> $LOG
mail somebody@somedomain.com -s "Backup Job Report" -v < $LOG

# Step - 6 All Done