[SOLVED] Rsync output to logfile, what's wrong with this script?

Toadman · 09-09-2018, 07:41 AM

This script to use rsync to back up my system wasn't written by me but someone on the rsync mailing list. From what I can see the script is supposed to write to a file ~/rsyncbackup.txt

Code:

rsync -aWSHPpl --delete-excluded --filter "merge ${EXC_FILE}" / "${BACKUP_DIR}" & > /home/chris/rsyncbackup.txt

Is there something missing on the line above? I see the -l option for log file is there. The entire script is below. It works fine as far as a whole system back up daily to my external USB drive.

Code:

#!/bin/sh
NOW=`date +%Y.%m.%d.%H.%M`
EXC_FILE=exclude.rules
BACKUP_DIR="/media/chris/backup2/snapshot/"
if [ ! -d ${BACKUP_DIR} ];then
	echo "Backup destination directory ${BACKUP_DIR} not exist."
        echo "run 'sudo mkdir ${BACKUP_DIR}' to create. "
	exit 1 
fi

if [ `id -u` != 0 ]; then #check user
        echo "Usage: sudo sh $0" 
        exit 1
fi

if [ ! -f "${EXC_FILE}" ];then #search exludes file
	echo "${EXC_FILE} not found.Exiting..."
	exit 1
fi

rsync -aWSHPpl --delete-excluded --filter "merge ${EXC_FILE}" / "${BACKUP_DIR}" & > /home/chris/rsyncbackup.txt
RETVAL=$?
if [ ${RETVAL} -ne 0 ];then
	echo "RSYNC ERROR: ${ERR} [ exit code: ${RETVAL} ]"
	exit 1 
fi
echo "Whole filesystem mirror in ${BACKUP_DIR} done at ${NOW}" 
exit 0

lougavulin · 09-09-2018, 08:07 AM

-l option is not for log file, but to copy links.

The only option which doesn't seem to be used is -P. I don't see the point to show progress within a script.

Toadman · 09-09-2018, 08:41 AM

You're right, -l is for links. Guess my eyes aren't opened quite all the way yet. Now I'm not sure what this portion of the script is supposed to do

Code:

& > /home/chris/rsyncbackup.txt

Can I add --log-file=~/rsyncbackup.txt here:

Code:

--delete-excluded --filter

and not have it affect the rest of the line

Code:

rsync -aWSHPpl --delete-excluded --filter "merge ${EXC_FILE}" / "${BACKUP_DIR}"

So the new line would read

Code:

rsync -aWSHpl --delete-excluded --filter --log-file=/home/chris/rsyncbackup.txt "merge ${EXC_FILE}" / "${BACKUP_DIR}"

I've removed the 'P' for progress and added the logfile.

Turbocapitalist · 09-09-2018, 08:48 AM

Quote:

Originally Posted by Toadman

Now I'm not sure what this portion of the script is supposed to do

Code:

& > /home/chris/rsyncbackup.txt

That part looks like a mistake. Take away the ampersand & if you wish to collect the output.

Code:

> /home/chris/rsyncbackup.log 2> /home/chris/rsyncbackup.errors.log

lougavulin · 09-09-2018, 08:48 AM

The purpose of :

Code:

& > /home/chris/rsyncbackup.txt

& : to call rsync in background. Which seems wrong when I read the whole script.

Code:

> /home/chris/rsyncbackup.txt

Rsync print messages on the stdouput when it is running. This write all the messages into the file /home/chris/rsyncbackup.txt.

And yes, you can add --log-file option as you did.

pan64 · 09-09-2018, 09:57 AM

not necessarily, probably it was only the space between them.

Code:

rsync .... &> /home/chris/rsyncbackup.txt

may work too (and probably that was written in the original script).

jlinkels · 09-09-2018, 10:42 AM

If your purpose is to back up your system, have a look at rsnapshot. It uses rsync, but someone already invented the wheel. Sooner or later you want your backup to do that anyway.

jlinkels

Toadman · 09-09-2018, 03:28 PM

Thank you all for the suggestions and pointing out the errors. The script now as written:

Code:

#!/bin/sh
NOW=`date +%Y.%m.%d.%H.%M`
EXC_FILE=exclude.rules
BACKUP_DIR="/media/chris/backup2/snapshot/"
if [ ! -d ${BACKUP_DIR} ];then
	echo "Backup destination directory ${BACKUP_DIR} not exist."
        echo "run 'sudo mkdir ${BACKUP_DIR}' to create. "
	exit 1 
fi

if [ `id -u` != 0 ]; then #check user
        echo "Usage: sudo sh $0" 
        exit 1
fi

if [ ! -f "${EXC_FILE}" ];then #search exludes file
	echo "${EXC_FILE} not found.Exiting..."
	exit 1
fi

# rsync -aWSHPpl --delete-excluded --filter "merge ${EXC_FILE}" / "${BACKUP_DIR}" &> /home/chris/rsyncbackup.txt
rsync -aWSHpl --delete-excluded --filter "merge ${EXC_FILE}" / "${BACKUP_DIR}" > /home/chris/rsyncbackup.log 2> /home/chris/rsyncbackup.errors.log 
RETVAL=$?
if [ ${RETVAL} -ne 0 ];then
	echo "RSYNC ERROR: ${ERR} [ exit code: ${RETVAL} ]"
	exit 1 
fi
echo "Whole filesystem mirror in ${BACKUP_DIR} done at ${NOW}" 
exit 0

This is my exclude.rules file

Code:

#Dont forget to add your backup mount point as exclude rule to avoid infinite loop
- /media/chris/backup/*
- /media/chris/backup2/*
- /var/backups/*
- /var/lib/named/proc/*
#Exclude non-local files
- /media/*
- /mnt/*
- /net/*
- /home/*/.gvfs
#Exclude virtual file systems
- /proc/*
- /sys/*
- /run/*
#Exclude temporary files
- /tmp/*
- /var/lock/*
- /var/run/*
- /var/tmp/*
# useless for me - udev create nodes as needed
- /dev/*

I made a run and everything worked perfectly. No errors noted to the ~/rsyncbackup.errors.log or the ~/rsyncbackup.log
I'll take a look at rsnapshot to replace what I have now.

As far as I'm concerned this thread is solved and I'll mark as such. Thanks again for all the assistance.

FredGSanford · 09-10-2018, 12:00 PM

What happens if the backup device is not mounted or turned on/unplugged?

Toadman · 09-10-2018, 03:29 PM

Good question and ever since I saw it I've been Googling for some kind of answer. Assuming it would be an I/O error. I did find mention of I/O errors in the rsysnc manpage

Code:

If the sending side detects any I/O errors, then the deletion of any files at the destination will be automatically disabled. This is to prevent temporary filesystem failures (such as NFS errors) on the sending side causing a massive deletion of files on the destination. You can override this with the --ignore-errors option.

Reading further in the manpage I found this which I should add to the command line

Code:

--timeout=TIMEOUT
This option allows you to set a maximum I/O timeout in seconds. If no data is transferred for the specified time then rsync will exit. The default is 0, which means no timeout.

so now my command line looks like

Code:

rsync -vaWSHpl --timeout=15 --delete-excluded --filter "merge ${EXC_FILE}" / "${BACKUP_DIR}" > /home/chris/rsyncbackup.log 2> /home/chris/rsyncbackup.errors.log

Which I believe should prevent rsync from just sitting there churning away trying to write to a dead USB drive.

jlinkels · 09-10-2018, 07:36 PM

It is much worse when an external device is not detected. Then the backup is written to the mounting directory on the local drive. That is really fun to try to write you 400 GB backup to the 10GB root partition.

You can use this to check if your drive is mounted:

Code:

#!/bin/bash

mount_point='/mnt/ext_daily'

df -h | grep $mount_point > /dev/null
if [ $? -eq 0 ]
then
    rsync -uav --exclude='/mnt' --exclude='/proc' --exclude='/sys' --exclude='/dev' --exclude='/data/movies' --exclude='/vmware' --delete / /mnt/ext_daily
#> /var/log/rsync_daily
    echo "mount point $mount_point exists, rsync started"
else
    echo "Error: mount point $mount_point does not exist, rsync operation skipped"
fi

You rsync command is different of course.

jlinkels

Toadman · 09-10-2018, 08:53 PM

Great, thanks very much for that. I'll modify the script tomorrow and give it a test run both ways. I forgot to ask last night. Can I integrate the commands

Code:

mount_point='/media/chris/backup2/snapshot/'

df -h | grep $mount_point > /dev/null
if [ $? -eq 0 ]
then

into the existing script? Or, are these lines that check for the existence of the destination directory and if not found exits

Code:

BACKUP_DIR="/media/chris/backup2/snapshot/"
if [ ! -d ${BACKUP_DIR} ];then
	echo "Backup destination directory ${BACKUP_DIR} not exist."
        echo "run 'sudo mkdir ${BACKUP_DIR}' to create. "
	exit 1 
fi

sufficient to stop the rsync process?