LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 11-12-2009, 10:15 AM   #1
tqz
Member
 
Registered: Jan 2008
Posts: 67

Rep: Reputation: 15
rsync incremental backups


Hello all

How do you get Rsync to do incremental backups rather than full backups? At the moment I have a script that will create a backup folder (if it doesnt already exist) then copy the source files into the backup directory with the command

rsync $VERBOSE --exclude=$TARGET/ $EXCLUDE --exclude '/Ls-wtgl1c8/**' -rt --delete $source/ $TARGET/$source/ >> $LOG_FILE

Target is where the files will be backed up to
Sources is the dir(s) to be backed up
Exclude files is the list of files not to backup
log file is where the output will be saved to.

At the moment it only does full backups, but I would only like to do incremental, how would this be achieved? Am I missing out an option in the Rsync that is required.

Many thanks in advance.

t.
 
Old 11-12-2009, 11:04 AM   #2
centosboy
Senior Member
 
Registered: May 2009
Location: london
Distribution: centos5
Posts: 1,137

Rep: Reputation: 116Reputation: 116
Quote:
Originally Posted by tqz View Post
Hello all

How do you get Rsync to do incremental backups rather than full backups? At the moment I have a script that will create a backup folder (if it doesnt already exist) then copy the source files into the backup directory with the command

rsync $VERBOSE --exclude=$TARGET/ $EXCLUDE --exclude '/Ls-wtgl1c8/**' -rt --delete $source/ $TARGET/$source/ >> $LOG_FILE

Target is where the files will be backed up to
Sources is the dir(s) to be backed up
Exclude files is the list of files not to backup
log file is where the output will be saved to.

At the moment it only does full backups, but I would only like to do incremental, how would this be achieved? Am I missing out an option in the Rsync that is required.

Many thanks in advance.

t.
the default action of rsync is to do incremental backups by doing a checksum comparison of source and destination and transferring only the content change in files...
 
Old 11-12-2009, 11:25 AM   #3
tqz
Member
 
Registered: Jan 2008
Posts: 67

Original Poster
Rep: Reputation: 15
See this is what I thought it should be doing by default - incremental backups. But even when no files have changed it seems to be deleting the files in my backup directory (I can see it in the trash can of my link station and what is being backed up in my log file), and backing up all the files again. So dont understand why a full backup is being performed. I'm going to take a look at my code again, if I cannot see anything obvious will post here fOR some more help!

Thanks centosboy for your rep!

Last edited by tqz; 11-12-2009 at 11:28 AM. Reason: Adding more detail
 
Old 11-13-2009, 05:15 AM   #4
tqz
Member
 
Registered: Jan 2008
Posts: 67

Original Poster
Rep: Reputation: 15
I cant see anything obvious wrong...so heres the code...any help would be greatly appreciated!

!/bin/sh
# backup.sh -- backup to a local drive using rsync

# Directories to backup.
SOURCES="/home /etc /root /usr /boot /var /"
#for testing uncomment
#SOURCES="/home/abc"

# Directory to backup to.


BACKUPDIR="dailybackup"
LOG_FILE="/etc/rsyncbackup.txt"



TARGET="/Linkstation/"$BACKUPDIR

#EXCLUDE_FILE tells rsync what NOT to backup.

EXCLUDE_FILE="/bin/exclude_file.txt"

# delete contents of log so it doesnt grow too large
echo > $LOG_FILE

#add date to the first line of the log file
string=`date +%m/%d/%y`
echo $string >> $LOG_FILE


# Comment out the following line to disable verbose output
VERBOSE="-v"
###########################

if [ ! -x $TARGET ]; then
echo "Creating directory"
mkdir $TARGET/$BACKUP_DATE
fi
if [ ! -x $TARGET ]; then
echo "Backup target does not exist or you don't have permission!"
echo "Exiting..."
exit 2
fi

echo "Verifying Sources..."
for source in $SOURCES; do
echo "Checking $source..."
if [ ! -x $source ]; then
echo "Error with $source!"
echo "Directory either does not exist, or you do not have proper permissions."
exit 2
fi
done

if [ -f $EXCLUDE_FILE ]; then
EXCLUDE="--exclude-from=$EXCLUDE_FILE"
fi

echo "Sources verified. Running rsync..."
for source in $SOURCES; do

# Create directories in $TARGET to mimick source directory hiearchy
if [ ! -d $TARGET/$source ]; then
mkdir -p $TARGET/$source
fi

rsync $VERBOSE --exclude=$TARGET/ $EXCLUDE --exclude '/Linkstation/**' -rt --delete $source/ $TARGET/$source/ >> $LOG_FILE
done

exit 0
 
Old 11-13-2009, 10:27 AM   #5
tqz
Member
 
Registered: Jan 2008
Posts: 67

Original Poster
Rep: Reputation: 15
Think I know what the problem is nothing to do with the code...will know for sure on Tuesday then will update the post if I rememeber.
 
Old 01-21-2010, 08:11 AM   #6
tqz
Member
 
Registered: Jan 2008
Posts: 67

Original Poster
Rep: Reputation: 15
Back to investigating this problem! It wasnt what i thought the problem was...so does anyone have any ideas??? I have done some expermimenting with different scripts still the same problem. Also I discoverd that If i was backing up a dir such as /home/user incremental backups will work. But if i was to backup a directory like /home/domain/user then it doesnt work...any suggestions as to why this may be???

Many thanks in advacnce

t.
 
Old 01-21-2010, 10:25 AM   #7
tqz
Member
 
Registered: Jan 2008
Posts: 67

Original Poster
Rep: Reputation: 15
Okay so I have done some more investigating and have found the following:-

It seems that the reason why a full backup is being performed rather than an incremental is because the modification times of the files are different when the files are copied over.

So if i had a file called test.doc on my ubuntu server its file properties may be:- Created 18 Jan 2010 10:45:11
Modified 18 Jan 2010 10:45:11

Rsync copies the file over to the NAS and the file properties for test.doc on there would be:- Created 18th April 2009 17:44
:- Modified 18 Jan 2010 10:45:10

The creation times are completely different but the modification times seem to be out by a second.

Is this simply a case of changing the NAS box date and time settings (I dont have the perms to do this, so cant just simply try this that why posting on here before getting others involved)...

Thanks for any responses in advance!
 
Old 01-21-2010, 06:26 PM   #8
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,311

Rep: Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040Reputation: 2040
The fact that the files created by rsync have an mtime < src files indicates you haven't got your systems time-synced; see ntp.
Should be the other way round if anything...
Try adding a few flags eg

-t, --times preserve modification times

http://linux.die.net/man/1/rsync
 
Old 01-25-2010, 08:20 AM   #9
tqz
Member
 
Registered: Jan 2008
Posts: 67

Original Poster
Rep: Reputation: 15
Hi chrism01 and thanks for your reply!

I have got someone to change the NAS boxes to the correct date and time and tried to synch the time between my NAS boxes and my ubuntu server by making the NTP SERVER - ntp.ubuntu.com for both. But still the problem remains. The only diff is that the
creation time is 1 second out just like the modification time now.

I have used flags in my rsync command - t to preserve mod times and r (recursive)...

I have a feeling that the full backups are being perfomed for when my files are being shared over the samba server so the files stored on my ubuntu server being accessed by windows pc....

Any other suggestions???

t.
 
Old 09-15-2012, 04:22 AM   #10
DeeDeeRamone
LQ Newbie
 
Registered: Jul 2008
Location: Santander, Spain
Distribution: Ubuntu 10.10 and Fedora 16 x86_64
Posts: 9

Rep: Reputation: 0
FAT fs saves times by 2sec. interval

Could be that your NAS boxes a FAT fs?

If this is the case you can see on the man page:
"In particular, when transferring to or from an MS Windows FAT filesystem (which represents times with a 2-second resolution), --modify-window=1 is useful (allowing times to differ by up to 1 second)."


Regards
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Incremental backups? arashi256 Linux - Newbie 9 07-06-2009 09:46 PM
Rsync for incremental backups Meson Linux - General 1 10-30-2007 10:44 AM
Image and Incremental backups with rsync kaplan71 Linux - Software 2 08-13-2007 04:42 PM
LXer: Create Incremental Snapshot-style Backups With rSync And SSH LXer Syndicated Linux News 0 08-13-2006 09:21 PM
Rsync for incremental backups? Phaethar Linux - Software 3 12-04-2003 02:27 PM


All times are GMT -5. The time now is 08:17 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration