LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 04-05-2017, 06:10 PM   #1
Enquirer
LQ Newbie
 
Registered: Apr 2017
Posts: 7

Rep: Reputation: Disabled
incremental backups with rsync -link-dest


Hi,
I realise that this is a very common subject but I cannot find a thread with the same specific setup and yes I have referred to rsync.man.

My situation is this;

I have a data server and a backup server which I am setting up to do backups of the media server. I have written a script to carry out the backups which works very nicely with the exception of an error on the link-dest saying that the directory does not exist when it clearly does.

Actual names changed but this is the setup;

The media server "MediaServer" contains a shared data drive "DataDrive" which I wish to backup. At the moment I wish to do full backups on a monthly basis (3 months on a 28 day monthly cycle) with daily incremental backups from the current month.

So the backup server "BackupServer" contains a directory "BackupDataDrive" with two sub directories "Monthly" and "Daily". "Monthly" contains 3 backup directories "Month1", Month2" and "Month3" while "Daily" contains the directories "Day1" through to "Day28".

Straight-forward enough!

There is a control file "daynum.txt" and "backup.log" is created by each backup and then moved to the correct backed up directory at the end of the backup process.

On each day 28 "Month3" is deleted, Months 1 and 2 renamed to 2 and 3 and a new "Month1" created, then a full backup is done to this. Finally day is set to 1 before a daily backup is done.

On each other day the "dayn" directories are rolled on in the same way and then a new backup done to "day1".

for the daily backup I am using;

rsync -a -y --link-dest=BackupDataDrive/Monthly/Month1/ -e ssh mediaserver_IP_address:/path_to_datadrive/DataDrive/ BackupDataDrive/Daily/Day1

This all works without problem except that when each daily (incremental) backup is performed I get an error telling me that link-dest: BackupDataDrive/Monthly/Month1 does not exist when it quite clearly does. I have checked and double checked the spelling etc.

As far as I can see this is all as it should be, I want it to the latest full monthly backup (BackupDataDrive/Monthly/Month1) as the comparison so that when it backs up MediaServerataDrive to BackupDataDrive/Daily/Day1 it should only backup those file which have changed since that full backup.

Again I apologise but all the threads I have seen have been client based backups rather than run from the server, using windows machines or whatever and none have had the problem of the link-dest directory not being found.

The only other thing I can think of is that it expects then link-dest to be on the remote machine, in this case the client, which would be stupid as that full backup should obviously be on the backup server, the local machine.

Any help would be greatly appreciated, thanks in advance.
 
Old 04-05-2017, 06:19 PM   #2
Enquirer
LQ Newbie
 
Registered: Apr 2017
Posts: 7

Original Poster
Rep: Reputation: Disabled
One thing I should maybe add is that having given the error message, it then performs a full backup to day1 as if the link-dest was not there.
 
Old 04-05-2017, 09:18 PM   #3
jailbait
LQ Guru
 
Registered: Feb 2003
Location: Mineral, Virginia
Distribution: Debian 11
Posts: 7,908

Rep: Reputation: 354Reputation: 354Reputation: 354Reputation: 354
I think that you are using a hard link to BackupDataDrive/Monthly/Month1. When you delete Month1 and then create a new Month1 the hard link is then pointing to a different location than the new Month1 directory.

I suggest either using a symbolic link or else deleting the Month1 contents without deleting and recreating the Month1 directory.

------------------------
Steve Stites

Last edited by jailbait; 04-05-2017 at 09:23 PM.
 
Old 04-06-2017, 06:18 AM   #4
Enquirer
LQ Newbie
 
Registered: Apr 2017
Posts: 7

Original Poster
Rep: Reputation: Disabled
Many thanks for that jailbait it made me think more about relativity. On going back to rsync.man I reread the details for compare-dest, link-dest and copy-dest and noticed the sentence;

Quote:
If DIR is a relative path, it is relative to the destination directory. See also --compare-dest and --copy-dest.
Of course "relative to the destination" directory. This means that it needs to start with "../../" to bring it back up from Daily/Day1 before going back down to Monthly/Month1. It couldn't find the directory Daily/Day1/Monthly/Month1 because it was correct, it didn't exist!

I am retesting it now, it hasn't shown the error message, and will let you know, hopefully confirm, that it is now ok.

Always something stupid and simple, in this case, me.

Will hope fully let you know soon.
 
Old 04-11-2017, 07:49 AM   #5
jmgibson1981
Member
 
Registered: Jun 2015
Location: Tucson, AZ USA
Distribution: Ubuntu
Posts: 747

Rep: Reputation: 241Reputation: 241Reputation: 241
This is what I'm using to keep my stuff backed up to my server. It's over the network but should help to understand how it works. Found this on google and modified to my needs.

Code:
#!/bin/bash
# tadaen sylvermane | jason gibson
# last edit date - 2016/9/20
# version 1.0

##### variables #####

NOW=$(date +%Y.%m.%d.%H.%M) # time when script is run
RSYNCOPT=auz # /usr/bin/rsync options
SERVER=failbox.mylan.home # server ip
FOLDER=$(basename "$1")
BACKUPPATH="$USER"/"$HOSTNAME"/"$FOLDER"

##### begin script #####

if ping -c 1 "$SERVER" ; then
	if [[ -e "$1" ]] ; then
		if ssh "$USER"@"$SERVER" "[[ ! -e ${2}/${BACKUPPATH} ]]" ; then
			ssh "$USER"@"$SERVER" "mkdir -p ${2}/${BACKUPPATH}"
		fi
		if ssh "$USER"@"$SERVER" "[[ -e ${2}/${BACKUPPATH}/current ]]" ; then
			rsync -"$RSYNCOPT" --link-dest="$2"/"$BACKUPPATH"/current "$1" "$USER"@"$SERVER":"$2"/"$BACKUPPATH"/"$FOLDER"."$NOW"
			ssh "$USER"@"$SERVER" "rm -f ${2}/${BACKUPPATH}/current"
		else
			rsync -"$RSYNCOPT" "$1" "$USER"@"$SERVER":"$2"/"$BACKUPPATH"/"$FOLDER"."$NOW"
		fi
		ssh "$USER"@"$SERVER" "ln -s ${2}/${BACKUPPATH}/${FOLDER}.${NOW} ${2}/${BACKUPPATH}/current"
		ssh "$USER"@"$SERVER" "find ${2}/${BACKUPPATH}/ -type d -maxdepth 1 -mtime +${3} -exec rm -rf {} \;"
	fi
fi

##### end script #####
 
Old 05-01-2017, 01:52 PM   #6
Enquirer
LQ Newbie
 
Registered: Apr 2017
Posts: 7

Original Poster
Rep: Reputation: Disabled
Right, firstly v.sorry for great delay, I've been very busy and this is unfortunately well down on the list at the moment but still important.

I've still not had any luck and am trying to keep it as simple as possible at least untilI've got it working, then I can go back to making it more usable on different backups with out changing it just the parameters passed to it.

So I looked at some other posts on the web and decided to try the most basic I could find and did this;

mkdir 1
mkdir 2
mkdir 3
echo "file1" >1/file1.txt
echo "file2" >1/file2.txt
cp 1/file1.txt 2

so I had file1.txt and file2.txt in 1, file1.txt in 2 and nothing in 3.

I then did rsync -a --compare-dest=../2 1/ 3/ and it copied both files from 1 to 3!

I emptied 3 and tried rsync -a --compare-dest=/fullpathto/2 1/ 3/ and it copied both files from 1 to 3 again!

I emptied 3 again and tried rsync -a --compare-dest=2 1/ 3/ and it errored "--compare-dest arg does not exist: 2" which I expected.

I have tried rsync -a --compare-dest=../2 1 3 and it produced a file named "1" in directory 3.

So I'm completely baffled as I can't even get this simple test to work and from the posts I've seen elsewhere this is a very common problem.

Even in my final setup I am not bothered about compression etc. as space will not be that much of a problem but I obviously don't want to do full backups every day.
 
Old 05-06-2017, 02:29 AM   #7
Beryllos
Member
 
Registered: Apr 2013
Location: Massachusetts
Distribution: Debian
Posts: 529

Rep: Reputation: 319Reputation: 319Reputation: 319Reputation: 319
Quote:
Originally Posted by Enquirer View Post
...So I looked at some other posts on the web and decided to try the most basic I could find and did this;
Code:
mkdir 1
mkdir 2
mkdir 3
echo "file1" >1/file1.txt
echo "file2" >1/file2.txt
cp 1/file1.txt 2
so I had file1.txt and file2.txt in 1, file1.txt in 2 and nothing in 3.

I then did
Code:
rsync -a --compare-dest=../2 1/ 3/
and it copied both files from 1 to 3!
rsync didn't use 2/file1.txt because that file had a different timestamp than 1/file1.txt. If you create 2/file1.txt instead by cp --preserve (or rsync -a), then the timestamps will match, --compare-dest will work, and rsync will copy only the other file, as you expected:
Code:
$ mkdir 1
$ mkdir 2
$ mkdir 3
$ echo "file1" >1/file1.txt
$ echo "file2" >1/file2.txt
$ cp --preserve 1/file1.txt 2
$ ls *
1:
file1.txt  file2.txt

2:
file1.txt

3:
$ rsync -av --compare-dest=../2 1/ 3/
sending incremental file list
./
file2.txt

sent 128 bytes  received 34 bytes  324.00 bytes/sec
total size is 12  speedup is 0.07
$ ls *
1:
file1.txt  file2.txt

2:
file1.txt

3:
file2.txt
$

Last edited by Beryllos; 05-06-2017 at 02:42 AM. Reason: realized --preserve is the simplest option for this application of cp
 
Old 05-06-2017, 03:25 AM   #8
Beryllos
Member
 
Registered: Apr 2013
Location: Massachusetts
Distribution: Debian
Posts: 529

Rep: Reputation: 319Reputation: 319Reputation: 319Reputation: 319
Quote:
Originally Posted by Enquirer View Post
... Of course "relative to the destination" directory. This means that it needs to start with "../../" to bring it back up from Daily/Day1 before going back down to Monthly/Month1. It couldn't find the directory Daily/Day1/Monthly/Month1 because it was correct, it didn't exist!

I am retesting it now, it hasn't shown the error message, and will let you know, hopefully confirm, that it is now ok.
Yes, using the correct relative directory solves the "--link-dest arg does not exist" problem.

Also I wanted to make sure you understand what the result of --link-dest looks like. It creates hardlinks in the destination directory, so it will appear (by ls, for example) that all of the files were copied. Use the rsync -v option to see which files are actually being transferred, or after the transfer, you can use ls -i to see that a hardlink in the destination points to the same inode (same disk space) as the corresponding file in the --link-dest directory.

In fact, there is no difference between the original "file" and the newly-created "hardlink." Technically they are both hardlinks to the same inode. If you use rsync --linkdest efficiently, you can have inodes (and therefore disk space) shared by hardlinks in many directories, all on a completely equal footing. Among other things, this means you can delete or prune older backups in any order without losing any files that are hardlinked in the remaining backups.

Last edited by Beryllos; 05-06-2017 at 03:38 AM.
 
Old 05-06-2017, 04:44 PM   #9
scasey
LQ Veteran
 
Registered: Feb 2013
Location: Tucson, AZ, USA
Distribution: CentOS 7.8.2003
Posts: 5,491

Rep: Reputation: 2115Reputation: 2115Reputation: 2115Reputation: 2115Reputation: 2115Reputation: 2115Reputation: 2115Reputation: 2115Reputation: 2115Reputation: 2115Reputation: 2115
I know little (nearly nothing) about rsync, but am using rsnapshot [rsnapshot.org] -- a perl script that uses rsync and and a configuration file to manage routine backups. HTH
Sean
 
Old 05-10-2017, 07:49 AM   #10
Enquirer
LQ Newbie
 
Registered: Apr 2017
Posts: 7

Original Poster
Rep: Reputation: Disabled
Very many thanks to Beryllos.
The timestamps were causing the problem, I had not even considered this.
During my initial writing and testing of the script, I had made the mistake with the relative path and had copied the directories without the --preserve option when resetting for the next test so subsequent tests were doomed to failure regardless of any changes I made.
Good old school-boy errors!
Thanks too for the info on the hardlinks. As you can see, I had decided, at least during testing, to not select the -h option for simplicity until I had the rest sorted out.
After retesting using the --preserve option I did the ls with the -i option and it did in fact show that every file, all directories, did have different inodes showing that there were no hardlinks.
So thanks to your help I can now go back to the "real" script and finish it off.
Once again, many thanks, I am eternally grateful.
 
Old 05-10-2017, 07:50 AM   #11
Enquirer
LQ Newbie
 
Registered: Apr 2017
Posts: 7

Original Poster
Rep: Reputation: Disabled
Thanks to scasey, I will have a look at rsnapshot which may prove useful in the future.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync with --link-dest: Question about interrupted rsync session bvz Linux - Server 9 09-10-2016 12:31 PM
rsync incremental backups tqz Linux - Newbie 9 09-15-2012 04:22 AM
using rsync for incremental backups cccc Linux - Server 5 01-29-2010 07:02 AM
Problem using rsync for backups: link-dest JosephS Linux - Software 0 02-21-2008 08:43 AM
Rsync for incremental backups? Phaethar Linux - Software 3 12-04-2003 02:27 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 03:12 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration