LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 02-19-2016, 12:05 PM   #1
bioshock
LQ Newbie
 
Registered: May 2015
Location: UK
Distribution: Centos 7.1
Posts: 8

Rep: Reputation: Disabled
rsync the right tool for the job?


Not sure if this should be in the programming section or here, however I'll see what happens.

Basically I'm trying to create a script using rsync and I want to confirm it's correct and make sure there's not a better way of doing it.

The gist is I have two directories, with the second one holding a copy of the first one.
The first copy will be a full copy.
Any deltas after that will be created in a subfolder of the second copy along with a list of what's changed.
If the first folder is empty then I want to throw an error.
If files are deleted from the first copy I want to log the deletions but not delete from the second copy.

So I have the empty check:-
empty= `rsync directory1/ wc -l`
if [ $empty = 1]; then
echo "Something has gone wrong, exiting."
exit 1
fi

logfile= output_$(date "+%d_%m-%Y_%H-%M-%S")

I do a deleted file from original check:-
rsync -avz --delete --dry-run directory1/ directory2 > $logfile

I then do delta copies into a new folder, first copy will be a full one, anything after that will be in a sub-folder:-
delta= output_$(date "+%d_%m-%Y")
if [ directory2 is empty]; then
rsync -avz directory1/ directory2 > $logfile
else
rsync -avz --backup --backup-dir=$delta directory1/ directory2 > $logfile

Hopefully that makes sense to someone :-)
Thanks for reading, please let me know if it's junk or where I could do it better.

Thanks
B
 
Old 02-19-2016, 12:32 PM   #2
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,573

Rep: Reputation: 2138Reputation: 2138Reputation: 2138Reputation: 2138Reputation: 2138Reputation: 2138Reputation: 2138Reputation: 2138Reputation: 2138Reputation: 2138Reputation: 2138
It sounds rather complicated. Why not just do a simple logged incremental backup? Nothing ever gets deleted until you manually do it, but each backup directory will be a complete copy of the original at that time, using only the space of the files that had changed since the previous run (the rest are hard-linked from the previous backup), and the log will tell you what changed.

eg:
Code:
symbackup=/path/to/backup-latest
prevbackup=$(readlink "$symbackup")
currbackup=/path/to/backup_$(date +%Y%m%d_%H%M%S)

if [[ -z "$prevbackup" ]]; then
   add=""
else
   add="--link-dest=$prevbackup"
fi

rsync -aAHXv $add /path/to/source/ "$currbackup/" &> "$currbackup.log"
errstat=$?
if [[ $errstat -eq 0 ]]; then
   rm -f "$symbackup"
   mv "$currbackup" "${currbackup}.complete"
   ln -s "${currbackup}.complete" "$symbackup"
fi
Then you can go back and delete the dated incremental backups whenever you want without affecting the others. This won't log deletions, since deleted files simply don't get copied/linked into the new backup, but you could throw your "--delete --dry-run" command in there as well to log those.
 
Old 02-19-2016, 02:49 PM   #3
sneakyimp
Senior Member
 
Registered: Dec 2004
Posts: 1,056

Rep: Reputation: 78
Have you considered using git? It's designed for this kind of thing and it's flexible and powerful.
 
Old 02-19-2016, 05:05 PM   #4
bioshock
LQ Newbie
 
Registered: May 2015
Location: UK
Distribution: Centos 7.1
Posts: 8

Original Poster
Rep: Reputation: Disabled
Thanks suicidaleggroll, I don't quite understand all the code hence why it's in the newbie section but I'll read up on it.
It's not for a backup, more to keep two directories in sync and log any changes between the two.

Don't know enough about git other than to pull and make commits sneakyimp. I don't have access to github so am not sure if that will make a difference?
I'd also need plenty of logging?
 
Old 02-20-2016, 04:03 AM   #5
Elizine
Member
 
Registered: Aug 2015
Posts: 54
Blog Entries: 1

Rep: Reputation: Disabled
`rsync' is the right tool for the job. If you need under one minute refreshing, you'll need to write a script to put `rsync' in a loop as `cron's most granular unit is per minute.
 
Old 02-20-2016, 03:27 PM   #6
sneakyimp
Senior Member
 
Registered: Dec 2004
Posts: 1,056

Rep: Reputation: 78
Quote:
Originally Posted by bioshock View Post
Don't know enough about git other than to pull and make commits sneakyimp. I don't have access to github so am not sure if that will make a difference?
I'd also need plenty of logging?
Git might be overkill for this -- and would likely require more storage because it keeps old copies of files when those files are changed.

I suggested git because I have found it useful for deploying changes from a local working directory to my web servers. It sounded like you might be doing something similar This command can list changes made:
Code:
git log --name-only --oneline
It accepts additional parameters to limit the output or by who committed, etc.

You don't need github for git to work. You can install it locally. On Debian or Ubuntu, you can install it using apt
Code:
sudo apt-get install git
The setup would be that you have a repo somewhere on your machine. You create your own working directory where you push changes to the repo. Then you could clone this repo to your "copy" directory and just pull changes periodically.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Finding the right tool for the job LXer Syndicated Linux News 0 12-01-2015 09:42 AM
rsync not running in cron job; please help Ravishankarappa Linux - Newbie 8 08-17-2011 01:11 PM
Cannot execute rsync as a cron job jdaniels73 Linux - Software 2 09-03-2006 06:03 AM
best tool for the job or how to tell what distro is best for me Desidarius Linux - Distributions 1 10-08-2004 10:55 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 04:12 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration