LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 05-09-2017, 06:45 PM   #1
haertig
Senior Member
 
Registered: Nov 2004
Distribution: Debian, Ubuntu, LinuxMint, Slackware, SysrescueCD, Raspbian, Arch
Posts: 2,331

Rep: Reputation: 357Reputation: 357Reputation: 357Reputation: 357
Question: rsync with dynamic directory determination?


I am trying to figure out a way to initiate an rsync on LOCAL that connects to a rsync daemon on REMOTE. REMOTE is Linux, LOCAL is Linux initially, but I'm hoping to add some Windows machines as LOCAL later.

With the following functions/features:

The rsync daemon on REMOTE defines a module for "LOCAL" and within that module's directory structure, I want snapshot directories named like this:

Backup_2017-05-07_00:00:01
Backup_2017-05-08_00:00:01
Backup_2017-05-09_00:00:01

These are not full backup directories - I will be using --link-dest to create hard links to previous backups instead. But each backup directory will APPEAR to be a full backup.

The specific directory names don't have to be exactly this, my point being that I want the snapshot directories named by date, not some arbitrary thing like Backup.1, Backup.2, Backup.3, etc. I prefer date-naming for less sophisticated users who might have trouble determining when an arbitrarily named "Backup.3" was actually created.

So I want to calculate --link-dest dynamically. 99% of the time the --link-dest will have a filename based on yesterdays date, but on some occasions maybe the backup didn't occur yesterday (an extended household power failure, etc.) So I want to dynamically figure out the most recent backup directory, whether that be one day ago, three days ago, a week ago, etc. If I were on the REMOTE machine (where rsync daemon runs), I would run a command like "ls -t Backup_* | head -1" to determine this most recent directory.

Is there a way I can dynamically determine --link-dest at run time by running a command on the destination (rsync daemon) machine? I know I could write a script on LOCAL to call ssh to REMOTE before calling rsync, and have that ssh command determine the --link-dest. But I was hoping to do it all within the rsync command itself. (Reason: eventually I hope to port this to Windows too, and I'd have to create a functionally equivalent Windows script to determine --link-dest ... and I hate scripting in Windows!)

An alternative to this would be to always create a symlink to the latest backup dir (the destination dir on REMOTE) and name that symlink "Backup_current". Then I could use --link-dest=Backup_current (I am assuming that --link-dest will follow a symlink, but I haven't tried that yet to verify). So in this alternate strategy, the question becomes, "Is there a way to have rsync daemon automatically update this symlink after completing a backup?" Again, I could write a script to do this on LOCAL, calling ssh to REMOTE after the rsync has competed. But I hope to do this all in a single rsync command. I could kludge this by running a cron job on REMOTE that asynchronously creates the symlink (coordinating that so the symlink isn't changed in the middle of a running backup, although that would probably be OK - would have to test for dure).

My description above is about as clear as mud. But if anybody actually understands what it is I'm trying to do, I'd love to hear any suggestions or alternate approaches. Thanks!
 
Old 05-10-2017, 02:34 PM   #2
scasey
LQ Veteran
 
Registered: Feb 2013
Location: Tucson, AZ, USA
Distribution: CentOS 7.9.2009
Posts: 5,727

Rep: Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211Reputation: 2211
rsnapshot [rsnapshot.org] which is a perl script running rsync will build a directory structure like this:
Code:
# ll /.snapshots/
total 56
drwxr-xr-x 4 root root 4096 May 10 01:39 daily.0
drwxr-xr-x 4 root root 4096 May  9 01:39 daily.1
drwxr-xr-x 4 root root 4096 May  8 01:40 daily.2
drwxr-xr-x 4 root root 4096 May  7 01:39 daily.3
drwxr-xr-x 4 root root 4096 May  6 01:39 daily.4
drwxr-xr-x 4 root root 4096 May  5 01:39 daily.5
drwxr-xr-x 4 root root 4096 May  4 01:39 daily.6
drwxr-xr-x 4 root root 4096 Apr  3 01:42 monthly.0
drwxr-xr-x 4 root root 4096 Feb 27 01:36 monthly.1
drwxr-xr-x 4 root root 4096 Jan 30 01:38 monthly.2
drwxr-xr-x 4 root root 4096 May  1 01:39 weekly.0
drwxr-xr-x 4 root root 4096 Apr 24 01:39 weekly.1
drwxr-xr-x 4 root root 4096 Apr 17 01:39 weekly.2
drwxr-xr-x 4 root root 4096 Apr 10 01:38 weekly.3
Each directory contains
Code:
# ll /.snapshots/daily.0
total 8
drwxr-xr-x 5 root root 4096 Jul 21  2012 localhost
drwxr-xr-x 7 root root 4096 Jul 25  2012 SCA
Where localhost is the snapshot of the local server and SCA is the snapshot from the remote server "pulled" over SSH.
This doesn't provide the directory naming you were asking for, but might be a serviceable given the timestamps on the directories.
It is certainly very easy to set up and configure.

HTH

Sean
 
Old 05-11-2017, 11:35 AM   #3
haertig
Senior Member
 
Registered: Nov 2004
Distribution: Debian, Ubuntu, LinuxMint, Slackware, SysrescueCD, Raspbian, Arch
Posts: 2,331

Original Poster
Rep: Reputation: 357Reputation: 357Reputation: 357Reputation: 357
Thanks - I am investigating a couple different backup solutions to suppliment what I'm currently doing. One, something I'd craft myself using rsync. Two, something like you mentioned, rsnapshot. Three, a more customizeable client/server backup solution like UrBackup.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync with --link-dest: Question about interrupted rsync session bvz Linux - Server 9 09-10-2016 11:31 AM
Dynamic mount of Active Directory Share ronw69 Linux - Enterprise 3 02-16-2012 06:07 PM
IP determination rapa Programming 5 06-22-2009 12:40 AM
Automatic rsync restart when dynamic IP changes joma Slackware 6 04-19-2006 05:50 AM
Dynamic route determination not working 10.0 tallthom Linux - Networking 21 09-27-2004 06:19 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 04:49 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration