Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum. |
Notices |
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
Are you new to LinuxQuestions.org? Visit the following links:
Site Howto |
Site FAQ |
Sitemap |
Register Now
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
|
 |
06-07-2014, 10:51 AM
|
#1
|
Senior Member
Registered: Aug 2007
Location: Zurich, Switzerland
Distribution: Gentoo
Posts: 1,001
Rep: 
|
SW to keep directory synchronized over multiple PCs
Hi
I'm looking for some software that keeps the contents of a directory (files & subdirs) in sync between multiple PCs and a central host.
Desired result:
- when I add/remove on PC 1 the file X to/from the directory and synchronize the dir with the host, the file gets added/removed to/from the repository on the host.
- when I synchronize PC 2 with the host, the new file gets added/removed to PC 2.
- when I modify the same file on both PCs and synchronize it, the host will consider the file's timestamp, keeping only the version of the file with the most recent timestamp or it will ask me what it shall do (e.g. overwrite the file or merge the contents of the file).
The files will normally be offline - a network connection will be possible only during the synchronization.
What would be the best approach/SW to do this?
Would for example a git-repository make sense (and if yes, is there a frontend specialized for this kind of job so that I don't have to fiddle around directly with git-commands)?
Thank you
|
|
|
06-07-2014, 05:01 PM
|
#2
|
ReliaFree Maintainer
Registered: Aug 2004
Location: Kalamazoo, Michigan
Distribution: Slackware 14.2
Posts: 2,815
|
Depending on the types of files, a version control system like git would work. rsync would also work and, based on your three requirements, is basically what you're describing. rsync would also be the better choice if you have binary files to keep in sync.
If you choose git, there are any number of GUI frontends as well as file browser plugins so you don't have to use that scary CLI. Same goes for rsync. Some of the rsync GUI's and/or backup solutions based on rsync are listed here.
rsync is the solution I use to keep a grunch of files sync'd between my workstation and my laptop. I use rsync in the following script and have cron execute the script. Obviously, I can manually execute the script when I want as well. Not a GUI solution, but I don't have to type in the command every time either. If it's helpful, please feel free to use it.
Code:
#!/bin/bash
# ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #
# Script to create incremental snapshot backups of a passed directory.
# Backs up using rsync to a remote machine.
#
# Copyright (C) 2008 Andrew "Weibullguy" Rowland
# [darowland <AT> ieee <DOT> com]
#
# By using this script, you agree that all damage to your system, loss of
# data, and other undesired effects are your problem. I make no warranty,
# explicit or otherwise. I make no guarantees, explicit or otherwise. I
# recommend you study the script, understand it, and independently validate
# it is the solution you desire and executing it on your machine will not
# result in the aforementioned bad juju.
#
# Since, there is no warranty, guarantee, promise, oath, contract, covenant,
# pledge, compact, treaty, pact, assurance, etc., I ask only that you keep
# the copyright statement above in the header. However, you may add, delete,
# or edit anything in the script itself to make it safer, more useful, more
# efficient, etc. You are under no obligation, obviously, to provide your
# changes. But you can, if you want, send your changes to me at the email
# address above.
#
# Enjoy....
#
# Change the values in the User-Defined Defaults section below for your
# particular setup.
# ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #
# ----- ----- ----- ----- - User-Defined Defaults ----- ----- ----- ----- #
CALEN=${CALEN:-"monthly"} # Snapshot periodicity.
EXCLUDES=${EXCLUDES:-"$HOME/.file.list"} # List of files to include/exclude from the transfer.
SOURCE=${SOURCE:-"$HOME/"} # Source directory.
HOST=${HOST:-""} # Host to backup to.
SNAPDIR=${SNAPDIR:-"/srv/samba/backups"} # Destination directory.
MODULE=${MODULE:-`hostname | tr '[a-z]' '[A-Z]'`}
PWDFILE=${PWDFILE:-"$HOME/.secret"}
RSYNCOPTS="--delete --delete-before --delete-excluded --delay-updates"
# ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ---- #
# ----- ----- ----- ----- -- System Commands --- ----- ----- ----- ----- #
DATE=`which date`
ECHO=`which echo`
ID=`which id`
NOTIFY=`which notify-send`
PING=`which ping`
RSYNC=`which rsync`
SSH=`which ssh`
# ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ---- #
rotate() {
# Function used to rotate old backups on the remote server.
if [ "x$DRYRUN" == "x1" ]; then
$ECHO -e "Execution would:"
$ECHO -e "\t1. Delete directory $SNAPDIR/$MODULE/$CALEN.3 on $HOST."
$ECHO -e "\t2. Move directory $SNAPDIR/$MODULE/$CALEN.2 to $SNAPDIR/$MODULE/$CALEN.3 on $HOST"
$ECHO -e "\t3. Move directory $SNAPDIR/$MODULE/$CALEN.1 to $SNAPDIR/$MODULE/$CALEN.2 on $HOST"
$ECHO -e "\t4. Copy directory $SNAPDIR/$MODULE/$CALEN.0 to $SNAPDIR/$MODULE/$CALEN.1 on $HOST"
else
# Step 1: delete the oldest snapshot, if it exists.
$SSH -t -t $RUSER@$HOST rm -fr $SNAPDIR/$MODULE/$CALEN.3
# Step 2: shift the middle snapshots(s) back by one, if they exist.
$SSH -t -t $RUSER@$HOST mv $SNAPDIR/$MODULE/$CALEN.2 $SNAPDIR/$MODULE/$CALEN.3
$SSH -t -t $RUSER@$HOST mv $SNAPDIR/$MODULE/$CALEN.1 $SNAPDIR/$MODULE/$CALEN.2
# Step 3: make a hard-link-only (except for dirs) copy of the latest snapshot, if that exists.
$SSH -t -t $RUSER@$HOST cp -al $SNAPDIR/$MODULE/$CALEN.0 $SNAPDIR/$MODULE/$CALEN.1
fi
}
backup() {
# Function to backup the local directory to the remote server.
if [ "x$DRYRUN" == "x1" ]; then
$ECHO -e "\t5. rsync the latest $DIRECTORY from $SOURCE to $HOST. The following will occur:"
fi
# Step 4: rsync from the system into the latest snapshot.
if [ "x$PROTO" == "x1" ]; # Use with servers running a rsync daemon.
then
/usr/bin/rsync $RSYNCOPTS $SOURCE rsync://$RUSER@$HOST/$MODULE/$DIRECTORY/
elif [ "x$PROTO" == "x2" ]; # Use with servers needing to SSH with rsync.
then
/usr/bin/rsync $RSYNCOPTS $SOURCE/ $RUSER@$HOST:$SNAPDIR/$MODULE/$DIRECTORY/
fi
if [ "x$CALEN" != 'x' ];
then
# Step 5: update the mtime of $CALEN.0 to reflect the snapshot time.
$SSH -t -t $RUSER@$HOST touch $SNAPDIR/$MODULE/$DIRECTORY
# Step 6: update the file with the backup date and time.
NOWD=`$DATE +%m-%d-%Y`
NOWT=`$DATE +%H%M`
if [ ! -f $TIMEFILE ];
then
echo "hourly.0 $NOWD $NOWT" > $TIMEFILE
echo "daily.0 $NOWD $NOWT" >> $TIMEFILE
echo "weekly.0 $NOWD $NOWT" >> $TIMEFILE
echo "monthly.0 $NOWD $NOWT" >> $TIMEFILE
# Step 7: copy the snapshot time file to the remote server.
scp $TIMEFILE $RUSER@$HOST:$SNAPDIR/$MODULE/
#else
# OLDD=`grep $CALEN.$SNAP $TIMEFILE | awk '{print $2}'`
# OLDT=`grep $CALEN.$SNAP $TIMEFILE | awk '{print $3}'`
# sed -i "/${CALEN}.$SNAP/ s/${OLDD}/${NOWD}/" ${TIMEFILE}
# sed -i "/${CALEN}.$SNAP/ s/${OLDT}/${NOWT}/" ${TIMEFILE}
fi
fi
}
restore() {
# Function to update the local directory from the remote server.
if [ "x$DRYRUN" == "x1" ]; then
$ECHO -e "\trsync the latest $CALEN.$SNAP from $HOST to $SOURCE. The following will occur:"
fi
# Step 1: rsync from the latest snapshot to the system.
if [ "x$PROTO" == "x1" ]; # Use with servers running a rsync daemon.
then
/usr/bin/rsync $RSYNCOPTS rsync://$RUSER@$HOST/$MODULE/$DIRECTORY $SOURCE
elif [ "x$PROTO" == "x2" ]; # Use with servers needing to SSH with rsync.
then
/usr/bin/rsync $RSYNCOPTS $RUSER@$HOST:$SNAPDIR/$MODULE/$DIRECTORY/ $SOURCE/
fi
}
CRON=0
DRYRUN=0
PROTO=1
RESTORE=0
SNAP=0
# Get command line arguments.
while [ $# -gt 0 ];
do
case "$1" in
-c|--calendar)
CALEN=$2
;;
-d|--cron)
CRON=1
;;
-e|--exclude)
EXCLUDES=$2
if [ ! -f $EXCLUDES ];
then
$ECHO "Error: No such file or directory $EXCLUDES."
exit 1
fi
;;
-h|--host)
HOST=$2
$PING -c 3 $HOST >/dev/null 2>&1
ROUT=$?
if [ $ROUT != 0 ];
then
$ECHO "Error: No route to $HOST."
exit 1
fi
;;
-l|--logfile)
LOGFILE=$2
;;
-m|--module)
MODULE=$2
;;
-n|--dry-run)
DRYRUN=1
;;
-p|--protocol)
PROTO=$2
;;
-r|--restore)
RESTORE=1
;;
-s|--source)
SOURCE=$2
if [ ! -d $SOURCE ];
then
$ECHO "Error: No such directory $SOURCE."
exit 1
fi
;;
-t|--target)
SNAPDIR=$2
;;
-u|--user)
RUSER=$2
;;
-z|--snapshot)
SNAP=$2
;;
--help)
$ECHO -e "-c, --calendar INT\t\twhether this is hourly, daily, weekly, or monthly snapshot. Leave blank for non-snapshot backup."
$ECHO -e "-d, --cron \t\t\texecute script using cron."
$ECHO -e "-e, --exclude FILE\t\tfile containing exclude patterns."
$ECHO -e "-h, --host HOST\t\t\thost name of rsync server."
$ECHO -e "-l, --logfile LOGFILE\t\tname and path of the file used to log ouput."
$ECHO -e "-m, --module MODULE\t\trsync module on rsync server."
$ECHO -e "-n, --dry-run\t\t\tperform a dry run of the snapshot."
$ECHO -e "-p, --protocol\t\t\tspecify the rsync protocol to use."
$ECHO -e "\t\t\t\t\t1 = rsync daemon on remote server (default)"
$ECHO -e "\t\t\t\t\t2 = rsync+ssh"
$ECHO -e "\t\t\t\t\t3 = http(s)"
$ECHO -e "-r, --restore\t\t\trestore from snapshot on remote host."
$ECHO -e "-s, --source SOURCE\t\tbase directory on local machine."
$ECHO -e "-t, --target TARGET\t\ttarget directory on remote server."
$ECHO -e "-u, --user USER\t\t\tuser on remote rsync server."
$ECHO -e "-z, --snapshot VERSION\t\twhich snapshot to restore from. Options are 0, 1, 2, or 3. Defaults to 0."
$ECHO -e "--help\t\t\t\tthis help."
exit 0
;;
esac
shift
done
TIMEFILE="$HOME/.local/var/log/$MODULE.time"
# If running as a cron job, then get the SSH password from the keychain.
LOCALHOST=`hostname -f`
if [ "x$CRON" == "x1" ];
then
source $HOME/.keychain/$LOCALHOST-sh
source $HOME/.keychain/$LOCALHOST-sh-gpg
else
RSYNCOPTS="${RSYNCOPTS} --progress"
fi
# Set rsync options depending on the connection protocol to use.
if [ "x$PROTO" == "x1" ]; # Use with servers running a rsync daemon.
then
RSYNCOPTS="-var ${RSYNCOPTS} --password-file=$HOME/.secret --exclude-from=$EXCLUDES"
elif [ "x$PROTO" == "x2" ]; # Use with servers needing to SSH with rsync.
then
RSYNCOPTS="-iramlHP ${RSYNCOPTS} -e $SSH --exclude-from=$EXCLUDES"
fi
# Log file stuff.
if [ "x$LOGFILE" == "x" ];
then
LOGFILE="$HOME/.local/var/log/$MODULE_$CALEN.log"
fi
if [ "x$LOGFILE" != "x" ];
then
if [ -f $LOGFILE ];
then
rm $LOGFILE
fi
RSYNCOPTS="${RSYNCOPTS} --log-file=$LOGFILE"
fi
if [ "x$DRYRUN" == "x1" ];
then
RSYNCOPTS="--dry-run ${RSYNCOPTS}"
fi
if [ "x$RESTORE" == "x1" ];
then
if [ "x$CALEN" == "x" ];
then
SNAP=""
DIRECTORY=""
else
DIRECTORY="$CALEN.$SNAP"
fi
restore
if [ "x$NOTIFY" != "x" ]; then
$NOTIFY 'SnapShot' "Update of $MODULE from $CALEN.$SNAP on $HOST is complete." -i dialog-information
fi
else
if [ "x$CALEN" == "x" ];
then
SNAP=""
DIRECTORY=""
else
DIRECTORY="$CALEN.$SNAP"
rotate
fi
backup
if [ "x$NOTIFY" != "x" ]; then
$NOTIFY 'SnapShot' "Snapshot $CALEN.$SNAP for $MODULE to $HOST is complete." -i dialog-information
fi
fi
exit 0
Last edited by weibullguy; 06-07-2014 at 05:20 PM.
|
|
|
All times are GMT -5. The time now is 09:00 AM.
|
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
|
Latest Threads
LQ News
|
|