LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 06-07-2014, 10:51 AM   #1
Pearlseattle
Senior Member
 
Registered: Aug 2007
Location: Zurich, Switzerland
Distribution: Gentoo
Posts: 1,001

Rep: Reputation: 142Reputation: 142
SW to keep directory synchronized over multiple PCs


Hi
I'm looking for some software that keeps the contents of a directory (files & subdirs) in sync between multiple PCs and a central host.

Desired result:
  • when I add/remove on PC 1 the file X to/from the directory and synchronize the dir with the host, the file gets added/removed to/from the repository on the host.
  • when I synchronize PC 2 with the host, the new file gets added/removed to PC 2.
  • when I modify the same file on both PCs and synchronize it, the host will consider the file's timestamp, keeping only the version of the file with the most recent timestamp or it will ask me what it shall do (e.g. overwrite the file or merge the contents of the file).
The files will normally be offline - a network connection will be possible only during the synchronization.


What would be the best approach/SW to do this?
Would for example a git-repository make sense (and if yes, is there a frontend specialized for this kind of job so that I don't have to fiddle around directly with git-commands)?


Thank you
 
Old 06-07-2014, 05:01 PM   #2
weibullguy
ReliaFree Maintainer
 
Registered: Aug 2004
Location: Kalamazoo, Michigan
Distribution: Slackware 14.2
Posts: 2,815
Blog Entries: 1

Rep: Reputation: 261Reputation: 261Reputation: 261
Depending on the types of files, a version control system like git would work. rsync would also work and, based on your three requirements, is basically what you're describing. rsync would also be the better choice if you have binary files to keep in sync.

If you choose git, there are any number of GUI frontends as well as file browser plugins so you don't have to use that scary CLI. Same goes for rsync. Some of the rsync GUI's and/or backup solutions based on rsync are listed here.

rsync is the solution I use to keep a grunch of files sync'd between my workstation and my laptop. I use rsync in the following script and have cron execute the script. Obviously, I can manually execute the script when I want as well. Not a GUI solution, but I don't have to type in the command every time either. If it's helpful, please feel free to use it.

Code:
#!/bin/bash

# ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #
# Script to create incremental snapshot backups of a passed directory.
# Backs up using rsync to a remote machine.
# 
# Copyright (C) 2008 Andrew "Weibullguy" Rowland 
# [darowland <AT> ieee <DOT> com]
#
# By using this script, you agree that all damage to your system, loss of
# data, and other undesired effects are your problem.  I make no warranty, 
# explicit or otherwise.  I make no guarantees, explicit or otherwise. I
# recommend you study the script, understand it, and independently validate
# it is the solution you desire and executing it on your machine will not
# result in the aforementioned bad juju.
#
# Since, there is no warranty, guarantee, promise, oath, contract, covenant,
# pledge, compact, treaty, pact, assurance, etc., I ask only that you keep
# the copyright statement above in the header.  However, you may add, delete,
# or edit anything in the script itself to make it safer, more useful, more 
# efficient, etc.  You are under no obligation, obviously, to provide your
# changes.  But you can, if you want, send your changes to me at the email
# address above.
# 
# Enjoy....
#
# Change the values in the User-Defined Defaults section below for your
# particular setup.
# ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- #

# ----- ----- ----- ----- - User-Defined Defaults ----- ----- ----- ----- #
CALEN=${CALEN:-"monthly"}                       # Snapshot periodicity.
EXCLUDES=${EXCLUDES:-"$HOME/.file.list"}        # List of files to include/exclude from the transfer.
SOURCE=${SOURCE:-"$HOME/"}                      # Source directory.
HOST=${HOST:-""}                                # Host to backup to.
SNAPDIR=${SNAPDIR:-"/srv/samba/backups"}        # Destination directory.
MODULE=${MODULE:-`hostname | tr '[a-z]' '[A-Z]'`}
PWDFILE=${PWDFILE:-"$HOME/.secret"}
RSYNCOPTS="--delete --delete-before --delete-excluded --delay-updates"
# ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ---- #

# ----- ----- ----- ----- -- System Commands --- ----- ----- ----- ----- #
DATE=`which date`
ECHO=`which echo`
ID=`which id`
NOTIFY=`which notify-send`
PING=`which ping`
RSYNC=`which rsync`
SSH=`which ssh`
# ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ---- #

rotate() {

	# Function used to rotate old backups on the remote server.
	if [ "x$DRYRUN" == "x1" ]; then
		$ECHO -e "Execution would:"
		$ECHO -e "\t1. Delete directory $SNAPDIR/$MODULE/$CALEN.3 on $HOST."
		$ECHO -e "\t2. Move directory $SNAPDIR/$MODULE/$CALEN.2 to $SNAPDIR/$MODULE/$CALEN.3 on $HOST"
		$ECHO -e "\t3. Move directory $SNAPDIR/$MODULE/$CALEN.1 to $SNAPDIR/$MODULE/$CALEN.2 on $HOST"
		$ECHO -e "\t4. Copy directory $SNAPDIR/$MODULE/$CALEN.0 to $SNAPDIR/$MODULE/$CALEN.1 on $HOST"
	else
		# Step 1: delete the oldest snapshot, if it exists.
		$SSH -t -t $RUSER@$HOST rm -fr $SNAPDIR/$MODULE/$CALEN.3

		# Step 2: shift the middle snapshots(s) back by one, if they exist.
		$SSH -t -t $RUSER@$HOST mv $SNAPDIR/$MODULE/$CALEN.2 $SNAPDIR/$MODULE/$CALEN.3
		$SSH -t -t $RUSER@$HOST mv $SNAPDIR/$MODULE/$CALEN.1 $SNAPDIR/$MODULE/$CALEN.2

		# Step 3: make a hard-link-only (except for dirs) copy of the latest snapshot, if that exists.
		$SSH -t -t $RUSER@$HOST cp -al $SNAPDIR/$MODULE/$CALEN.0 $SNAPDIR/$MODULE/$CALEN.1
	fi

}

backup() {

	# Function to backup the local directory to the remote server.
	if [ "x$DRYRUN" == "x1" ]; then
		$ECHO -e "\t5. rsync the latest $DIRECTORY from $SOURCE to $HOST.  The following will occur:"
	fi

	# Step 4: rsync from the system into the latest snapshot.
	if [ "x$PROTO" == "x1" ];		# Use with servers running a rsync daemon.
	then
		/usr/bin/rsync $RSYNCOPTS $SOURCE rsync://$RUSER@$HOST/$MODULE/$DIRECTORY/
	elif [ "x$PROTO" == "x2" ];		# Use with servers needing to SSH with rsync.
	then
		/usr/bin/rsync $RSYNCOPTS $SOURCE/ $RUSER@$HOST:$SNAPDIR/$MODULE/$DIRECTORY/

	fi

	if [ "x$CALEN" != 'x' ];
	then
		# Step 5: update the mtime of $CALEN.0 to reflect the snapshot time.
		$SSH -t -t $RUSER@$HOST touch $SNAPDIR/$MODULE/$DIRECTORY

		# Step 6: update the file with the backup date and time.
		NOWD=`$DATE +%m-%d-%Y`
		NOWT=`$DATE +%H%M`
		if [ ! -f $TIMEFILE ];
		then
			echo "hourly.0 $NOWD $NOWT" > $TIMEFILE
			echo "daily.0 $NOWD $NOWT" >> $TIMEFILE
			echo "weekly.0 $NOWD $NOWT" >> $TIMEFILE
			echo "monthly.0 $NOWD $NOWT" >> $TIMEFILE
			# Step 7: copy the snapshot time file to the remote server.
			scp $TIMEFILE $RUSER@$HOST:$SNAPDIR/$MODULE/
		#else
		#	OLDD=`grep $CALEN.$SNAP $TIMEFILE | awk '{print $2}'`
		#	OLDT=`grep $CALEN.$SNAP $TIMEFILE | awk '{print $3}'`
		#	sed -i "/${CALEN}.$SNAP/ s/${OLDD}/${NOWD}/" ${TIMEFILE}
		#	sed -i "/${CALEN}.$SNAP/ s/${OLDT}/${NOWT}/" ${TIMEFILE}
		fi
	fi

}

restore() {

	# Function to update the local directory from the remote server.
	if [ "x$DRYRUN" == "x1" ]; then
		$ECHO -e "\trsync the latest $CALEN.$SNAP from $HOST to $SOURCE.  The following will occur:"
	fi

	# Step 1: rsync from the latest snapshot to the system.
	if [ "x$PROTO" == "x1" ];		# Use with servers running a rsync daemon.
	then
		/usr/bin/rsync $RSYNCOPTS rsync://$RUSER@$HOST/$MODULE/$DIRECTORY $SOURCE
	elif [ "x$PROTO" == "x2" ];		# Use with servers needing to SSH with rsync.
	then
		/usr/bin/rsync $RSYNCOPTS $RUSER@$HOST:$SNAPDIR/$MODULE/$DIRECTORY/ $SOURCE/
	fi

}

CRON=0
DRYRUN=0
PROTO=1
RESTORE=0
SNAP=0

# Get command line arguments.
while [ $# -gt 0 ]; 
do
	case "$1" in

		-c|--calendar)
			CALEN=$2
			;;
		-d|--cron)
			CRON=1
			;;
		-e|--exclude)
			EXCLUDES=$2
			if [ ! -f $EXCLUDES ];
			then
				$ECHO "Error: No such file or directory $EXCLUDES."
				exit 1
			fi
			;;
		-h|--host)
			HOST=$2
			$PING -c 3 $HOST >/dev/null 2>&1
			ROUT=$?
			if [ $ROUT != 0 ];
			then
				$ECHO "Error: No route to $HOST."
				exit 1
			fi
			;;
		-l|--logfile)
			LOGFILE=$2
			;;
		-m|--module)
			MODULE=$2
			;;
		-n|--dry-run)
			DRYRUN=1
			;;
		-p|--protocol)
			PROTO=$2
			;;
		-r|--restore)
			RESTORE=1
			;;
		-s|--source)
			SOURCE=$2
			if [ ! -d $SOURCE ];
			then
				$ECHO "Error: No such directory $SOURCE."
				exit 1
			fi
			;;
		-t|--target)
			SNAPDIR=$2
			;;
		-u|--user)
			RUSER=$2
			;;
		-z|--snapshot)
			SNAP=$2
			;;
		--help)
			$ECHO -e "-c, --calendar INT\t\twhether this is hourly, daily, weekly, or monthly snapshot.  Leave blank for non-snapshot backup."
			$ECHO -e "-d, --cron \t\t\texecute script using cron."
			$ECHO -e "-e, --exclude FILE\t\tfile containing exclude patterns."
			$ECHO -e "-h, --host HOST\t\t\thost name of rsync server."
			$ECHO -e "-l, --logfile LOGFILE\t\tname and path of the file used to log ouput."
			$ECHO -e "-m, --module MODULE\t\trsync module on rsync server."
			$ECHO -e "-n, --dry-run\t\t\tperform a dry run of the snapshot."
			$ECHO -e "-p, --protocol\t\t\tspecify the rsync protocol to use."
			$ECHO -e "\t\t\t\t\t1 = rsync daemon on remote server (default)"
			$ECHO -e "\t\t\t\t\t2 = rsync+ssh"
			$ECHO -e "\t\t\t\t\t3 = http(s)"
			$ECHO -e "-r, --restore\t\t\trestore from snapshot on remote host."
			$ECHO -e "-s, --source SOURCE\t\tbase directory on local machine."
			$ECHO -e "-t, --target TARGET\t\ttarget directory on remote server."
			$ECHO -e "-u, --user USER\t\t\tuser on remote rsync server."
			$ECHO -e "-z, --snapshot VERSION\t\twhich snapshot to restore from.  Options are 0, 1, 2, or 3.  Defaults to 0."
			$ECHO -e "--help\t\t\t\tthis help."

			exit 0
			;;
	esac
	shift
done

TIMEFILE="$HOME/.local/var/log/$MODULE.time"

# If running as a cron job, then get the SSH password from the keychain.
LOCALHOST=`hostname -f`
if [ "x$CRON" == "x1" ]; 
then
	source $HOME/.keychain/$LOCALHOST-sh
	source $HOME/.keychain/$LOCALHOST-sh-gpg
else
	RSYNCOPTS="${RSYNCOPTS} --progress"
fi

# Set rsync options depending on the connection protocol to use.
if [ "x$PROTO" == "x1" ];		# Use with servers running a rsync daemon.
then
	RSYNCOPTS="-var ${RSYNCOPTS} --password-file=$HOME/.secret --exclude-from=$EXCLUDES"
elif [ "x$PROTO" == "x2" ];		# Use with servers needing to SSH with rsync.
then
	RSYNCOPTS="-iramlHP ${RSYNCOPTS} -e $SSH --exclude-from=$EXCLUDES"
fi

# Log file stuff.
if [ "x$LOGFILE" == "x" ];
then
	LOGFILE="$HOME/.local/var/log/$MODULE_$CALEN.log"
fi

if [ "x$LOGFILE" != "x" ];
then

	if [ -f $LOGFILE ];
	then
		rm $LOGFILE
	fi

	RSYNCOPTS="${RSYNCOPTS} --log-file=$LOGFILE"

fi

if [ "x$DRYRUN" == "x1" ];
then
	RSYNCOPTS="--dry-run ${RSYNCOPTS}"
fi

if [ "x$RESTORE" == "x1" ];
then

	if [ "x$CALEN" == "x" ];
	then
		SNAP=""
		DIRECTORY=""
	else
		DIRECTORY="$CALEN.$SNAP"
	fi

	restore

	if [ "x$NOTIFY" != "x" ]; then
		$NOTIFY 'SnapShot' "Update of $MODULE from $CALEN.$SNAP on $HOST is complete." -i dialog-information
	fi

else

	if [ "x$CALEN" == "x" ];
	then
		SNAP=""
		DIRECTORY=""
	else
		DIRECTORY="$CALEN.$SNAP"
		rotate
	fi
	
	backup

	if [ "x$NOTIFY" != "x" ]; then
		$NOTIFY 'SnapShot' "Snapshot $CALEN.$SNAP for $MODULE to $HOST is complete." -i dialog-information
	fi

fi

exit 0

Last edited by weibullguy; 06-07-2014 at 05:20 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Linux PCs no longer can view other PCs on LAN SheltonMac Linux - Wireless Networking 2 03-04-2014 06:40 AM
Connect multiple PCs and use as one jdwilder General 13 06-27-2006 05:51 PM
Run a program on multiple pcs mesh2005 Linux - Networking 4 09-18-2005 02:33 PM
Connecting a single ethernet port to multiple PCs? beejayzed Linux - Networking 14 09-08-2005 04:14 PM
Old PC as a Router/Fire for Multiple PCs AxXium Slackware 21 08-17-2005 10:02 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 09:00 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration