[SOLVED] Auto-synchronizing a user's document directory.

Rupadhya · 10-21-2012, 06:17 PM

Hello,

I have a user who has used one Fedora 17 workstation. Now she will use one of two machines. I currently do not have a server where both machines could have a share to. They machines might not be up at the same time, so I don't want to symbolic link the directories together.

I would like to automatically synchronize the two directories by a cron job. If a file is created on one machine, I would like to see it show up on the other machine. If a file is updated on one machine, I would like to see the new version propagated to the other machine and vice versa. If a file is deleted on one machine, I don't necessarily want it deleted (right away) on the other machine in case it was accidentally deleted.

Do you think I can do this with RSYNC from one machine? Anyone with experience doing this?

- Raj Upadhyaya

aetratus · 10-21-2012, 07:53 PM

This is trivially easy to do, but you should take the time to read the rsync documentation so that you understand what's going on. You can always resort to google for many examples of how others do this.

Rupadhya · 10-21-2012, 08:52 PM

Looks like I might have it.. I mounted the Workstation 1 file system home share under the /home/raj/c directory. Here is what I have. First, I will sync from workstation 2 to workstation 1.

Code:

rsync -a -s -t -u --numeric-ids --no-owner --no-group --progress /home/virginia/Documents /home/raj/c/virginia

Then immediately after that, I will sync from workstation 1 to workstation 2 if there are any new files on the workstation 1.

Code:

rsync -a -s -t -u --numeric-ids --no-owner --no-group --progress /home/raj/c/virginia/Documents /home/virginia

Does that make sense? I think if I schedule it from cron I will remove the --progress tab and replace it with -q (which is quiet mode).

- Raj

suicidaleggroll · 10-21-2012, 10:50 PM

Add the Documents directory onto the end of the command:

Code:

rsync -a -s -t -u --numeric-ids --no-owner --no-group --progress /home/virginia/Documents/ /home/raj/c/virginia/Documents/

And vice versa. The trailing slash here IS important.

Also, make sure your user knows that if they ever want to delete OR RENAME a file, they must do so on BOTH workstations in between cron jobs.

You'll also want a check to make sure the remote mount is up before running these commands.

Rupadhya · 10-21-2012, 11:15 PM

From my root crontab.

Code:

5 0 * * * /bin/rsync -a -s -t -u --numeric-ids --no-owner --no-group  /home/virginia/Documents/ /home/raj/c/virginia/Documents/
#### Rest for a few minutes
10 0 * * * /bin/rsync -a -s -t -u --numeric-ids --no-owner --no-group  /home/raj/c/virginia/Documents/ /home/virginia/Documents/

Does that look good? It should kick off at 5 minutes after midnight tonight, and then 10 minutes after midnight. I always make mistakes with the crontab schedule.

- Raj

suicidaleggroll · 10-21-2012, 11:31 PM

I would put both rsync commands in a single script and run that from the cron, that way you don't run into any race condition where the second rsync starts before the first has finished. Depending on usage, you may want to run it more often than that, especially if the user will be modifying documents in both places during the day.

Rupadhya · 10-22-2012, 07:36 AM

I got up early this morning, and took your advice. Instead of just wrapping the two rsync lines in a simple script, I made it more robust.. You can now execute it from the command line and increase its verbosity. Uggh. I have to get more sleep or a better hobby. Here it is. If you have a suggestion for improvement, I would appreciate it.
- Raj

Code:

#!/bin/bash
# By Raj Upadhyaya
# BSD License
#
# Redistribution and use in source and binary forms, with or
#   without modification, are permitted provided that the following
#   conditions are met:
#
# *  Redistributions of source code must retain the above
#    copyright notice, this list of conditions and the following
#    disclaimer.
#
# *  Redistributions in binary form must reproduce the above
#    copyright notice, this list of conditions and the following
#    disclaimer in the documentation and/or other materials
#    provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
# PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR
# BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
# TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
# THE POSSIBILITY OF SUCH DAMAGE.

##  globals
verbosity=""
# Display help
function usage(){
    cat << EOF

    syncDirectory [options] username directory
        rsync a directory from one server to the other.

        -v     : verbose (number) 1 or 2
        -h     : display help


EOF
}
#  ___  _ __ | |_(_) ___  _ __  ___
# / _ \| '_ \| __| |/ _ \| '_ \/ __|
#| (_) | |_) | |_| | (_) | | | \__ \
# \___/| .__/ \__|_|\___/|_| |_|___/
#      |_|

# Parsing options
while getopts "v:h" option
do
    case $option in
        v)
        if [[ $2 -ne 2 ]]
        then
          verbosity="--progress"
        else
          verbosity="--progress -vv"
        fi
        ;;
        h)
        usage
        exit 0
        ;;
        *)
        ;;
    esac
done


# Mainline...

shift $((OPTIND-1));

# Checks number of options correct
if [[ $# -ne 2 ]]
then
    usage
    exit 1
fi

myUser=$1
myDirectory=$2
server1Location=/home/$myUser/$myDirectory/
server2Location=/home/raj/c/$myUser/$myDirectory/
rsyncOptions="-a -l -t -u --numeric-ids -g -o -p"

if id -u $myUser > /dev/null 2>&1;
then
        echo "user exists" > /dev/null
else
        echo "Who is $myUser?  I can't seem to find that user on this system."
        exit 1
fi

if [[ ! -d "$server1Location" ]]
then
    echo "Couldn't find the directory: $server1Location.  Are you sure it exists
?"
    usage
    exit 1
fi

## Here we go!!!
/bin/rsync $rsyncOptions $verbosity $server1Location $server2Location
/bin/rsync $rsyncOptions $verbosity $server2Location $server1Location

chrism01 · 10-22-2012, 08:02 PM

You might want to read this discussion of rsync vs unison.
(its quite short; see the latter posts)

Rupadhya · 10-23-2012, 12:32 AM

Chris,

I found this discussion on http://www.linuxquestions.org. http://www.linuxquestions.org/questi...8/#post4509973

I installed Unison to test it out, and it is very nice. The only difference that I saw with the RSYNC option is I could create a cron job to execute the above script a few times a day. The Unison product is a GUI implementation which nice and easy to set up. The difference that I saw that was cool was if a file was updated on both sides. If you use Unison, it prompts you to resolve the conflict and it won't automatically copy either side over the other. The RSYNC option will find the newer file (even if it is by a second) and choose that version to sync with both sides. I think given the use case of one user using one of two workstations at a time, this shouldn't happen.

Well, I learned alot about Rsync, and created a BASH script that is probably 104 lines too long, but what the hey.

- Raj

chrism01 · 10-23-2012, 05:51 AM

Well done; sorry I seem to have dropped the link there ...

Anyway, it sounded like something that might be applicable and certainly worth knowing for future ref

Rupadhya · 10-23-2012, 11:27 AM

Looks like you can run Unison from the command line as well without the GUI.

Quote:

unison -ui text -auto -batch batchpref

With this, you could potentially run unison from a cron job.

I will look into this and document my findings for others reference.

- Raj

Rupadhya · 10-23-2012, 11:33 AM

Looks like it will work as a cron job. Here is my output of the first run.

Code:

Videos$ unison -ui text -auto -batch batchpref
Contacting server...
Looking for changes
Warning: No archive files were found for these roots, whose canonical names are:
/home/raj/Documents
/home/raj/c/raj/Documents
This can happen either
because this is the first time you have synchronized these roots,
or because you have upgraded Unison to a new version with a different
archive format.

Update detection may take a while on this run if the replicas are
large.

Unison will assume that the 'last synchronized state' of both replicas
was completely empty. This means that any files that are different
will be reported as conflicts, and any files that exist only on one
replica will be judged as new and propagated to the other replica.
If the two replicas are identical, then no changes will be reported.

If you see this message repeatedly, it may be because one of your machines
is getting its address from DHCP, which is causing its host name to change
between synchronizations. See the documentation for the UNISONLOCALHOSTNAME
environment variable for advice on how to correct this.

Donations to the Unison project are gratefully accepted:
http://www.cis.upenn.edu/~bcpierce/unison

Reconciling changes
file <-?-> file output 2.pdf
home/raj/... : file modified on 2012-10-23 at 8:04:25 size 17850 rw-rw-r--
c/raj/Doc... : file modified on 2012-10-22 at 4:39:55 size 19198 rw-rw-r--
file <-?-> file output 3.pdf
home/raj/... : file modified on 2012-10-23 at 8:06:59 size 93902 rw-rw-r--
c/raj/Doc... : file modified on 2012-10-22 at 4:36:11 size 93523 rw-rw-r--
file <-?-> file output 4.pdf
home/raj/... : file modified on 2012-10-23 at 8:08:07 size 34097 rw-rw-r--
c/raj/Doc... : file modified on 2012-10-22 at 4:32:32 size 33754 rw-rw-r--
file <-?-> file output.pdf
home/raj/... : file modified on 2012-10-23 at 8:01:25 size 6055 rw-rw-r--
c/raj/Doc... : file modified on 2012-10-22 at 4:38:56 size 6032 rw-rw-r--
file ----> HR 3962 - Medical Devices.doc
Propagating updates
UNISON 2.40.63 started propagating changes at 11:30:58.42 on 23 Oct 2012
[CONFLICT] Skipping output 2.pdf
[CONFLICT] Skipping output 3.pdf
[CONFLICT] Skipping output 4.pdf
[CONFLICT] Skipping output.pdf
[BGN] Copying HR 3962 - Medical Devices.doc from /home/raj/Documents to /home/raj/c/raj/Documents
[END] Copying HR 3962 - Medical Devices.doc
UNISON 2.40.63 finished propagating changes at 11:30:58.53 on 23 Oct 2012
Saving synchronizer state
Synchronization complete at 11:30:58 (1 item transferred, 4 skipped, 0 failed)
skipped: output 2.pdf
skipped: output 3.pdf
skipped: output 4.pdf
skipped: output.pdf

chrism01 · 10-23-2012, 09:08 PM

Looks good; try amending a file and creating a new one and see what happens.
Also try deleting a test(!) file...