LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 11-28-2007, 10:04 AM   #1
BadTim
LQ Newbie
 
Registered: May 2007
Location: Alberta, Canada
Distribution: Ubuntu v7.04 Desktop
Posts: 14

Rep: Reputation: 0
How can I use rsync to keep multiple clients in sync?


Hello everyone,

I know that there will be several of you that roll your eyes in disgust at what you are about to read but please; humor me! (I'm still somewhat new to Linux - I'm sure it shows)

The situation:

- I need to synchronize data on several Windows XP clients.
- The clients are used by a single user but are in multiple locations across North America.
- All client locations have cable (or better) connections to the internet.
- The user needs to be able to create, modify, or delete a document on one of his XP clients and have the change reflected on each of the other clients.
- I'm cheap and I don't want to pay for a subscription service this will do this for me (not to mention that I don't trust them).
- This needs to be completely transparent to the user (because if the user can tinker, the user *will* tinker and undoubtedly mess it up).

My plan (thus far):

- Using Ubuntu create a "server" in a centralized location and synchronize over VPN.
- Apps I'm using to do this:
- Rsync (will look after the transfers and the versioning of the files)
- Hamachi VPN (will give me a secure pipe to each of the clients that will traverse NAT securely so I don't have to go and punch holes in any firewalls)
- Cygwin (enables me to run Rsync on the clients as a service)

Progress:

I've followed this handy guide but I must be missing something somewhere because I'm having problems on the server side. The author of that page also didn't assume that his readers would be synchronizing the data to/from multiple clients.

Right now I'm calling rsync from a batch file from the task scheduler (for now) and that part seems to be working well.

Here is what my rsync command looks like on the XP clients:
Code:
rsync -vrutzh --stats --progress --password-file=c:\cygwin\secret --delete "cygwindrive/c/Documents and Settings/<username>/My Documents" <username>@<server's Hamachi IP>::<module name>
Here is what my /etc/rsyncd.conf looks like on the Ubuntu "server":
Code:
# GLOBAL OPTIONS
log file=/var/log/rsyncd
pid file=/var/run/rsyncd.pid

# MODULE OPTIONS

[mod1]

        path = /home/<username>/backup
        uid = <username>
        gid = <username>
        read only = false
        auth users = <username>
        secrets file = /etc/rsyncd.secrets
        list = yes
        transfer logging = yes
        log format = %t: host %h (%a) %o %f (%l bytes). Total %b bytes.
        timeout = 600
        dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz

Questions:

- Am I headed down the right path or is there something that is bigger/better/easier/cooler that will do everything that I need?
- Are my client side rsync options correct or am I going to cause myself data loss at some point? Will this keep all of the computers up to date without risking unwanted deletes?
- Where (how) should I be automatically executing "rsync --daemon" on the Ubuntu machine?
- Where (how) should I be automatically executing "hamachi start" on the Ubuntu machine?

Please be gentle... I'm kinda/sorta new still! How-tos appreciated.

Thanks for reading this far!
 
Old 11-28-2007, 10:17 AM   #2
kotnik
Member
 
Registered: Nov 2004
Location: Novi Sad, Serbia
Distribution: Debian, Slackware, Gentoo, openSuSE
Posts: 254

Rep: Reputation: 31
Did you see unison? It might make it easier.
 
Old 11-29-2007, 09:20 AM   #3
BadTim
LQ Newbie
 
Registered: May 2007
Location: Alberta, Canada
Distribution: Ubuntu v7.04 Desktop
Posts: 14

Original Poster
Rep: Reputation: 0
I've played with Unison but it was a while ago... I couldn't seem to get the WIN32 side to work consistently. I see by your link that they've been doing some work on it (I think that I was using v1.x) so I might give it another try.

There is no mention on the Unison site about multiple clients syncing to the same "server" so I'm a little wary.

Thanks for your reply!
 
Old 12-01-2007, 02:29 AM   #4
BadTim
LQ Newbie
 
Registered: May 2007
Location: Alberta, Canada
Distribution: Ubuntu v7.04 Desktop
Posts: 14

Original Poster
Rep: Reputation: 0
Haven't had much luck with Unison... Most of the documentation and how-tos seem to be Win32/Win32 or *nix/*nix and not Win32/*nix like what I'm doing here.

The problem I'm having is evoking the command through ssh from the Win32 side (as prescribed in most tutorials).

...might be back to the Rsync idea...
 
Old 12-01-2007, 07:56 PM   #5
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,251

Rep: Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025
The way you're describing it, maybe use Samba to create a shared dir instead. That way it's more 'instant', instead of eg rsync+cron.
 
Old 12-03-2007, 07:29 PM   #6
BadTim
LQ Newbie
 
Registered: May 2007
Location: Alberta, Canada
Distribution: Ubuntu v7.04 Desktop
Posts: 14

Original Poster
Rep: Reputation: 0
Thought of that

Hi Chris, thanks for your reply.

I thought of that as well but what happens when the client is disconnected from the VPN?

Cheers,

Tim
 
Old 12-03-2007, 10:27 PM   #7
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,251

Rep: Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025
Not sure i understand... if it's not connected, no data can flow, regardless of rsync/unison/samba..

Can you clarify a bit more please.
Once setup, Samba gives an MS like env, so should be good for the user.
 
Old 12-04-2007, 09:01 AM   #8
BadTim
LQ Newbie
 
Registered: May 2007
Location: Alberta, Canada
Distribution: Ubuntu v7.04 Desktop
Posts: 14

Original Poster
Rep: Reputation: 0
A little clarification...

The user (who uses a couple of desktop computers as well as a laptop computer) creates documents from any of his computers without paying much mind to which computer he is creating on. This user also imports photos from his digital camera on to his laptop while away from a network connection. Everything is stored in "My Documents" so that part makes things a wee bit easier.

When the user returns to one of his offices (or a hotel room), he will inevitably power on his laptop which will automatically connect to his wireless network and his VPN connection to the "server" will be restored.

<This is the point where the sync will ideally occur>

Then, when the user logs in to one of his desktop computers, all of the new photos and changed (or newly created) documents will be waiting for him (and all of the other trash he keeps in "My Documents".

Currently we're using Samba to offer up a network share on the VPN and then Microsoft's Synctoy does the sync but it is a terrible bandwidth abuser and (worse than that) Synctoy (to my knowledge) only runs in the foreground (so the user sees what is going on and will tinker with it causing me headaches). Did I mention that my user is a terrible tinker-er?

I know this is a colossal p.i.t.a. so I thank you for your assistance. In a perfect world he would never be away from his internet connection (and it would be a 10 gigabit connection since we're dreaming here)!

Thanks again for your reply,


Tim
 
Old 12-04-2007, 06:01 PM   #9
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,251

Rep: Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025
So, only one physical user, so file locking isn't an issue; good.
You need to make his MyDocs the Samba share, the same as you would on a LAN eg Z: drive. That way his docs/photos are on server always. Prob is he'd need to be online to use them.
If he wants offline use as well rsync would be better. Only sends file differences, so bandwith usage is low (generally). Note that rsync can use the ssh protocol.
On Linux there is a tool (dnotify ?) that lets you know when a file has changed. Don't know if there's an MS equiv.
If it auto-mounts the Samba share to the server at logon, you could do an rsync straightaway and possibly use the dnotify thing from then on.
I'm getting a bit out of my depth, never having done this stuff as such.
You might want to peruse here: http://us1.samba.org/, http://samba.anu.edu.au/rsync/.
I thought LQ had a 'How I Did It' page, but I can't find it.
Please post your eventual soln, I'm sure it'll greatly appreciated.
 
Old 12-06-2007, 09:14 AM   #10
BadTim
LQ Newbie
 
Registered: May 2007
Location: Alberta, Canada
Distribution: Ubuntu v7.04 Desktop
Posts: 14

Original Poster
Rep: Reputation: 0
Hi Chris,

I'm planning to call rsync or Unison from the Task Scheduler. I'll look into dnotify but I think that is chrome at this point. Since we're using cygwin on the XP client machines dnotify shouldn't be an issue if and when I get to that point.

I've read that rsync will do SSH but I still like the idea of being able to remotely administer via VPN so I think that I'm going to stick with Hamachi for the security.

At this point I just need some uninterrupted time to see if I can get Unison to work locally (I have two of his clients and the server at my location currently) and then make it work through the Hamachi IP eventually. If I can't wrap my brain around Unison then it will be back to rsync.

I ran into version compatibility issues with the Unison for Windows GUI version and the version of Unison that I have installed on my Ubuntu 7.10 "server" so I switched over to cygwin on the XP client and the issue vanished. (I tried to install the latest version of Unison (v2.26.x) on the Ubuntu box but I ran into issues doing it from the CLI and couldn't find a repository that contained the latest version so I gave up and decided to simply run v2.13.x instead.)

I think that I'm going work on rsync a little more and see if I can get it to fire... my problem there seems to be in the config on the "server".
 
Old 12-06-2007, 06:02 PM   #11
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,251

Rep: Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025
I generally find that the underlying tools eg rsync are slightly more reliable / up to date than tools on top eg Unison, although if it works, sometimes the tools are easier to work with for a human.
At least you know that rsync can use ssh and does compression and only txfrs differences by default, so you've always got the basic soln avail. Also, simpler tech breaks less often.
Maybe you should post your rsync server issues here or in another thread.
 
Old 12-06-2007, 06:10 PM   #12
MasterC
Guru
 
Registered: Mar 2002
Location: Salt Lake City, UT - USA
Distribution: Gentoo ; LFS ; Kubuntu
Posts: 12,612

Rep: Reputation: 64
What about Flash or removable HD? I know it's not nearly as cool as a network option (and BTW, I was thinking Samba all the way up until I realized the laptop might now always have connectivity) but it's highly mobile. An added benefit might that the user is responsible for their own files; fewer admins to blame

-Chad
 
Old 12-06-2007, 07:36 PM   #13
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,251

Rep: Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025Reputation: 2025
If you're looking at a scheduler, you'll prob still need to supply a submit/do_backup button for the user, or you could lose the last update before he disconnects.
You could try wrapping the disconnect so it does a final backup first, but only if you trust the user not to pull the plug in impatience.
IOW, does it need to be invisible or will he do what he's told?
 
Old 12-07-2007, 12:03 PM   #14
BadTim
LQ Newbie
 
Registered: May 2007
Location: Alberta, Canada
Distribution: Ubuntu v7.04 Desktop
Posts: 14

Original Poster
Rep: Reputation: 0
Thanks Chris and Chad for your replies...

Chad, another one of the underlying factors that I'm dealing with here is disaster recovery... "disaster" could also mean the theft of the laptop, which in this case has happened before. So, the idea of the removable media traveling with the laptop might be a bad idea as well - at least in this case it is a risk that we're not willing to take.

Chris, I was thinking the same thing... if left to a "time of day" schedule we might miss that last backup before he jumps on a plane with the laptop or something similar. I don't see a way around putting an icon on his desktop and then doing some "user education".


Tim
 
Old 12-07-2007, 12:07 PM   #15
BadTim
LQ Newbie
 
Registered: May 2007
Location: Alberta, Canada
Distribution: Ubuntu v7.04 Desktop
Posts: 14

Original Poster
Rep: Reputation: 0
Note: I plan to edit this post and keep it current with the solution that is working for me

I've achieved partial success using Hamachi (connectivity), SSH (more connectivity), Unison (the heavy lifting of synchronization), and Cygwin to make my life tremendously easier.

A small task-list of what I did (subject to further editing as I refine the process):

Ubuntu 7.10 "Server" (an old piece of crap AMD 1.3 with ~160GB of hard drive space)
1. Download and install Hamachi

Install and configure to start at boot without the user logging in
Create a Hamachi Network with a strong password
2. Install Unison package (v2.13.x at this time)
3. Create a user that will connect via ssh and sync

Windows XP Professional "Clients"

1. Download and install Hamachi
Install with defaults, change nickname to something more descriptive
Join network created in Server step 1 above using strong password
Configure Hamachi to start at login
2. Install Cygwin with OpenSSH, OpenSSL, Unison, and nano
Add C:\cygwin and C:\cygwin\bin to PATH variable for all users
(I did this so that I can run Unison from the Windows CLI thus allowing me to run it as a scheduled task)
3. Configure SSH to use RSA key

(On the "client" in Cygwin shell)
Code:
ssh-keygen -t rsa
cat ~/.ssh/id_dsa.pub | ssh <hamachi IP of "server"> 'cat - >>~/.ssh/authorized_keys'
Now test to make sure it works:
Code:
ssh <username>@<Hamachi IP of server>
It should log you in without a password now.

4. Configure Unison profile on XP client using nano
Code:
unison
nano ~/.unison/<your profile name here>.prf
Here is my profile that I used:
Code:
root = C:\Documents and Settings\<username>\My Documents
root = ssh://<Hamachi IP of Server>//home/<username>/mydocs

ignore = Name Thumbs.db
ignore = Name *~
ignore = Name {desktop.ini}
ignore = Name *.tmp
ignore = Path .unison
ignore = Path {My Music/Sample Music.lnk}
ignore = Path {My Pictures/Sample Pictures.lnk}

auto=true
fastcheck=true
ignorecase=false
retry=3
rshargs=-C
sortnewfirst=true
times=true
#EOF
5. Set up a Scheduled Task on the XP Clients
Start > Programs > Accessories > System Tools > Scheduled Tasks
-Add Scheduled Task
...browse to C:\cygwin\bin\unison-2.13.exe
-Select your schedule, enter your user credentials
-Edit the Advanced Properties of the task you just created:
-Run: C:\cygwin\bin\unison-2.13.exe <your profile name>
-Windows will now make you re-enter your credentials.

6. Create an icon on the user's desktop so they can manually sync
-In Windows XP...
Right click the user's desktop and choose New > Shortcut
-Choose C:\cygwin\bin\unison-2.13.exe
-Give it a fancy name for the user's sake
-Right click your newly created shortcut, choose Properties
-Add <your profile name> on to the end of the target line

Conclusions (in testing)
-Everything seems to be working as planned at the moment... but, with that being said, it is still really early in this project so I haven't had a chance to see if it will work reliably and without data loss. I've backed up to another location in preparation for any sort of disaster that may occur due to shortsightedness or my general unfamiliarity with any of the above.

To Do
-Need to test to see what happens when one of the clients is left powered up (without any documents open) and changes are made on one of the other clients. Will it cause problems when the scheduled task runs? Do I need to offset schedules so they don't collide?

Links

I plan to post the links to the resources I've used on this project here.

Here's a site that I just stumbled upon that pretty much sums up the whole project (except the author uses rsync rather than Unison):
http://justinsomnia.org/2007/02/how-...u-using-rsync/

Notes
-The Windows path to the .unison directory is C:\cygwin\home\<username>\.unison
-- this is where your profile (*.prf) files should reside

-I know that Hamachi isn't needed because SSH will look after encrypting the connection. I choose to use Hamachi because I can remotely administer the XP client as well. If you don't need this capability then omit the Hamachi steps in favor of good ol' SSH connecting directly to your server.

Last edited by BadTim; 12-07-2007 at 02:02 PM.
 
  


Reply

Tags
sync, syncronization, ubuntu, unison, xp


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync daemon or not - multiple folders yull Linux - General 4 02-24-2009 05:09 AM
ALSA - multiple clients problem Limpid Arch 2 11-06-2007 08:02 AM
RSYNC Automatic sync script penguinpages Fedora 0 05-01-2007 11:52 AM
Rsync between multiple machines Maeltor Linux - General 7 07-31-2006 11:22 PM
Multiple Clients One Authentication jpbarto Linux - Networking 4 10-02-2003 01:50 PM


All times are GMT -5. The time now is 05:02 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration