LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 09-13-2008, 06:59 AM   #1
hymerman
LQ Newbie
 
Registered: Sep 2008
Location: Loughborough
Distribution: Ubuntu
Posts: 3

Rep: Reputation: 0
Question How would I keep files on two servers synchronised?


Summary for those with little time: How would you keep two large sets of binary files stored on separate machines synchronised over the Internet, where either machine may change those sets?

My friend and I have always had a large shared collection of music (all legal of course). We used to live together and would keep this collection on one machine, available to others over our local network. Now we live apart and each have a local version of this collection, but would like to keep it synchronised (i.e. his changes, additions and removals would be reflected in my local version and the other way round). Changes occur frequently since we're both very anal about organisation!

Every article I've read on backing up seems to focus on backing up one set of files to another set, rather than allowing changes to flow both ways (since I suppose this is quite atypical). Rsync keeps coming up but I haven't read anything that says it can be used when both machines are authorities.

Ideally we'd like some simple solution such as a script that will run every night, send changes to the other's machine and merge those, either waiting for user input when there are collisions (say, we both fix a tag in the same MP3 file or both add the same new album) or accepting one machine as the authority and logging collisions.

One problem that immediately strikes me about this is that for either machine to be able to send differences, it would need to keep an 'old' version to compare the current version against, requiring double the storage space.

Another option I've considered is putting the collection into some version control system hosted on one machine (or distributed), but I'm worried that the size of the repository will become huge if it's keeping a full history of changes. We only need the latest 'revision' on each machine - it's for backup purposes rather than for history.

Please help out by posting whatever ideas pop into your head, or by letting us know if there exists a common solution to this! Thanks!
 
Old 09-13-2008, 08:06 AM   #2
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian + kde 4 / 5
Posts: 6,845

Rep: Reputation: 2006Reputation: 2006Reputation: 2006Reputation: 2006Reputation: 2006Reputation: 2006Reputation: 2006Reputation: 2006Reputation: 2006Reputation: 2006Reputation: 2006
Check out unison. It's similar to rsync, but with bidirectional capability. I believe it can do what you want.

There's also a (new?) multiple-host sync tool called Tra. It's in the debian repositories, but the description says it's still in the early stages of development. It might be worth checking out though.

Last edited by David the H.; 09-13-2008 at 08:08 AM.
 
Old 09-13-2008, 08:15 AM   #3
TwinReverb
Member
 
Registered: Sep 2008
Location: Misawa AB, Japan
Distribution: Slackware
Posts: 191
Blog Entries: 2

Rep: Reputation: 40
Quote:
Originally Posted by hymerman View Post
Summary for those with little time: How would you keep two large sets of binary files stored on separate machines synchronised over the Internet, where either machine may change those sets?

My friend and I have always had a large shared collection of music (all legal of course). We used to live together and would keep this collection on one machine, available to others over our local network. Now we live apart and each have a local version of this collection, but would like to keep it synchronised (i.e. his changes, additions and removals would be reflected in my local version and the other way round). Changes occur frequently since we're both very anal about organisation!

Every article I've read on backing up seems to focus on backing up one set of files to another set, rather than allowing changes to flow both ways (since I suppose this is quite atypical). Rsync keeps coming up but I haven't read anything that says it can be used when both machines are authorities.
Use rsync. If one of you changes something, the one changing something should be the one to run rsync. Just keep each other informed as to when one of you is messing with their copy so the other can wait until they're done.

However, I've got a question. Your music may all be legal, but are you splitting the cost together (i.e. joint ownership)? If not, you may need to reconsider. If you buy a CD, that becomes your property, but not your friends' property, so sharing in that scenario is illegal. The same applies to DVDs. My advice: either keep a paper trail proving that you're buying these items jointly, or don't do this any more. Beyond which, if it is discovered that you are doing this, those prosecuting you may ignore the whole "joint ownership". Just a heads up.
 
Old 09-13-2008, 08:22 AM   #4
billymayday
LQ Guru
 
Registered: Mar 2006
Location: Sydney, Australia
Distribution: Fedora, CentOS, OpenSuse, Slack, Gentoo, Debian, Arch, PCBSD
Posts: 6,678

Rep: Reputation: 122Reputation: 122
That said, I think rsync will do exactly what you want. You'd run a cron job or similar at each end, and it will only copy updates files.

It doesn't need to keep old versions - they get overwritten once they are determined to be "old".

You could have an issue if you both update some file between syncs, but for music this probably isn't a problem.

Have a look at man rsync. The benefit of rsync is that it's absolutely standard on all systems
 
Old 09-14-2008, 08:34 AM   #5
hymerman
LQ Newbie
 
Registered: Sep 2008
Location: Loughborough
Distribution: Ubuntu
Posts: 3

Original Poster
Rep: Reputation: 0
Thanks chaps, those are some very good suggestions. It looks like Unison and Tra do exactly what I want so I'll trial them locally to see which I prefer.

"Joint ownership" seems very fuzzy to me. Would we have to have a receipt for each purchase where each item has been paid for half on my credit card and half on my friend's? Or would cash transfers (albeit with no proof) suffice? Thanks for the heads-up, but I don't think I'll worry about this. It's not like they're going to raid my house if my girlfriend watches a DVD I paid for while I'm not there.
 
Old 09-14-2008, 11:55 PM   #6
nny0000
Member
 
Registered: Aug 2003
Distribution: Slackware, Ubuntu
Posts: 158

Rep: Reputation: 30
Why don't you use an old computer as a file server? Allow both you and your friend to have access to it. Both of you can store your music on the file server and add files that way as opposed to dealing with a version control system.

I would suggest Unison or Rsync if the file transfer was one-way, but both need add/delete files to the system.

The only drawbacks that I see is accidental deletion of files, so you might want to break the hard drive into a read-only/read-write sections.
 
Old 09-15-2008, 04:36 AM   #7
hymerman
LQ Newbie
 
Registered: Sep 2008
Location: Loughborough
Distribution: Ubuntu
Posts: 3

Original Poster
Rep: Reputation: 0
Well, we both have dedicated server machines as well as desktops anyway, so we could well do that. We considered it a while ago since it would be pretty much the same as when we lived together albeit over a much longer wire! We figured it'd be quite a waste of bandwidth though, and wouldn't be as reliable as we'd want. It'd be annoying to have your music interrupted due to a bad connection (mine is fine but his ISP has a lot of outages). Thanks for the suggestion though
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Is there a tool for synchronised port switching? a.tsilfidis Linux - Networking 1 05-31-2007 08:36 AM
Streaming synchronised MP3s? 144419855310001 Linux - Software 3 10-01-2006 03:41 PM
Replicating files between servers. cpgeorge Linux - Networking 6 12-10-2004 11:07 AM
Audio on Multiple Machines Synchronised ar_howard Linux - Software 0 05-29-2004 05:58 PM
How to keep 2 directory trees synchronised? hrr Linux - General 2 07-02-2003 01:01 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 11:22 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration