Best daemon/program for keeping directories synchronised?

mat3j · 02-06-2013, 04:31 PM

Hello!

The goal is simple: have /home/user/directoryONE always synced with /home/user/directoryTWO with an interval around 10 seconds. Keep in mind though that there are a lot of subdirectories and files in directoryONE, so some kind of caching would be nice.

Any ideas?

Habitual · 02-06-2013, 04:37 PM

Quote:

Originally Posted by mat3j

Hello!

The goal is simple: have /home/user/directoryONE always synced with /home/user/directoryTWO with an interval around 10 seconds.

Any ideas?

Yeah, lower your expectation of 'interval'.

mat3j · 02-06-2013, 04:47 PM

np, just looking for suggestions.

chrism01 · 02-06-2013, 05:54 PM

rsync ?
It figures out the diffs between src & target and only txfrs diffs, so after the initial copy (which you could just use cp for), its pretty quick unless you change a LOT of stuff quickly.

Habitual · 02-06-2013, 05:57 PM

are you familiar with rsync?

or now "what he said".

chrism01 · 02-06-2013, 05:58 PM

me or the op?

mat3j · 02-07-2013, 04:52 AM

Quote:

Originally Posted by chrism01

rsync ?
It figures out the diffs between src & target and only txfrs diffs, so after the initial copy (which you could just use cp for), its pretty quick unless you change a LOT of stuff quickly.

Though rsync may be part of a solution, it's not a whole solution.

gdejonge · 02-07-2013, 09:24 AM

Quote:

Originally Posted by mat3j

Though rsync may be part of a solution, it's not a whole solution.

That's correct. But it is not the complete solution, because you didn't tell us what the complete problem was.
You had already figured out your "solution" and asked us how to implement it.

Maybe if you told us why you need to sync two directories with an interval of ~10sec. And what kind / amount of data we're talking about. And how often this data changes.
Maybe then we could help you to find a complete solution.

mat3j · 02-07-2013, 09:43 AM

ok, here are the numbers

Size: ~ 400 MB.
Number of files: ~ 25'000
Number of directories: ~ 3'000

Changes in /home/user/directoryONE are usually small, less than a few MB, though the update interval must be less than 1 minute.

whizje · 02-07-2013, 10:13 AM

You could use lsyncd.

Habitual · 02-07-2013, 10:49 AM

Quote:

Originally Posted by chrism01

me or the op?

the OP.

Philip Lacroix · 02-07-2013, 01:07 PM

Unison could also be interesting, it can handle bi-directional syncronization as well.

http://www.cis.upenn.edu/~bcpierce/unison/
http://www.cis.upenn.edu/~bcpierce/u...on-manual.html

Philip

chrism01 · 02-07-2013, 07:24 PM

I agree with whizje, given the extra information. http://linux.die.net/man/1/lsyncd looks like the solution.

It depends on exactly why you are doing this. If you are worried about HW failure, I'd go with RAID1 (or 5 or 6).

whizje · 02-08-2013, 12:28 AM

My first thought was also a raid 1, why do you want to sync those dirs?

schwing · 02-15-2013, 09:10 AM

I second the recommendation by whizje. Use lsyncd. I use it in a server environment to keep 3 servers in sync. It uses inotify to watch for new/changed inodes and then fires off rsync to copy those over. If I remember right, it has about a 15 second lag, but it should be pretty close to what you need.

https://code.google.com/p/lsyncd/