I have been looking for some time now to find more information on how to replicate data between servers. What I'm facing is a hardware load balancer to pull from two or more servers. Since NFS would present a single point of failure, I do not want to use it; redundancy, which is the main goal would be compromised here. I could use Rsync, but I need to implement real-time replication so that clients will be able to view their uploads/updates instantly and not be confused. Also, I have GBs of data that Rsync would need to synchronize on a timely basis (very resource intensive.) I have looked at FAM and IMON but this is a very old tutorial and the SGI::FAM is so old that installation is no longer possible (I really don't want to take up the torch either as I'm no perl developer.) I understand FAM provides an API so that apps can use it to monitor file changes, but I am no C developer either! My point being there's got to be someone out there who has done real-time file replication for linux servers in a load balanced environment. Any tips will be well appreciated!

I'm not exactly sure if this is what you are looking for or if you want to have fail over capabilities as well. But for simple file synchroniztion this will work http://www.cis.upenn.edu/~bcpierce/unison/

Many thanks!

I did look at unison. I discounted it after it seemed to me that it really isn't for real-time file replication. I will read up on it to see if possibly I can get it to work in the fashion I need it to. In the meantime, do you know of a way to implement it in a real-time (or as close as I can get) fashion?

Have you looked at
http://www.acronis.com/enterprise/products/ATISLin/
the catch? $700

http://www.redhat.com/software/rha/gfs/

http://www.linuxfocus.org/English/Ma...ticle199.shtml
old... uses fam

http://www.alphaworks.ibm.com/tech/vitalfilebackup

If you find a solution, please post it, because I am looking for the same thing, though my real-time requirements are lower.

No, but this seems to be backup software and not mirroring. Please let me know if I'm wrong.

I have looked at GFS. It is my understanding that GFS is wonderful but expensive (extra hardware to buy.) However, I am not familiar with Redhat's flavor of GFS. Also, this may not represent a high availability solution. Again... let me know if I'm wrong.

I actually posted this link in my original post. It looked like it was just what I was looking for. Since I have a 2.4.x kernel, IMON (the kernel feature has a different name but does the same thing as I recall) gives files the ability to fire an event that FAM listens for. The problem is, of course, that the perl module mentioned that hooks into FAM (via API) does not install due to dependencies that cannot be met now (again, I forgot the details.) I suppose if I were really desperate (or ambitious), I would attempt to write my own. Another issue is the fact that the perl module was not updated for so long (is this a less-than-ideal way to go about this?)

This one is very interesting. Replication was not the end goal but it may work. Allthough, it's not open-source and therefore there may be better non-free alternatives, such as:

Repliweb
or...
PeerSync

However, I just can't believe with Load balancers out there (including LVS ) there is no implementation for doing realtime file synchronization for your web server farm.

I was just looking for the same thing and I found this http://www.drbd.org/. Sounds very promising, when I start working with it I'll post my results.

Awesome, this looks the most promising so far. It may be tricky to implement with my existing production servers (I will need to scrounge up some boxes to test with.) However, this looks like it could be THE ANSWER I will also post my experience.

Thanks all!

DAMN!

Got DRBD working on Debian servers. After all the toil, it dawns on me that the secondary block device cannot be mounted. Instead this system is meant to do a switchover in the case the primary goes down (via heartbeat.) As for the ro on secondary, this is from the drbd website:

So it seems, this IS NOT the answer for file synchronization for one or more load balanced servers. Unless, I am mistaken, I'm back to the drawing board {SIGH}

Okay, I found some NON-free software that works very well. Only problem, it's fairly expensive: PeerFS. It does what I hoped drbd would do: that is, the files in a network block device are distributed to all nodes simultaneously and they are all instantly available for reading. The setup was extremely easy. The only downside is the dough (WAY cheaper than hardware based solutions though!) Does anybody know of opensource alternatives?

In case any thread subscribers are interested on how things turned out...well, they didn't really. PeerFS was a nightmare and another solution (that seems to work well) costs anywhere from 2-4 times as much as PeerFS (Constant Replicator.)

Here's a more detailed recent post

How abt RSYNC !

http://sunsite.dk/info/guides/rsync/...mirroring.html

http://amitsharma.linuxbloggers.com/how_to_rsync.htm

I do love rsync! BUT...it's not an alternative for realtime synchronization. I use it currently for my secondary peers that HAVE to be hot spares because I can only feasibly replicate every 2 hours or so (takes like 5-10 minutes just to count my 1.5 million files!) and with websites constantly changing, my clients are gonna get pissed they upload and can't see changes on a secondary load balanced peer for up to 2 hours!

Now Linux has a function like FAM, maybe more powerful, that is
inotify.

Recently I started an open source project "cutils" on the
sourceforge:
https://sourceforge.net/projects/crablfs/

The document can be found at:
http://crablfs.sourceforge.net/#ru_data_man

This project's mirrord/fs_mirror tools is a near realtime file system
mirroring application across 2 or more hosts, to mirrors the many
small files from one to another as soon as possible when there is
any modification.

mirrord/fs_mirror makes use of inotify, which is a function
afforded by the recent Linux kernel (from 2.6.12). I think it is a
counterpart of FAM, since Linux FAM has stopped so long.

I started these projects on LFS first, but these tools can also be
applied on other distributions, for instance, I used them also on the
RHEL4 environment.

I hope these tools can be useful to you.

Thanks.