There has got to be sysadmins out there who deploy high availability clusters serving Apache. I can do this, I even have a very expensive load balancer, however the caveat is real-time (not scheduled or batched like rysnc or unison) file replication. I have been down various roads. I have demo-ed commercial versions (spent alot of time and toil doing so.) The following is my experience with two vendors, it is not just a rant and I do have questions after (so if you don't wanna read my experiences skip ahead):

PeerFS: Great concept. Uses proprietary filesystem and kernel module to make network drive function like RAID 1 over Ethernet.

CONS: It was literally an oxymoron. I almost got fired because our cluster was down so often. When one server would go (for no apparent reason) the other's i/o would freeze up. That bug was dealt with, however, after that, when you brought a failed server up (oh yes, inexplicable crashes) it does this initial datacheck which effectively brings the working peer's i/o to a screeching halt from 1 minute to 3 minutes (I had 50GB to sync). What I really didn't like was when I added a remote peer (I was about to bring it into a GSLB cluster together with the primary peers.) When the peers on my primary production site went down (inexplicably. The developers constantly assured me this was not *normal* behavior.), the remote peer became the data initiator (kahuna.) This constituted a data consistency check over remote networks. This meant that my primary peers mounted their peerfs drives in a "remote" state. In other words, all my files were being pulled from the remote peer to serve Apache requests during the 20-30 hours this consistency check took to complete. I could do nothing. The servers had already been down for an unacceptable amount of time. The end result, SUPER slow website performance for almost a day! This inconvenience (final straw) prompted me to consider another vendor: Constant Replicator.

Constant Replicator or CR doesn't use a proprietary filesystem (which seems to make it much more stable.) Getting it to work on Debian was a days worth of tech support, but I was able to get it done. Then I asked for a quote. Let's just say that I could buy 4 more servers for the price to put CR on 3 servers. It was roughly about 20-25 times more expensive than PeerSync costs for Windows servers (which makes my bosses think that *NIX is inconvenient as we are moving closer and closer to a total Microsoft shop all the time.)

Does anyone out there have some experience setting up real-time file replication (Debian experience preferred) for HA clusters? I know there are hardware solutions, but they are just a little more expensive than CR and that's too much money. Plus the hardware solutions represent a single point of failure scenario to me also (that and the cost would never be feasible for my bosses.) My bosses want to turn my same-lan secondary peer and remote peer into rainy day paper-weights (hot standby.)

Another question, why hasn't the open source community embraced this? I know file replication can be a barrel of worms and all, but I'm surprised there aren't more midsized companies (like us) with this business requirement. DRBD was the closest project I found. However, secondary peers could not be mounted r/w or even read (making load balancing useless.)

I'll appreciate any help. I'd even consider paying consultation fees.

Thanks!!!

Its open source so you're free to write your own solution and share it with everyone else.

Unfortunately it does cost money to do sophisiticated things - for specific niches my company uses Linux or BSD but for some things we go to commercial Unix (which makes Windows look cheap - Unix however is more stable and scalable than Windows by quite a bit). Linux is more viable in commercial environments than it was when I first ran Caldera years ago but it still has a long way to go. I hate to say it but for some things Windows is the correct choice.

Oracle has OCFS for file sharing but its really written for their RAC database so I don't think it would help you. We are running that successfully on RH EL AS 3.

You could check out Veritas' website (or Symantec's since they bought Veritas) for capabilities of their Veritas Cluster Suite and Veritas Filesystem. Not sure they're available for Linux but seems likely they are. It won't be free though.

Also on another thread earlier today saw mention of Redhat Cluster Suite. Probably only available for Redhat distros but it may be worth your while to check out RedHat for that.

The problem is likely to be that whatever you do is going to cost money and if you figure out a way (short of writing your own code) to avoid spending money to run a business please let me know.

Trust me, if I had the "skills", I would certainly write my own.

I agree, I know it's a niche market and all. That ALWAYS means you pay significantly more. To retort, I would just say that it seems every other part of the complexity of HA linux clusters is supported by the open source community: From load balancing, failover (to name a few.) These are NOT simple things either. Seems like just that one aspect is missing from the puzzle. Even the author of DRBD mentions...

Again, I'd love to jump into that effort.

Thanks, for your response. I actually did look into the solutions you suggested. At the time they did no pan out. Maybe it's time to take another look.

P.S. I'd be willing to pay for the software (I do appreciate the labors developers take to bring us their "magic"), I just don't think backing up data from one server to another in realtime should cost the same as an entry-level car!

I've been using peersync7.2 at my work place. Its pretty flexible including the option for realtime replication. We use it for bi-directional realtime mirroring between two locations. It also comes with a utility called peerlock which can lock files being modified at one location from being opened for writing at the other end.

I'm not saying its a plug&play solution, we've had our share of problems with files dissapearing , files being concurrently edited at both locations, files not opening and such. But all of those problems were due to lack of implementation documentation from the vendor. Otherwise, with all the kinks worked out now, it seems to be showing no signs of messing up anymore.

The same software is also being used to generate a weekly backup mirror of all drives onto a backup server. The target is to eventually make use of it in a real-time disaster recovery solution with automated fail-over switching.

Also, in my opinion it didnt cost as much as an entry level car, but that just depends on how much entry-level car prices have fluctuated over the past 6 months

I guess you can check out their website peersoftware.com for more info.

PS: Right now it runs on Windows Server 2003 and I'm not sure if it is even available for any *nix distribution :P

Thanks for your response netah.

My first experience with file replication comes from PeerSync actually. We use it for Windows file replication. We also had some problems with it (until we upgraded that is.) It has worked very well for us.

I spoke with PeerSync and they said that for *NIX to use samba.

*NIX -> Windows w/ PeerSync -> *NIX

I didn't care to use another machine to do this replication. They told me that are working on a Linux version (or they were considering it...can't remember.) To date I see nothing about a Linux version on their Website--other than the scenario aforementioned.

I have heard wind of DRBD version 8 or 0.8 in the works. They are touting read-write mirrored peers (looks like good prospect.)