High Availability - Thinking out loud
 

Hi folks.

I've been tasked with setting up a HA setup as follows. I think I've planned this right, but bear with me as I explain the background behind this and what I've done. I've also attached a very crude diagram of what I'm planning.

Basically, we're setting up a nation wide "cloud" for the different branches of the local Red Cross (about 25 branches, spread out across the country) Since this is a hybrid cloud, mixed between our datacenter and other VPSs online (Linode most probably), balancing and replication has to be happening behind the scenes.

There is one change to the diagram on the PDF and that there is now a second load balancer, local to our datacenter.

Replicating the data between the MySQL servers should not be an issue. The one that worries me the most are the web servers. Do I use DRBD? But this is not a "Raid-1" on the net. Through puppet, I can control the different applications that are being stored on any of the machines, however, what about the actual files (not worried about the stuff that gets stored in the DB)? Do I RSync them? Is there another way?

WHERE would I place my heartbeat monitor (no, not over my heart thank you :))?

Suggestions? Plans? Ideas?

I'd really appreciate them :)

Is the remotely hosted service acting like a hot-backup or is the load balancer distributing evenly? .. trying to determine the timeframe for data replication. Not sure on your scale, maybe you could use a cdn ?

Well, what I'm aiming for is a "cloud" feel. In the sense that the user should not care where they connect (on to one of the hosted servers or to one at our DC), in the end, they can utilize whatever the service/application they're trying to access as well as the latest data. Much like any web based email, you can arbitrarily be connected to any available server that all correspond to mail.google.com (for example) and find all of your information. So I guess in short this is supposed to be distributing evenly.

How would the CDN help? Wouldn't that be perfect for distributing static items across the globe?

The scale is not as big as you think, but I'm terrified about losing data, so all my backups are backups of backups that have backups of originals. You can tell I was hurt bad by data loss one time.

[edit]
I think this is along the lines of what I want: http://www.howtoforge.com/efficient-high-available-loadbalanced-cluster-on-centos-5.3-direct-routing-method but I'm not sure if my design/plan is right or good enough.

There's no obvious reason why it shouldn't work but you will be running this in POC/test mode for a while first correct? .. so you'll be able to iron out any bugs. There are many ways to approach content replication/distribution from a simple rsync to a commercial CMS - try to find one that fits your budget and the content management staff.