I am in the process of setting up a high availibility solution

The end result should be:
A single IP address to access all services
No noticeable downtime when one node goes down

It will have a couple of services which should use HA
tomcat
named
dhcpd
postgresql
subversion
nagios
mail (have not decided on which server)

There are I believe two scenarios

Either I create a virtual system by using kerrighed, openssi or perhaps beowulf or I use pgcluster, some sort of load balancer,tomcat clustering, drbd for synchronizing dhcp leases and dns zones, I have no idea as to how I should set up mail, nagios or subversion HA

Either sound like they could do the job

What I would like to know.

Which one people here think is 'better' meaning less maintenance, stable, reliable, scalable, free
And where I would find alternative resources as to what and how I would set this up.

Time to set up isn't important compared to low maintenance

Thanks a lot in advance

I don't see where a virtual system enters into the picture. You get no benefit from installing one OS on top of another and you get slower performance, less stability, and additional maintenance work by doing so.

-------------------------
Steve Stites

I think you will need a cluster. Stacking VM's on will not help if you base machine dies.

Problem statement:
Example Architecture:

-----------------
| LOAD BALANCER |
-----------------
|
|
----------------------------------
| |
----------------- Heartbeat -----------------
| NODE1 |---------------| NODE2 |
----------------- -----------------
| |
----------------------------------
|
-----------------
| SHARED |
| STORAGE |
-----------------

Possible Solution:
1. Both the nodes must have identical configuration at any time. All the updates must be synced properly.
2. Application binaries and data should be kept on different partitions.
3. Application binaries should be kept on local partition on each node.
4. Application data must reside on Shared storage.
5. Shared storage must provide fault tolerance to single/multiple disk failure, possibly RAID 1 or RAID 5.
6. Node 1 and Node 2 must be configured as Heartbeat master/slave nodes.
7. Your single IP address should be assigned to Load balancer, Load balancer will redirect requests to nodes. In normal condition, both nodes should provide service, in case of one node failure all the services will be transferred to second node.
8. You can setup DMZ configuration by changing subnet of nodes and restricting traffic using iptables. Do not try if you are not comfortable with firewall settings, as this will be complicated configuration.
9. Nagios should be kept outside this configuration to monitor the nodes efficiently.
10. You can host all the applications depending upon your hardware configuration. IF not possible, divide you configuration in Tomcat/named/dhcpd/postgressql & Subversion/Mail/Nagios.
11. DHCP provides inbuilt functionality for load balancing. Google for exact option. So, it will autotmatically take care of lease.
12. We can setup Nagios for HA using same configuration and making apache as HA service.
12. Mail services can be in HA using multiple MX records in DNS configuration.

Softwares:
For load balancing, lvs (linux-ipvs)
For Heartbeat, linux-ha
For RAID, mdadm or raidtools

Note:
I also dont have much idea about subversion HA. Let me search and update.

That looks very interesting, thank you.

Basically you're saying: create a fail over load balancer and install separate instances of each service on each node.
Heatbeat can be done using a separate NIC and through RS-232
This seams like it can really work...

Just a few questions arise then
Can the load balancer be load balanced and protected by fail over.
How many nodes/devices/hosts would this require (in the drawing storage and load balancer look SPOF)
Is DRBD suitable to synchronize data or is it better to use some sort of clustered file system(alternatively I have a 4disk RAID5 NAS, which I rather not use, since I have only one).
Can all applications that require that share sessions.
Are the specs, 2 Servers, XEON 3220, 8GB, 80GB RAID1 (hardware), 4NICS suffice for the setup (the expected clients will start very low (about 8 internally and remotely about 50(tomcat, mail, nagios))
Is it possible to use nagios on the same 2 nodes (perhaps both monitoring each other) and either being a distributed server for the other one.

Thanks a lot everyone so far.

Well,

I have done some research and have found a couple of things.

It looks like I have to set up each service on each node separately.
The components as they seem suitable so far:

• Hearbeat, IPVS and ldirectord
• Tomcat native clustering
• Cybercluster for multimaster PostgreSQL
• Subversion not possible
• Shared filesystem for DNS, Mail and DHCP
• Nagios unfeasible

Hearbeat, IPVS and ldirectord
Heartbeat serves for checking whether the other host is 'up', by doing this through both RS-232 and TCP/IP it is very unlikely for split-brain to occur.
IPVS as I understand assures that when a connection is set up to a certain host, that connection stays active between the client and the host, as long as it is available and the connection lasts, also it seems to make it possible to assign the same IP to multiple nodes
ldirectord serves as the load balancer which handles the direction of traffic to individual nodes (this sounds as if it may conflict in functionality with IPVS), also it is presently not clear if this is suitable for failover or if this would be a SPOF by itself.
Tomcat clustering is part of the default functionality all I have to implement is a load balancer and IP sharing functionality, tomcat detects when the other one is down but this behavior can be changed, combining heartbeat, ldirectord and IPVS should solve this.
Subversion seems an issue since locking the files would be required to update the data, otherwise if the same files get updated twice on different nodes version conflicts can arise.
Shared filesystem seems a harder choice. It looks like there are benefits to a clustered file system, DRBD and NFS. As of yet I am not sure which is better.
DNS does not need to be a serious problem because zone files will only be read when started or by manual reload, this can easily be solved by using some sort of storage sharing or synchronization (clustered filesystem, drbd, nfs)
Mail will be stored on disk so the same as with DNS can be used to share data
DHCP only shares leases and this can be done either the same way as with DNS and mail or through functionality inside the dhcp server

Any ideas?

I wrote a simple HA howto for Cent here for apache. Might be a help.
http://orangespike.ca/?q=node/62

I also noted reading this that there isn't anything related to the security of your servers. I'm not entirely sure on what network perimeter your deciding on, but feel free to ask if you need a hand.

Thanks for the link, very helpful

I'll tell a little more about the (current idea of the) topology
Two nodes, 4NICs
eth0 heartbeat
eth1 LAN
eth2 WAN
eth3 DRBD
I have a NAT router which will connect on the WAN network, which will be on a separate vlan.
I am not entirely sure I need heartbeat or DRBD.
I was thinking the following IP addresses
eth0 192.168.0.0/24
eth1 192.168.1.0/24
eth2 192.168.2.0.24
eth3 192.168.3.0/24

RS-232 for hearbeat
When I use cybercluster for synchronizing PostgreSQL data I cannot use a shared resource for that I believe (or perhaps by using drbd with a clustered file system and a distributed lock manager.
As far as I can tell Tomcat can easily handle a shared resource for file data.
Alternatively I could use MySQL multi master replication (it has an auto_increment_increment and auto_increment_offset configuration setting which prevent corruption of data by duplicate ID fields)
The other files can easily be synchronized by a cron job I think.

On security
I read a couple of things:
Perhaps port knocking is an option at least for ssh.
SMTP should be able to secure -I do not think other servers on the internet can send mails to SMTP servers that only have secure connections) and such information cannot be set in DNS (that I know of)-
IPTables to allow all traffic from the lan interface to anywhere
IIRC IPTables can protect against DDoS
All traffic to the other heartbeat node will be allowed from either node

I am wondering if a gateway can be configured as failover (making the/a cluster IP the gateway, otherwise I would have to eliminate the WAN network

I am still a little worried about the load balancer, since that should also be configured load balancing and fail over.

Nagios seems possible since both nodes seem to be able to be distributed as well as master, by running two instances of nagios, this would suggest that I need heartbeat to change the configuration in some way so agents can still connect to the distributed server and the master server (on the same node) will still be updated. I think I need to create some one way 8-cycle of data flow, unless nagios reloads on disk/database data constantly, then I need just a shared resource for the data
Subversion can be 'protected' by the distributed lock manager and a shared data resource.

I think I am getting the hang of it

Further research has helped me to discover a few things

Instead of the gateway, I could use squid for connecting outside the network.
For the load balancer ldirectord can be configured to perform fail-over while running active/passive.
Altough keepalived would be an alternative to heartbeat+lvs+ldirectord, I hear support isn't as good and it seems to be dead (can anyone confirm this?)
Squid might nog work properly when I create HTTPs connections through it (something to do with changing source IPs)
I could not find complete documentation and how things work on the inside of cybercluster.
I read at the mailinglist about a two node setup for lvs, has anyone tried this?

Any input is greatly appreciated

Hi Again,

You should better get a separate system for Nagios and Subversion. Nagios do not require to be setup in HA & Subversion you can't (May be your Load balancer itself).

I faced problem with https connection, when i was using squid in transparent mode. If you are changing LAN settings on the boxes
using (IE->Options->Connections->LAN settings) on windows
using HTTP_PROXY in /etc/profile for linux
You shouldn't face any problems with https.
Upto my understanding, transparent proxy can be setup for single port, thats why it creates probem with https(port 443) connections.


Yes, I tried on a fedora core 4 system. It is working fine.
Steps for testing:
1. Configure lvs on load balancer node for apache service on two backend nodes.
2. For testing, start apache services on both the backend nodes. In index.html file, put the server name on each of the backend nodes.
3. Send a request to http://<loadbalancer IP>/index.html
4. Depending upon algorithm you choose (Round robin, weighted), it will display index page from backend nodes. And you can cross check as it will display server name.

Suggestions
1. You can separate out both the backend nodes ( your actual servers ) and load balancer into different n/w altogether. This way your actual configuration will be converted into a DMZ, which will be much secure.
2. Instead of synchronizing data between two systems, better go for Shared storage.

I think I have an idea what I want by know.

Just one more thing I need te determine:
What mail server is the most suitable for HA

It's been a while since the last update and I reached the point where I actually start setting up the environment.
The most suitable mail server seems to be postfix, because it is easy to setup and is able to share data between instances.
The first server I am setting up is partitioned with 4 partitions:
/boot ext3 128mb
swap 12288mb
/ ext3 4096mb
VG:data lvm ~40GB (0 50% ext3, 1 50% gfs2)
What I need to know now is what to put in /etc/cluster/cluster.conf. Most examples and explanations I can found which seem to apply to my environment are based on some sort of master/slave scenario.
I compiled DRBD into the kernel, but now the init-script don't work anymore (I think), I get an error that the module cannot be loaded (which makes sense, since I patched the kernel). Also when I start drbd (using the init-scripts) both nodes are in the secondary role and the only way to get them both in primary is to change the role using drbdadm.
Currently I have two subnets configured
eth0: 192.168.1.0/24
eth3: 10.0.0.0/29
I am thinking about bonding eth2 and eth3 and using the bonded interface for both drbd sync and heartbeat.
My IO subsytem does at best 50MB/s so that should not pose any issues

Thanks in advance