I have been trying for about 2 years here and there (and failing I might add) to create a "SAN/NAS system" (see my description below) that can export a data partition that can be mounted r/w by multiple cluster nodes. I have tied to use all of these technologies at one point or another in various combinations: drbd, nfs, gfs, ocfs, aoe, iscsi, heartbeat, ldirectord, round robin DNS, pvfs, cman, clvm, and several fencing solutions. I finally "gave up" trying to do it myself, and e-mailed about 10 SAN/NAS appliance re-sellers and only 2 said they could meet my specs. Their price range . . . $10,000 - $80,000. Well that priced me right out of the market.
So back to trying to do it myself. I came across glusterfs, it was super easy to set-up and test and is almost exactly what I want/need. There are only 2 issues.
#1. When one of the servers goes down the client hangs at least for a little while (more testing is needed) to be sure it comes back at all.
#2. The read/write tests I performed came in at 1.6, NFS on all the same machines came in at 11, and a direct test on the data server came in at 111. How do I improve the performance?

###############################################
My gluster set-up:
2 supermicro dual Xeon 3.0 ghz CPUs, 8gb ram, 4 @ 750gb seagate sata HDs, 3 in raid5 with 1 hot spare. (data servers)
1 supermicro dual AMD 2.8 ghz CPUs, 4gb ram, 2 @ 250gb seagate sata HDs in raid 1. (client server)
gluster is set-up with round robin DNS to handle the load balancing of the 2 data servers.
I also tried heartbeat with a virtual IP and ldirectord to redirect the traffic, but when I took down server 1, there was like a 10 second delay before server 2 took over the virtual IP and the client hung.

###############################################
Description of my dream "SAN/NAS system"
Data server:
2 units (appliances/servers) that each have a 4 drives in a raid5 disk set (3 active, 1 hot spare) (they can be active/passive or active/active I don't care)
These two units should mirror each other in in real-time.
If 1 unit fails for any reason the other picks up the load and carries on.
On a failure I want the data to be re-synced automatically when the failed unit comes back on-line.

Data clients:
Each cluster node machine (the clients) in the server farm (CentOS 5.4 OS) will mount 1 or more data partitions provided by the data server(s).
If the active server goes down (multiple HD failure, network issue, power supply, etc) the 2nd server takes over and the client machines never know.
All clients will mount r/w simultaneously, so some type of network file system that supports network file locks is required.

It would be even more ideal if the data servers could be 1-N instead of just 1-2.

To re-cap:
A. 2 SAN/NAS data servers mirroring each other in real time.
B. Auto fail over to the 2nd server if 1st fails (without the clients needing to be restarted, or even being interrupted if possible).
C. Auto re-sync of the data if a failed unit comes back on-line, when the sync is done the unit goes active again (assuming its normal state is active).
D. Multiple machines mounting the same partition in read/write mode (some kind of network file system).
E. Linux CentOS will be used on the cluster nodes.

Can anyone help?

You can use a third system to export raw devices (GNBD) which can be shared by your two servers but dont know exactly how reliable it is or you can use openfiler (Linux distro) to export file system as iSCSI which can be shared by your servers

Hye,

this is a highly complex configuration you wish to create here. We can discuss this for weeks. Also it is required to know what this cluster should serve. For this purpose there are several different cluster file system available like gfs, ocfs, etc.

Not sure if there is a software available for data mirroring between boxes (your NAS). This kind of mirroring is normally done by vendor specific storage software or as host based mirror with tools like Veritas Storage Foundation. Both are real expensive. I have not found such software for Linux yet, but if someone knows one he should contact me

If the storage part is solved, your cluster should not be the problem.

I should have had GNBD on the list of technologies I have tried.
I was using it for fencing.

As far as I know iscsi is a 1 to 1 technology. Do you know of a way to mount a single iscsi target on multiple machines? What network file system would be used to do the file locking?

Mainly it is serving web content, html, images, perl scripts. It also have asterisk nodes in it so config files, and voice recordings.

There are several mirroring solutions, I mentioned some of them.
Here is a short list (and I am sure it is not complete)
drbd
glusterfs
gfs
ocfs
hdfs
pvfs

However that is only part of the solution, you still need a way to fail-over if one of the servers goes down, and a way to mount the partition multiple times read/write.

I intend to write a detailed how-to if I ever figure this out (of if someone figures it out for me).

Yes iscsi target can be mounted on multiple nodes but it depends on the file system if shared storage is required. You can try GFS for that. As GFS is a shared storage mechanism you don't need to configure fail-over for that. For NFS you can format the file system as ext3 and configure HA cluster for fail-over scenarios.