High availablility server: Suggestions on linking two public servers
 

Hello

I have two public servers hosting some data for my clients. One of them is a primary one and one is a backup of the other. In order to make the backup, I need to rsync between them once a day. This isn't ideal for me as I'm dealing in ~1Tb of data. Therefore, the rsyncing takes a while to generate its list.


I have been reading http://www.howtoforge.com/high_avail...drbd_heartbeat about using software to turn two servers into one "virtual server", so that the client just seems one IP address, but the two servers work exactly in unison (just like RAID 1 on a local level). Thus, if one server breaks, the other instantly takes over and the client has no idea of the change.

My servers are in the public domain. The howto above is using NFS to share a certain partition on each server so that both srevers can write to each other's data partition. Is it safe to use NFS to link them if they have public IPs? If so, what security precautions can I take? I know that NFS has a number of daemons running, which don't always run on the same ports. I beleive that I can force these daemons to run on the same ports. Can I use MAC filtering on my firewalls too so that it is harder to spoof the IP address?

Basically, my key questions are:
1. Is using NFS the correct way to go about doing this?
2. Will it be safe?

Thanks in advance for all your advice
Hamish

Hi.

I would be wary of running NFS over public IPs. You might be better off with FUSE and SSH, if it's just a couple of servers:
http://fuse.sourceforge.net/sshfs.html

That would also simplify firewall configuration, as everything is done over the normal SSH ports and it should be as secure as SSH normally is. You'll need FUSE support in your kernel, though. Late 2.6 kernels should already support this.

Dave

Hiya

but that won't support the high availability which I need, will it?

Hamish

In fact, even if I did set this up, all connections between my servers would be transferred in plain text, and that isn't good for me. Is there a way to set this up AND encrypt all data transfer between the two?

Hamish

A couple of questions:

Are the servers geographically separated?
How do your clients connect to the servers?

they are in two separate data centres. Clients connect with ssh over rsync.

Basically, I want to make a folder on each server (eg /mnt/data) which uses the heartbeat program to create a high availability server (as per the howto) so that even if one server goes down, clients can still connect to the othre server. I will also need to find a way of keeping the user accounts synchronised on both machines.

Have looked into NFS tunneled through SSH for the encryuption, but I don't know how this will work with the high availability.

H

In fact, is there a system which would automatically encrypted every single packet of informatin sent between two servers? For example, would IPsec do this?

That way, I could just set up the HA system as it is - with unencrypted NFS - and then set up IPsec to encrypted every single packet sent between the two machines.

Any thoughts?
H

Good idea. Yes, you should be able to set up IPsec to encrypt just the data going between the servers.

Incidentally, I don't think you actually need NFS for this - the example you linked just happened to be an NFS server implementation. If your clients are connecting via SSH/rsync, then DRBD/heartbeat should be enough to mirror the data.

Dave

My servers run encyrpted RAID 0+1 hard drives (6 500gb hard drives which are mirrored, then stripped and the stripped array is encrypted). Would DRBD support this?

H

Possibly. Possibly not. If you have a single device which refers to the array, and which you can write to as a raw device (i.e. a device which you can run mkfs on to create a filesystem), then you might get away with it.

Also, from what I can see, you'll probably be creating your filesystem from scratch - don't know if that would be a problem for you though:
http://www.gossamer-threads.com/lists/drbd/users/7734

Dave

once I have made the stipe, I end up with a single device which relates to the array, for example /dev/md7.

mmmm:

"You too seem to have created the filesystem on the raw disk and then put
drbd on top. wrong. You should have put drbd on first and then created
the fs on top. "

Suggests that I make /dev/drbd0 and then make the RAID array on top. Don't know how that would work.

This (http://www.drbd.org/drbd-howto.html) also suggests that you make the drbd device, link it across the network and then create the filesystem on top

Hamish

Looks like it should be OK.

You *do* make the drdb device first, then put the filesystem on that, but in this case you'll have your /dev/mdX device underneath the drdb device instead of a /dev/hdX or /dev/sdX device.

So:

Create your RAID array (you've already done this, since you're using it right now)
Create / configure your /dev/drdbX device / software to use /dev/mdX as the raw device
Create your filesystem on top of the /dev/drdbX device

This will blow away all the data on the /dev/mdX array slice though, but I don't think you have a choice if you want to use drdb

Yeah. I was in the pub thinking about this too. I think you are right. However, what about encryption? Can I make the drbd device encrypted? Basically, I need to know that if someone pinches the serverthey cannot read the data.

H

Hmmm.

You can create encrypted filesystems in a file via a loopback devices, so you might be able to do this using /dev/loopX as the raw device for drdb (instead of /dev/mdX or /dev/hdX), where loop0 is the loopback device for the encrypted filesystem file. This involves creating a file to hold the encrypted filesystem, then mounting that file through the loop device. To be honest, though, this is a few too many layers of abstraction than I would be happy with for a production system. Debugging problems on a:
hd -> md -> file -> crypto -> loop -> drdb -> mount
filesystem is the stuff of nightmares.

In theory it should work Just Fine, but if customer data is at stake, I wouldn't like to risk it.

do you know of any other file systems which one can use across a network for high availabibility? For example, I have been reading about Lustre.

H