Network Disk Mirroring of >16TB XFS Volumes
 

I'm setting up two media servers - one will be primary and the second will be the failover box if the first one dies.

My original plan was to use Heartbeat and DRBD to replicate the first to the second, but it turns out DRBD has a limitation of 16TB per volume, and my RAID 6 is 30TB in size.

I'm trying to figure out some options.

One is to split the partition into two, so it's under the 16TB limit of DRBD, share both partitions, and then use LVM to join them as a single volume at the OS level. I'm not sure what kind of performance hit that will make on read/write access since now it will need to go through LVM *and* DRBD.

Two is to use something like GlusterFS or XtreemFS. However, I am wary of both of those as the first is a FUSE-based system and isn't really tuned for performance, and XtreemFS is still kind of beta-ish.

Three is good old rsync, but that is a bit too asynchronous in my opinion. I'd like to be able to fail over virtually instantly and not lose any content.

This is a production system and performance of the disk systems are paramount, followed by reliability.

Whether or not rsync is "a bit too asynchronous" is a function of the rate of change for the content. If you are occasionally adding videos to your media server, you can have an inotifywait script kick off rsync as soon as a new video is added. Or run it every hour if that makes more sense. If you're only changing a few gigabytes per hour, maybe you want to wait longer. They won't be in perfect sync, but then errors won't be propagated immediately either, allowing for the failover box to be your 'warm recovery'.

Well, I'm trying to design for thousands of video files per second. Small files, mind you - 5mb or so - but still many thousands.

Even a one minute difference between the primary and secondary could mean missing tens of thousands of files.