UCARP IP address failover creates multiple masters

duffrecords · 04-11-2012, 10:27 PM

I installed UCARP on four nodes running Gluster, which is then exported over NFS. The idea is to assign a VIP to one of the nodes so that if that node fails, the VIP will be assigned to one of the others. Since it's an active/passive system, /var/lib/nfs is shared between the nodes so NFS can be started on the new node without any issues. However, when I simulate a failure by unplugging the network cable, more than one node becomes the new master. Sometimes it's all three, sometimes just two. This will cause data corruption. Here is the network configuration for the first node:

Code:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
	address 10.80.80.1
	netmask 255.255.0.0
	network 10.80.0.0
	broadcast 10.80.255.255
	gateway 10.80.10.1
	# dns-* options are implemented by the resolvconf package, if installed
	dns-nameservers 10.80.90.250 4.2.2.2 8.8.8.8
	dns-search lax.xen.com

ucarp-vid 10
ucarp-vip 10.80.80.100
ucarp-password secret
ucarp-advskew 0
ucarp-advbase 1
ucarp-facility local1 
ucarp-master yes 
ucarp-upscript /usr/share/ucarp/vip-up 
ucarp-downscript /usr/share/ucarp/vip-down 
ucarp-nomcast yes

iface eth0:ucarp inet static 
	address 10.80.80.100
	netmask 255.255.0.0

The only difference between the configuration files are the advskew values. I've tried all sorts of combinations but the failover never results in a single master taking over. I did see some errors such as this one

Code:

Apr 11 18:46:44 gluster1 ucarp[13488]: [ERROR] exiting: pfds[0].revents = 8

but I don't know what they mean. Did I miss something in the installation?

kbp · 04-12-2012, 07:42 PM

I haven't played with ucarp, but have you tried modifying the advbase values? .. like:

Master: advbase 1
2nd preferred: advbase 2
3rd preferred: advbase 3
4th preferred: advbase 4

A node with a lower base should be more likely to become a master from what I can see. You may also be able to delay a node becoming master by playing with deadratio settings.

duffrecords · 04-12-2012, 11:19 PM

I tried that too. I decided to simply use a two-node failover system and that works, at least the IP address part. NFS doesn't fail over so smoothly because of file locking by the client. If I pull out the network cable during a file transfer, eventually it times out because of the stale NFS file handle, even though the IP address fails over and nfs-kernel-server starts on the new node.

kbp · 04-13-2012, 02:11 AM

I usually go with keepalived .. in regard to the stale nfs handles, I believe you can mitigate that by specifying a filesystem id in the nfs servers export options, that way it's not dynamically generated:

Code:

/shared      *(rw,sync,no_wdelay,mp=/shared,no_root_squash,fsid=1234)