cluster (corosync, pacemaker, drbd, mysql) lost communication between nodes

arrals.vl · 05-10-2012, 03:28 AM

Hi all,

we have setup one cluster with two nodes.
-drbd
-corosync
-pacemaker
-mysql
-piranha

sometimes, we loose connection between the two nodes (node 1 and node 2). When this happens, the two nodes starts both the services as masters,because they don't see each other.
is there any way to tell to one node when connection is lost, for example " see if sec_vip of mysql is up. If so, do start the service drbd but not as master, do not start mysql, and do not start sec_vip. So in such way, only one node will be the master and we wouldnt have duplicated logs on databases or lost transactions.

Thank u very much for your time!!

Ervin

EricTRA · 05-10-2012, 03:53 AM

Hi and welcome to LinuxQuestions,

AFAIK that would take away the advantage of having high availability for your service. I think you should first look at why the connection gets lost. Not sure on how you can have a cluster with two nodes but if connection get lost, don't use one of them.

Looking forward to your participation in the forums. Have fun with Linux.

Kind regards,

Eric

BRHellman · 05-10-2012, 10:09 AM

Hi Ervin,

I agree with Eric; figuring out why your network is dropping is a wise idea, check the logs.

Additionally I would like to add that you should have a redundant network link, preferably in the form of a crossover cable. This will reduce the likelihood of these split brain situations. Using a recent version of Corosync you can setup redundant rings. Here is an example corosync.conf snippet to guide you in the right direction.

Code:

.....
	interface {
		# This is the back-channel subnet, which is the primary network
		# for the totem protocol.
		ringnumber: 0
		bindnetaddr: 10.0.1.0
		mcastaddr: 226.94.1.1
		mcastport: 5405
	}
	interface {
		# This is the storage network, which acts as a secondary, backup
		# network for the totem protocol.
		ringnumber: 1
		bindnetaddr: 10.0.0.0
		mcastaddr: 227.94.1.2
		mcastport: 5405
	}
....

Please let us know how it works out for you, or if you have any more questions.

Brian