Redhat CLuster fence failed - problem
 

Hi I have a RED-HAT 5.3 and configured cluster RHCS on two nodes.
Everythings works ok,services are relocated when one of the active nodes shutdown or reboot.
But I have a problem when I unpluded power cord of the active node. Recources cannot relocate. In logs I see that node2 cannot fence node1 (whitch is power off) -fening failed.
How to resolve that problem, thanks in advance.

Hi,

A cluster needs a minimum of three nodes. If only 2 physical nodes are present a so called quorum disk is needed.

Why? With an even amount of nodes the cluster cannot determine which one is faulty and needs to be taken out of service (a so calles 'split-brain' situation can arise).

Take a look here: RedHat - 2.5. Considerations for Using Quorum Disk and here: RedHat - Enhancing cluster quorum with QDisk

Hope this helps.

fence failed
 

I Add the quorum disk but the result is the same ,when I unpluge teh power cord from the active node,the second one cannot fence
Icreated quorum disk and added:

<cman expected_votes="3" two_node="0"/>
<quorumd interval="3" tko="23" device="/dev/mapper/quorum_diskp1" votes="1"/>
<cman deadnode_timeout="135" expected_nodes="3"/>

the result is the same node cannot facing

Hi,

Are you using (external) power fencing or the dummy fencing (fence_manual)? The last one (fence_manual) doesn't fence by itself, it needs human intervention.

Take a look here: Red Hat GFS 6.0: Administrator's Guide - 10.2 Fencing Methods

fence failed
 

Hi there is any method to automate manual fencing?

After I added quorum disk I observed that situation.
When I for example shutdown node2, and reboot node1
The node1 gets up correctyly,but after that I power node2 ,and when It comes up the cman deamon on node1 is killed.And that node doesnt reboot.

Before adding quorum disk when I had in cluster.conf
<cman expected_votes="1" two_node="1"/> it worked.
The node reboots.
Where is the problem?

Hi,

Not that I know of, that is one of the down sides of manual fencing (which should not be used other then testing/evaluation or be used as a last resort fallback next to 'real' fencing). The human intervention is part of the mechanism (fence_ack_manual must be used to acknowledge the failed node).

Did you read the fence_manual and fence_ack_manual man page?

I'm not sure which docs you are using as a reference, but this one is a good start: Red Hat Cluster Suite for Red Hat Enterprise Linux (Red Hat 5.2, 5.3 version is not available yet).

Hope this gets you going again.

cluster fencing problem
 

Do I have to have fencing configured to quorum disk work properly?

Hi,

That depends.

- If this is 'just' an exercise and meant to get some understanding in (Red Hat) clustering: No you don't need it, as long as you know/understand what manual fencing does (and doesn't!) and its place in the bigger picture. At certain points your cluster will not react as described in the docs/manuals (that is a given! and very annoying at times) because manual fencing isn't set up and/or isn't configured properly.

- If this cluster is for real: Yes you do. A quorum disk is needed to end up with an uneven amount of nodes, without an uneven amount of nodes fencing cannot properly decide which is the 'bad node'. You could decide, if you have the resources, to add a third physical node and drop the quorum disk. But that leaves you with manual fencing. As I stated before, manual fencing should not be used other then testing, evaluation or be used as a last resort fallback next to 'real' fencing.