LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices



Reply
 
Search this Thread
Old 07-24-2008, 01:07 PM   #1
Tiago Cruz
Member
 
Registered: Jan 2003
Location: Brasil - São Paulo
Distribution: RHEL, SuSE, *BSD, Debian
Posts: 36

Rep: Reputation: 15
Unhappy fence_gnbd failed


Hello,

I have one machine (hotsite-bsb-la-1) exporting GNBD to two machines (hotsite-bsb-la-2 and "-3")

The cluster with RHEL 5.2 x86_64 and GFS was working very well, util I reboot the hotsite-bsb-la-2:

Code:
Jul 23 18:56:38 hotsite-bsb-la-1 openais[3082]: [CLM  ] CLM CONFIGURATION CHANGE 
Jul 23 18:56:38 hotsite-bsb-la-1 openais[3082]: [CLM  ] New Configuration: 
Jul 23 18:56:38 hotsite-bsb-la-1 openais[3082]: [CLM  ]         r(0) ip(10.65.13.30)  
Jul 23 18:56:38 hotsite-bsb-la-1 openais[3082]: [CLM  ]         r(0) ip(10.65.13.33)  
Jul 23 18:56:38 hotsite-bsb-la-1 openais[3082]: [CLM  ] Members Left: 
Jul 23 18:56:38 hotsite-bsb-la-1 kernel: dlm: closing connection to node 2
Jul 23 18:56:38 hotsite-bsb-la-1 openais[3082]: [CLM  ]         r(0) ip(10.65.13.31)  
Jul 23 18:56:38 hotsite-bsb-la-1 openais[3082]: [CLM  ] Members Joined: 
Jul 23 18:56:38 hotsite-bsb-la-1 openais[3082]: [CLM  ] CLM CONFIGURATION CHANGE 
Jul 23 18:56:38 hotsite-bsb-la-1 openais[3082]: [CLM  ] New Configuration: 
Jul 23 18:56:38 hotsite-bsb-la-1 fenced[3099]: hotsite-bsb-la-2.com not a cluster member after 0 sec post_fail_delay
Jul 23 18:56:38 hotsite-bsb-la-1 openais[3082]: [CLM  ]         r(0) ip(10.65.13.30)  
Jul 23 18:56:38 hotsite-bsb-la-1 openais[3082]: [CLM  ]         r(0) ip(10.65.13.33)  
Jul 23 18:56:38 hotsite-bsb-la-1 fenced[3099]: fencing node "hotsite-bsb-la-2.com"
Jul 23 18:56:38 hotsite-bsb-la-1 openais[3082]: [CLM  ] Members Left: 
Jul 23 18:56:38 hotsite-bsb-la-1 openais[3082]: [CLM  ] Members Joined: 
Jul 23 18:56:38 hotsite-bsb-la-1 openais[3082]: [SYNC ] This node is within the primary component and will provide service. 
Jul 23 18:56:38 hotsite-bsb-la-1 openais[3082]: [TOTEM] entering OPERATIONAL state. 
Jul 23 18:56:38 hotsite-bsb-la-1 openais[3082]: [CLM  ] got nodejoin message 10.65.13.30 
Jul 23 18:56:38 hotsite-bsb-la-1 fenced[3099]: fence "hotsite-bsb-la-2.com" failed
Jul 23 18:56:38 hotsite-bsb-la-1 openais[3082]: [CLM  ] got nodejoin message 10.65.13.33 
Jul 23 18:56:38 hotsite-bsb-la-1 openais[3082]: [CPG  ] got joinlist message from node 1 
Jul 23 18:56:38 hotsite-bsb-la-1 openais[3082]: [CPG  ] got joinlist message from node 3 
Jul 23 18:56:43 hotsite-bsb-la-1 fenced[3099]: fencing node "hotsite-bsb-la-2.com.br"
Jul 23 18:56:43 hotsite-bsb-la-1 fenced[3099]: fence "hotsite-bsb-la-2.com.br" failed
Jul 23 19:00:57 hotsite-bsb-la-1 last message repeated 50 times
Why fence was failing? Follow the cluster.conf:

Code:
<?xml version="1.0"?>
<cluster alias="hotsites" config_version="18" name="hotsites">
        <fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
        <clusternodes>
                <clusternode name="hotsite-bsb-la-1.com" nodeid="1" votes="1">
                <fence/>
                </clusternode>
                <clusternode name="hotsite-bsb-la-2.com" nodeid="2" votes="1">
                <fence>
                   <method name="single">
                        <device name="gnbd" nodename="hotsite-bsb-la-2.com"/>
                   </method>
                </fence>
                </clusternode>
                <clusternode name="hotsite-bsb-la-3.com" nodeid="3" votes="1">
                <fence>
                   <method name="single">
                        <device name="gnbd" nodename="hotsite-bsb-la-3.com"/>
                   </method>
                </fence>
                </clusternode>
        </clusternodes>
        <cman/>
        <fencedevices>
                <fencedevice agent="fence_gnbd" name="hotsite" servers="hotsite-1.com"/>
        </fencedevices>
        <rm>
                <failoverdomains/>
                <resources>
                        <clusterfs device="/dev/gnbd/hotsite" force_unmount="1" fsid="5666" fstype="gfs" mountpoint="/data" name="data" self_fence="1"/>
                </resources>
        <totem consensus="4800" join="60" token="10000" token_retransmits_before_loss_const="20"/>
</cluster>


Code:
# cman_tool status
Version: 6.1.0
Config Version: 18
Cluster Name: hotsites
Cluster Id: 27589
Cluster Member: Yes
Cluster Generation: 184
Membership state: Cluster-Member
Nodes: 2
Expected votes: 3
Total votes: 2
Quorum: 2  
Active subsystems: 8
Flags: Dirty 
Ports Bound: 0 177  
Node name: hotsite-bsb-la-1.com
Node ID: 1
Multicast addresses: 239.192.107.49 
Node addresses: 10.65.13.30
Thanks
 
  


Reply

Tags
cluster, gfs, rhel


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Bind9: NDC command failed : rndc: connect failed: connection refused Boudewijn Linux - Networking 19 01-02-2014 08:19 AM
udevd - rmdir(/dev/.udev/failed) failed: Permission denied pbhj Slackware 20 03-21-2008 11:46 AM
online_update failed - ERROR(Media:connection failed)[Connect failed] rover Suse/Novell 8 02-22-2005 08:57 AM
unpacking of archive failed: cpio: read failed-input/output error rafc Linux - Newbie 0 04-21-2004 10:03 AM
Loosing CD Drive while installing Mandrake 9.2 (ldconfig failed or idconfig failed) sjzabel Linux - General 3 02-26-2004 05:35 PM


All times are GMT -5. The time now is 03:55 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration