LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 03-16-2012, 12:22 PM   #1
calicowboy54
LQ Newbie
 
Registered: Jan 2012
Location: Huntington beach CA
Distribution: RH, CentOS
Posts: 6

Rep: Reputation: Disabled
Red Hat Cluster Problem


I have a problem with my RedHat 5.6 2 node cluster and fencing....

problem is at random time the cluster de-clusters and then shuts down the system, powers it off.... from what i can tell its in the fencing process when this happens. here is sample from log, other server logs just say system is going down at same time of time stamp:

Mar 15 16:41:13 sys1-cmdb2 openais[3955]: [TOTEM] entering GATHER state from 11.
Mar 15 16:41:13 sys1-cmdb2 openais[3955]: [TOTEM] Creating commit token because I am the rep.
Mar 15 16:41:13 sys1-cmdb2 openais[3955]: [TOTEM] entering RECOVERY state.
Mar 15 16:41:13 sys1-cmdb2 openais[3955]: [TOTEM] position [0] member 10.10.10.2:
Mar 15 16:41:13 sys1-cmdb2 openais[3955]: [TOTEM] previous ring seq 188 rep 10.10.10.1
Mar 15 16:41:13 sys1-cmdb2 openais[3955]: [TOTEM] aru 631 high delivered 631 received flag 1
Mar 15 16:41:13 sys1-cmdb2 openais[3955]: [TOTEM] Did not need to originate any messages in recovery.
Mar 15 16:41:13 sys1-cmdb2 openais[3955]: [TOTEM] Sending initial ORF token
Mar 15 16:41:13 sys1-cmdb2 openais[3955]: [CLM ] CLM CONFIGURATION CHANGE
Mar 15 16:41:13 sys1-cmdb2 openais[3955]: [CLM ] New Configuration:
Mar 15 16:41:13 sys1-cmdb2 openais[3955]: [CLM ] r(0) ip(10.10.10.2)
Mar 15 16:41:13 sys1-cmdb2 kernel: dlm: closing connection to node 1
Mar 15 16:41:13 sys1-cmdb2 fenced[3974]: sys1-cmdb1. not a cluster member after 0 sec post_fail_delay
Mar 15 16:41:13 sys1-cmdb2 openais[3955]: [CLM ] Members Left:
Mar 15 16:41:13 sys1-cmdb2 fenced[3974]: fencing node "sys1-cmdb1."
Mar 15 16:41:13 sys1-cmdb2 openais[3955]: [CLM ] r(0) ip(10.10.10.1)
Mar 15 16:41:13 sys1-cmdb2 openais[3955]: [CLM ] Members Joined:
Mar 15 16:41:14 sys1-cmdb2 openais[3955]: [CLM ] CLM CONFIGURATION CHANGE
Mar 15 16:41:14 sys1-cmdb2 openais[3955]: [CLM ] New Configuration:
Mar 15 16:41:14 sys1-cmdb2 openais[3955]: [CLM ] r(0) ip(10.10.10.2)
Mar 15 16:41:14 sys1-cmdb2 openais[3955]: [CLM ] Members Left:
Mar 15 16:41:14 sys1-cmdb2 openais[3955]: [CLM ] Members Joined:
Mar 15 16:41:14 sys1-cmdb2 openais[3955]: [SYNC ] This node is within the primary component and will pr
ovide service.
Mar 15 16:41:14 sys1-cmdb2 openais[3955]: [TOTEM] entering OPERATIONAL state.
Mar 15 16:41:14 sys1-cmdb2 openais[3955]: [CLM ] got nodejoin message 10.10.10.2
Mar 15 16:41:14 sys1-cmdb2 openais[3955]: [CPG ] got joinlist message from node 2
Mar 15 16:41:14 sys1-cmdb2 openais[3955]: [TOTEM] entering GATHER state from 9.
Mar 15 16:41:14 sys1-cmdb2 openais[3955]: [TOTEM] Storing new sequence id for ring c4
Mar 15 16:41:14 sys1-cmdb2 openais[3955]: [TOTEM] entering COMMIT state.
Mar 15 16:41:14 sys1-cmdb2 openais[3955]: [TOTEM] entering RECOVERY state.
Mar 15 16:41:14 sys1-cmdb2 openais[3955]: [TOTEM] position [0] member 10.10.10.1:
Mar 15 16:41:14 sys1-cmdb2 openais[3955]: [TOTEM] previous ring seq 192 rep 10.10.10.1
Mar 15 16:41:14 sys1-cmdb2 openais[3955]: [TOTEM] aru 11 high delivered 11 received flag 1
Mar 15 16:41:14 sys1-cmdb2 openais[3955]: [TOTEM] position [1] member 10.10.10.2:
Mar 15 16:41:14 sys1-cmdb2 gfs_controld[3986]: cluster is down, exiting
Mar 15 16:41:14 sys1-cmdb2 clurgmgrd[10489]: <warning> #67: Shutting down uncleanly
Mar 15 16:41:14 sys1-cmdb2 dlm_controld[3980]: cluster is down, exiting

Last edited by calicowboy54; 03-16-2012 at 02:21 PM.
 
Old 03-18-2012, 01:24 AM   #2
John VV
LQ Muse
 
Registered: Aug 2005
Location: A2 area Mi.
Posts: 17,090

Rep: Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474Reputation: 2474
Quote:
and then shuts down the system, powers it off
have you checked the cpu temp ?
it might be over heating
 
Old 03-19-2012, 11:15 AM   #3
calicowboy54
LQ Newbie
 
Registered: Jan 2012
Location: Huntington beach CA
Distribution: RH, CentOS
Posts: 6

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by John VV View Post
have you checked the cpu temp ?
it might be over heating
yes i checked the CPU Temp on the ILO port and no over heating.... only thing thats showing in ILO logs is power failure on power supply 02 but that is not even hooked up.... and no power failure on power supply 01 just simply stopped and powered down....
 
Old 03-19-2012, 12:17 PM   #4
Satyaveer Arya
Senior Member
 
Registered: May 2010
Location: Palm Island
Distribution: RHEL, CentOS, Debian, Oracle Solaris 10
Posts: 1,415

Rep: Reputation: 305Reputation: 305Reputation: 305Reputation: 305
Since you are using RedHat and paying for that, right? So you can take RedHat support: https://www.redhat.com/wapps/sso/log...port/cases/new you need to login first.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Red Hat Cluster Problems fcfury Red Hat 1 09-02-2011 12:54 PM
Red hat cluster YaaY Linux - Server 3 03-17-2011 03:54 AM
Red Hat Cluster and fence problem gontolal Red Hat 2 02-02-2010 02:51 PM
Red Hat Cluster Issue? your_shadow03 Linux - Newbie 3 08-19-2008 01:11 AM
Red Hat Cluster ??? your_shadow03 Linux - Newbie 1 08-13-2008 11:30 AM


All times are GMT -5. The time now is 01:21 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration