LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices



Reply
 
Search this Thread
Old 09-30-2009, 01:01 PM   #1
tmccord1
LQ Newbie
 
Registered: Sep 2009
Posts: 2

Rep: Reputation: 0
File locking issues with clustering, GFS, and NFS


We are attempting to implement high-availability NFS services using Red Hat Cluster Suite and GFS file systems, shared out to a number of NFS clients. This is one of the patterns described in Red Hat documentation, wherein a pair of servers act as the NFS servers in an active/passive configuration. A virtual IP address is created as a secondary logical network interface and moved from server to server by cluster failover scripts.

The current configuration defines the NFS shares using the traditional NFS configuration file /etc/exports on the currently-active NFS server and and /etc/fstab on the NFS clients. In some versions of Red Hat clustering documents, this is pointed out as an invalid configuration, saying that the proper way to do this is to define the HA services entirely within the cluster.conf file (or to use the cluster UI to do the same). Later versions of GFS and clustering documents no longer mention this constraint.

When an NFS client mounts the NFS share, which, again, is a GFS file system mounted directly via a SAN on the two clustered GFS/NFS servers, most file I/O operations seem to be fine. It is only when an NFS client attempts to obtain an exclusive lock on a file using the fcntl system call that problems are observed. We have seen two behaviors in two different test systems, despite an attempt to make them identical at the outset:

Scenario 1) The first NFS client to attempt to get a lock succeeds and is able to release the lock explicitly via a system call or implicitly at exit. As long as only this first client is used to run the test program, it can be executed any number of times in a row with no errors. As soon as any other client or one of the GFS server nodes attempts to get a lock, the fcntl system hangs forever. The first client, recall, has released the lock and we ensured that 30 seconds or so had passed to clear any caching delays. The second process is non-interruptible once it hangs. If the second machine was the GFS/NFS server, it will have left a process on that GFS server that cannot be killed by any user, including root. If the second machine happened to be an NFS client, it will also have created a process on the active GFS/NFS server that cannot be killed by any user, including root. A reboot is required to remove the PID in either case.

Scenario 2) In our second test bed, no NFS client is able to obtain a lock at all. However, even the attempt causes the same behavior as described above on the GFS server.

When we mount the NFS file systems using the primary (physical) address of the the active NFS server, the locking issues go away. The use of a virtual IP address, which is necessary to achieve transparent failover, seems to be the critical factor.

NOTE: This has been misinterpreted in some forums as a Java issue, relating to the java.nio.channels.FileChannel locking interface. However, using strace on the Java program shows that in any reasonably modern JVM, the Java implementation wraps the fcntl system call. Thus, the problem can be generalized to the fcntl call. The use of the BSD flock() system call may not reliably reproduce the problem.

My first question is whether this hybrid approach to GFS/NFS high availability is supported or should be expected to work? We have not yet tried configuring NFS services through the cluster confguration tool, but plan to do that. The problem is that we have deployed the hybrid solution into a system that cannot be taken off line and reconfigured easily.

The second question is whether the particular combination of clustering, GFS distributed locking, NFS locking, and Virtual IP addressing for high availability is valid?
 
Old 10-01-2009, 09:58 PM   #2
kbp
Senior Member
 
Registered: Aug 2009
Posts: 3,758

Rep: Reputation: 644Reputation: 644Reputation: 644Reputation: 644Reputation: 644Reputation: 644
Hi,

I don't think your configuration will work correctly without configuring RHCS properly, have you read sources.redhat.com/cluster/doc/nfscookbook.pdf ?

cheers,

kbp
 
Old 11-18-2009, 10:43 AM   #3
tmccord1
LQ Newbie
 
Registered: Sep 2009
Posts: 2

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by kbp View Post
Hi,

I don't think your configuration will work correctly without configuring RHCS properly, have you read sources.redhat.com/cluster/doc/nfscookbook.pdf ?

cheers,

kbp
Could you please elaborate? While I personally don't have root access to do the configuration, my reading of the at least some versions of Cluster Suite/GFS documentation said specifically not to use /etc/exports and tradtional NFS configuration, but rather to configure the NFS service via cluster configuration. But later versions of similar docs don't say that as clearly. What part of what I am describing strikes you as improper user of RHCS?
 
Old 05-05-2010, 04:38 PM   #4
wallaceg
LQ Newbie
 
Registered: Sep 2007
Posts: 4

Rep: Reputation: 3
This may be your problem? ...

https://bugzilla.redhat.com/show_bug.cgi?id=500653
 
  


Reply

Tags
cluster, gfs, locking, nfs


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Samba on NFS File Locking Cottsay Linux - Software 4 01-26-2012 07:15 AM
Clustering & Fencing in GFS Brad.Scalio@noaa.gov Linux - Software 1 07-08-2009 05:09 AM
GFS and Clustering - general question akvino Linux - Enterprise 2 07-28-2008 09:56 AM
Can GFS used without clustering Guy Grenier Linux - Newbie 3 06-14-2008 04:29 AM
How to enable NFS file locking? ahz10 Linux - Networking 2 05-22-2006 12:34 PM


All times are GMT -5. The time now is 09:46 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration