How to properly configure a 2 node glusterfs volume ?

anindyameister · 02-02-2016, 07:37 AM

I am trying to make a 2 node linux servers with high available apache using glusterfs 3.7.6 for data replication and pacemaker+corosync as resource manager. However i'm seeing a problem with gluster in the specific scenario when both nodes get shutdown and one of them comes online first. Even though there's a brick on that node and gluster service is running, there's no brick process.

Code:

[root@node1 ~]# gluster volume status data 
Status of volume: data
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick node1:/gluster_data                   N/A       N/A        N       N/A  
NFS Server on localhost                     N/A       N/A        N       N/A  
NFS Server on localhost                     N/A       N/A        N       N/A  

Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks

And when I start the other node, everything seems to work and I can mount the volume.

Code:

[root@node1 ~]# gluster volume status data
Status of volume: data
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick node1:/gluster_data                  49152      0          Y       2674 
Brick node2:/gluster_data                  49152      0          Y       3086 
NFS Server on localhost                     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       2796 
NFS Server on node2                         N/A       N/A        N       N/A  
Self-heal Daemon on node2                   N/A       N/A        Y       3085 

Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks

And at this point if I shut down node2, node1 brick process remains active so atleast I can mount and use it.

Code:

[root@node1 ~]# gluster volume status data

Status of volume: data
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick node1:/gluster_data                   49152     0          Y       2674 
NFS Server on localhost                     N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       2796 

Task Status of Volume data
------------------------------------------------------------------------------
There are no active volume tasks

So my observation is that for the gluster volume to work, both nodes need to be online atleast for a moment so the bricks can start and then if one node goes down, it won't affect the volume operation. So how can I make it work when one of the nodes come online from a total blackout ?

fjennings · 02-02-2016, 08:58 AM

Technically only one node needs to be up, when the other comes up, it should enter a healing state to begin syncing.

Make sure that when you are entering the configuration in the client's fstab, that use the following (or similar) layout:

gluster1:/brick1 /mnt/brick1 glusterfs defaults,_netdev,backupvolfile-server=gluster2 0 0

Using the backupvolfile will ensure that if the client does not see gluster1, that it will probe 2 as well. This provides you with your fault tolerance.