LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Virtualization and Cloud
User Name
Password
Linux - Virtualization and Cloud This forum is for the discussion of all topics relating to Linux Virtualization and Linux Cloud platforms. Xen, KVM, OpenVZ, VirtualBox, VMware, Linux-VServer and all other Linux Virtualization platforms are welcome. OpenStack, CloudStack, ownCloud, Cloud Foundry, Eucalyptus, Nimbus, OpenNebula and all other Linux Cloud platforms are welcome. Note that questions relating solely to non-Linux OS's should be asked in the General forum.

Notices

Reply
 
Search this Thread
Old 10-01-2013, 12:14 AM   #1
shfyang
Member
 
Registered: Jun 2013
Location: metro Detroit
Distribution: RHEL, SuSE
Posts: 45

Rep: Reputation: Disabled
Angry GlusterFS failed to form trusted storage pool


I have 4 Centos 6.4 (x86_64) VM servers installed onto VMWork Station 7.1.4, the host machine is Windows 7 Ultimate 64-bit, each with 1GB memory and 1 CPU. The networking is through VMnet0(Bridged) using the WiFi card of the Windows 7 host. All computers get static IP through the Netgear router

The fours node have ip: 192.168.1.4, 192.168.1.7, 192.168.1.8, 192.168.1.9, the starting machine is 192.168.1.4, I don't have DNS server,so I just use the ip address. Each machine, I have 15GB dedicated for storage, the partition is always /dev/sda3, each mounted to /mnt/exp4 (or 7,8,9) folder. Firewall is setup as follows. Port 24007-24012 (tcp/udp), Port 34865-34867 (tcp/udp), 111(tcp/udp), the problem persis even I turn off firewall for all the nodes.

This is what I get:
*********************************************
[root@stor4 ~]# gluster peer status
peer status: No peers present
[root@stor4 ~]# gluster peer probe 192.168.1.7
peer probe: success
[root@stor4 ~]# gluster peer probe 192.168.1.8
peer probe: success
[root@stor4 ~]# gluster peer probe 192.168.1.9
peer probe: success
[root@stor4 ~]# gluster peer status
Number of Peers: 3

Hostname: 192.168.1.7
Port: 24007
Uuid: cc76c4e9-3ef2-4dc9-b704-cb160c404503
State: Accepted peer request (Connected)

Hostname: 192.168.1.8
Port: 24007
Uuid: 2bbd2a39-ef93-4b5d-aa36-00a9fad1775b
State: Accepted peer request (Connected)

Hostname: 192.168.1.9
Port: 24007
Uuid: 97297223-2927-4d39-a1e4-039663c0190a
State: Accepted peer request (Connected)
******************************************

The question is this is not right, I am supposed to get:
State: Peer in Cluster (Connected)

because the peering clearly does not work well. I go to each host and go to /var/lib/glusterd/peers/ and delete all the peering info files, such as these (1.4 has three files, other nodes have only one in this /var/lib/glusterd/peers/ folder )

-rw-------. 1 root root 72 Oct 1 00:51 2bbd2a39-ef93-4b5d-aa36-00a9fad1775b
-rw-------. 1 root root 72 Oct 1 00:51 97297223-2927-4d39-a1e4-039663c0190a
-rw-------. 1 root root 72 Oct 1 00:51 cc76c4e9-3ef2-4dc9-b704-cb160c404503

I'm ware of the attributes can be problematic, so I used the following command
[root@stor4 mnt]# getfattr -m- -d /mnt/exp4
[root@stor4 mnt]# setfattr -x trusted.glusterfs.volume-id /mnt/exp4
[root@stor4 mnt]# setfattr -x trusted.gfid /mnt/exp4

Aftre these, I restarted each nodes and start the peer search, once again I have the same peering issues. I am really buffled.

BTW, I did the same thing 3 weeks ago and did not experience these problems, this is just driving me crazy, as I have seen again and again, RHEL is just WAY TOO complicated and extremely prone to issues! :-(

Last edited by shfyang; 10-01-2013 at 12:16 AM.
 
Old 10-01-2013, 04:26 PM   #2
Habitual
Senior Member
 
Registered: Jan 2011
Distribution: Undecided
Posts: 3,314
Blog Entries: 5

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
How was gluster installed?

version?

Code:
rpm -qa | grep glusterfs
 
Old 10-01-2013, 08:16 PM   #3
shfyang
Member
 
Registered: Jun 2013
Location: metro Detroit
Distribution: RHEL, SuSE
Posts: 45

Original Poster
Rep: Reputation: Disabled
Thanks, this is the result:

[root@stor4 ~]# rpm -qa | grep glusterfs
glusterfs-libs-3.4.0-8.el6.x86_64
glusterfs-cli-3.4.0-8.el6.x86_64
glusterfs-server-3.4.0-8.el6.x86_64
glusterfs-fuse-3.4.0-8.el6.x86_64
glusterfs-geo-replication-3.4.0-8.el6.x86_64
glusterfs-3.4.0-8.el6.x86_64

I have 4 nodes, stor4, stor7, stor8 and stor9, they worked w/o any problem 4 weeks ago, from then, I have kept 4,7,8 intact, never bother with them since they exists solely for practising Gluster FS, for stor9, since it has extra space, I practised partitioning, building LVM, expanding swap, that sort of things using command lines (these are easy for me, but I still struggle with command line because I can't remember all of them)

It is really bizaare as nothing has really changed since then!
 
1 members found this post helpful.
Old 10-02-2013, 10:07 AM   #4
Habitual
Senior Member
 
Registered: Jan 2011
Distribution: Undecided
Posts: 3,314
Blog Entries: 5

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
First off, I would create 4 identical entries on each "brick" (node) in /etc/hosts using this format:
Code:
192.168.1.4 <tab> node1
192.168.1.7 <tab> node2
192.168.1.8 <tab> node3
192.168.1.9 <tab> node4
and then read and understand the techniques listed here and then troubleshoot using this document.

I have never had to "go to /var/lib/glusterd/peers/ and delete all the peering info files" and why would you do that?
You may have to delete the peers and re-add them to get it working again.
Never delete files in gluster-related directories using the command-line.
Use the gluster-related c-line tools instead.

http://gluster.org/community/documen....php/Main_Page
 
1 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: GlusterFS: Distributed Replicated Storage Across 4 Storage Nodes On CentOS 6.3 LXer Syndicated Linux News 0 01-08-2013 12:00 PM
LXer: Distributed Replicated Storage Across Four Storage Nodes With GlusterFS 3.2.x On Ubuntu 12.04 LXer Syndicated Linux News 0 06-21-2012 03:10 AM
LXer: Distributed Replicated Storage Across Four Storage Nodes With GlusterFS 3.2.x On Ubuntu 11.10 LXer Syndicated Linux News 0 04-20-2012 12:30 PM
LXer: Distributed Storage Across Four Storage Nodes With GlusterFS On Mandriva 2010.0 LXer Syndicated Linux News 0 04-28-2010 06:30 PM
LXer: Distributed Storage Across Four Storage Nodes With GlusterFS On Debian Lenny LXer Syndicated Linux News 0 06-25-2009 02:40 PM


All times are GMT -5. The time now is 02:30 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration