GlusterFS failed to form trusted storage pool

I have 4 Centos 6.4 (x86_64) VM servers installed onto VMWork Station 7.1.4, the host machine is Windows 7 Ultimate 64-bit, each with 1GB memory and 1 CPU. The networking is through VMnet0(Bridged) using the WiFi card of the Windows 7 host. All computers get static IP through the Netgear router

The fours node have ip: 192.168.1.4, 192.168.1.7, 192.168.1.8, 192.168.1.9, the starting machine is 192.168.1.4, I don't have DNS server,so I just use the ip address. Each machine, I have 15GB dedicated for storage, the partition is always /dev/sda3, each mounted to /mnt/exp4 (or 7,8,9) folder. Firewall is setup as follows. Port 24007-24012 (tcp/udp), Port 34865-34867 (tcp/udp), 111(tcp/udp), the problem persis even I turn off firewall for all the nodes.

This is what I get:
*********************************************
[root@stor4 ~]# gluster peer status
peer status: No peers present
[root@stor4 ~]# gluster peer probe 192.168.1.7
peer probe: success
[root@stor4 ~]# gluster peer probe 192.168.1.8
peer probe: success
[root@stor4 ~]# gluster peer probe 192.168.1.9
peer probe: success
[root@stor4 ~]# gluster peer status
Number of Peers: 3

Hostname: 192.168.1.7
Port: 24007
Uuid: cc76c4e9-3ef2-4dc9-b704-cb160c404503
State: Accepted peer request (Connected)

Hostname: 192.168.1.8
Port: 24007
Uuid: 2bbd2a39-ef93-4b5d-aa36-00a9fad1775b
State: Accepted peer request (Connected)

Hostname: 192.168.1.9
Port: 24007
Uuid: 97297223-2927-4d39-a1e4-039663c0190a
State: Accepted peer request (Connected)
******************************************

The question is this is not right, I am supposed to get:
State: Peer in Cluster (Connected)

because the peering clearly does not work well. I go to each host and go to /var/lib/glusterd/peers/ and delete all the peering info files, such as these (1.4 has three files, other nodes have only one in this /var/lib/glusterd/peers/ folder )

-rw-------. 1 root root 72 Oct 1 00:51 2bbd2a39-ef93-4b5d-aa36-00a9fad1775b
-rw-------. 1 root root 72 Oct 1 00:51 97297223-2927-4d39-a1e4-039663c0190a
-rw-------. 1 root root 72 Oct 1 00:51 cc76c4e9-3ef2-4dc9-b704-cb160c404503

I'm ware of the attributes can be problematic, so I used the following command
[root@stor4 mnt]# getfattr -m- -d /mnt/exp4
[root@stor4 mnt]# setfattr -x trusted.glusterfs.volume-id /mnt/exp4
[root@stor4 mnt]# setfattr -x trusted.gfid /mnt/exp4

Aftre these, I restarted each nodes and start the peer search, once again I have the same peering issues. I am really buffled.

BTW, I did the same thing 3 weeks ago and did not experience these problems, this is just driving me crazy, as I have seen again and again, RHEL is just WAY TOO complicated and extremely prone to issues! :-(

How was gluster installed?

version?

rpm -qa | grep glusterfs

Thanks, this is the result:

[root@stor4 ~]# rpm -qa | grep glusterfs
glusterfs-libs-3.4.0-8.el6.x86_64
glusterfs-cli-3.4.0-8.el6.x86_64
glusterfs-server-3.4.0-8.el6.x86_64
glusterfs-fuse-3.4.0-8.el6.x86_64
glusterfs-geo-replication-3.4.0-8.el6.x86_64
glusterfs-3.4.0-8.el6.x86_64

I have 4 nodes, stor4, stor7, stor8 and stor9, they worked w/o any problem 4 weeks ago, from then, I have kept 4,7,8 intact, never bother with them since they exists solely for practising Gluster FS, for stor9, since it has extra space, I practised partitioning, building LVM, expanding swap, that sort of things using command lines (these are easy for me, but I still struggle with command line because I can't remember all of them)

It is really bizaare as nothing has really changed since then!

First off, I would create 4 identical entries on each "brick" (node) in /etc/hosts using this format:

192.168.1.4 <tab> node1
192.168.1.7 <tab> node2
192.168.1.8 <tab> node3
192.168.1.9 <tab> node4

and then read and understand the techniques listed here and then troubleshoot using this document.

I have never had to "go to /var/lib/glusterd/peers/ and delete all the peering info files" and why would you do that?
You may have to delete the peers and re-add them to get it working again.
Never delete files in gluster-related directories using the command-line.
Use the gluster-related c-line tools instead.

http://gluster.org/community/documen....php/Main_Page