connect: Network is unreachable
I'm somewhat of a newbie and am building a set of 4 servers for a clustering project. I've installed RHEL 5. Each server has 2 nics, one for the private and one for the public network connections. I've set static ip's all servers. All 4 servers are connected to a single switch. After all 4 installs, each server can see the other. I've ssh'd to each without any problems. So far, so good.
I added a 5th server to utilize iSCSI for common storage. I went through the same installation steps. The install was successful.
Now here's the rub. All 4 servers of the original servers can see the new sever (dev5). I can ssh, ping, etc. to the new sever from one of the original servers. I can't however, get out to ping any of the original 4 servers from the new one (dev5). I get the "connect: Netowrk is unreachable" message.
The /etc/sysconfig/network file on all the servers looks like this:
HOSTNAME=devx.domainname.suffix (where x=1,2,3,4 or 5)
An example of the /etc/sysconfig/network-scripts/ifcfg-eth0 files looks like this:
HWADDR= (some MAC address)
Any help on this one will be deeply appreciated.
On your newest server (dev5), is there an entry in the route table for 192.168.0.x? If this is missing (I could not hazard a guess as to why), you would get the "Network Unreachable" message when that server tried to reach 192.168.0.x by way of the public Internet (which will refuse to forward packets addressed to a private network).
Look at the result of
one for network 192.168.0.x, one for your public Internet address, and a default route (0.0.0.0) for everything else.
Netstat -rn results
The output from the netstat -rn on dev5 is as follows:
Destination Gateway Genmask Flags MSS Windows irtt Iface
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
Interestingly enough, the other servers (dev1-4) output from netstat -rn is:
Destination Gateway Genmask Flags MSS Windows irtt Iface
192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1
My confusion is that the bottom routing table works fine for the other servers even though it looks the same except for the extra nic.
If I understand you correctly, all I have to do is add the IP address of the server to the routing table and this should be o.k.?
Is it true that your iSCSI server (#5) has only a single interface (eth0)? And are all 5 of these machines connected to network 192.168.0.x? If both are true, then you ought to be able to get pings, etc to work in either direction among all 5 machines.
Let's step back a second, though, and ask precisely how did you test the ping function? There are two ways to specify the target of a ping:
BTW, the route line for 169.254.0.0 is bogus on all your machines. This network number is reserved for unattached networks; IMHO, it just causes confusion. And none of your 5 machines seem to have a default route specified; it should specify a destination of 0.0.0.0, and a gateway address, which should be the address to which otherwise unrouteable traffic should be sent. This is not any address belonging to any of your machines; it is usually the address of the other end of your link to the Internet.
To answer your questions, yes dev5 has only one nic. The others have dual nics. (I had to shift gears on my test and that was the only machine available to me).
Also yes, they all are connected to the 192.168.0.x network. Interestingly enough, the other 4 machines can communicate with dev5, via ping/ssh.
I performed each test. Both returned the same response of "connect: Network is unreachable". I also tried to ping the server name and it's fully qualified name, i.e. dev5 and then dev5.foo.net and both returned the same result.
The asymmetry is not surprising; your dev5 machine is doing what it is supposed to do when it receives a PING packet: swap source and destination IP addresses, and send it back out the interface it came in on.
The problem seems to be that traffic that originates on dev5 cannot find its way out of the box. Yet your reported route table says clearly that the way out is eth0 for network 192.168.0.x. Therefore, there must be something else that is stopping the traffic. Do you have any sort of firewall on dev5? That is one possible blockage.
If you cannot think of any reason that the packets should be blocked, try using tcpdump to report on all traffic that comes and goes through eth0. I can see several possible results of such a test:
1. No packets are leaving on eth0. This means that there is actually a blockage of some sort keeping traffic from finding its way out of the box. In addition to the firewall rules I alluded to above, it might be that there is some NAT going on, which would change the IP address so that there was no route for the revised address.
2. Packets are leaving, but nothing is coming back. There might be lots of reasons for this. The most obvious is not your case: the target might refuse to honor a PING. (Since dev1-4 can all ping one another, this is not the problem.) A less obvious reason is possible firewall rules on dev1-4 that prohibit dev5 from communicating with these machines.
3. Packets are leaving, and replies are arriving, but they are not being recognized for some reason.
Thanks for your help. Your last post put it all together for me. As suggested, I ran the tcpdump comand and found that nothing was getting out on eth0. I also made a connection with the firewall discussion in item #2 of your last post. Here's what I learned. The host name and IP address that I was pinging was on network 192.168.2.0. After reviewing this discussion thread I realized that the route table on dev5 didn't have the 192.168.2.0 network represented. Once I added that route to the table, all started working as I would have expected.
Thanks for your help in resolving this.
|All times are GMT -5. The time now is 01:56 PM.|