Red Hat Network Ping issue

farooqnasim · 08-12-2009, 03:03 AM

tcpdump output details
Pinging from 192.168.1.5(problematic machine) to 192.168.1.254(Gateway)
(destination host unreachable)
Tcpdump output at 192.168.1.5(problematic machine)[root@cosmo5 ~]# tcpdump -vn -i eth1 arp
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
09:48:50.533727 arp who-has 192.168.1.254 tell 192.168.1.5
09:48:52.534360 arp who-has 192.168.1.254 tell 192.168.1.5
09:48:53.534177 arp who-has 192.168.1.254 tell 192.168.1.5
09:48:54.534994 arp who-has 192.168.1.254 tell 192.168.1.5
09:48:55.536810 arp who-has 192.168.1.254 tell 192.168.1.5
09:48:56.536629 arp who-has 192.168.1.254 tell 192.168.1.5
09:48:57.536446 arp who-has 192.168.1.254 tell 192.168.1.5
09:48:57.579118 arp who-has 192.168.2.254 tell 192.168.2.11
09:48:59.537082 arp who-has 192.168.1.254 tell 192.168.1.5
09:49:00.536897 arp who-has 192.168.1.254 tell 192.168.1.5
09:49:01.536715 arp who-has 192.168.1.254 tell 192.168.1.5
09:49:01.572261 arp who-has 192.168.2.254 tell 192.168.2.11
09:49:05.558445 arp who-has 192.168.2.254 tell 192.168.2.11

13 packets captured
13 packets received by filter
0 packets dropped by kernel
Tcpdump output at 192.168.1.254(gateway)
[root@cosmosrv1 ~]# tcpdump -vn -i eth0 arp
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
11:04:09.372381 arp who-has 192.168.1.254 tell 192.168.1.5
11:04:09.372433 arp reply 192.168.1.254 is-at 00:1a:4b:b0:54:68
11:04:12.372753 arp who-has 192.168.1.254 tell 192.168.1.5
11:04:12.372759 arp reply 192.168.1.254 is-at 00:1a:4b:b0:54:68
11:04:15.912753 arp who-has 192.168.2.254 tell 192.168.2.11
11:04:16.374920 arp who-has 192.168.1.254 tell 192.168.1.5
11:04:16.374925 arp reply 192.168.1.254 is-at 00:1a:4b:b0:54:68
11:04:17.374717 arp who-has 192.168.1.254 tell 192.168.1.5
11:04:17.374721 arp reply 192.168.1.254 is-at 00:1a:4b:b0:54:68
11:04:18.374511 arp who-has 192.168.1.254 tell 192.168.1.5
11:04:18.374515 arp reply 192.168.1.254 is-at 00:1a:4b:b0:54:68
11:04:20.376100 arp who-has 192.168.1.254 tell 192.168.1.5
11:04:20.376105 arp reply 192.168.1.254 is-at 00:1a:4b:b0:54:68
11:04:21.375892 arp who-has 192.168.1.254 tell 192.168.1.5
11:04:21.375896 arp reply 192.168.1.254 is-at 00:1a:4b:b0:54:68
11:04:22.755886 arp who-has 192.168.2.254 tell 192.168.2.11
11:04:24.376275 arp who-has 192.168.1.254 tell 192.168.1.5
11:04:24.376279 arp reply 192.168.1.254 is-at 00:1a:4b:b0:54:68
11:04:25.376066 arp who-has 192.168.1.254 tell 192.168.1.5
11:04:25.376070 arp reply 192.168.1.254 is-at 00:1a:4b:b0:54:68
11:04:26.375860 arp who-has 192.168.1.254 tell 192.168.1.5
11:04:26.375864 arp reply 192.168.1.254 is-at 00:1a:4b:b0:54:68

22 packets captured
22 packets received by filter
0 packets dropped by kernel

Pinging from 192.168.1.5(problematic machine) to 192.168.1.2(old machine on the network which can ping to gateway and ping back to problematic machine)
(pinging ok)
Tcpdump output at 192.168.1.5(problematic machine)
[root@cosmo5 ~]# tcpdump -vn -i eth1 arp
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
09:50:25.743454 arp who-has 192.168.1.5 tell 192.168.1.2
09:50:25.743464 arp reply 192.168.1.5 is-at 00:1a:4d:45:5d:d1
09:50:41.101824 arp who-has 192.168.2.254 tell 192.168.2.11
09:50:42.046476 arp who-has 192.168.2.254 tell 192.168.2.11
09:50:42.937909 arp who-has 192.168.2.254 tell 192.168.2.11
09:50:53.498888 arp who-has 192.168.2.254 tell 192.168.2.11
09:50:57.482055 arp who-has 192.168.2.254 tell 192.168.2.11
09:51:00.586401 arp who-has 192.168.2.254 tell 192.168.2.11

8 packets captured
8 packets received by filter
0 packets dropped by kernel

Pinging from 192.168.1.2 to 192.168.1.5(problematic machine)
(pinging ok)
Tcpdump output at 192.168.1.2
[root@cosmo2 ~]# tcpdump -vn -i eth0 arp
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
11:26:05.558835 arp who-has 192.168.2.254 tell 192.168.2.11
11:26:07.115163 arp who-has 192.168.1.2 tell 192.168.1.5
11:26:07.115172 arp reply 192.168.1.2 is-at 00:1a:4d:46:22:d9
11:26:08.503962 arp who-has 192.168.2.254 tell 192.168.2.10
11:26:13.357358 arp who-has 192.168.2.254 tell 192.168.2.11
11:26:24.013964 arp who-has 192.168.1.230 tell 192.168.1.10
11:26:25.013777 arp who-has 192.168.1.230 tell 192.168.1.10
11:26:26.013590 arp who-has 192.168.1.230 tell 192.168.1.10
11:26:35.236833 arp who-has 192.168.2.254 tell 192.168.2.10
11:26:36.181292 arp who-has 192.168.2.254 tell 192.168.2.10
11:26:36.867693 arp who-has 192.168.2.254 tell 192.168.2.10

11 packets captured
11 packets received by filter
0 packets dropped by kernel

Pinging from 192.168.1.2 to 192.168.1.254(gateway)
(pinging ok)
Tcpdump output at 192.168.1.2
[root@cosmo2 ~]# tcpdump -vn -i eth0 arp
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
11:29:09.023082 arp who-has 192.168.1.230 tell 192.168.1.10
11:29:10.022895 arp who-has 192.168.1.230 tell 192.168.1.10
11:29:11.022708 arp who-has 192.168.1.230 tell 192.168.1.10
11:29:26.873604 arp who-has 192.168.2.254 tell 192.168.2.10
11:29:30.902103 arp who-has 192.168.2.254 tell 192.168.2.10
11:29:31.925150 arp who-has 192.168.2.254 tell 192.168.2.10
11:29:42.024910 arp who-has 192.168.1.230 tell 192.168.1.10
11:29:43.024722 arp who-has 192.168.1.230 tell 192.168.1.10
11:29:44.024536 arp who-has 192.168.1.230 tell 192.168.1.10

9 packets captured
9 packets received by filter
0 packets dropped by kernel
Tcpdump output at 192.168.1.254
[root@cosmosrv1 ~]# tcpdump -vn -i eth0 arp
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
11:00:44.807297 arp who-has 192.168.1.230 tell 192.168.1.10
11:00:45.807045 arp who-has 192.168.1.230 tell 192.168.1.10
11:00:46.806818 arp who-has 192.168.1.230 tell 192.168.1.10
11:00:55.168895 arp who-has 192.168.2.254 tell 192.168.2.10
11:00:59.029044 arp who-has 192.168.2.254 tell 192.168.2.10
11:01:03.204153 arp who-has 192.168.2.254 tell 192.168.2.10
11:01:03.804903 arp who-has 192.168.1.254 tell 192.168.1.10
11:01:03.804910 arp reply 192.168.1.254 is-at 00:1a:4b:b0:54:68
11:01:17.807688 arp who-has 192.168.1.230 tell 192.168.1.10
11:01:18.807459 arp who-has 192.168.1.230 tell 192.168.1.10
11:01:19.807227 arp who-has 192.168.1.230 tell 192.168.1.10
11:01:31.737440 arp who-has 192.168.2.254 tell 192.168.2.10

------------------------------
is this the way to show the output? does ip6tables make any difference coz working machines have both iptables and ip6tables enabled in services while this prolematic one has just iptables. plz note hardware/software details if needed
Gateway/Server(multipurpose server,(storage server,nfs,samba)

HP Proliant DL380, Generation 5, Enhanced
2xDual Core Intel Xeon, 3.0 Ghz each, L2 Cache=4MB/dual core processor, FSB=1333 MHz
4GB DDR2, 667 Mhz, HP brand
Running RHEL 4.4 and VMWare Server 1.0.3 Master Node

Clients(all wtih same config)

Intel Core 2 Duo 6600, 2.4 Ghz, L2 Cach=4MB, FSB=1066 MHz
Water cooled Gigbyte motherboard, PXE boot capable, built in RAID Controller
4GB DDR2, 800 Mhz
Running RHEL 4.4 and VMWare Server 1.0.3 Client Nodes(the problematic one doesn't have VMWare Server 1.0.3)

acid_kewpie · 08-12-2009, 04:10 AM

OK, so the arp reply from the gateway just never gets back inside the original machine. Odd. What does the switching setup look like here? It's possible that if you have slightly cranky switches they might refuse to play ball with their FDB tables or something, certainly getting odd now. I'd be intested to know if the arp reply is seen on the wire, but as the reply is unicast you couldn't just sniff it from a third box, so let's leave that for now.

What I would also query is the .2.11 IP. Why is that appearing on the same network interface? Could well be something of interest there.

Finally, let's try bypassing arp and see if it pings once you have a static arp entry in your machine... run "arp -s 192.168.1.254 00:1a:4b:b0:54:68" and try pinging once more. With tcpdumps again if it doesn't work.

farooqnasim · 08-12-2009, 06:50 AM

192.168.2.11 could be virtual computer's node IP on the same interface.i have found a document regarding current network which i am attaching as an image. it might elaborate 192.168.2.11 more. i tried "arp -s" and after that when i ping 254, it didn't give me any answer for aournd 20 minutes and i had to press ctl+c to quit. i configured eth0 instead of eth1 but again i can't ping to 254 and still can ping to 2 and others

acid_kewpie · 08-12-2009, 07:09 AM

Hmm, with VMware in the equation as well it seems, that's getting pretty messy, and without the base network infrastructure being consistent and reliable, I wouldn't really want to speculate what else is going on there.

farooqnasim · 08-13-2009, 12:23 AM

no further advice?

unSpawn · 08-14-2009, 06:21 AM

Try filling the ARP cache (/etc/ethers) with MAC IP pairs on 192.168.1.5(problematic machine) then 'arp -f /etc/ethers' to see if that works?
Lines have one MAC and IP per line, values are space separated and look like:
00:1A:4D:46:229 192.168.1.2
00:1A:4D:46:3A 192.168.1.3

acid_kewpie · 08-14-2009, 07:01 AM

Personally from what I can see, your layer 2 networking is most likely to be the issue, you clearly have issues with subnets getting mixed up that .2.11 IP intruding in .1.0/24 so as I can't have any real confidence in the switching, the only advice I can give from here is to fix your switching.