[SOLVED] servers on same VLAN can't communicate

duffrecords · 07-18-2014, 12:35 PM

I recently set up some tagged VLANs on a managed switch. I configured a server that has two NICs to have a unique address and gateway on each, thus connected to two VLANs simultaneously (see http://www.linuxquestions.org/questi...es-4175510522/ for more details). It appears to be working correctly, as the server can ping both of its gateways and I can SSH to both of the server's addresses from my workstation (which is on a different VLAN entirely: 192.168.10.0/24). However, now I've noticed that if I set up another server on the same VLAN as the dual-NIC server (192.168.101.0/24), neither one can reach the other via that network. They can reach each other via the alternate VLAN (192.168.102.0/24). As long as they're sending traffic via different networks, the packets go out through the gateway and arrive at their destination but when sending traffic on the same network, the packets mysteriously disappear.

Actually, they don't quite disappear. I monitored the first NIC with tcpdump while I tried to SSH to it from the other server on the same network and I do see the packets arriving. However, the SSH connection is never established and it times out. Here is some sample tcpdump output from the dual-NIC server:

Code:

[root@virthost1 ~]# tcpdump -i eth0 src not 192.168.10.166 and dst not 192.168.10.166
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

10:19:28.228053 IP 192.168.101.3.56293 > 192.168.101.4.ssh: Flags [S], seq 3343503303, win 14600, options [mss 1460,sackOK,TS val 79326555 ecr 0,nop,wscale 7], length 0
10:19:28.228104 IP 192.168.101.4.ssh > 192.168.101.3.56293: Flags [S.], seq 803956727, ack 3343503304, win 14480, options [mss 1460,sackOK,TS val 744499908 ecr 79326555,nop,wscale 7], length 0
10:19:29.227711 IP 192.168.101.3.56293 > 192.168.101.4.ssh: Flags [S], seq 3343503303, win 14600, options [mss 1460,sackOK,TS val 79327555 ecr 0,nop,wscale 7], length 0
10:19:29.227741 IP 192.168.101.4.ssh > 192.168.101.3.56293: Flags [S.], seq 803956727, ack 3343503304, win 14480, options [mss 1460,sackOK,TS val 744500908 ecr 79326555,nop,wscale 7], length 0
10:19:29.627362 IP 192.168.101.4.ssh > 192.168.101.3.56293: Flags [S.], seq 803956727, ack 3343503304, win 14480, options [mss 1460,sackOK,TS val 744501308 ecr 79326555,nop,wscale 7], length 0
10:19:31.227707 IP 192.168.101.3.56293 > 192.168.101.4.ssh: Flags [S], seq 3343503303, win 14600, options [mss 1460,sackOK,TS val 79329555 ecr 0,nop,wscale 7], length 0
10:19:31.227738 IP 192.168.101.4.ssh > 192.168.101.3.56293: Flags [S.], seq 803956727, ack 3343503304, win 14480, options [mss 1460,sackOK,TS val 744502908 ecr 79326555,nop,wscale 7], length 0
10:19:31.627371 IP 192.168.101.4.ssh > 192.168.101.3.56293: Flags [S.], seq 803956727, ack 3343503304, win 14480, options [mss 1460,sackOK,TS val 744503308 ecr 79326555,nop,wscale 7], length 0
10:19:33.227706 ARP, Request who-has 192.168.101.4 tell 192.168.101.3, length 46
10:19:33.227732 ARP, Reply 192.168.101.4 is-at 00:25:90:c9:6d:30 (oui Unknown), length 28
10:19:35.227703 IP 192.168.101.3.56293 > 192.168.101.4.ssh: Flags [S], seq 3343503303, win 14600, options [mss 1460,sackOK,TS val 79333555 ecr 0,nop,wscale 7], length 0
10:19:35.227735 IP 192.168.101.4.ssh > 192.168.101.3.56293: Flags [S.], seq 803956727, ack 3343503304, win 14480, options [mss 1460,sackOK,TS val 744506908 ecr 79326555,nop,wscale 7], length 0
10:19:35.627372 IP 192.168.101.4.ssh > 192.168.101.3.56293: Flags [S.], seq 803956727, ack 3343503304, win 14480, options [mss 1460,sackOK,TS val 744507308 ecr 79326555,nop,wscale 7], length 0
10:19:43.227627 IP 192.168.101.3.56293 > 192.168.101.4.ssh: Flags [S], seq 3343503303, win 14600, options [mss 1460,sackOK,TS val 79341555 ecr 0,nop,wscale 7], length 0
10:19:43.227661 IP 192.168.101.4.ssh > 192.168.101.3.56293: Flags [S.], seq 803956727, ack 3343503304, win 14480, options [mss 1460,sackOK,TS val 744514908 ecr 79326555,nop,wscale 7], length 0
10:19:43.627346 IP 192.168.101.4.ssh > 192.168.101.3.56293: Flags [S.], seq 803956727, ack 3343503304, win 14480, options [mss 1460,sackOK,TS val 744515308 ecr 79326555,nop,wscale 7], length 0
10:19:59.227802 IP 192.168.101.3.56293 > 192.168.101.4.ssh: Flags [S], seq 3343503303, win 14600, options [mss 1460,sackOK,TS val 79357555 ecr 0,nop,wscale 7], length 0
10:19:59.227838 IP 192.168.101.4.ssh > 192.168.101.3.56293: Flags [S.], seq 803956727, ack 3343503304, win 14480, options [mss 1460,sackOK,TS val 744530908 ecr 79326555,nop,wscale 7], length 0
10:19:59.627367 IP 192.168.101.4.ssh > 192.168.101.3.56293: Flags [S.], seq 803956727, ack 3343503304, win 14480, options [mss 1460,sackOK,TS val 744531308 ecr 79326555,nop,wscale 7], length 0

What's up with that ARP request in the middle of the connection attempt? It looks like the servers are confused about the routing. Also, note that iptables and SELinux have both been temporarily disabled while I troubleshoot this problem, so it's not a firewall issue. I don't see anything related to the SSH attempt in the logs either. Do you think the switch or gateway is configured incorrectly? That was my first impression but, if that were the case, why would tcpdump see the packets?

duffrecords · 07-18-2014, 08:59 PM

After capturing some packets with a Cisco tech, I've observed some strange behavior with these servers. For example, if 192.168.101.3 tries to ping 192.168.101.4, the ICMP echo request is sent through the switch directly to 192.168.101.4 because it is on the same subnet. However, the ICMP echo reply is sent through the switch's uplink port to the Cisco firewall, which then drops the packet because it's a reply to a request that it never saw in the first place. So for some reason, the replies from 192.168.101.4 are being sent to the default gateway instead of directly to 192.168.101.3.

If I try to do the opposite and ping 192.168.101.3 from 192.168.101.4 it works as expected. The routing table on each server looks correct:

Code:

[root@virthost1 ~]# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.101.0   *               255.255.255.0   U     0      0        0 eth0
192.168.102.0   *               255.255.255.0   U     0      0        0 eth1
192.168.122.0   *               255.255.255.0   U     0      0        0 virbr0
link-local      *               255.255.0.0     U     1002   0        0 eth0
link-local      *               255.255.0.0     U     1003   0        0 eth1
default         192.168.101.1   0.0.0.0         UG    0      0        0 eth0

Code:

[root@cloudstack ~]# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.101.0   *               255.255.255.0   U     0      0        0 p4p1
link-local      *               255.255.0.0     U     1002   0        0 p4p1
default         192.168.101.1   0.0.0.0         UG    0      0        0 p4p1

Why does 192.168.101.4 send traffic directly to 192.168.101.3 when it initiates it but send traffic to the default gateway when replying to a request?

duffrecords · 07-18-2014, 09:05 PM

Code:

[root@virthost1 ~]# ip route
192.168.101.0/24 dev eth0  proto kernel  scope link  src 192.168.101.4 
192.168.102.0/24 dev eth1  proto kernel  scope link  src 192.168.102.4 
192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1 
169.254.0.0/16 dev eth0  scope link  metric 1002 
169.254.0.0/16 dev eth1  scope link  metric 1003 
default via 192.168.101.1 dev eth0

Code:

[root@virthost1 ~]# ip rule
0:	from all lookup local 
32762:	from all to 192.168.102.4 lookup eth1table 
32763:	from 192.168.102.4 lookup eth1table 
32764:	from all to 192.168.101.4 lookup eth0table 
32765:	from 192.168.101.4 lookup eth0table 
32766:	from all lookup main 
32767:	from all lookup default

Code:

[root@virthost1 ~]# cat /etc/iproute2/rt_tables 
#
# reserved values
#
255	local
254	main
253	default
0	unspec
#
# local
#
#1	inr.ruhep
11	eth1table
10	eth0table

duffrecords · 09-23-2014, 12:11 PM

I realized later that for this particular application, I did not need multiple gateways. In fact, multiple gateways are probably not a good idea in the majority of cases.