LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 06-10-2015, 04:42 PM   #1
systemlordanubis
Member
 
Registered: Jun 2010
Distribution: Debian, Ubuntu, Win
Posts: 143

Rep: Reputation: 16
Unexpected ARP communication between two different subnets.


Hi All,

I have a server (Server A - Debian) which has been having difficulty getting to our DNS servers (Server B - Debian); it works and then randomly stops. Upon investigating further, it appears that both servers are communicating directly, although they should not be; they exist in different subnets.

Topology:
Server A (in co-lo environment) > Firewall > Server B (DNS)

Now, Server A and Server B both have IP addresses in /27 blocks adjacent to one another; for example:
Server A: 10.0.0.40/27
Server B: 10.0.0.2/27

Now, as far as I am aware, when Server A wants to talk to Server B; it shouldn't send an 'ARP' on the network because server B doesn't exist in the same network segment; but this is exactly what it's doing.

Server A is sending an ARP directly to the wire for 10.0.0.2 and Server B is answering directly, instead of routing through the firewall.

Now, I can get around this, by separating the two networks onto different interfaces/switches, but I'm more interested to know 'why' and better yet, how stop the servers from doing it.

Doing some researching I came across "arp_ignore" and "arp_announce" but the combinations of these didn't seem to achieve what I am looking for. It's as if the servers are explicitly ignoring the /27 specification of their subnet.

Thanks
Anubis.
 
Old 06-10-2015, 09:44 PM   #2
netnix99
Member
 
Registered: Jun 2011
Distribution: redhat, CentOS, OpenBSD
Posts: 298

Rep: Reputation: 99
systemlordanubis,

You are correct in the fact that these two machines should not be using ARP (layer 2 protocol) because they are in different subnets (layer 3 boundary). However, I don't understand what you mean by:

Quote:
Now, I can get around this, by separating the two networks onto different interfaces/switches
Even if these two subnets are on a single, physical interface that has two sub-interfaces, there wouldn't be any layer 2 communication between these two hosts. That said, each server will send an ARP request for the other server, but there would be no reply because of the layer 3 boundary between the servers, forcing the server to send the IP packet to its default gateway, which would then route the IP packet to the second server.

Make sense??
 
Old 06-10-2015, 09:51 PM   #3
systemlordanubis
Member
 
Registered: Jun 2010
Distribution: Debian, Ubuntu, Win
Posts: 143

Original Poster
Rep: Reputation: 16
Quote:
Originally Posted by netnix99 View Post
Even if these two subnets are on a single, physical interface that has two sub-interfaces, there wouldn't be any layer 2 communication between these two hosts. That said, each server will send an ARP request for the other server, but there would be no reply because of the layer 3 boundary between the servers
That's exactly what I thought; however some trusty tcpdumping found that 'ServerB' was infact answering the ARP request that wasn't in its subnet range. This is what lead me through searching the issue to discover the "arp_ignore" and "arp_announce" flags, but setting those to what I thought would be applicable, this only broke communication between the hosts.

I had set:
sysctl -w net.ipv4.conf.all.arp_announce=1 (default is 0)
sysctl -w net.ipv4.conf.all.arp_ignore=2 (default is 0)

But to no avail.
 
Old 06-10-2015, 10:03 PM   #4
netnix99
Member
 
Registered: Jun 2011
Distribution: redhat, CentOS, OpenBSD
Posts: 298

Rep: Reputation: 99
Is your firewall providing proxy ARP? It would be easy to tell as the MAC address in the ARP reply packet would be that of the firewall and not the second server's MAC.
 
Old 06-10-2015, 10:15 PM   #5
systemlordanubis
Member
 
Registered: Jun 2010
Distribution: Debian, Ubuntu, Win
Posts: 143

Original Poster
Rep: Reputation: 16
No, no proxy ARP; the second server is responding directly to the first and what's more, it seems to be bi-directional.
Both servers are the same OS/version (Debian 7).

A ping from either server goes directly to that server and never passes through the default gateway.

Thanks.
Anubis.
 
Old 06-10-2015, 10:24 PM   #6
netnix99
Member
 
Registered: Jun 2011
Distribution: redhat, CentOS, OpenBSD
Posts: 298

Rep: Reputation: 99
That's crazy! Even if you were using a layer 3 switch and had inter-vlan routing turned on, that would explain the firewall bypass, but NOT the bi-directional ARP replies between subnets!

The only other logical conclusion that I can come up with is if you made a typo on the first subnet's mask, making it include the second subnet. Crazy to ask, but you are using a mask of 255.255.255.224, right?? That's all I have left! Someone smarter than me will have to help!! : )

Last edited by netnix99; 06-10-2015 at 10:28 PM.
 
Old 06-10-2015, 11:44 PM   #7
systemlordanubis
Member
 
Registered: Jun 2010
Distribution: Debian, Ubuntu, Win
Posts: 143

Original Poster
Rep: Reputation: 16
Hi Netnix99,

Ok, glad it's just not me!

Yep, check and re-checked and again re-checked the subnets and they're correct for /27s.

Below is the info from the servers as well as the PCAP lines showing the ARP from 'Server B' to 'Server A'.

Thanks
Anubis.



SERVER A:

root@XXXXXX:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether XX:XX:XX:9e:26:52 brd ff:ff:ff:ff:ff:ff
inet XXX.XXX.210.40/27 brd XXX.XXX.210.63 scope global eth0
inet6 fe80::4018:32ff:fe9e:2652/64 scope link
valid_lft forever preferred_lft forever


SERVER B:

root@XXXXXX:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether XX:XX:XX:39:30:ba brd ff:ff:ff:ff:ff:ff
inet XXX.XXX.210.2/27 brd XXX.XXX.210.31 scope global eth0
inet6 fe80::34b2:41ff:fe39:30ba/64 scope link
valid_lft forever preferred_lft forever


PCAP:

No. Time Source Destination Protocol Info
100 283.624027 XX:XX:XX:39:30:ba Broadcast ARP Who has XXX.XXX.210.40? Tell XXX.XXX.210.2
101 283.624226 XX:XX:XX:9e:26:52 XX:XX:XX:39:30:ba ARP XXX.XXX.210.40 is at XX:XX:XX:9e:26:52
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Strange ARP behavior : A linux server responds to all ARP requests Hdvd21 Linux - Networking 4 10-24-2013 05:02 AM
fail2ban.server : ERROR Unexpected communication error roy-arne Linux - Server 4 04-12-2009 04:43 PM
Fault in KDE processes communication: Could not read network communication list Magnus Johansson MEPIS 0 03-30-2008 12:50 PM
Disabling ARP probes after receiving an ARP request AltecLansingMan Linux - Networking 1 03-30-2004 01:25 PM
How to create an proxyarp entry in arp table by using arp command? himalayas Linux - Networking 0 06-04-2003 04:14 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 02:45 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration