iptables dropping "in=eth0 out=eth0" on same interface

I am not sure why I am seeing some dropped connections going in and out on the external network card for the same connection! But the source is a client computer on the Internal subnet and out to the Internet. So it should be going in eth1 and out eth0.

My setup is I have a firewall with two network cards. I will use random, simple IP Addresses and port numbers to illustrate.

Internal Eth1 - 192.168.1.1 (Internal Subnet)
External Eth0 - 192.168.2.1 (External Subnet

I have a postrouting rule to allow traffic to route from Internal Subnet to External Subnet:

iptables -t nat -A POSTROUTING -o eth0 -j SNAT --192.168.2.1

I have Drop Policies for Input, Output, Forward. I have established, related Accept Rules for Input, Output and Forward traffic.

I have dropped invalid packets so don't any of that in the logs.

# TCP Port 1199 Remote Access Software

iptables -A FORWARD -i eth1 -o eth0 -d 8.8.8.8 -p tcp --dport 1199 -m state --state NEW -j ACCEPT

I have rules in place for email, web traffic, NTP etc. No issues with any of that. However, I seem to be getting a lot of dropped connections for this specific preogram using port 1199 connecting to the server that runs the Remote Access Software.

Everything works as expected with no issues. All the forward rules forward traffic from client computers on the Internal Lan so the firewall sees incoming traffic on eth1 (which I have allowed) and outgoing traffic on eth0 - which I have allowed in the rule above. Same for all other traffic. They are all using - FORWARD -i eth1 -o eth0 and everything else works as expected with no dropped traffic.

I can connect to the clients using my Remote Access Software, I can see the clients on the dashboard. So they are clearly connecting to the Remote Access Software Service on the Server at 8.8.8.8 - Yes I know that is Google's IP Address. I am just using a random destination IP address.

The two odd things about this issue is that not all computers/Servers on the Lan, are experiencing dropped connections. Overnight - when there is no traffic and the Server are obviously online - there are no dropped connections. The second odd thing is that the dropped connections might only be for a minute or two (perhaps when client computers are switching on and have not fully loaded network), but in other cases, the dropped connection from a specific PC will go on for several hours!

My understanding of the rules that I have put in above is that all NEW traffic on port 1199 going to destination 8.8.8.8 using TCP will be forwarded through the firewall going into eth1 (internal Network card) and out eth0 (external network card). The postrouting rule will send all non local traffic to the external network card of the firewall (otherwise the firewall doesn't know where to send the forward traffic to for connecting to the Internet). So all NEW traffic forwarded from client machines should be -i eth1 -o eth0, yet while that is obviously working, I am also seeing a lot of dropped traffic "in=eth0 out=eth0".

Why would any traffic new traffic from internal lan be going IN to eth0. It should be going IN eth1 only.

Originally Posted by dalacor Why would any traffic new traffic from internal lan be going IN to eth0. It should be going IN eth1 only.

I could just be packets from the Internet with spoofed source addresses, although this is getting increasingly uncommon with most ISPs implementing reverse-path filtering. It's still possible, though, especially with a forwarded port that attracts attention.

It could also be a local host generating these packages. Could there be a host/PC with a 192.168.1.x address connected to the external network?

If these packets are really originating from a host in your internal LAN, then there must be a (layer 2) connection between eth0 and eth1, and there really shouldn't be. You can easily check this with arp -an | grep 'on eth0' - if you see any ARP entries for IP addresses in the 192.168.1.0/24 network, either your networks aren't wired right or a host with a static IP is connected to the wrong network.

Thank you for replying.

It is definitely legitimate and would not be coming from the Internet. - I have investigated that. Further investigation does reveal that some computers are not showing up on the Remote Support dashboard when they are dropping connections and then suddenly they connect later. I believe that the issue is DNS related to how the client program syncs with the server. I don't think it's a firewall mis-configuration!

I have asked the Remote Access Support Program developer to confirm my findings. I believe that the problem might be that the IP address of the server resolves to a couple of different names as I have the Remote Access Server Software installed on that Server, but I also have a couple of other services running on that Server. So I have setup different domain names such as service1.domain.co.uk and service2.domain.co.uk. What I have noticed on machines that seem to be dropping connections is that netstat says that they are connecting to service1.domain.co.uk when it should say service2.domain.co.uk - even though both addresses resolve to the same IP address and same actual server - just a different service on the server. When it does connect up on the dashboard, it then says service2.x. However, some computers show up on the dashboard and yet are still dropping connections.

So I think what is happening is that the program doesn't like the fact that several domain names resolve to the same IP address and when it's using the "wrong" domain name, it's creating some kind of loop which is dropping the connections. I am not sure why it sometimes uses the wrong name as the client config is configured for the service2.domain.co.uk address.

I might be completely wrong, but this looks like what the problem might be. I am not seeing any problems with any other service. I will update once I hear back from the program developer. But I have come to the conclusion it is not a firewall configuration issue.

The DNS name reported by netstat is a red herring. When two names resolve to the same IP, a simple reverse lookup will return one name. Which name this happens to be depends on the DNS setup, but it will not affect packet flow in any way.

In short, you cannot tell by looking at the output from netstat which hostname was used by the client to initiate the connection.

Did you run the arp command I suggested on the firewall?

Yes, I have now realised that netstat is a red herring as it shows the dns name the IP address resolves to. I assumed it defined what dns name it was using to connect to the service. However, I still believe a dns issue may be the cause as more and more services are requiring server Name to match to prevent MITM etc. This is what the software developer believes as well.

I have run arp -an | grep 'on eth0' and am getting quite a few entries - presumably any Lan machine that is online. I have also run arp -an | grep 'on eth1' and I get a few more entries than on eth0.

I presume the postrouting rule be the reason why it's showing on the eth0 interface as well as on the eth1 interface? Or it could be to do with the proxy server installed on this Linux Firewall which filters Internet Traffic. I am not sure.

I am going to wait until the Remote Support Software developer gets back to me as I don't think it's a firewall issue and neither do they. They are looking at the logs and will come back to me. Once I have that information, I can troubleshoot the firewall further if necessary. But for the moment, they and I are proceeding on the assumption that the firewall is not the problem.

I had a similar sort of issue with Activesync dropping source port 443 traffic on the Mail Server Firewall last year and without changing the firewall - that problem suddenly disappeared on the 13th December last year. So I assume that the issue was something to do with IOS which got fixed when the Ipads updated to the next IOS version. I am working on the same assumption with the Remote Support Software. It doesn't seem to be a firewall issue as everything else is working 100%. I am only experiencing issues with the Remote Support Software and only randomly.

I will update when I hear back from the developers regarding the Support Program. I will also investigate if I should be seeing connections from local lan on both interfaces - but my understanding is this is what the postrouting rule does. I will check if my mail server firewall does the same.

Just checked the mail server firewall. The Host machine and the Virtual Machine on the Internal Lan are showing on both Eth1 and Eth0 using that arp command. Again, I presume this is to do with postrouting rule. The Host Server and Mail server are definitely on the Internal Lan, with the Internal eth1 interface as the gateway.

Originally Posted by dalacor I have run arp -an | grep 'on eth0' and am getting quite a few entries - presumably any Lan machine that is online. I have also run arp -an | grep 'on eth1' and I get a few more entries than on eth0.

Filtering for eth0 should only ever show IP addresses in the external network (192.168.2.0/24 in your case), and conversely, filtering for eth1 should only show internal IP addresses (192.168.1.0/24).

If any entries appear on the wrong interface, there's bridging going on between the networks. If on the other hand you see no rogue ARP entries, then you can safely assume that the packets arriving at eth0 does not originate from an internal host, but are instead spoofed packets from the Internet.

Originally Posted by dalacor I presume the postrouting rule be the reason why it's showing on the eth0 interface as well as on the eth1 interface? Or it could be to do with the proxy server installed on this Linux Firewall which filters Internet Traffic. I am not sure.

It's neither. ARP is the glue between Layer 2 (MAC addresses) and Layer 3 (IP addresses). It is in no way affected by firewall rules or applications.

Originally Posted by dalacor I will also investigate if I should be seeing connections from local lan on both interfaces - but my understanding is this is what the postrouting rule does.

Not to be mean or anything, but you seem to have a very limited understanding of how ARP, NAT, routing, and IP in general, works. And not operating with a sufficiently accurate conceptual model makes it exceedingly difficult to perform anything resembling proper troubleshooting, as you're basically stabbing in the dark much of the time.

Even if the firewall were to receive packets from hosts on the internal network (192.168.1.0/24) destined for the external IP of the firewall (192.168.2.1), such packets would still never arrive via eth0 or even access eth0 in any way. That's why I'm dragging ARP into the conversation: No hosts on the internal network should ever be asking for the MAC address of a host in the external network, or vice versa.

I had to reboot the Mail Server Linux system this afternoon as I was updating something. I did another arp -an | grep 'on eth0' after reboot and now it only shows the Public IP Address and eth1 only shows the two internal servers, which is exactly what it should be showing. Not sure if maybe I mistyped the command last time or something is causing the ip addresses to show up on the wrong interface. Will monitor and see whether the mail server firewall system ends up showing internal IP addresses on eth0 over time.

Don't know what the issue with the Remote Support Software is, but will wait to hear back from developers on that as it's probably unrelated to the issue you have found.

But it would appear that (with your help) I have discovered a mis-configuration issue where internal IP addresses are showing up on the wrong interface, presumably as you say because of bridging. I will investigate that as there is clearly some issue with the client firewalls. As far as I am aware, I only have forwarding setup. I did not explicitly setup any bridging, but maybe the distro has something enabled by default that it should not.

No, unfortunately my understanding of arp, routing and Nat is not very deep. I don't do a lot of work with routing, firewalls etc. Most of my work is with Windows Servers and Clients. But I am happy to research the matter while investigating this issue to understand what is happening here and get it fixed. Might take me a bit longer than others, but I get there in the end.

I have a test distro that I can use for troubleshooting. I will update this post if and when I find out what is happening.