I am not sure why I am seeing some dropped connections going in and out on the external network card for the same connection! But the source is a client computer on the Internal subnet and out to the Internet. So it should be going in eth1 and out eth0.

My setup is I have a firewall with two network cards. I will use random, simple IP Addresses and port numbers to illustrate.

Internal Eth1 - 192.168.1.1 (Internal Subnet)
External Eth0 - 192.168.2.1 (External Subnet

I have a postrouting rule to allow traffic to route from Internal Subnet to External Subnet:

iptables -t nat -A POSTROUTING -o eth0 -j SNAT --192.168.2.1

I have Drop Policies for Input, Output, Forward. I have established, related Accept Rules for Input, Output and Forward traffic.

I have dropped invalid packets so don't any of that in the logs.

# TCP Port 1199 Remote Access Software

iptables -A FORWARD -i eth1 -o eth0 -d 8.8.8.8 -p tcp --dport 1199 -m state --state NEW -j ACCEPT

I have rules in place for email, web traffic, NTP etc. No issues with any of that. However, I seem to be getting a lot of dropped connections for this specific preogram using port 1199 connecting to the server that runs the Remote Access Software.

Everything works as expected with no issues. All the forward rules forward traffic from client computers on the Internal Lan so the firewall sees incoming traffic on eth1 (which I have allowed) and outgoing traffic on eth0 - which I have allowed in the rule above. Same for all other traffic. They are all using - FORWARD -i eth1 -o eth0 and everything else works as expected with no dropped traffic.

I can connect to the clients using my Remote Access Software, I can see the clients on the dashboard. So they are clearly connecting to the Remote Access Software Service on the Server at 8.8.8.8 - Yes I know that is Google's IP Address. I am just using a random destination IP address.

The two odd things about this issue is that not all computers/Servers on the Lan, are experiencing dropped connections. Overnight - when there is no traffic and the Server are obviously online - there are no dropped connections. The second odd thing is that the dropped connections might only be for a minute or two (perhaps when client computers are switching on and have not fully loaded network), but in other cases, the dropped connection from a specific PC will go on for several hours!

My understanding of the rules that I have put in above is that all NEW traffic on port 1199 going to destination 8.8.8.8 using TCP will be forwarded through the firewall going into eth1 (internal Network card) and out eth0 (external network card). The postrouting rule will send all non local traffic to the external network card of the firewall (otherwise the firewall doesn't know where to send the forward traffic to for connecting to the Internet). So all NEW traffic forwarded from client machines should be -i eth1 -o eth0, yet while that is obviously working, I am also seeing a lot of dropped traffic "in=eth0 out=eth0".

Why would any traffic new traffic from internal lan be going IN to eth0. It should be going IN eth1 only.

I could just be packets from the Internet with spoofed source addresses, although this is getting increasingly uncommon with most ISPs implementing reverse-path filtering. It's still possible, though, especially with a forwarded port that attracts attention.

It could also be a local host generating these packages. Could there be a host/PC with a 192.168.1.x address connected to the external network?

If these packets are really originating from a host in your internal LAN, then there must be a (layer 2) connection between eth0 and eth1, and there really shouldn't be. You can easily check this with arp -an | grep 'on eth0' - if you see any ARP entries for IP addresses in the 192.168.1.0/24 network, either your networks aren't wired right or a host with a static IP is connected to the wrong network.

Thank you for replying.

It is definitely legitimate and would not be coming from the Internet. - I have investigated that. Further investigation does reveal that some computers are not showing up on the Remote Support dashboard when they are dropping connections and then suddenly they connect later. I believe that the issue is DNS related to how the client program syncs with the server. I don't think it's a firewall mis-configuration!

I have asked the Remote Access Support Program developer to confirm my findings. I believe that the problem might be that the IP address of the server resolves to a couple of different names as I have the Remote Access Server Software installed on that Server, but I also have a couple of other services running on that Server. So I have setup different domain names such as service1.domain.co.uk and service2.domain.co.uk. What I have noticed on machines that seem to be dropping connections is that netstat says that they are connecting to service1.domain.co.uk when it should say service2.domain.co.uk - even though both addresses resolve to the same IP address and same actual server - just a different service on the server. When it does connect up on the dashboard, it then says service2.x. However, some computers show up on the dashboard and yet are still dropping connections.

So I think what is happening is that the program doesn't like the fact that several domain names resolve to the same IP address and when it's using the "wrong" domain name, it's creating some kind of loop which is dropping the connections. I am not sure why it sometimes uses the wrong name as the client config is configured for the service2.domain.co.uk address.

I might be completely wrong, but this looks like what the problem might be. I am not seeing any problems with any other service. I will update once I hear back from the program developer. But I have come to the conclusion it is not a firewall configuration issue.

The DNS name reported by netstat is a red herring. When two names resolve to the same IP, a simple reverse lookup will return one name. Which name this happens to be depends on the DNS setup, but it will not affect packet flow in any way.

In short, you cannot tell by looking at the output from netstat which hostname was used by the client to initiate the connection.

Did you run the arp command I suggested on the firewall?

Yes, I have now realised that netstat is a red herring as it shows the dns name the IP address resolves to. I assumed it defined what dns name it was using to connect to the service. However, I still believe a dns issue may be the cause as more and more services are requiring server Name to match to prevent MITM etc. This is what the software developer believes as well.

I have run arp -an | grep 'on eth0' and am getting quite a few entries - presumably any Lan machine that is online. I have also run arp -an | grep 'on eth1' and I get a few more entries than on eth0.

I presume the postrouting rule be the reason why it's showing on the eth0 interface as well as on the eth1 interface? Or it could be to do with the proxy server installed on this Linux Firewall which filters Internet Traffic. I am not sure.

I am going to wait until the Remote Support Software developer gets back to me as I don't think it's a firewall issue and neither do they. They are looking at the logs and will come back to me. Once I have that information, I can troubleshoot the firewall further if necessary. But for the moment, they and I are proceeding on the assumption that the firewall is not the problem.

I had a similar sort of issue with Activesync dropping source port 443 traffic on the Mail Server Firewall last year and without changing the firewall - that problem suddenly disappeared on the 13th December last year. So I assume that the issue was something to do with IOS which got fixed when the Ipads updated to the next IOS version. I am working on the same assumption with the Remote Support Software. It doesn't seem to be a firewall issue as everything else is working 100%. I am only experiencing issues with the Remote Support Software and only randomly.

I will update when I hear back from the developers regarding the Support Program. I will also investigate if I should be seeing connections from local lan on both interfaces - but my understanding is this is what the postrouting rule does. I will check if my mail server firewall does the same.

Just checked the mail server firewall. The Host machine and the Virtual Machine on the Internal Lan are showing on both Eth1 and Eth0 using that arp command. Again, I presume this is to do with postrouting rule. The Host Server and Mail server are definitely on the Internal Lan, with the Internal eth1 interface as the gateway.

Filtering for eth0 should only ever show IP addresses in the external network (192.168.2.0/24 in your case), and conversely, filtering for eth1 should only show internal IP addresses (192.168.1.0/24).

If any entries appear on the wrong interface, there's bridging going on between the networks. If on the other hand you see no rogue ARP entries, then you can safely assume that the packets arriving at eth0 does not originate from an internal host, but are instead spoofed packets from the Internet.
It's neither. ARP is the glue between Layer 2 (MAC addresses) and Layer 3 (IP addresses). It is in no way affected by firewall rules or applications.
Not to be mean or anything, but you seem to have a very limited understanding of how ARP, NAT, routing, and IP in general, works. And not operating with a sufficiently accurate conceptual model makes it exceedingly difficult to perform anything resembling proper troubleshooting, as you're basically stabbing in the dark much of the time.

Even if the firewall were to receive packets from hosts on the internal network (192.168.1.0/24) destined for the external IP of the firewall (192.168.2.1), such packets would still never arrive via eth0 or even access eth0 in any way. That's why I'm dragging ARP into the conversation: No hosts on the internal network should ever be asking for the MAC address of a host in the external network, or vice versa.

I had to reboot the Mail Server Linux system this afternoon as I was updating something. I did another arp -an | grep 'on eth0' after reboot and now it only shows the Public IP Address and eth1 only shows the two internal servers, which is exactly what it should be showing. Not sure if maybe I mistyped the command last time or something is causing the ip addresses to show up on the wrong interface. Will monitor and see whether the mail server firewall system ends up showing internal IP addresses on eth0 over time.

Don't know what the issue with the Remote Support Software is, but will wait to hear back from developers on that as it's probably unrelated to the issue you have found.

But it would appear that (with your help) I have discovered a mis-configuration issue where internal IP addresses are showing up on the wrong interface, presumably as you say because of bridging. I will investigate that as there is clearly some issue with the client firewalls. As far as I am aware, I only have forwarding setup. I did not explicitly setup any bridging, but maybe the distro has something enabled by default that it should not.

No, unfortunately my understanding of arp, routing and Nat is not very deep. I don't do a lot of work with routing, firewalls etc. Most of my work is with Windows Servers and Clients. But I am happy to research the matter while investigating this issue to understand what is happening here and get it fixed. Might take me a bit longer than others, but I get there in the end.

I have a test distro that I can use for troubleshooting. I will update this post if and when I find out what is happening.

I think your over-simplifying your situation.

if you have a server with multiple network cards acting as a gateway for other machines perhaps also with multiple cards using SNAT to share static IP address for the connection to the internet ?

Then: some rules have to be on the client machines. the host machine cannot do it because it doesn't have the SNAT the way the clients do

(your situation isn't as simple as you are supposing it is)

No, they don't.Yes, it most certainly can.

@xlfs-0.2

You just a need a firewall rule like this.

iptables -t nat -A POSTROUTING -o eth0 -j SNAT --192.168.2.1

This forwards traffic sent to the gateway (internal interface of firewall) to the external interface. The only setting that you need to define on the clients is the gateway address which is the internal interface of the firewall. You don't need to setup forwarding or snat on the clients.

@ser Olmy. I have just checked the mail server today. It is still only showing internal IP addresses on the Internal Interface only. Which means two things. I obviously was checking the wrong interface last week on the mail server and more importantly, the problem is not with the Linux Server configuration, as the mail server firewall and client firewalls are setup pretty much identically. The only real difference between the mail server firewall and the client Linux systems is what forwarding rules are in place on the Linux Firewall.

I did however look at route rules, interface configuration comparing between the mail server system and the client systems. Everything seems to be in order there. When I have time, I will investigate this further, but for the time being I have no idea why internal IP addresses are showing up on external interface.

The only thing that comes to mind is the network switches and how the interfaces are connected. The firewall is a virtual machine and while it has two interfaces - there is only one physical cable connection between the firewall and the switch. Having said that, the mail server firewall is also virtual and also connected to a switch via one physical cable. The only difference is that the Hosted Datacentre probably vlan each port on their switches to isolate each client server in the datacentre.

I will come back to this when I have fixed the Remote Support Program issue and I when I have more time. I am not sure what the problem is, but it doesn't seem to be causing any issues. The only issue I am having is with the Remote Support Software which is probably completely unrelated to the firewall.

I believe that I have managed to fix this issue. What I have done is put eth0 and the Internet Router on a separate vlan, whilst keeping eth1 and the internal network on the native vlan. Running arp -an | grep 'on eth0' now only shows the Internet Router in the arp cache - which is exactly what I should be seeing!

I will have to wait until Monday when the teachers come back from their half term holiday, to see if I still get the issue with the Remote Support Software trying to go through eth0. I believe that the issue was never with the Remote Support Software - it just happened to be a program that connects quite frequently from a large number of computers so it appeared that the issue was something to do with the Remote Software. The developer of the Remote Support Software looked at the logs and could not see anything that could explain the problem.

Now that eth0 is on a completely isolated vlan from the Internal network, I will be very surprised if I see further connections being dropped on eth0 coming from the internal network. It is likely that most networks either have the two interfaces on two completely separate vlans or on completely separate isolated switches, which is why most people don't experience this problem. Although possibly because my firewall is quite strict in terms of restricting with interface is allowed what traffic as it were.

I will update next week, once the staff come back. But I am reasonably certain that I have fixed the issue and the solution was to put the interfaces on separate vlans or logically and physically separate switches with no connection between the two switches - other than through the firewall.

I can confirm that the issue is resolved. Putting the interfaces on separate vlans ensures that ARP caching works as expected for each subnet and ultimately all outbound traffic is now going in Eth1 and out Eth0 as expected. Thank you Ser Olmy for providing the solution by pointing me in the direction of ARP caching.