[SOLVED] Iptables Syn, ACK, PSH Handshake and process to troubleshoot dropped connections

dalacor · 10-08-2022, 10:32 AM

Beginning this year, I have started using Iptables for firewalls. Once I learnt the basics, I have had no issues with setting up connections such as ssh, ftp, websites etc and blocking all other traffic.

Where I seem to be consistently floundering (on several firewall systems) is where I am seeing a lot of dropped connections for established or related traffic for specific ports. Most ports - no issues. The problem seems to be services such as activesync or a remote support agent where there is a 24/7 constant connection.

For example on my Mail Server system, I see a lot of the following dropped connections:

Source Port 443 ACK, PSH, URGP=0 dropped. The Source IP is the mail server and it is obviously trying to send back something back to the client - which I have determined is an activesync client.

I have the following rules in place in the order shown. I drop all traffic by default. I allow all established and related traffic by default. I allow all incoming traffic from port 443 to be redirected to mail server which is behind firewall. I allow all outgoing port 443 from mail server. I also drop any invalid packets. Finally, I am only logging dropped traffic.

Code:

# Policy Rules

# Ipv4 traffic

iptables -P INPUT DROP
iptables -P OUTPUT DROP
iptables -P FORWARD DROP

# Established and Related Rules

iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT


# TCP Port 443 Internet Access from Internal Lan and Mail Server

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j DNAT --to IP Mail server
iptables -A FORWARD -p tcp --dport 443 -j ACCEPT

# Invalid Packets

iptables -A INPUT -p ALL -m state --state INVALID -j DROP
iptables -A OUTPUT -p ALL -m state --state INVALID -j DROP
iptables -A FORWARD -p ALL -m state --state INVALID -j DROP

# Logging

# Log IPv4 Dropped Traffic

iptables -N LOGGING
iptables -A INPUT -j LOGGING
iptables -A OUTPUT -j LOGGING
iptables -A FORWARD -j LOGGING
iptables -A LOGGING -m limit --limit 2/min -j LOG --log-prefix "IPTables-Dropped: " --log-level 4
iptables -A LOGGING -j DROP

Question 1:

Is the handshake process (for Activesync):
1. Ipad connects to the mail server with Syn
2. Mail Server responds to Ipad with ACK
3. Ipad finishes up with a Fin command
4. Mail server responds to Ipad with ACK, PSH and it's failing because Ipad has already done a Fin command? I am guessing the Server is trying to close the connection which the Ipad has already closed?

Question 2:

I have set the firewall to drop invalid packets. So if the connection has been closed (which I assume is the case), why am I still seeing dropped connections for ACK, PSH traffic?

Question 3:

Is ACK, PSH part of Established or part of Related traffic? I am wondering if maybe I need to enable something in Linux for Related Traffic because one of the articles I read said something about loading a module for that? I have no issues with websites which I presume is Established Traffic. I only seem to have problems with programs running on Servers or Activesync where there is a constant 24 hour connection as it were.

Question 4:

How do I find out what is causing this? It doesn't look like a timeout issue as Established timeout is 5 days according to my sysctl setting. I want to preferably fix the underlying cause of why these ACK, PSH connections are being dropped. If this is normal and expected behaviour, then just to get these dropped connections out of my log file. I don't know if what I am seeing is normal, or if there is a problem somewhere. With my mail server for example, I am only seeing the problem with source port 443 from the mail server to the activesync client. Everything else is working as expected.

I have used the example of my mail server, but other networks with a firewall in place, I am seeing the same problem, with a lot of established/related traffic being dropped. Again I assume that the underlying cause is the established connection has been closed by one side or another and the other side is still sending traffic on that established connection? Again it always seems to be services that are running like Datto. It doesn't seem to be like website traffic.

dalacor · 10-09-2022, 09:11 AM

I have made one change and will see if it fixes it. I have just realised that I have the Drop invalid Packets rule at the end of all the rules. So the Invalid Rule is probably not being applied. I assume that the postrouting rule which is at the top of the ruleset is being applied to the sport 443 ack, psh packets. I will update once I have seen whether the Drop Invalid Packets Rule at the top has any effect on the issue.

I have also realised that I need to look at how my prerouting, postrouting and forwarding of port 443 works to ensure that it is correct. I currently have the postrouting rule right at the top (below the established, related rules). I forgot to show the postrouting rule in my first post.

dalacor · 10-09-2022, 01:49 PM

I am still getting the ACK, PSH packets dropped by the Mail Server firewall despite moving drop invalid packets to the front of the ruleset. In addition, I did some work on the client firewalls and I stand corrected. A lot of ACK,PSH or ACK, FIN or RST packets are dropped as well. I suspect most of them are connections from apps on the Ipads as it doesn't seem to be actual websites visited that are the problem.

For the clients, I have just simply allowed source port 443 and source port 80 through the firewall because there is far too much stuff in the logs to make it viewable. I will do the same for the Mail Server firewall of just allowing source port 443 from the Mail server to the Internet if I can't find the underlying cause. I doubt that I will, because there seem to be lots of posts on various sites on the Internet going back decades, where people are reporting ACK, PSH and Fin packets being dropped and there never seems to be any solution other than vague pointers to look at timeout values.

It looks like either Established/Related (or more specifically some Established/Related) traffic just does not work properly through Iptables for reasons unknown. The only "solution" is to allow the source port traffic through even though it's established/related traffic not new sync traffic. Otherwise you just end up with hundreds of dropped ack, PSH connections in the logs making it harder to focus on the logs that you actually want to review.

dalacor · 10-10-2022, 11:03 AM

I have found that dropping invalid packets on the client firewalls does clear the logs of all the dropped 443/80 traffic. So it would seem that the client firewalls ACK, Fin and RST would seem to be expired connections, invalid packets or else is in some way different from the Mail Server. The client firewall dropped connections is largely ACK, FIN or RST whereas the Mail Server Firewall is always ACK, PSH connections.

I am guessing that the Mail server dropped connections may not be an expired connection or invalid connection, because otherwise the Drop Invalid Packets would clear the logs of these connections. I am wondering if the drop invalid packets only applies to Established Connections which I think the Client Firewall dropped packets probably are, and the Mail Server PSH is Related traffic and unaffected by Invalid Drop? I can't think of any other reason why drop invalid works for the Client Firewalls but not the Mail Server one, apart from a difference in how Established/Related packets work.

So hopefully I have fixed the issue for client firewalls, although I had to move the invalid rules to the bottom of the firewall set because it broke my remote assistance program, backup program and remote desktop between sites! So I need to check all the rules on the client firewalls to see why RDP etc is being dropped when Drop Invalid Packets is at the top of the ruleset. I am guessing the problem is the return packets, not the outgoing packets.

brebs · 10-10-2022, 11:26 AM

It could be caused by your blocking of ICMP - http://shouldiblockicmp.com/

Can troubleshoot by capturing the traffic, then analyzing using Wireshark.

dalacor · 10-10-2022, 12:25 PM

No ICMP is allowed. I have just not shown every rule that I created for the Mail Server firewall. ICMP is enabled outgoing. It's not enabled incoming for obvious reasons.

I am not very familiar with TCDump and Wireshark. I might need to learn how to use them if all else fails. However, I think I might have worked out what the problem is with the Mail Server, not quite sure how to fix it yet though. But I think that I am on the right track.

The client firewall is on the local networks of each client behind a Draytek router. So I setup these systems to have only one network card as they are not internet facing. The Mail Server Firewall on the other hand has two key differences. It has two network cards - an external Internet facing one and an internal LAN one. Even more importantly, it has prerouting rule in place to redirect dst port 443 traffic coming from the Internet to the Mail Server IP Address (Internal IP Address). Then a postrouting rule that sends all packets from the mail server back to the end client, whilst making it appear as if it is from the external IP address (not the Internal one). Assuming I have the rule set correctly. I also have a Forward rule for all syn traffic from the actual Mail Server to the Internet for things like Windows Update.

I can now see that the problem is not a timeout issue or connection expired issue because it would be dropped by the Drop invalid packets rule. I suspect that the real problem must be my prerouting, postrouting and forwarding rules are not setup correctly for Related Traffic which I suspect that ACK, PSH traffic would be. Reviewing The client firewalls logs again, I see that they do have some ACK, PSH traffic which is being dropped by the Drop Invalid Rule so clearly, the Drop Invalid Packets is working and does apply to Related as well as Established traffic.

I suspect that the connection is simply being blocked based on either eth0 or eth1 as a result of my prerouting, postrouting and forwarding rules.

Code:

# NAT Rules (Prerouting, Postrouting and Output Chains)

# Allows Internal Lan Traffic to be Routed to Internet

iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to external IP Address

# TCP Port 443 Internet Access from Internal Lan and Mdaemon Webmail

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j DNAT --to Mail Server IP Address
iptables -A FORWARD -p tcp --dport 443 -j ACCEPT

The exact error message I am seeing is this:

Quote:

Oct 10 17:40:12 Darkstar kernel: IPTables-Dropped: IN=eth1 OUT=eth0 SRC=(Mail Server Internal IP Address) DST=(End Client IP Address) LEN=71 TOS=0x00 PREC=0x00 TTL=127 ID=49917 DF PROTO=TCP SPT=443 DPT=58532 WINDOW=256 RES=0x00 ACK PSH URGP=0

I am guessing the connection is being dropped because of IN=eth1 which is the Internal network adapter on the firewall. I could try moving Postrouting to below all the prerouting and forward rules to see if that makes any difference? I am not really sure because obviously all the incoming traffic for say port 25 etc, is working correctly going back to the original requestor. So not sure why the prerouting, postrouting would work fine for 95% of the traffic coming in and going back out again.

dalacor · 10-10-2022, 03:27 PM

I have created a new topic - https://www.linuxquestions.org/quest...00#post6385600 and focused the question on prerouting, postrouting and forwarding rules as I believe this is where the problem is. This topic is based on the assumption that the problem was timeouts or issues with established related traffic. So I thought it would make sense to start a brand new topic and relate the information necessary to Nat and Postrouting etc.