[SOLVED] Why is UFW/iptables blocking packets it shouldn't be?
Linux - NetworkingThis forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Introduction to Linux - A Hands on Guide
This guide was created as an overview of the Linux Operating System, geared toward new users as an exploration tour and getting started guide, with exercises at the end of each chapter.
For more advanced trainees it can be a desktop reference, and a collection of the base knowledge needed to proceed with system and network administration. This book contains many real life examples derived from the author's experience as a Linux system and network administrator, trainer and consultant. They hope these examples will help you to get a better understanding of the Linux system and that you feel encouraged to try out things on your own.
Click Here to receive this Complete Guide absolutely free.
Why is UFW/iptables blocking packets it shouldn't be?
Definitely a head-scratcher, here.
I'd been seeing some [UFW BLOCK] messages in the syslog among some torrent traffic like these (with 192.168.1.100 representing my local IP address, and 18321 representing my torrent client's listening port, which *is* open in UFW/iptables):
And there's also blocks happening on high random ports that should be allowed through as normal passive connection traffic (or whatever the proper terminology is... sorry), though those ports are "blocked" against "incoming connections" by UFW:
The TL;DR is that some packets were overly-large and getting rejected by netfilter as INVALID. On a mailing list thread linked in a comment there about a separate but similar issue, the cause of the INVALIDs were different and various ( http://www.spinics.net/lists/netfilter/msg51408.html ).
From that, I at least managed to figure out that I should turn on logging for invalid packets, by doing:
**If there's someone out there who can explain the actual commands to do some of the other types of debugging they talk about there, I'd greatly appreciate it! In particular, "When enabling logging in the conntrack module for what causes the invalid packet" - I would love to know this.**
So anyway now, lo and behold, I see a bunch of packets being marked invalid in my syslog:
Pretty much all the inbound invalids seem to be RSTs, just like our high-port UFW blocks above, but waitasecond - that last one there is from me, outbound! But I have absolutely no UFW rules regarding outbound packets... Let's check our full set of iptables rules:
# iptables -S
-P INPUT DROP
-P FORWARD DROP
-P OUTPUT ACCEPT
-A INPUT -j ufw-before-logging-input
-A INPUT -j ufw-before-input
-A INPUT -j ufw-after-input
-A INPUT -j ufw-after-logging-input
-A INPUT -j ufw-reject-input
-A INPUT -j ufw-track-input
-A FORWARD -j ufw-before-logging-forward
-A FORWARD -j ufw-before-forward
-A FORWARD -j ufw-after-forward
-A FORWARD -j ufw-after-logging-forward
-A FORWARD -j ufw-reject-forward
-A OUTPUT -j ufw-before-logging-output
-A OUTPUT -j ufw-before-output
-A OUTPUT -j ufw-after-output
-A OUTPUT -j ufw-after-logging-output
-A OUTPUT -j ufw-reject-output
-A OUTPUT -j ufw-track-output
-A ufw-after-input -p udp -m udp --dport 137 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 138 -j ufw-skip-to-policy-input
-A ufw-after-input -p tcp -m tcp --dport 139 -j ufw-skip-to-policy-input
-A ufw-after-input -p tcp -m tcp --dport 445 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 67 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 68 -j ufw-skip-to-policy-input
-A ufw-after-input -m addrtype --dst-type BROADCAST -j ufw-skip-to-policy-input
-A ufw-after-logging-forward -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw-after-logging-input -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw-before-forward -j ufw-user-forward
-A ufw-before-input -i lo -j ACCEPT
-A ufw-before-input -m state --state RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-input -m state --state INVALID -j ufw-logging-deny
-A ufw-before-input -m state --state INVALID -j DROP
-A ufw-before-input -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 4 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 12 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A ufw-before-input -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A ufw-before-input -j ufw-not-local
-A ufw-before-input -d 18.104.22.168/32 -p udp -m udp --dport 5353 -j ACCEPT
-A ufw-before-input -d 22.214.171.124/32 -p udp -m udp --dport 1900 -j ACCEPT
-A ufw-before-input -j ufw-user-input
-A ufw-before-output -o lo -j ACCEPT
-A ufw-before-output -m state --state RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-output -j ufw-user-output
-A ufw-logging-allow -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW ALLOW] "
-A ufw-logging-deny -m state --state INVALID -m limit --limit 3/min --limit-burst 10 -j RETURN
-A ufw-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw-not-local -m addrtype --dst-type LOCAL -j RETURN
-A ufw-not-local -m addrtype --dst-type MULTICAST -j RETURN
-A ufw-not-local -m addrtype --dst-type BROADCAST -j RETURN
-A ufw-not-local -m limit --limit 3/min --limit-burst 10 -j ufw-logging-deny
-A ufw-not-local -j DROP
-A ufw-skip-to-policy-forward -j DROP
-A ufw-skip-to-policy-input -j DROP
-A ufw-skip-to-policy-output -j ACCEPT
-A ufw-track-output -p tcp -m state --state NEW -j ACCEPT
-A ufw-track-output -p udp -m state --state NEW -j ACCEPT
-A ufw-user-input -p udp -m multiport --dports 18320:18323 -j ACCEPT
-A ufw-user-input -p tcp -m multiport --dports 18320:18323 -j ACCEPT
-A ufw-user-limit -m limit --limit 3/min -j LOG --log-prefix "[UFW LIMIT BLOCK] "
-A ufw-user-limit -j REJECT --reject-with icmp-port-unreachable
-A ufw-user-limit-accept -j ACCEPT
Ok, so it will reject INVALID packets from me, too, that's fine. Except it's finding one about every 10 seconds, even when there's very little traffic overall... hmmmm... Is my wireless card borked and mangling packets, or is something else going on? I suspect that netfilter might be using an overly-strict definition of "INVALID."
So what's going on, here, and what can I do to get more information? Thanks so much to any and all who read this massive post!
So, after a lot of reading about "invalid" packets, TCP and conntrack, I'm getting a clearer picture, but I'm still not sure how to fix this. But I've ruled some things out.
At first, I thought it might be a timeout problem - that these packets were "invalid" because they were late arrivals from peers that I was once connected to, but conntrack had decided I was done with them, even though they had not. I also read this: http://www.smythies.com/~doug/network/iptables_notes/ which talks about the ip_conntrack_tcp_timeout_close_wait setting (in /proc/sys/net/ipv4/netfilter/) potentially causing some problems.
So, I raised every one of the TCP timeouts in that /proc directory by a factor of 4 or more (after backing the originals up), and after almost two hours, I'm still getting just as many INVALIDs, perhaps even more.
Then I tried disabling conntrack's checksumming, in case these packets were getting rejected for bad checksums, by doing
What's a bit frustrating is that these options are all very poorly documented, and I know that raising some of these timeouts might have some undesireable effects, while it's possible that disabling checksumming might not actually prevent checksum failures from being marked as invalid.
Most of the invalid packets are still Resets (RST), but about one in thirty of them is an ACK, SYN, or ACK SYN. That ratio is true for both incoming and outgoing invalid packets. There are also about 30 times as many incoming invalid packets as outgoing.
I'm really mystified as to how my own outgoing packets are getting labeled invalid before they even leave the kernel... Please folks, what the heck is going on here?
I'm beginning to think that this might even be an injected RST attack, like the ISP Comcast was railed against for carrying out years ago in 2007, and eventually ordered by a U.S. judge to halt in 2008 ( https://en.wikipedia.org/wiki/TCP_re...st_Controversy ). I am in fact a ComCast customer, and this might make sense... except I feel like a paranoid talking about a practice that was supposedly halted 6 years ago...
There is in fact many non-torrent anomalies going on, as well, including some invalid packets coming in from google.com on ports 80 and 443, so I have to imagine that there's something going on in my configuration that's causing it, but I can't rule anything out at this point...
For systems using IP connection tracking (this includes many routers as well as most operating systems, including Linux), a long-running and hence high-volume DHT server can produce an unnecessary strain because each incoming query creates a new connection, even though typically no other packets will follow after the first one.
When used in addition to firewall rules which drop invalid packets, this may cause genuine non-DHT connections to be dropped as well if the kernel runs out of memory for tracking connections.
There are several strategies to mitigate this:
Turn off the DHT server when it is not needed (default). This quickly causes incoming queries to stop when there is no response.
Reduce the DHT query volume. Since one cannot directly control incoming queries, this can be done by throttling the outgoing DHT bandwidth to discourage too many incoming queries.
Turn off connection tracking for DHT packets.
To turn off DHT connection tracking on Linux, use the NOTRACK target in the "raw" netfilter table. For instance, when using UDP port 6881 for DHT, execute these iptables commands:
Note that this causes all DHT packets to have a state of "UNTRACKED", keep that in mind if you have any state-based rules in the regular netfilter tables. Similarly, ICMP packets received in reply to untracked outgoing DHT packets (for instance "Host Unreachable" or "Time Exceeded") will have a state of "INVALID" because they do not belong to any known connection.
So, I added those rules as relevant to my connection:
And now my torrent client is indeed performing much better, and far fewer torrent packets are being dropped, only about one per minute (exluding the many ICMP packets that are now being dropped, unfortunately, which I know no decent way to make a specific allowance for). And every single one that is being dropped is now a RST packet, again excluding ICMP. I still don't quite know what to draw from that, but it's info.
I would still like to have an arrangement where these RSTs are not being dropped (or perhaps not being sent in the first place, if they're not appropriate?), and where the ICMP traffic is able to do its job, but for now, this is a step up.
I think I may have finally got it, although I had to open up the firewall a little more than I'd prefer. Fortunately for this case, doing that doesn't really open up any risks, but if I had more services running, it might be a problem.
So, I should mention that the torrent client I'm using is Deluge, which uses libtorrent-rasterbar as its back end. I finally got all my torrent traffic unblocked (DHT, peer connections, mystery "invalid" RST packets, peer ICMP packets, etc.), it seems. I essentially did this by disabling connection tracking with conntrack for all of the relevant torrent traffic, including on the ephemeral ports (or as deluge calls them, "Random ports").
I did this by disabling the "Use Random Ports" checkbox in deluge for all connections, incoming and outgoing, and specifying acceptable port ranges; for instance, 11000-11020 for incoming connections and 11021-11400 for outgoing connections. After setting the ports Deluge will use in that way, I closed Deluge and set my iptables rules - I'll include all of them here as entered commands, which will then need to be saved to preserve them on subsequent boots.
It may be sloppy to just insert each rule as rule #1 over and over, but I don't know iptables well enough to do it another way. When I tried using the -A flag to append them initially, I found stuff was still getting blocked as the desired packets triggered some rule before the appended ones.
The ICMP allowances are desirable because once connection tracking is disabled on those ports, those important ICMP messages (types 3 (Destination Unreachable), 4 (Source Quench), 11 (Time Exceeded) and 12 (Parameter Problem)) would otherwise be declared INVALID themselves (as netfilter would not know of any connections that they are associated with), and they would potentially be blocked.
I don't really understand why netfilter/iptables was having issues with the RST packets on ephemeral ports, but it was, and I can only assume that there's something about the way that a torrent swarm (and especially a DHT torrent swarm) sends traffic around that's atypical, and conntrack doesn't know how to handle it. My guess is that some of the traffic is just too infrequent, and conntrack "forgets" about those peers, but I can't be sure - after all, when I raised the sysctl timeouts, I was still getting those blocks. But of course I wasn't really sure what I was doing.
So, with rules like the ones above in place, I think I can finally use torrents effectively with restrictive iptables in effect. Indeed, my torrents are seeding much more often, now - before they were hardly seeding at all, which was very obnoxious. I hope that this thread can help others with the same problem (and I imagine that almost anybody using bittorrent and iptables/netfilter is affected, which is a lot of people!). The only reason I discovered it is because of UFW's default configuration choosing to log some blocks in the syslog - hooray for logs!
Make sure to get the word out about how iptables/netfilter likes to call lots of torrent traffic "invalid," and let's keep those swarms healthy!
Last edited by d2d2; 04-13-2014 at 02:08 PM.
Reason: Updated ICMP rules and reasoning
A quick update. I've continued to try to learn about netfilter_conntrack and its behavior toward "invalid" packets, and I've learned some more important things. Because there seems to be EXCEEDINGLY little documentation about any of this stuff, I'd like to at least put it online here.
Firstly: when invalid packet logging is enabled, as by issuing the command mentioned above:
This is **NOT** a packet being dropped! I had to look at the freakin' linux kernel source code to find this out, but "ignored" in this case means that the packet is accepted! At least, I'm pretty sure, as a totally novice programmer. If anyone would like to verify, the relevant source (for a 3.2 kernel) is in the file linux-3.2.xx/net/netfilter/nf_conntrack_proto_tcp.c within the standard linux kernel source tree. It's available online at http://lxr.free-electrons.com/source...ck_proto_tcp.c
You can search/grep for "invalid packet ignored," the section it's in concludes with "return NF_ACCEPT;" as opposed to some others that are obvious blocks that end with "return -NF_ACCEPT;" (note the - sign). So this seems to strongly suggest that these packets are in fact getting through. I really wish this log message (and even the comments in the source) were a little less ambiguous about that.
In other words, if you see a log message like the line above, DON'T PANIC! Your traffic isn't being blocked, it's just that conntrack is having a senile moment, is aware of that, and lets the traffic through.
In all this struggling with netfilter_conntrack, I am really wishing that this module had much better documentation. I may try to submit some, but it's daunting, and a lot of it is still stuff I don't really understand. If anyone does, please contribute some documentation (to the manpages and online)!