I have a small server (running 24/7) with a bunch of services running on it:
- apache2 serving a few static web pages (reachable from outside)
- proftpd (currently disabled)
- yacy (currently disabled)
- ktorrent or transmission up- and downloading some torrents
- vncserver for the graphic desktop (used to maintain the server), secured with a strong password.

The server is plugged into an ADSL-Modem/Router, this setup on Ubuntu 8.0/8.10/9.04/9.10/10.04 worked greatly for years.

However, months ago it seemed ktorrent goes haywire sending massive amounts of some torrent ACK-Packets blocking my whole network and the DSL-Modem/Router.
I suspected various things like bad torrents, messed up configurations el al...
I even erased the whole Ubuntu Linux a few months ago to replace it with Arch Linux, used transmission instead of ktorrent (KDE), Arch installed from scratch. And guess what? The problem persisted!! *sigh*
I messed around with that for a few more days, trying to pinpoint the problem, then giving up.

Anyway, a few things I can tell for sure:
- The problem occurs when ktorrent or Transmission (or any other torrent client, as I suspect) is running with active torrents
- It's not all the time, only about 5 - 10% of the time.
- The sending of these ACK-Packets takes place in intervals of 1-3 minutes, them pauses for 5 minutes up to several hours.
- During that intervals, it seems always to consume a constant amount of bandwidth (according to gkrellm monitor), between 0.5 and 5 Megabytes per second (yes, you read right!)
- when I close the torrent client, the problem may persist for a few minutes, or may not
- ksoftirqd has an elevated consumption of CPU time, around 3-15% (if the problem occours)
- the address of the ACK-packets is (according wireshark) always an outside address, it may be some other torrent client which is/was connected to mine. The address seems to be the same for that few minutes while the system is sending the packets, after a while it goes back to normal. Next time it goes haywire, the destination IP of the torrent ACK-Packets completely differs from the one before.

I appreciate any Ideas how to track down this beast.

Thanks in advance, Ice

P.S. Ther server is a single-core CPU, 64-bit AMD with 2 GB of RAM, running the 64-bit-version of Arch Linux (software is always kept up-to-date, updating the system every 4 weeks)

Greetingz!

Two possible scenarios come to mind;

Most Likely: A recent update may have enabled or installed either a service (or a cron job, or an xinetd entry) that's checking for something. Things like samba broadcasting out every few moments, looking for any SMB/CIFS responses.

Possible Cause: ACK packets, by their nature, are initiated when someone shoots you a SYN packet. So something is trying to establish a connection to your system. Do the IP address have any consistency? My firewall stops traiffc from the netblocks of many, many, many outside countries (and certain corporations, too), which really cuts down on the traffic I see bouncing against things like my webserver.

Unlikely Cause: You're ISP is trying to bottleneck your connection because they've detected BitTorrent traffic from you, so they're SYN-bombing you with spoofed IPs to slow you down. You wouldn't happen to have ComCast, would you?

Hope this helps!

P.S: If it does, then do me a solid and click the "Thanks" button way on the right of my signature.
P.S.S: "Thanked" your post for being so well worded.

Maybe you are just scanned. The ACK packets are normal - more details needed: source and destination ports, if other flags are also set or alone, how many torrents you have active simultaneously and how do you known about these packets?

@xeleema:
Thanks again for the suggestions.
I can rule out #3 and #1 samba (I deinstalled server and client, I anyway use it very rarely) on the machine causing the ACK-packets. Despite no samba running, the problem remains.
That leaves me with #2, the "possible cause". I think a cheap solution weould be eliminating the yamptom...
Question: So how do I make a rule using iptables that either blocks "undesired" SYN packets?
Or as an alternative, better - how do I limit (or eliminate) multiple ACK-packets to the same IP within a certain time period?

Sorry for the question about iptables, I messed around years ago with this and are not *THIS* familiar with its possibilities.


Thanks in advance,
Ice

Well, I've seen two kinds of iptables configurations;
The first is a really tiny rule set. Typically easy to fix, especially if comments are included and you save a backup copy of the rules before and after each modification.

However, then there's the Rube Goldberg's machine of iptables rules;
one false move, and you break a half dozen rules (without really knowing it).

If you have a very simple ruleset, here's what I would suggest;

iptables -A INPUT -p tcp --syn -m limit --limit 1/s --limit-burst 5 -j ACCEPT
iptables -A INPUT -p tcp --syn -j DROP

If you have a Rube Goldberg machine setup, try this one-liner;

iptables -A INPUT -p tcp --syn -m iplimit --iplimit-above 5 -j REJECT

Note that the former will limit SYN packets to 5 per second, then drop any "extras". If you're running a semi-popular website (~100 unique IP hits per month), then you may need to increase that number.

If you're not familiar with iptables, I would *strongly* suggest you ensure you have physical access to the server and read the man page prior to implementing.

As a general rule of thumb, I don't implement anything I do not 100% understand (also helps if you have a separate firewall running in front of your network, too).

P.S: This will not work if someone's trying to SYN flood you from a number of spoofed IP addresses (or otherwise multiple IPs).

1 members found this post helpful.

@xeleema:
Hey, many thanks!
I do NOT habe a Rube Goldberg setup, especially in things like iptables, which I am not 100% familiar with. But I always act like my computers ARE Rube Goldberg things

Now, just to make sure if I understood correctly...

This
iptables -A INPUT -p tcp --syn -m iplimit --iplimit-above 5 -j REJECT
would prevent an outside IP from flooding me with SYN packets, right? Maybe I will have to increase the number from 5 to 25 or 100, because I also run a small webserver (apache2) on this server.

But - I would now prefer to prevent any program - especially P2P software like torrent - to SEND a lot of ACK packets to any outside IP address. (I know, everything I use is open source, so why not fixing the root cause? But no, I just don't WANT to fiddle around with something as complex as KTorrent or Transmission, especially trying to fix a problem which may be anything from a race condition to a simple glitch and even cannot be reproduced at will)
So how do I prevent my machine from sending a lot of OUTGOING ACK packets? Will the following line do the job?
iptables -A OUTPUT -p tcp --ack -m iplimit --iplimit-above 50 -j DROP

If yes, then it looks like progress towards a solution...
Thanks a lot again for your help with this beast of a problem!


Ice

P.S I have a script in /etc/init.d/... which does all the local magic like these iptables stuff, and I can clear out all rules using iptables -F ... and reset if things tend to get messy.

OK, next problem...

I tried to issue the following iptables command to limit outgoing ACK packets (torrent ACK (TCP) which otherwise occasionally flood my intranet - see original problem):
This results in an error saying that I do not have the module "dstlimit".
I use Arch Linux, there is als nothing about this in the repositories.
-> How do I get that module? Do I have to download it separately? Compile it? Enable it in the kernel (the arch wikis say, all their precompiled kernels are iptables-ready, whatever that means...


Thanks in advance,
Ice

You are correct. However, remember I mentioned the following for "Simple" rulesets;

iptables -A INPUT -p tcp --syn -m limit --limit 1/s --limit-burst 5 -j ACCEPT
iptables -A INPUT -p tcp --syn -j DROP

This would rate-limit them to 5 every 1 second. Keep in mind that the "DROP" rule should go near the bottom of your overal configuration.

Correct again! Since you're trying to stop what appears to be a flood of some kind, try setting it to a higher value (like 100, or 50) at first, watching your network traffic, then adjusting the value lower and lower until you reach a happy balance.

Keep in mind that Bittorrent ("BT") requires a lot of connections to different hosts. That's the beauty of BT (and sometimes a curse). You may want to check out a few guides on tweaking your BT client.

If you're unfamiliar with the inner workings of BT, there's a very well written guide that explains things quite well. Admitiedly it reads as a "BitTorrent 101" type of guide. But it's quite good.

Ah, yes. However, like most Open Source applications, BT clients are usually highly configurable. You'll have to check the specifics for the exact client you're using, but I'm postitive you wouldn't have to resort to something as rash as modifying the source code.

Granted, some Linux distributions do not set the most desireable compilation flags when they build packages. The most I usually have to do is obtain the source code and recompile things. It's rare (nowdays) that I actually have to dust off my programming bookcase and start tweaking code.

Well, again keep in mind that your local Bittorrent client is supposed to send out ACK packets, as that's what answers all the SYN packets (and starts a connection). So the big question would be "How do you stop a flood of SYN packets from hitting everything in your network except your BitTorrent client?" The answer to that may be a combination of tweaking your BitTorrent client, and some sort of FireWall at the actual "edge" of your network.

Good, that should be there. Take a look at that script and see how it saves and loads the rulesets. Your Linux distribution may have a method of adding/removing rules that they'd like you to use, so you may want to double-check their documentation.

However, one word to the wise; as long as you have physical access to the system in question, you should be able to undo anything that goes wrong; provided that you have backed-up a copy of the iptables rulesets before you start making changes.

There are many, many tricks you can do with iptables. But keep in mind that in most cases (like this one), you'll have to have a good understanding of how the protocols you're trying to control or block function.

There's a tale about an Email Administrator that tried to block all of the Spam coming in to his corporate network with a shiny new iptables configuration. The short version of the story basically says he was super-effective...nothing ever hit the email servers again.

(Before you ask, that's not the voice of experience talking. However, I was there when the email stopped.)

Happy computing!

1 members found this post helpful.

Okay, "dstlimit" is an iptables module that works directly with the networking stack of the Linux Kernel. However, "dstlimit" was for the 2.4.x versions of the kernel.

For any 2.6.x-based Linux distributions, you'll need to read up on "hashlimit". It behaves a bit differently, but it's still possible to do what you're trying to accomplish.

Please read my previous post (I saw this post of yours after I replied with that really big post), as I'm not sure you want to start chopping away at your Bittorrent client via iptables first. A bit of time on the configuration for the BT client will probably save you quite a bit of headaches.

@xeleema
Thanks a lot for your feedback - I see, I have to dig deeper into this thing...

But first, one question: Is it possible to cause multiple ACK if a host sends only one SYN? If yes, how?

Another piece of information: I found out why these outgoing floodings can exceed the the upload speed limit which my provider places on my ADSL. Apparently ACK-packets are part of the IP-protocol, so they do not count as "payload", so an outgoing flood of ACK-packets can exceed any bandwith limit placed by the ISP.

Some more information - I managed to get a live capture while the flooding took place:


I am not *this* of a Expert in networking, but it seems to me like
FIN,ACK-packets outgoing to an IP (which seems to run an BT over port 80) are the problem. More than 5000 of these packets (as far as I can see identical to each other, so redundant) are sent to the same IP.
Oddly, I do not see a lot of SYN-Packets coming in from the outside (in fact i've seen NONE at all, but maybe I just missed it in the log), so it seems to be a glitch in Transmission and the same in KTorrent - somewhow unlikely I think.

I now save the wireshark log and are going to investigate further in the evening...

Not that I'm aware of. SYN/ACK connection handshakes are a one-for-one deal as far back as I can remember.

Wait, what? It's rather late for me, so maybe I read that wrong;
Your ISP is counting all ACK packets as a part of your bandwidth limit?
Or is it just the incoming? (or just the outgoing?)

The reason I ask is that traffic is traffic as far as most bandwidth monitoring software goes...however, ACK packets are usually quite small.

You might want to make *sure* you're not getting SYN-flooded from outside somewhere. Check RFC4987:"TCP SYN Flooding Attacks and Common Mitigations" for way more information than you'd ever want to know on the subject.
(NOTE: RFCs, or "Request for Comments" are your friend.)


The highlighted-in-blue packet (FIN,ACK) in your screenshot shows you closing a connection to 95.143.193.5. That IP doesn't resolve for me, so it *might* be a BitTorrent peer.

I would advise (if possible) that you also log the peers your BitTorrent client connects to. Compare the time/date stamps of your wireshark output with the traffic you're getting from BitTorrent.

If need be, you might have to set your 'total upload connections' to something like "2" (not per torrent upload connections).

See if that drops things down a peg or two.

@xeleema:
No, my ISP doesn't. Maybe my wording was misleading.
I just *wondered* the first few times when my monitoring on the server showed me outgoing traffic *WAY* beyond my ISP's bandwith limit. "That can't be" I first thought, but it can, as my ISP does NOT count all ACK packets as part of the bandwidth limit.
As ACK-packets are small, there has to be a huge amount to be sent out to make something like 3Mbit/s or even 8Mbit/s. The upload limit of my connection is 1Mbit/s.

The difficult part is now - it doesn't occur constantly, it doesn't even happen frequently - that's why I think of a soloution with iptables. Currently all iptables rule sets are empty.

I am already using a blocklist with Transmission. The problem with wireshark is, that after a few minutes of logging it consumes 100% CPU time and virtually locks my server up to the point of where it won't respond for minutes. The best way of reducing the flooding will probably be just reducing the number of active torrents.

The next few days when I find some time, I'll investigate further into this and try some possible solutions.

Thanks for all your help so far, I will keep posting any progress!
Ice

So, I almost fixed it. In fact, I didn't fix it at all, only the symptom (massive number of outgoing ACK-Packets) I managed to marginalize. Here's how I did it:

Adding these two rules did the trick, and a quick glance at the stats after having some torrents active for ca 36 hours revealed the size of the impact. A
returned

OMG - roughly 20% of my outgoing torrent traffic would be some redundant ACK-packets! Something smells fishy here - but now with the pressure gone, I can concentrate on investigating the cause of it in the near future. I'll post it here when I find something.

Anyway - I currently have no idea what the cause may be. Either it's due an flaw in the torrent spec or in the most common implementation or these are DoS-like attacks. We'll see...


Greets, Ice