Linux - NetworkingThis forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
network card receives a signal (packet) -> hardware interrupt handler handles it BUT if there is too many interrupts ksoftirqd starts dealing with it. Now I wonder what exactly happens.
Does ksoftirqd "puts" the packet in something like queue and simply forgets about it (deals with another interrupt?). What happens if application can not handle this packet?
What happens if packet has a wrong checksum? Is it dropped? When?
My machine is currently being flooded with a huge number of packets, I'm receiving around 250k packets/sec, ksoftirqd (rx) goes crazy (full CPU usage) and the whole machine works badly. I wonder if application (www server) might slowly process packets and therefore slow down the whole ksoftirqd process.
Is this a normal behaviour for such number of packets?
I'm using Intel 82574L network card and 2.6.35.5 kernel.
My machine is currently being flooded with a huge number of packets, I'm receiving around 250k packets/sec, ksoftirqd (rx) goes crazy (full CPU usage) and the whole machine works badly.
The flood comes from random spoofed IP addresses, all packets have wrong checksum, no flags, port 80. Here is a short log from tcpdump. If you need some specyfic details, let me know, I will try to post it here.
So basically what you've got is "-m tcp" packets with to "--dport 80" which are "! --syn" (no flags) and "--state INVALID" (checksum) and "-m ttl --ttl-eq 249" which you could ("-m limit --limit 1/second" and then) "-j REJECT" or "-j DROP". If unsure post your unabbreviated rule set (please use BB code tags or attach plain text "iptables.conf" from running 'iptables-save > /tmp/iptables.conf').
I got a rule to drop these packets:
iptables -A INPUT -p tcp ! --syn -m state ! --state ESTABLISHED,RELATED -j DROP
Unfortunately it doesn't deal with my problem. Interrupt handler goes crazy (ksoftirqd for rx transmition) and uses the whole CPU thread (99% cpu usage) which causes connection problems, random connection loses - basically the whole machine becomes unstable. I'm not sure if I can block these kind of attack on server-level. Since ksoftirqd eats the whole CPU thread I was even considering disabling Hyper Threading in bios to get additional CPU power for ksoftirqd process.
Maybe there is a way to somehow tweak ksoftirqd or tweak network driver (to somehow drop such "bad packets" as soon as possible?). I found an interesting post about Facebook and their memcached servers, where they were struggling similar problem. Unfortunately, I don't really understand how they dealt with it... http://www.facebook.com/note.php?note_id=39391378919
Quote:
Another issue we saw in Linux is that under load, one core would get saturated, doing network soft interrupt handing, throttling network IO. In Linux, a network interrupt is delivered to one of the cores, consequently all receive soft interrupt network processing happens on that one core. Additionally, we saw an excessively high rate of interrupts for certain network cards. We solved both of these by introducing “opportunistic” polling of the network interfaces. In this model, we do a combination of interrupt driven and polling driven network IO. We poll the network interface anytime we enter the network driver (typically for transmitting a packet) and from the process scheduler’s idle loop. In addition, we also take interrupts (to keep latencies bounded) but we take far fewer network interrupts (typically by setting interrupt coalescing thresholds aggressively). Since we do network transmission on every core and since we poll for network IO from the scheduler’s idle loop, we distribute network processing evenly across all cores.
Maybe there is a way to somehow tweak ksoftirqd or tweak network driver
Does this happen with other kernels? If unknown maybe try an earlier or later one than 2.6.35.5?
Is your Intel 82574L network card swappable for another card for you to test?
Does /proc/interrupts show any other devices with high counts?
In some cases unloading unneeded kernel modules could help. Tried that?
(BTW, while old, this might describe what you're seeing in a slightly easier to read way.)
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
