LinuxQuestions.org - How does kernel handles packets?

- Linux - Networking (https://www.linuxquestions.org/questions/linux-networking-3/)

- - How does kernel handles packets? (https://www.linuxquestions.org/questions/linux-networking-3/how-does-kernel-handles-packets-915344/)

How does kernel handles packets?

How does kernel handles packets?

So far I came to this:

network card receives a signal (packet) -> hardware interrupt handler handles it BUT if there is too many interrupts ksoftirqd starts dealing with it. Now I wonder what exactly happens.
Does ksoftirqd "puts" the packet in something like queue and simply forgets about it (deals with another interrupt?). What happens if application can not handle this packet?
What happens if packet has a wrong checksum? Is it dropped? When?

My machine is currently being flooded with a huge number of packets, I'm receiving around 250k packets/sec, ksoftirqd (rx) goes crazy (full CPU usage) and the whole machine works badly. I wonder if application (www server) might slowly process packets and therefore slow down the whole ksoftirqd process.

Is this a normal behaviour for such number of packets?

I'm using Intel 82574L network card and 2.6.35.5 kernel.

Quote:

Originally Posted by rfire (Post 4532838)

My machine is currently being flooded with a huge number of packets, I'm receiving around 250k packets/sec, ksoftirqd (rx) goes crazy (full CPU usage) and the whole machine works badly.

Could you post details?

Sure.

The flood comes from random spoofed IP addresses, all packets have wrong checksum, no flags, port 80. Here is a short log from tcpdump. If you need some specyfic details, let me know, I will try to post it here.

Quote:

16:33:43.434282 IP (tos 0x0, ttl 249, id 24886, offset 0, flags [none], proto TCP (6), length 40)
172.209.1.67.7407 > ip_address.80: Flags [.], cksum 0xbb5e (incorrect -> 0x625e), ack 0, win 65535, length 0
16:33:43.434284 IP (tos 0x0, ttl 249, id 24295, offset 0, flags [none], proto TCP (6), length 40)
117.217.232.1.58157 > ip_address.80: Flags [.], cksum 0x3367 (incorrect -> 0xda66), ack 0, win 65535, length 0
16:33:43.434286 IP (tos 0x0, ttl 249, id 5530, offset 0, flags [none], proto TCP (6), length 40)
125.156.144.72.24990 > ip_address.80: Flags [.], cksum 0xb1fe (incorrect -> 0x58fe), ack 0, win 65535, length 0
16:33:43.434289 IP (tos 0x0, ttl 249, id 41470, offset 0, flags [none], proto TCP (6), length 40)
15.151.140.8.10304 > ip_address.80: Flags [.], cksum 0xdc20 (incorrect -> 0x8320), ack 0, win 65535, length 0
16:33:43.434292 IP (tos 0x0, ttl 249, id 59347, offset 0, flags [none], proto TCP (6), length 40)
219.143.44.24.50861 > ip_address.80: Flags [.], cksum 0xf60f (incorrect -> 0x9d0f), ack 0, win 65535, length 0
16:33:43.434294 IP (tos 0x0, ttl 249, id 49477, offset 0, flags [none], proto TCP (6), length 40)
70.68.131.66.41743 > ip_address.80: Flags [.], cksum 0x309a (incorrect -> 0xd799), ack 0, win 65535, length 0
16:33:43.434297 IP (tos 0x0, ttl 249, id 18639, offset 0, flags [none], proto TCP (6), length 40)
114.247.219.54.52758 > ip_address.80: Flags [.], cksum 0xf3fe (incorrect -> 0x9afe), ack 0, win 65535, length 0
16:33:43.434301 IP (tos 0x0, ttl 249, id 60799, offset 0, flags [none], proto TCP (6), length 40)
115.136.12.121.51718 > ip_address.80: Flags [.], cksum 0x11b0 (incorrect -> 0xb8af), ack 0, win 65535, length 0
16:33:43.434303 IP (tos 0x0, ttl 249, id 35806, offset 0, flags [none], proto TCP (6), length 40)
85.106.232.13.28012 > ip_address.80: Flags [.], cksum 0x189b (incorrect -> 0xbf9a), ack 0, win 65535, length 0
16:33:43.434307 IP (tos 0x0, ttl 249, id 38148, offset 0, flags [none], proto TCP (6), length 40)
96.118.9.114.375 > ip_address.80: Flags [.], cksum 0xc983 (incorrect -> 0x7083), ack 0, win 65535, length 0
16:33:43.434310 IP (tos 0x0, ttl 249, id 17476, offset 0, flags [none], proto TCP (6), length 40)
44.203.96.85.1229 > ip_address.80: Flags [.], cksum 0x3789 (incorrect -> 0xde88), ack 0, win 65535, length 0
16:33:43.434313 IP (tos 0x0, ttl 249, id 48699, offset 0, flags [none], proto TCP (6), length 40)
2.86.80.118.18902 > ip_address.80: Flags [.], cksum 0x983d (incorrect -> 0x3f3d), ack 0, win 65535, length 0
16:33:43.434316 IP (tos 0x0, ttl 249, id 21016, offset 0, flags [none], proto TCP (6), length 40)
159.201.164.112.38427 > ip_address.80: Flags [.], cksum 0xa5f1 (incorrect -> 0x4cf1), ack 0, win 65535, length 0
16:33:43.434319 IP (tos 0x0, ttl 249, id 34014, offset 0, flags [none], proto TCP (6), length 40)
175.153.100.47.59298 > ip_address.80: Flags [.], cksum 0x7178 (incorrect -> 0x1878), ack 0, win 65535, length 0
16:33:43.434323 IP (tos 0x0, ttl 249, id 18514, offset 0, flags [none], proto TCP (6), length 40)
183.14.120.30.64113 > ip_address.80: Flags [.], cksum 0x3444 (incorrect -> 0xdb43), ack 0, win 65535, length 0
16:33:43.434326 IP (tos 0x0, ttl 249, id 1969, offset 0, flags [none], proto TCP (6), length 40)

So basically what you've got is "-m tcp" packets with to "--dport 80" which are "! --syn" (no flags) and "--state INVALID" (checksum) and "-m ttl --ttl-eq 249" which you could ("-m limit --limit 1/second" and then) "-j REJECT" or "-j DROP". If unsure post your unabbreviated rule set (please use BB code tags or attach plain text "iptables.conf" from running 'iptables-save > /tmp/iptables.conf').

I got a rule to drop these packets:
iptables -A INPUT -p tcp ! --syn -m state ! --state ESTABLISHED,RELATED -j DROP

Unfortunately it doesn't deal with my problem. Interrupt handler goes crazy (ksoftirqd for rx transmition) and uses the whole CPU thread (99% cpu usage) which causes connection problems, random connection loses - basically the whole machine becomes unstable. I'm not sure if I can block these kind of attack on server-level. Since ksoftirqd eats the whole CPU thread I was even considering disabling Hyper Threading in bios to get additional CPU power for ksoftirqd process.

Maybe there is a way to somehow tweak ksoftirqd or tweak network driver (to somehow drop such "bad packets" as soon as possible?). I found an interesting post about Facebook and their memcached servers, where they were struggling similar problem. Unfortunately, I don't really understand how they dealt with it...
http://www.facebook.com/note.php?note_id=39391378919

Quote:

Another issue we saw in Linux is that under load, one core would get saturated, doing network soft interrupt handing, throttling network IO. In Linux, a network interrupt is delivered to one of the cores, consequently all receive soft interrupt network processing happens on that one core. Additionally, we saw an excessively high rate of interrupts for certain network cards. We solved both of these by introducing “opportunistic” polling of the network interfaces. In this model, we do a combination of interrupt driven and polling driven network IO. We poll the network interface anytime we enter the network driver (typically for transmitting a packet) and from the process scheduler’s idle loop. In addition, we also take interrupts (to keep latencies bounded) but we take far fewer network interrupts (typically by setting interrupt coalescing thresholds aggressively). Since we do network transmission on every core and since we poll for network IO from the scheduler’s idle loop, we distribute network processing evenly across all cores.

Quote:

Originally Posted by rfire (Post 4539340)

Unfortunately it doesn't deal with my problem.

Bummer.

Quote:

Originally Posted by rfire (Post 4539340)

Maybe there is a way to somehow tweak ksoftirqd or tweak network driver

Does this happen with other kernels? If unknown maybe try an earlier or later one than 2.6.35.5?
Is your Intel 82574L network card swappable for another card for you to test?
Does /proc/interrupts show any other devices with high counts?
In some cases unloading unneeded kernel modules could help. Tried that?

(BTW, while old, this might describe what you're seeing in a slightly easier to read way.)