Help in stopping DNS requests (DNS Amplification)

accessthecloud · 02-18-2013, 03:56 AM

I lease a few servers from 1and1. Up until the beginning of last week, everything was functioning normally.

Starting last week one of my servers started receiving a massive increase in DNS traffic. There were between 100-200 requests per second with logs containing similar information to the following lines:

Code:

Feb 17 06:22:37 serv01 named[2563]: client xxx.xxx.xxx.xx#60797: query (cache) 'isc.org/ANY/IN' denied
Feb 17 08:23:37 serv01 named[2563]: client xx.xx.xxx.xx#53: query (cache) 'ripe.net/ANY/IN' denied

After some research I discovered that during a recent upgrade to Plesk, the DNS recursion settings were changed (by the upgrade script) to Any host! This resulted in my server acting as an open DNS answering all requests. This I changed immediately.
Further investigation identified huge log files and a massive amount of traffic. The traffic all requesting the above information, while spoofing the source ip. Blocking individual addresses required constant monitoring.

In order to reduce the log file size and CPU time, I implemented a set of iptables rules designed to:

Limit the number of connections to UDP 53
Match the strings; ripe.net and isc.org (discovered using Wireshark)

I have successfully managed to drop all these requests at the firewall and releasing the strain on my servers' I/O and CPU time, however using iptraf and 'iptables -vnL --line-numbers' the traffic is still continuing with the following statistics after 24 hours:

Code:

num   pkts bytes target     prot opt in     out     source               destination
1      22M 1387M            udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:53 state NEW recent: SET name: DNSLF side: source
2      21M 1382M DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:53 state NEW recent: UPDATE seconds: 1 hit_count: 6 name: DNSLF side: source
3    82190 5269K            udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:53 state NEW recent: SET name: DNSHF side: source
4     4522  289K DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:53 state NEW recent: UPDATE seconds: 7 hit_count: 20 name: DNSHF side: source
5    4797K  316M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

1     1927  118K DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0           STRING match "|076c796e6c656f6e03636f6d00|" ALGO name bm TO 65535 udp dpt:53
2    2052K  135M DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0           STRING match "|0472697065036e657400|" ALGO name bm TO 65535 udp dpt:53
3    2371K  152M DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0           STRING match "|00000000000103697363036f726700|" ALGO name bm TO 65535 udp dpt:53
4        0     0 DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0           STRING match "|00000000010372697065036e657400|" ALGO name bm TO 65535 udp dpt:53
5    3576K  276M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Although my firewall rules are working, I would prefer to find a more permanent method to block/reject. Anybody have any advice?

My server specs:
AuthenticAMD, AMD Phenom(tm) II X6 1055T Processor (6 CPU)
16GB RAM
1TB HDD
100Mbps/sec full duplex
Hardware Firewall
Linux 2.6.32-279.5.1.el6.x86_64 #1 SMP Tue Aug 14 23:54:45 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
Plesk 11.0.9 (fully up-to-date)
'iptables' and 'ossec-hids'
8 IP addresses
authoritative name server (12 zones)

Thanks in advance,

Mark

unSpawn · 02-18-2013, 09:30 AM

Quote:

Originally Posted by accessthecloud

After some research I discovered that during a recent upgrade to Plesk, the DNS recursion settings were changed (by the upgrade script) to Any host! This resulted in my server acting as an open DNS answering all requests. This I changed immediately.

0) I hope you filed a bug report? 1) Changed how (as in allowing now recursion from where exactly: details)?

Quote:

Originally Posted by accessthecloud

In order to reduce the log file size and CPU time, I implemented a set of iptables rules designed to: Limit the number of connections to UDP 53 Match the strings; ripe.net and isc.org (discovered using Wireshark)

First of all if you're using ISC BIND for dual reasons, caching name server plus authoritative name server, then you should split those up: providing yourself and your LAN with BIND as caching name server requires it to have recursion set, your publicly accessible authoritative name server does not.

Please read links / advice posted in: http://www.linuxquestions.org/questi...on-4175417284/, http://www.linuxquestions.org/questi...es-4175450008/ and maybe http://www.openlogic.com/wazi/bid/18...-DNS-with-BIND. You may find you have already implemented all of that but it never hurts to ensure you have. After checking / implementing that Netfilter rate limiting is still useful but you should not need IP blocking anymore (if you do use ipset instead) and having Netfilter do string matches is computationally way expensive (use a Snort rule instead). If you have ensured your NS can't be used for amplification attacks anymore you probably want to make BIND log less.

accessthecloud · 02-18-2013, 03:43 PM

Thanks for the reply.

0) & 1) Bug report? : The Plesk Administrator's Guide explains! Link

Quote:

After your install Parallels Plesk Panel, the built-in DNS server defaults to serving recursive queries only from your own server and from other servers located in your network. This is the optimal setting. If your upgraded from earlier versions of Parallels Plesk Panel, your DNS server defaults to serving recursive queries from any host.

I run an authoritative name server for my domains (and my customers). When I query my name server for one of my own domains, I get the right answer. Querying for anything else replies with Query refused. After the upgrade and before I realised it, I was running an open DNS

! My server was obviously found and abused. What annoys me is that they are continuing with the isc and ripe requests, and these are dropped at the firewall.

While I do believe that the rate limiting I added to my firewall is helpful, I am concerned about any performance hit. The string matching is, as you stated; expensive but without this with the rate limiting on, named is using around 10% of 3/4 cores constantly.

Quote:

If you have ensured your NS can't be used for amplification attacks anymore you probably want to make BIND log less.

Definitely cannot be used for amplification attacks. Would altering the logging change the CPU usage?

Snort looks useful and ipset is exactly what I need. The links you provided are also very good for reference. Thanks

I will update what I do in due course.

FYI; statistics of iptables after 36 hours

Code:

num   pkts bytes target     prot opt in     out     source               destination
1      32M 2086M            udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:53 state NEW recent: SET name: DNSLF side: source
2      32M 2080M DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:53 state NEW recent: UPDATE seconds: 1 hit_count: 6 name: DNSLF side: source
3    98105 6292K            udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:53 state NEW recent: SET name: DNSHF side: source
4     4918  315K DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:53 state NEW recent: UPDATE seconds: 7 hit_count: 20 name: DNSHF side: source
5    4819K  318M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

1     2957  183K DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0           STRING match "|076c796e6c656f6e03636f6d00|" ALGO name bm TO 65535 udp dpt:53
2    2052K  135M DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0           STRING match "|0472697065036e657400|" ALGO name bm TO 65535 udp dpt:53
3    2373K  152M DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0           STRING match "|00000000000103697363036f726700|" ALGO name bm TO 65535 udp dpt:53
4    4070K  337M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0