A few hours ago, I received a Munin alert that traffic had exceeded a threshold I had set. When I checked my munin page on my webserver I saw something very strange. I saw the exact same traffic passing over all IP addresses on my webserver. The box handles 8 IP addresses and for each IP address, munin showed the exact same outgoing traffic. Here are two IP addresses that normally have no traffic at all:

http://i515.photobucket.com/albums/t...1970/munin.gif

As you can see, the traffic is exactly the same. If I look at my apache logs, I don't see anything strange going on. When I do a tcpdump, I see a lot of outgoing UDP traffic like this:

(I replaced my real domain for "my.domain.com")

I downloaded and installed the latest version of rkhunter on my system and that found nothing.

I also see a lot of this with tcpdump:

Edit: It's mostly the request from the first window above that generate the bulk of the traffic. Those are apparently DNS requests.

So what's going on? Or how can I find out what's going on?

Looks to me like either a vulnerability scanner or DOS script. Do you perchance run a public DNS server?

I'm running Bind (9) if that's what you mean.

Yes, but how are you running it? Do you make it accessible to the world or is it local only? If it is accessible, is it "open" or designed to only resolve your public facing servers? For example, I too run Bind and it is the authoritative name server for my domains and if you query it for one of them on their public facing address, it will give you the proper address. If you instead query for something like google.com, it will refuse to answer that for you.

There are two things that I think you need to consider and review. One is whether or not your DNS is configured as it should be. BTW, don't run an open DNS. Two, if you are getting hit with excessive queries finding a proper counter measure to discourage the practice OR switching to a different DNS such as your registrar's which may be better equipped to handle this sort of thing.

I think it's local only. I do know recursion is disabled and that it's setup to answer on random ports.

I really am not very well known with bind so I can't be sure it's configured as it should be. Here are my config files:

/etc/bind/named.conf:
/etc/bind/named/conf.local:
(empty)

/etc/bind/named.conf.options:

I am not a Bind expert, more of a medium level user and perhaps one of the experts (e.g. Bathory) will chime in, but I am pretty sure that: is allowing public access to your DNS. If your DNS is public and recursive, it is only a matter of time before it will be abused, which seems to be your problem. Listening on non standard or random ports will not stop the problems as a port scan will quickly reveal the service. It only stops the lowest of the low who are too lazy or stupid to look for it elsewhere.

I would either configure an ACL to allow recursion ONLY from your private LAN side, and / or have it listen on port 53 and block this traffic from outside of your network.
For example:

I'm no bind expert either but I'm pretty sure what you did is pretty much the same as what I did. :D

This is your standard bill-of-fare DNS amplification attack. The attack is directed at the site whose information is being requested, not you (directly) - but they're using your DNS (and many others) as a means of attacking the target.

As you've probably guessed, the requesting IP addresses are forged. Your DNS requests the record from the target. The cumulative effect of all those requests from all the venue DNS servers is designed to overwhelm the target DNS. (Secondarily, the real version of the forged address receives the non-authoritative response from your DNS.)

If you look up ways to deal with amplification attacks, you'll find an answer to suit you. The answers I've found involve turning off recursion, which I cannot implement, since I require recursion. Alternatively, I have done two things:
First, I set up a fake entry for the target address, as if I'm authoritative for it. This way I'm no longer helping the attack on the primary target.
Second, I wrote a quick-n-dirty perl script that reads a tail of the DNS log, every second, then adds the forged IP to the firewall (ipfw for me using FreeBSD - iptables for you Linux folks.) Of course, my DNS logging includes a "category queries" and "category security" logs, so I have the full info of the requests, and I'm not stuck having to do a tcpdump when I _happen_ to notice a traffic spike.

Now all I'll have to do is clean up the firewall list whenever the attack is over. In my experience, these attacks are short-lived; a few days to a few weeks, at most.


Next, I'll cut-and-paste a security e-mail I got from one of my hosts on how to preclude recursion, which they say solves the problem.

This is from the e-mail I got from hivelocity/noc4hosts whom is one of the companies I have the misfortune of being a customer of. (Therefore, the IP's mentioned are specific to their service - you'll have to tailor to your own services. You may wish to contact your upstream provider for assistance and details.)
-=-=-=-

1. Linux Servers running Bind you need to modify file /etc/named.conf. (Please back up the file before making any changes)

Notice the first line is setup for 127.0.0.1. This will allow the local Linux machine to query locally if the linux
machine has nameserver 127.0.0.1 in there to be able to query locally. The example here adds additional Hivelocity IP
blocks which allows these blocks to use your DNS for recursive and caching lookups. You could edit these lines and put
only your required or preferred subnets to lock down your DNS even further.

options {
recursion yes;
allow-recursion { 127.0.0.1/32; 199.193.112.0/21;
91.209.57.0/24; 38.112.194.0/24;
199.119.100.0/22; 216.109.224.0/20;
199.167.144.0/21; 68.233.224.0/19;
96.31.64.0/19; 74.50.96.0/19;
206.51.232.0/21; 206.51.224.0/21;
69.46.0.0/19; 66.96.80.0/20; };
allow-query-cache { 127.0.0.1/32; 199.193.112.0/21;
91.209.57.0/24; 38.112.194.0/24;
199.119.100.0/22; 216.109.224.0/20;
199.167.144.0/21; 68.233.224.0/19;
96.31.64.0/19; 74.50.96.0/19;
206.51.232.0/21; 206.51.224.0/21;
69.46.0.0/19; 66.96.80.0/20; };
};

After the change you need to restart Bind with command "service named restart" or "/etc/init.d/named restart"

I missed the "acl recurseallow { 1.2.3.4; 127.0.0.1; };" line. Like I said, no expert!
It looks like subfolder is thinking along the same lines, except they add the allow-query-cache block. Rereading these posts a few times brought me to two questions:
1 - are you sure that the DNS response, i.e. the part where TCP dump says: TXT[|domain] that it isn't sending back an error message like REFUSED rather than a DNS record. This would still be data in an amplification attack that the recipient would either have to block or process.
2 - If it is not an error message, is your network set up so that data arriving on your public IP 1.2.3.4 appears to your DNS as coming from your IP and hence is allowed? One of the differences between your ACL and mine is that I don't allow recursion on my public IP (or even on the DMZ IP that it translates to). Instead, I specifically allow queries on it, with the allow query any, but not recursion.

For the third time, the configuration stub I cited (above) was a direct cut-n-paste from an email from my provider. I cannot verify its veracity. I did not implement it, nor can I, since my DNS's must be unilaterally recursive. Since it's directed at Linux users (I use FreeBSD), I suspect it's likely it may be a usable example in this forum.

At subfolder: I understand that it is a cut and paste with IP addresses designated for another system. What I am saying is that if I am understanding it correctly, your suggesting the same thing that we are: to use an ACL to stipulate only which IP addresses are allowed to perform recursive queries. My follow up post from this morning was to double check that 1) the existing ACLS may in fact be working and the resulting traffic is a REFUSED message. If this is the case, there is little more that can be done, other than to use a tactic to limit the number of refusals that will be sent before the server simply ignores the query, b) that the fact that the public IP address is listed in the ACL isn't somehow invalidating the ACL.

Hi,

According to your configuration your dns is not an open resolver. You can omit the "recursion yes;" as the "allow-recursion..." supersedes it.
Anyway, from the logs in your 1st post looks like host86-128-99-49.range86-128.btcentralplus.com is using your server as a resolver. Is this host part of your LAN? I guess it is because the recursion ACL.
So if you get much traffic from it, you should check if it's compromised somehow and sends out spam or something like that, that make extensive use of dns.

Regards

It was.

When I first noticed the traffic, I checked if bind was setup to disallow recursion from outside my network. It wasn't. So I added the lines for disallowing recursion.

At first I thought it made no difference because I still saw the same lines in my log files. What I didn't notice however, was that data traffic dropped to almost nothing because those recursive lookups were denied. So in fact disallowing recursion was the solution (even though I didn't notice it straight away).

Anyway, the attack stopped as abruptly as it started after a day or so. Thank you all for your help. It helped me understand a little bit more how bind works. :D

Apparently, default logging settings of bind, in Linux, don't give information that can be easily discerned for viewing the details of an attack. I rely on the logs to see the data commonly associated with the more common attacks. Also, I have a quick-n-dirty perl-script that continually monitors the logs to automatedly add a firewall entry, as they are found. Additionally, for a server with multiple IP's, I have a separate bind process for each of the IP's, each with their own configuration and logging.

Here's the relevant snippet of one of them - you may find useful;

......

listen-on { 111.222.33.44; };
};

logging{
channel dns_log {
file "/var/log/dns.111.222.33.44.log" versions unlimited size 10m;
severity debug 99;
print-severity yes;
print-time yes;
print-category yes;
};
channel dns_loh {
file "/var/log/dns.111.222.33.44.loh" versions unlimited size 10m;
severity debug 99;
print-severity yes;
print-time yes;
print-category yes;
};
channel dns_loi {
file "/var/log/dns.111.222.33.44.loi" versions 99 size 10m;
severity debug 99;
print-severity yes;
print-time yes;
print-category yes;
};
category queries {
dns_log;
};
category security {
dns_loh;
};
category default {
dns_loi;
};
};

........

"category queries" produces a log with entires like this;

23-Jul-2012 16:37:23.147 queries: info: client 176.227.198.146#24983: query: isc.org IN ANY +E
23-Jul-2012 16:37:23.199 queries: info: client 176.227.198.146#16996: query: isc.org IN ANY +E
23-Jul-2012 16:37:23.350 queries: info: client 176.227.198.146#16326: query: isc.org IN ANY +E
23-Jul-2012 16:37:23.600 queries: info: client 176.227.198.146#24983: query: isc.org IN ANY +E
23-Jul-2012 16:37:23.670 queries: info: client 176.227.198.146#16996: query: isc.org IN ANY +E
23-Jul-2012 16:37:23.851 queries: info: client 176.227.198.146#16326: query: isc.org IN ANY +E
23-Jul-2012 16:37:23.986 queries: info: client 176.227.198.146#24983: query: isc.org IN ANY +E
23-Jul-2012 16:37:24.426 queries: info: client 176.227.198.146#24983: query: isc.org IN ANY +E
23-Jul-2012 16:37:24.625 queries: info: client 176.227.198.146#16996: query: isc.org IN ANY +E
23-Jul-2012 16:37:24.999 queries: info: client 176.227.198.146#16996: query: isc.org IN ANY +E
23-Jul-2012 16:37:25.078 queries: info: client 176.227.198.146#16996: query: isc.org IN ANY +E

"category security" produces a log that looks like this;

23-Jul-2012 16:37:02.373 security: debug 3: client 176.227.198.146#16996: request is not signed
23-Jul-2012 16:37:02.373 security: debug 3: client 176.227.198.146#16996: recursion available
23-Jul-2012 16:37:02.373 security: debug 3: client 176.227.198.146#16996: query (cache) 'isc.org/ANY/IN' approved
23-Jul-2012 16:37:02.908 security: debug 3: client 176.227.198.146#16996: request is not signed
23-Jul-2012 16:37:02.908 security: debug 3: client 176.227.198.146#16996: recursion available
23-Jul-2012 16:37:02.908 security: debug 3: client 176.227.198.146#16996: query (cache) 'isc.org/ANY/IN' approved
23-Jul-2012 16:37:02.932 security: debug 3: client 176.227.198.146#16326: request is not signed
23-Jul-2012 16:37:02.932 security: debug 3: client 176.227.198.146#16326: recursion available
23-Jul-2012 16:37:02.932 security: debug 3: client 176.227.198.146#16326: query (cache) 'isc.org/ANY/IN' approved
23-Jul-2012 16:37:03.030 security: debug 3: client 176.227.198.146#24983: request is not signed
23-Jul-2012 16:37:03.030 security: debug 3: client 176.227.198.146#24983: recursion available
23-Jul-2012 16:37:03.030 security: debug 3: client 176.227.198.146#24983: query (cache) 'isc.org/ANY/IN' approved
23-Jul-2012 16:37:03.416 security: debug 3: client 176.227.198.146#16996: request is not signed
23-Jul-2012 16:37:03.417 security: debug 3: client 176.227.198.146#16996: recursion available
23-Jul-2012 16:37:03.417 security: debug 3: client 176.227.198.146#16996: query (cache) 'isc.org/ANY/IN' approved