I run my own DNS server - standard bind and current with all updates...

I have access restricted to the few zones I am authoritative for so no one can poison the cache. This seems to be working fine as I get occasional entries (maybe a dozen or less a day on average) in the message file like:

Mar 20 05:11:28 alpha named[13963]: client 96.247.61.192#1794: query (cache) '89.156.184.220.broad.hz.zj.dynamic.163data.com.cn/NS/IN' denied
Mar 20 05:11:28 alpha named[13963]: client 96.247.61.192#1795: query (cache) '89.156.184.220.broad.hz.zj.dynamic.163data.com.cn/NS/IN' denied

all fine and the world is a happy place *until*...

in the past week or two I have started getting message log entries that look like *this*:

Mar 20 10:22:10 alpha named[660]: client 127.0.0.1#60884: query (cache) '47.199.64.125.broad.yb.sc.dynamic.163data.com.cn/A/IN' denied
Mar 20 10:22:25 alpha last message repeated 3 times
Mar 20 11:33:31 alpha named[660]: client 127.0.0.1#60884: query (cache) '24.106.154.61.broad.qz.fj.dynamic.163data.com.cn/A/IN' denied
Mar 20 11:33:46 alpha last message repeated 3 times
Mar 20 12:31:00 alpha named[660]: client 127.0.0.1#60884: query (cache) '207.199.165.60.broad.pl.gs.dynamic.163data.com.cn/A/IN' denied
Mar 20 12:31:15 alpha last message repeated 3 times
Mar 20 23:29:38 alpha named[660]: client 127.0.0.1#60884: query (cache) '149.28.160.60.broad.zt.yn.dynamic.163data.com.cn/A/IN' denied


I have searched high and low and can't come up with anything explaining what is happening? at first I thought they queries had to be coming from the local host but why would I reject my own DNS queries (every other query seems to work fine) and why would my box be looking up an "A" record for what appears to be a reverse address lookup formatted address?

To check out my own address I did a dig of the address that was rejected and it returns information just fine so I am assuming this is a request from "outside".

Is this just someone sending a packet with a spoofed source address? Is there any risk (I seem to be rejecting these as I would expect)? Can anyone clue me in at all on this?

Many thanks in advance to all of the gurus here who actually know what they are doing and can set me right on this.

-Tom

Ever evaluated djbdns? Just asking, OK?


Then it would IMHO be more precise to say that only those inside those zones stand a chance at poisoning the cache, right?


So what changed between the world being a happy place and then?


Malformed queries aren't odd. For example root servers get at least twenty percent due to Microsoft, Apple, misconfiguration and malice. IIRC if this would be a simple poisoning attack against a target domain, you would not only see many requests for that domain but also a lot of fake replies that try to match the request ID and source port. Dissecting packet captures would show more details than BIND log can. Noticing the requested domain being .dynamic.163data.com.cn: are there more services running on the machine or is it a single service box? And you don't run by any chance a MTA that's been accessed by or for .dynamic.163data.com.cn?

thanks for the reply - it has given me some real help. Let me reply to your response point by point - after an intro.

I am far from an expert in this area which is why I turned to the gurus on here. I did research and I talked to one person who is reasonably knowledgeable on the subject but just cannot resolve my problem.


Quote:
Ever evaluated djbdns? Just asking, OK?

no I haven't but I will certainly check into it.


Quote:
Then it would IMHO be more precise to say that only those inside those zones stand a chance at poisoning the cache, right?

I guess I used the wrong terms there or else did not understand your reply. I assumed (I know - I *assumed* - go ahead and flog me now and get it over with) that my acl statement would allow my subnet to access everything and the rest of the immediate universe would only be able to access the zone information for those zones I am authoritative for. But you are absolutely right that theoretically, assuming I have the config correct, that only my own network could be poisoning anything.


Quote:
So what changed between the world being a happy place and then?

Excellent question. After about 40 years in the field I can tell you that I do not take "I didn't change anything - it just started happening by itself" as an answer from *anyone* including myself. I know better. But in this case I can't *remember* making any changes recently and I do not recall seeing any 127.0.0.1 entries in the logs until the past week or so. Mind you I do not rule out either possibility as being impossible. All I can do now is go forward on the assumption that there is a problem/issue of some sort of my making.


Quote:
Malformed queries aren't odd. For example root servers get at least twenty percent due to Microsoft, Apple, misconfiguration and malice. IIRC if this would be a simple poisoning attack against a target domain, you would not only see many requests for that domain but also a lot of fake replies that try to match the request ID and source port. Dissecting packet captures would show more details than BIND log can. Noticing the requested domain being .dynamic.163data.com.cn: are there more services running on the machine or is it a single service box? And you don't run by any chance a MTA that's been accessed by or for .dynamic.163data.com.cn?

Yes indeed the machine is a multi tasker and I do run sendmail on it. DNS and Sendmail have been co-habitating peacefully living side by side for a number of months now - maybe 7 or 8. Your question did raise the level of issue to higher relevance in my mind and added itself to my pondering.


Summary: I started out with an issue - finding 127.0.0.1 source IP DNS denials for something.163data.com.cn in my message log. After trying to resolve the issue I found myself in a mental either/or box. Either it *was* an internal request (impossible I say because I am resolving lots of domains successfully). Or it *was* from the outside word - and properly rejected because I am not authoritative for that domain (impossible I say because 127.0.0.1 cannot be an outside address). Those impossible statements were tongue firmly in cheek obviously because one of them *has* to be happening but I'll be damned if I can get my mind out of that box. What I need is someone to spot my obvious flaw and get me out of this mental trap. This is not a first - over the years I have seen and committed the blunder of seeing what I *thought* I coded instead of what I *did* many times. Getting out of that trap is the problem.

Since I first posted I broke out of the box just far enough to come up with one idea to try to eliminate a possibility - I assumed (I know I know - just beat me quickly and get it over with) that 127.0.0.1 should *never* be the source IP in a packet coming in over eth0 so I put a rule into iptables to drop any incoming packet with that source address and sat back to wait. Then I sat back to wait and voila (that's either a large violin or a French word, I forget which) I got no drops but a couple more log entries for 163data.com.cn again. This eliminated the "outside" impossible condition and left me with the remaining "impossible" condition - it is coming from my side.

Having jumped out of the impossible A or impossible B mindset trap I simply found myself in a new box - impossible situation B1 or impossible situation B2. Since it *is* on my side the 127.0.0.1 "problem" goes away but now I am confronted with "why didn't I get successful resolution?" from the query? with sendmail active and several client machines using this as their primary DNS server why isn't a lot more failing? A simple "dig" to any of the failed 163data.com.cn address was quickly resolved. Now I can't get myself out of the "either nothing should be resolving" or "163data stuff should also resolve". Can you give me *any* leads for where to look or what to test to figure out why just *one* domain is failing?

config snippet:

acl house_net {
206.174.132.57/28;
};
options {
directory "/etc";
pid-file "/var/run/named/named.pid";
allow-recursion {
house_net;
};
allow-query {
house_net;
};
allow-transfer {
house_net;
};
};


Please sir - I don't really want *more* - well not an entire bowl full, just a mere morsel, a tidbit as it were.

Thanks
-Tom

Quite an elaborate reply ;-p

Let's see what others think of say 163data.com.cn and maybe we'll find failures. Look up the DN 163data.com.cn (one of my favourite starting points). Notice the hints under "subdomains", follow "!dynamic" and find recursing ends at broad.hz.zj.dynamic.163data.com.cn. Nothing to be learnt from there, back to the DN. Query WHOIS (also notice the crap in the "hostnames beginning with" part) and compare with other source (or dnsgoodies or whois.ws). Notice how the NS is inconsistent: "tianyan1.cnnic.net.cn" vs the (nonexistent) "ns.jkflkadjfkldjfkljdfljdfljdfljldfjlf.com". The DNS Sleuth shows violations as well. Is this the registrar (sfn.cn) fscking up?.. Since you're running an MTA you can use your favourite searchengine or Aboutus to see if the DN is named for spam. Aboutus is kinda blisteringly subjective since it's a Wiki (but still an indication) so it's good to know the DN is in at least one RBL: URIBL (or use surblhost or see SURBL+ Checker).

Nice. But how about 89.156.184.220.broad.hz.zj.dynamic.163data.com.cn?
...which results in this (Notice doing reverse lookups at dnsstuff and dnsgoodies will show warnings like there's no authoritative NS for this IP. Nice.) with these RBL entries. Maybe not a complete answer and maybe I should have wedged in something about the MTA doing lookups over loopback and how to trace those but I hope it gives you an idea your resolving problems for this particular domain are not problematic. At least from my POV but I'm no expert. Corrections welcome.

I've just started getting a bunch of these as well, and all of mine are coming from a particular company in Marietta, GA.

log excerpt:
I did a reverse whois lookup and got this on the ip they are coming from:
NationalNet is a hosting company, so I can't tell if they are making these attempts or if they have been exploited or one of their clients is doing it.

I was about to blacklist their domain though, as I had 21,000 requests from them over the christmas holidays. All of them were between 4:14AM and 5:18AM and then it stopped. Hopefully they just misconfigured their name servers or something...

I've been seeing something similar:

Jan 16 20:52:36 shrub named[17982]: client 69.50.142.11#9956: view external: query (cache) './NS/IN' denied
Jan 16 20:52:37 shrub named[17982]: client 69.50.142.11#40042: view external: query (cache) './NS/IN' denied
Jan 16 20:52:39 shrub named[17982]: client 69.50.142.11#27433: view external: query (cache) './NS/IN' denied
Jan 16 20:52:39 shrub named[17982]: client 69.50.142.11#52686: view external: query (cache) './NS/IN' denied
Jan 16 20:52:39 shrub named[17982]: client 69.50.142.11#23171: view external: query (cache) './NS/IN' denied
Jan 16 20:52:41 shrub named[17982]: client 69.50.142.11#9746: view external: query (cache) './NS/IN' denied

Once again from NationalNet.com, and I believe they've done this to me before. I think one, or more, of their clients aren't too reputable and have something that is getting them blocked from accessing the regular DNS servers.

I used dnstop (good program) to watch what they were looking for, and it was always the '.' aka 'root' domain, *nothing else*. To me that seems pretty fishy, after all I'm only on an ADSL link, my IP is nothing like theirs, and I'm the other side of the continent and another country away. Why are they doing this? Either this is a DNS poisoning attemp, or they are likely doing something not quite right and trying to hide their queries... perhaps they're running a bot controller or something? (Pure speculation..)

I suggest you use IPTables to block access from them(substituting the offending IP);

iptables -I external -s 69.50.142.11 -j DROP

Cheers,
TJ99

UPDATE: Heard back from the abuse folks. The traffic is apparently a side-effect of a DDOS attack on port 53, so it's not NationalNet's fault, although I do wonder if they could filter some of the outgoing DNS traffic better so the rest of us don't get the scattershot.

I emailed the abuse contact and this is their response:

This actually is not one of our clients. This client is having a traditional
DDOS attack on PORT 53 UDP.

As you are aware there is no three way handshake with UDP traffic, poor
design perhaps, hence forth there are two steps to this attack.

A) Flood the server as listed below IE: 69.50.137.175
B) Modify the header and send the request to thousands of unexpected Name
Servers to make them think they need to respond.

This leads to two things.

A) Causing the server to be flooded in hopes to shut it down. This was
prevented by placing DDOS filters in place and setting an ACL to deny any
and all UDP traffic to and from this server.

B) Service providers such as you to make contact with the hosting company to
say there being attacked.

The server in question will never have a reason to do a UDP/53 request
directly as it's simply an HTTP server and not even running DNS services. I
would suggest to protect yourself by simply placing a NULL route on your
core router or set a rule in IPTables to drop connections from this IP
address.

We apologize for the inconvenience but DDOS attacks are never under the
recipients control.

I'm receiving the same traffic from the same address. It started hitting my nameservers about 36 hours ago so I've just filtered the IP entirely.

security: info: client 69.50.142.11#59209: query (cache) './NS/IN' denied

Ive been receiving DNS queries since approx the 7th of January from a mix of IPs and providers networks (NationalNet, ATM Link, nac).

Ive only just realised, and have added an ACL rule to block traffic from this IP from reaching my DNS servers.

According to a post I read in the "news.admin.net-abuse.email" newsgroup, the IP these DNS queries are being spoofed from might have hosted some questionable content...

I am seeing 4-5-6 packets/sec hit my deny ACL rule. Was wondering why my quota usage seemed a bit higher than usual this month. :-)

Tom

Hi,

If I'm understanding this attack correctly (which I've been seeing on both of our named servers for the past several days), filtering the following 2 IP blocks won't help: 69.50.137.175 & 69.50.142.11. The attacker is spoofing these IP addresses so the request is really not originating from these IP blocks.

Has anyone found out what the actual source IP addresses that are forging these requests to bind? I've looked at my bind logs and they only show the same spoofed ip addresses I listed above as the source so I assume bind is 'blind' to the actual source and just assumes whatever an attacker says is the source.

thx,

SW

EDIT: I was made aware of this by our egress filtering which seems to have worked because I saw lots of 'OUT_UDP DROP' orginating from our name servers to the above ip addresses.

I'd certainly run DNS or DHCP on a gateway but not a MTA. If you can, move the MTA off to another host inside your domain. It does look like the MTA is querying over the localhost. Could be some attempted external mayhem. Have you looked at sendmail logs? Might want to configure iplog or fwlogwatch for more information on the internal goings-on.

I've been seeing the same thing on my nameservers. I grew tired of manually adding IPs to my firewall so I wrote a script that tails the system log at intervals and automatically adds the offending (or spoofed) IP to the server's firewall. Seems to be affective for my uses. If anyone else is interested in checking it out, it's available here:

http://rob.pectol.com/dnsexploit.txt

It's very simple and could be improved upon but I hardly see the point in spending a lot of time on it for an exploit that will probably disappear relatively soon anyway. But it does work and maybe someone else will find it handy.

Rob

No, you're not understanding the issue correctly. If the source is forged, that's the only address you're going to see in the header and it is effective to filter that spoofed source. Of course, any legitimate traffic from those IPs will be blocked too, but it's generally impossible to determine the difference between spoofed and legitimate requests (hence the reason why spoofing is popular).

Only the attacker's service provider can possibly know the real source, but even then it won't show the IP address (since they aren't actually using their own IP), it will just show the MAC address of whatever piece of gear the attacker has connected to their ISP (cable modem, etc).

There's no way anyone downstream can filter on the actual source.

You could filter egress traffic going to those IPs, but why let your nameservers even go through the work to create those queries in the first place? Just block the spoofed sources on their way in.

If the service provider(s) whos network these bogus packets are being generated from had half competant admins they would:

a) have URPF enabled on their customer aggregation routers such that packets sourced from IPs that are not known to be routed to a particular customer would not even be able to enter the service providers network from the customer

b) or at least filter outbound packets from their network such that packets sourced from IPs that do not even exist on that SPs network could not leave their network and enter the Internet

Unfortuntely theres a lot of poorly run ISPs out there, and these types of attacks are what we have to live with as a result.

So a tip to any network admin who reads this, if you havnt already done so, filter all traffic leaving your network so that only traffic sourced from IPs that are actually routed into your network are allowed to leave it. Its your contribution to stopping this type of thing happening in the future by not allowing it to happen from your network! :-)

Tom

Sorry to pester about this, but I was wondering if you could clarify this statement for me so I can see about implementing it on my network. Specifically the "sourced from IPs that are routed in to your network" part. I still get the query "(cache) './NS/IN' denied" errors reletively frequently, and so far am only able to add the offending IP to my filter rules.

Granted, I found this thread after very little sleep, and may have an epiphany be the time you read this (don't count on it).

Andrew