Random DNS outages

Red Squirrel · 05-09-2016, 04:49 PM

I have a fairly newly setup local DNS server running CentOS 6.7 and whatever is the latest named available for that distro.

It acts as a local DNS server and also a cache/forwarder for rest of my internet.

Every now and then, it just stops working. All name resolutions fail. I need to restart it then everything starts working again. This is quite unacceptable, I should not have to restart a service all the time like that.

Is there a way to figure out why it's doing this, and to stop it? Does it create a log anywhere?

Emerson · 05-09-2016, 08:02 PM

Does it crash and leave behind a stale pid file?

Red Squirrel · 05-10-2016, 01:59 AM

Not that I'm aware of, just doing named restart fixes the issue, I don't get any unusual errors, like failing to stop the service. I looked for a log but can't seem to find where it puts it or what it names it, if there is any.

bathory · 05-10-2016, 02:34 AM

@OP

Looks like you hit this bug.
Since there is no solution mentioned there, try to use a different bind version. Upgrade if possible or downgrade (according to the guy that opened the bug report downgrading bind solves the problem).
BTW as you can see at the bug report, the named logs are written in /var/named/data/named.run

Regards

Red Squirrel · 05-10-2016, 11:51 AM

Hmmm wonder if my version is affected, it's only a few numbers off:

BIND 9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.7

I did a yum update but it's as high as it goes. I'm not sure how to downgrade and even if I did the next time I do a system update it will just update back anyway. I found the named.run file, but it's empty. There's a couple dated ones but they're all empty as well.

lazydog · 05-10-2016, 02:05 PM

You should be able to downgrade and then use the '--exclude=' option to exclude bind from being updated.

bathory · 05-10-2016, 04:07 PM

Quote:

it will just update back anyway. I found the named.run file, but it's empty. There's a couple dated ones but they're all empty as well.

Run:

Code:

rndc trace 3

to start logging at debug level 3. Or use 99 as in the bug report suggestion, for more detailed logging

Red Squirrel · 05-10-2016, 04:21 PM

Quote:

Originally Posted by lazydog

You should be able to downgrade and then use the '--exclude=' option to exclude bind from being updated.

Is there a way to make that permanent somewhere? I just go on all my servers and run "yum update" every now and then, I'll surely forget to put that flag. :P

But first I'll try to confirm if that really is the bug I have. I just tried that rndc command so next time it happens I'll go check that log.

lazydog · 05-10-2016, 08:47 PM

Quote:

Originally Posted by Red Squirrel

Is there a way to make that permanent somewhere? I just go on all my servers and run "yum update" every now and then, I'll surely forget to put that flag. :P

You could always make a alias for it, for example:

Code:

alias yud='yum update --exclude=bind'

Then when the alias is no longer needed you can simple delete it.

Red Squirrel · 05-15-2016, 07:30 PM

Did it again with logging on. Where do I find the log file? And is there a way to change it so it's in /var/log, that's where all the other logs are.

Edit: NM noticed it was already posted. Found it. That log is HUGE. I will sift through that and try to see if there's anything relevant to post. There are no time stamps so really it's kind of useless.

Red Squirrel · 05-15-2016, 08:31 PM

Log is just filling up with errors, there's definitely something wrong.

Code:

error (no valid RRSIG) resolving 'rambler.ru.dlv.isc.org/DS/IN': 208.67.222.123#53
error (insecurity proof failed) resolving 'rambler.ru.dlv.isc.org/DLV/IN': 208.67.222.123#53
validating @0x7faa4853c8b0: . NS: got insecure response; parent indicates it should be secure
error (insecurity proof failed) resolving './NS/IN': 208.67.220.123#53
  validating @0x7faa280048e0: net SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'sourceforge.net/DS/IN': 208.67.220.123#53
validating @0x7faa44034730: . NS: got insecure response; parent indicates it should be secure
error (insecurity proof failed) resolving './NS/IN': 208.67.222.123#53
validating @0x7faa30013bf0: . NS: got insecure response; parent indicates it should be secure
error (insecurity proof failed) resolving './NS/IN': 208.67.220.123#53
  validating @0x7faa30022840: net SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'minecraft.net/DS/IN': 208.67.220.123#53
validating @0x7faa34050620: . NS: got insecure response; parent indicates it should be secure
error (insecurity proof failed) resolving './NS/IN': 208.67.220.123#53
validating @0x7faa380151c0: . NS: got insecure response; parent indicates it should be secure
error (insecurity proof failed) resolving './NS/IN': 208.67.220.123#53
validating @0x7faa380151c0: . NS: got insecure response; parent indicates it should be secure
error (insecurity proof failed) resolving './NS/IN': 208.67.220.123#53
error (unexpected RCODE REFUSED) resolving '57.27.55.47.list.dsbl.org/TXT/IN': 8.8.8.8#53
error (unexpected RCODE REFUSED) resolving '57.27.55.47.list.dnswl.org/A/IN': 8.8.8.8#53
validating @0x7faa38027d00: . NS: got insecure response; parent indicates it should be secure
error (insecurity proof failed) resolving './NS/IN': 208.67.220.123#53
  validating @0x7faa48527f90: org SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'open-whois.org/DS/IN': 208.67.220.123#53
validating @0x7faa34056c00: . NS: got insecure response; parent indicates it should be secure
error (insecurity proof failed) resolving './NS/IN': 208.67.222.123#53
  validating @0x7faa28036c70: org SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'ahbl.org/DS/IN': 208.67.220.123#53
validating @0x7faa48527300: org DNSKEY: got insecure response; parent indicates it should be secure
error (insecurity proof failed) resolving 'org/DNSKEY/IN': 208.67.220.123#53
error (unexpected RCODE REFUSED) resolving '57.27.55.47.list.dsbl.org/TXT/IN': 8.8.4.4#53
  validating @0x7faa3c0312e0: org SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'spamhaus.org/DS/IN': 208.67.220.123#53
validating @0x7faa28013c50: org DNSKEY: got insecure response; parent indicates it should be secure
error (insecurity proof failed) resolving 'org/DNSKEY/IN': 208.67.222.123#53
  validating @0x7faa380299a0: org SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'ahbl.org/DS/IN': 208.67.222.123#53
  validating @0x7faa30024160: org SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'dnswl.org/DS/IN': 208.67.220.123#53
  validating @0x7faa44037cf0: org SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'bondedsender.org/DS/IN': 208.67.220.123#53
  validating @0x7faa30025170: org SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'w3.org/DS/IN': 208.67.220.123#53
  validating @0x7faa280048e0: com SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'habeas.com/DS/IN': 208.67.220.123#53
  validating @0x7faa34057890: com SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'isipp.com/DS/IN': 208.67.220.123#53
  validating @0x7faa3c02ed30: org SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'dnswl.org/DS/IN': 208.67.222.123#53
  validating @0x7faa28032330: org SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'open-whois.org/DS/IN': 208.67.222.123#53
  validating @0x7faa44037060: net SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'spamcop.net/DS/IN': 208.67.220.123#53
  validating @0x7faa44034730: org SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'w3.org/DS/IN': 208.67.222.123#53
  validating @0x7faa48536f60: org SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'spamhaus.org/DS/IN': 208.67.222.123#53
  validating @0x7faa4003d700: net SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'sorbs.net/DS/IN': 208.67.222.123#53
  validating @0x7faa3c02fd40: com SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'isipp.com/DS/IN': 208.67.222.123#53
  validating @0x7faa44034730: com SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'habeas.com/DS/IN': 208.67.222.123#53
  validating @0x7faa4003aa50: net SOA: got insecure response; parent indicates it should be secure
error (no valid RRSIG) resolving 'spamcop.net/DS/IN': 208.67.222.123#53
error (network unreachable) resolving 'dns1.njabl.org/AAAA/IN': 2001:500:e::1#53
error (network unreachable) resolving 'dns2.njabl.org/AAAA/IN': 2001:500:e::1#53

Is this due to that bug, or something else?

bathory · 05-16-2016, 07:25 AM

Quote:

Is this due to that bug, or something else?

This is a dnssec validation error.
Perhaps due to this error the cache gets full, resulting in named failure.
This is also mentioned in this bug report, with the workaround to disable dnssec, so you can try it to see if it solves your problem:

Code:

dnssec-enable no;
dnssec-validation no

Red Squirrel · 05-16-2016, 12:41 PM

Yeah just tried to disable dnssec as per the suggestion to see if it helps. This is just a local dns server so don't think it really matters, it's not exposed to the internet.

v4r3l0v · 05-17-2016, 12:34 AM

It is very important to discern who is actually writing to resolv.conf on your machine. Is it maybe the home router taking over with its own DHCP, and with it- its own suggestions for using DNS servers instead of what you prescribed?

Red Squirrel · 05-21-2016, 12:01 PM

Quote:

Originally Posted by v4r3l0v

It is very important to discern who is actually writing to resolv.conf on your machine. Is it maybe the home router taking over with its own DHCP, and with it- its own suggestions for using DNS servers instead of what you prescribed?

Not sure what you're tryng to say? There's only really one person with access to the server and resolv.conf is set to correct IP and having it set to a different IP would cause it to fail consistently not just randomly...unless I've been compromised which I can't see how given it's not public facing and all my public facing stuff is on a separate vlan, though that vlan does have access to the DNS server... is there any known exploits for named where there is remote code execution? It would still require that one of the public facing servers is compromised though.

So far so good with DNSSEC off though.... but is that a real solution? Turning something security related off does not sound like a good long term plan, or does it not really matter for a local server? I'll admit I never really read up much on it so not sure of the exact details on how it works.