I setup bind 9.10.4-P3 as a caching DNS server. It is very slow to respond.

There seem to be two problems.

When a query result is not in the cache it takes a very long time to query other servers, often returning SRVRFAIL. The overall response time is over two seconds

Even when the result already is cached it somtimes takes a long time for it to response.

The machine it runs on is a quad-core and is not busy at all.

Below is rndc stats report. I think it shows decent response times, so I don't understand why the actual response time is so different.

I suspect you are set up to try IPv6, timeout, then try IPv4.

I think the root cause is a bad DNS peer (that returns SERVFAIL) or a network/link problem (truncated responses, query timeouts).
Please run ping to the peers (i.e. the configured forwarders) and watch the errors.

Thanks smallpond, I re-configured the server by adding OPTIONS="-4" to /etc/sysconfig/named and filter-aaaa-on-v4 yes; to the options section in /etc/named.conf. It already was not listening for ipv6 since that line was commented out (# listen-on-v6 port 53 { ::1; }. I restarted the server and will see if it works better.

Thanks MadeInGermany, this server does not forwards requests, rather it resolves them by working its way from the root servers.

One more data tidbit. With the new configuration and an empty cache, notice how running dig from another machine (connected on the same network, it takes dig 7 seconds to return the result, even though it reports the query took 1.7 seconds.

I was running tcpdump on the machine performing the query to see the actual request response times to the server
As you can see it did take 7 seconds for the response to arrive (the requesting machine asked a second time after waiting 5 seconds for a response to the first request)

On the server side, looking in named log file, you can see the first request arrives immediately after it was sent from the client machine, nothing happens for 5 seconds, the second request arrives (client machine retries), named detects it is a duplicate.

Then suddenly named wakes up smells the coffee and goes to work

As you can see from the moment named went to work, it took 168 milliseconds for it to return a result. However, named reported the query time as 1.7 seconds, that is, from the time of the second request. (That's a named bug imho.) But what was it doing for the previous almost 1.5 seconds from the second request and almost 7 seconds from the first request?

smallpond, I made the changes, no observable improvement. I found a workaround which helps some: I enabled prefetch. That keeps cached entries up to date, helps with frequent requests.

Still, when named needs to query an external DNS server, it always takes 6-7 seconds. Interestingly, if I query that same server directly, for example using dig, it returns the answer quite quickly. For example, from named log when looking for pop.googlemail.com

As you can see named spent over 6 seconds doing something, logging "dns_adb_destroyfind" before it bothered to query google's name server.

Querying the same name server directly

It could be a firewall and source port randomization.

I ran wireshark on the dns machine looking for communication between it, the client machine and the outside world
Below see the request comming from 10.0.0.16 (client) to 10.0.0.2 (dns server) The server immediately queries the resolver for cnn.com and receives a response. It then queries another resolve and recieves the same response, etc. This goes on for a while (seems like named bug?) until the client times out and retires the request, at which point named sends back SRVFAIL (another bug?)

This seemed to have solved the problem

http://serverfault.com/questions/722...recursion-slow

I do not run the OS in a VM, however, the default setting was to offload checksum calculations

Before
Disable offloading
After

Thanks for sharing this!
If your NIC produces wrong checksums, you can as well go fo a NIC firmware update.