Very informative post Scowles!

And Red Squirrel: Have you considered using hosts files on all the clients just for your internal network? Just until you get all your DNS setup the way you intend.
Or maybe using a proxy server for caching content/names you intend to collect.

Based on your last reply, it sounds like all you need is a simple name server configuration that loads a couple of zones for your private tld (not queried from the internet, just hosts behind your firewall) and the root hints zone so that it can also resolve (and cache) non-authoritative queries. If that's the case, then the dns/bind webmin interface should allow you to configure and create zones without any problems.

Exactly what I need. And to avoid confusion I'll keep my custom domain/zone as another issue once I get the cache part working properly.

Right now my dns only caches for like 8 hours and it keeps it in memory... not really efficient nor reliable. So I want it to cache it as a dns entry like any other dns server would do, and only refetch it ever 3 days or so. But main thing is, I want it to be stored on the hard disk, just like a normal dns entry would so that if I go to bed or reboot the server, I can type in an address and it will still resolve it locally.

For example, last night I went to bed, so this morning I came back, and it had to all refetch everything, and not only that but half the hosts did not even want to resolve properly the first time. Basically the server had to "warm up" before it got useful, took about an hour or so for things to pick up. That's not acceptable. I need something that even if all the root servers go down, that it will keep the records forever, until they come back up and that it's been over NN days. Of course that won't happen, but that's how I want it, so that it rarely has to access the root servers for queries.

When someone changes their domain name it takes up to 3 days to change anyway, and it's because it has to propatigate. So I want my server to be one of those servers it propatigates to (in my case when I go to the address for the first time it will create the entry).

Is this even possible or is it asking for too much?

I just never realized DNS was this complicated. I was hoping for it to be like editing a host file, and to me it should be that way, no use making a simple protocol complicated. I guess there must be a reason behind it.

Thanks for the help! I also have webmin installed, but when I go in the dns section I have no clue about anything.

DNS changes propagation involves zone transfer. As zone transfer, when possible, is a method used by crackers to get network fingerprints, authoritative DNS servers should *not* allow zone transfer to any host. You may try to get a zone transfer with the command : run against a name server (yours for instance).

The general mechanism is : authoritative server for domain x will notify (parameter in zone file) secondary servers. Secondary servers may then transfer zone file, and update. Notifications and downloads do not occur on-the-fly but at intervals that are not set too short to avoid overloading the servers and to save bandwidth, which explain that propagation can take time.

Servers can compare zone transfer with the serial number (that should be incremented with each modification) and the timestamp.

Thus, if you are dreaming about getting a copy of the zones your computer queries, I'm afraid it will remain that : a dream ;-)

So it's impossible to even create entries as I visit sites? Since I only have a 120GB hard drive, not a 120TB one, so I realize I can't store the entire internet on here int erms of domain info, but is it possible to simply make it so the time out is like 3 days, and that it caches it on the hard disk instead of memory? My ISP's DNS server has been going up and down non stop, so I'm basically relying on my own DNS server, which sucks at the moment because it refetches the info every 8 hours, so when I go to bed and come back the next day it takes bloody ages to resolve everything. And I don't have enough ram to store them all. (assuming I may need to store about 100 domains/subdomains per day).

Also if there's a completly different DNS server software that would serve this purpose, I'd be willing to switch, as long as that software also allows making a custom top level domain, since I'll want to do that later.

Well, the problem is, if your DNS server caches stuff for three days, it will be up to three days before you can successfully resolve that address again. It seems that the Bind crew made the decision not to support this idea, leaning in favour of 'correctness'. I agree it should be possible to cache things as long as you want, but there needs to be a mechanism for forcing an update or it will cause you problems.

BUT! If your DNS server is set up to query the root servers and move on from there, you will not be relying on your ISP's DNS servers at all, so I think there really is no problem.

If you still want to go down this path, I can't see software for Linux, but I can see a few DNS servers for Windows that will let you do this. Bind-PE seems to be one, as does one called "Monolith". Caveat: I have not tried either of these, I only found them searching for DNS and "persistent cache".

I meant to say "It COULD be up to three days", i.e if the DNS info actually changes just after you have refreshed your cache, you will not see another refresh for three days because your server will believe its own information correct.

Apparently I am an idiot - Monolith does run under Linux.