Linux - SoftwareThis forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
If I have an address, say example.com, and I want to run n number of wgets on it. How can I do this? I'm curious for the reason of checking how wgets caches DNS.
‘--no-dns-cache’
Turn off caching of DNS lookups. Normally, Wget remembers the IP addresses it looked up from DNS so it doesn't have to repeatedly contact the DNS server for the same (typically small) set of hosts it retrieves from. This cache exists in memory only; a new Wget run will contact DNS again.
The last part confuses me. " a new Wget run will contact DNS again." This means if I run a for-loop to call wget on an address, it will just make a new call to DNS every time. How do I avoid this?
In that case wget will save the pages as page.html, page.html.1, page.html.2 and so on. But it won't contact DNS again because it already resolved for the first page.
So then it will be making a new DNS request per page which won't help me. I need to know how long it caches an entry.
It will make new DNS requests each time you invoke wget. If you run it once, with several urls, then it will use the DNS cache, which lasts for the duration of the execution of the program.
It will also use the DNS cache if page A links to B, and page C also links to B. When downloading page C, it won't make a DNS request for page B, which is already in the cache.
The option you mention, --no-dns-cache, will disable this behavior. Take a look at the manpage again:
Code:
--no-dns-cache
Turn off caching of DNS lookups. Normally, Wget remembers the IP
addresses it looked up from DNS so it doesn't have to repeatedly contact
the DNS server for the same (typically small) set of hosts it retrieves
from. This cache exists in memory only; a new Wget run will contact DNS
again.
However, it has been reported that in some situations it is not desirable
to cache host names, even for the duration of a short-running application
like Wget. With this option Wget issues a new DNS lookup (more precisely,
a new call to "gethostbyname" or "getaddrinfo") each time it makes a new
connection. Please note that this option will not affect caching that
might be performed by the resolving library or by an external caching
layer, such as NSCD.
If you don't understand exactly what this option does, you probably won't
need it.
I see now! Well aside from inlining the entire is there another way? The truth of the matter is that I need to compare the file produced in between each call, but I also need to not make a new call to DNS each time...
Last edited by YaMaHaBoB; 06-28-2011 at 07:41 PM.
Reason: typo
I see now! Well aside from inlining the entire is there another way? The truth of the matter is that I need to compare the file produced in between each call, but I also need to not make a new call to DNS each time...
Another way? Hmm... Can't think of one right now. Why is it you can't run it with several urls? Each downloaded file gets a different name, so comparing them afterwards seems possible. The issue with the DNS cache is that it lives for the duration of the wget run, each new invocation creates a new cache. It's not persistent.
It seems to me that you'll need to adjust your script to the behavior of the software. Or change the software. In any case, good luck!
Well, I'm testing a dynamic DNS setup and I'm wanting to see how my system acts when the cached DNS is invalid. I'm trying to redirect the old traffic to a different computer.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.