simple wget questions

YaMaHaBoB · 06-28-2011, 05:54 PM

If I have an address, say example.com, and I want to run n number of wgets on it. How can I do this? I'm curious for the reason of checking how wgets caches DNS.

‘--no-dns-cache’
Turn off caching of DNS lookups. Normally, Wget remembers the IP addresses it looked up from DNS so it doesn't have to repeatedly contact the DNS server for the same (typically small) set of hosts it retrieves from. This cache exists in memory only; a new Wget run will contact DNS again.

The last part confuses me. " a new Wget run will contact DNS again." This means if I run a for-loop to call wget on an address, it will just make a new call to DNS every time. How do I avoid this?

Diantre · 06-28-2011, 06:21 PM

Passing several urls to wget in the same command line, eg: wget url1 url2 url3

YaMaHaBoB · 06-28-2011, 06:36 PM

Even if the URLs are the same?

Diantre · 06-28-2011, 06:51 PM

In that case wget will save the pages as page.html, page.html.1, page.html.2 and so on. But it won't contact DNS again because it already resolved for the first page.

YaMaHaBoB · 06-28-2011, 07:10 PM

So then it will be making a new DNS request per page which won't help me. I need to know how long it caches an entry.

Diantre · 06-28-2011, 07:23 PM

Quote:

Originally Posted by YaMaHaBoB

So then it will be making a new DNS request per page which won't help me. I need to know how long it caches an entry.

It will make new DNS requests each time you invoke wget. If you run it once, with several urls, then it will use the DNS cache, which lasts for the duration of the execution of the program.

It will also use the DNS cache if page A links to B, and page C also links to B. When downloading page C, it won't make a DNS request for page B, which is already in the cache.

The option you mention, --no-dns-cache, will disable this behavior. Take a look at the manpage again:

Code:

       --no-dns-cache
           Turn off caching of DNS lookups.  Normally, Wget remembers the IP
           addresses it looked up from DNS so it doesn't have to repeatedly contact
           the DNS server for the same (typically small) set of hosts it retrieves
           from.  This cache exists in memory only; a new Wget run will contact DNS
           again.

           However, it has been reported that in some situations it is not desirable
           to cache host names, even for the duration of a short-running application
           like Wget.  With this option Wget issues a new DNS lookup (more precisely,
           a new call to "gethostbyname" or "getaddrinfo") each time it makes a new
           connection.  Please note that this option will not affect caching that
           might be performed by the resolving library or by an external caching
           layer, such as NSCD.

           If you don't understand exactly what this option does, you probably won't
           need it.

YaMaHaBoB · 06-28-2011, 07:39 PM

I see now! Well aside from inlining the entire is there another way? The truth of the matter is that I need to compare the file produced in between each call, but I also need to not make a new call to DNS each time...

Diantre · 06-28-2011, 07:56 PM

Quote:

Originally Posted by YaMaHaBoB

I see now! Well aside from inlining the entire is there another way? The truth of the matter is that I need to compare the file produced in between each call, but I also need to not make a new call to DNS each time...

Another way? Hmm... Can't think of one right now. Why is it you can't run it with several urls? Each downloaded file gets a different name, so comparing them afterwards seems possible. The issue with the DNS cache is that it lives for the duration of the wget run, each new invocation creates a new cache. It's not persistent.

It seems to me that you'll need to adjust your script to the behavior of the software. Or change the software. In any case, good luck!

YaMaHaBoB · 06-28-2011, 08:02 PM

Well, I'm testing a dynamic DNS setup and I'm wanting to see how my system acts when the cached DNS is invalid. I'm trying to redirect the old traffic to a different computer.