BLFS: wget --spider reports broken link even though the link is working?

McZ · 03-06-2013, 03:37 AM

I am using wget in my scripts to download the source packages before I build. Each package has a list of mirrors, the scripts try with wget --spider to see if the file is available.

This is working on all packages except git. Running

Code:

wget --spider http://git-core.googlecode.com/files/git-1.8.1.5.tar.gz

reports broken link, but running

Code:

wget http://git-core.googlecode.com/files/git-1.8.1.5.tar.gz

downloads the file. Anyone knows why?

EDIT: I am using wget --spider before the actual download because adding lots of switches to the actual download is destroying my % parsing

Code:

wget --progress=dot $url | grep --line-buffered "%" | sed -u -e "s,'.,,g' | awk '{ printf("\b\b\b\b%4s", $2)}'

Keith Hedger · 03-06-2013, 04:37 AM

As the man page for wget says spider needs more work you may be able to use this:

Code:

 wget --no-clobber --no-check-certificate https://git-core.googlecode.com/files/git-1.8.1.5.tar.gz
File 'git-1.8.1.5.tar.gz' already there; not retrieving.

For example it will download the file if it needs to or not, or maybe try a specially made spider program.

McZ · 03-07-2013, 12:40 AM

Quote:

Originally Posted by Keith Hedger

As the man page for wget says spider needs more work you may be able to use this:

Code:

 wget --no-clobber --no-check-certificate https://git-core.googlecode.com/files/git-1.8.1.5.tar.gz
File 'git-1.8.1.5.tar.gz' already there; not retrieving.

For example it will download the file if it needs to or not, or maybe try a specially made spider program.

if I use the above line wget will download the file before I use my % parsing line to download the file. thus the file will be downloaded twice, or not at all if I add -nc to my % parsing line.

for some reason

Code:

wget --progress=dot $url | grep --line-buffered "%" | sed -u -e "s,'.,,g' | awk '{ printf("\b\b\b\b%4s", $2)}'

is always returning success, so I can't rely on this failing to test the next mirror. That is why I use the wget --spider function, to check if the URL is valid before downloading or moving on to next mirror.