LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Linux From Scratch
User Name
Password
Linux From Scratch This Forum is for the discussion of LFS.
LFS is a project that provides you with the steps necessary to build your own custom Linux system.

Notices

Reply
 
Search this Thread
Old 03-06-2013, 03:37 AM   #1
McZ
Member
 
Registered: Aug 2012
Posts: 42

Rep: Reputation: Disabled
BLFS: wget --spider reports broken link even though the link is working?


I am using wget in my scripts to download the source packages before I build. Each package has a list of mirrors, the scripts try with wget --spider to see if the file is available.

This is working on all packages except git. Running
Code:
wget --spider http://git-core.googlecode.com/files/git-1.8.1.5.tar.gz
reports broken link, but running
Code:
wget http://git-core.googlecode.com/files/git-1.8.1.5.tar.gz
downloads the file. Anyone knows why?

EDIT: I am using wget --spider before the actual download because adding lots of switches to the actual download is destroying my % parsing

Code:
wget --progress=dot $url | grep --line-buffered "%" | sed -u -e "s,'.,,g' | awk '{ printf("\b\b\b\b%4s", $2)}'

Last edited by McZ; 03-06-2013 at 03:42 AM.
 
Old 03-06-2013, 04:37 AM   #2
Keith Hedger
Member
 
Registered: Jun 2010
Location: Devon,UK
Distribution: Linux From Scratch, Slackware64,
Posts: 836

Rep: Reputation: 167Reputation: 167
As the man page for wget says spider needs more work you may be able to use this:
Code:
 wget --no-clobber --no-check-certificate https://git-core.googlecode.com/files/git-1.8.1.5.tar.gz
File 'git-1.8.1.5.tar.gz' already there; not retrieving.
For example it will download the file if it needs to or not, or maybe try a specially made spider program.
 
Old 03-07-2013, 12:40 AM   #3
McZ
Member
 
Registered: Aug 2012
Posts: 42

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Keith Hedger View Post
As the man page for wget says spider needs more work you may be able to use this:
Code:
 wget --no-clobber --no-check-certificate https://git-core.googlecode.com/files/git-1.8.1.5.tar.gz
File 'git-1.8.1.5.tar.gz' already there; not retrieving.
For example it will download the file if it needs to or not, or maybe try a specially made spider program.
if I use the above line wget will download the file before I use my % parsing line to download the file. thus the file will be downloaded twice, or not at all if I add -nc to my % parsing line.

for some reason
Code:
wget --progress=dot $url | grep --line-buffered "%" | sed -u -e "s,'.,,g' | awk '{ printf("\b\b\b\b%4s", $2)}'
is always returning success, so I can't rely on this failing to test the next mirror. That is why I use the wget --spider function, to check if the URL is valid before downloading or moving on to next mirror.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
wget indirect link downloading lamachine Linux - Server 2 04-02-2012 08:39 PM
Submitting a link with Wget and Proxy? acctman Linux - General 1 08-10-2011 01:03 PM
wget - force link conversion for all links? allasso Linux - General 2 02-15-2009 08:34 AM
wget - problems making it ignore a link jlpktnst Linux - Software 2 06-09-2006 06:31 AM
link (broken) newlin Linux - Software 1 11-22-2003 02:20 PM


All times are GMT -5. The time now is 07:08 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration