LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 12-08-2010, 11:43 AM   #1
diamond_D
Member
 
Registered: Jul 2005
Posts: 51

Rep: Reputation: 0
wget question


Hi,

I'm new to using the wget command to download files from the internet.

I was wondering if it's possible to search a website for the files if you know some details but not exactly what you are looking for.

For example:

On website http://download.mysite.com/linux
There is a file called testA.0710.tar.gz There are also other older versions posted on the website, labelled testA.0702.tar.gz and testA.0614.tar.gz

Can I use wget to list all available posted versions so I can choose to download the latest?

Can it be used to perform a recursive search? Using the above example if I knew the file existed on http://download.mysite.com but did not know about "/linux"
 
Old 12-08-2010, 12:09 PM   #2
sag47
Senior Member
 
Registered: Sep 2009
Location: Orange County, CA
Distribution: Kubuntu x64, Raspbian, CentOS
Posts: 1,850
Blog Entries: 36

Rep: Reputation: 455Reputation: 455Reputation: 455Reputation: 455Reputation: 455
Like many commands in a Unix based OS wget does one thing but it does it damn good. Wget downloads files. That's it.

There are other ways of doing what you want such as using Python. Check out urllib.

Why work hard when someone else has already done it?

You can even use a Python Regular Expression to see if you have the latest version of the software and if not, automatically download it and run a script to install it.

Last edited by sag47; 12-08-2010 at 12:11 PM.
 
1 members found this post helpful.
Old 12-08-2010, 01:56 PM   #3
Mr. Alex
Senior Member
 
Registered: May 2010
Distribution: No more Linux. Done with it.
Posts: 1,238

Rep: Reputation: Disabled
I'm sure it's possible to create a script for this work. But I'm not a shell programmer.
 
Old 12-08-2010, 03:21 PM   #4
sag47
Senior Member
 
Registered: Sep 2009
Location: Orange County, CA
Distribution: Kubuntu x64, Raspbian, CentOS
Posts: 1,850
Blog Entries: 36

Rep: Reputation: 455Reputation: 455Reputation: 455Reputation: 455Reputation: 455
I slightly modified that guys scripts to show more output. I didn't change its functionality though.

fetch-music.py
Code:
#!/usr/bin/env python
# @source: http://ubuntuforums.org/showthread.php?t=1542894
# This script downloads a group of web pages, parses all the music links,
# and then downloads all of the music by artist and title.
import urllib, re

def do_page(url):
    f = urllib.urlopen(url)
    html = f.read()
    pattern = r'title="Download free MP3: (.*?) \(.*?\)" href="(.*?)"'
    hits = re.findall(pattern, html)
    return hits

if __name__ == '__main__':
    hits = []
    for i in range(1, 26):
        url = 'http://www.last.fm/music/+free-music-downloads?page=%d' % i
        print "Fetching " + url
        hits.extend(do_page(url))

    fh = open('links', 'wb')
    for hit in hits:
        fh.write('%s\t%s\n' % (hit[1], hit[0]))
    fh.close()
download.sh
Code:
#!/bin/bash
while read url title
do
  wget "$url" -O "$title".mp3
done
So you could easily run the command like this.
Code:
chmod 755 ./fetch-music.py ./download.sh
./fetch-music.py && ./download.sh < ./links
It is a pretty interesting script and simple!

Last edited by sag47; 12-08-2010 at 03:24 PM.
 
1 members found this post helpful.
Old 12-09-2010, 09:26 AM   #5
diamond_D
Member
 
Registered: Jul 2005
Posts: 51

Original Poster
Rep: Reputation: 0
Wow! It pays to know Python.

Thanks for the input guys.
 
Old 12-09-2010, 08:39 PM   #6
sag47
Senior Member
 
Registered: Sep 2009
Location: Orange County, CA
Distribution: Kubuntu x64, Raspbian, CentOS
Posts: 1,850
Blog Entries: 36

Rep: Reputation: 455Reputation: 455Reputation: 455Reputation: 455Reputation: 455
No problem! Here's another example of that same code.

I modified it to grab a proxy list.

Last edited by sag47; 12-12-2010 at 10:48 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
wget question kpachopoulos Linux - General 1 07-23-2005 11:00 AM
wget question Yohhan Linux - Software 1 05-05-2004 08:49 PM
wget question ziggie216 Linux - Software 1 10-12-2003 02:10 PM
wget question ziggie216 Linux - Software 0 10-12-2003 01:02 PM
wget question satimis Linux - Software 4 07-14-2003 05:25 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 01:41 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration