wget mirroring

LinuxQuestions.org

Review your favorite Linux distribution.

		LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
wget mirroring

Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Welcome to LinuxQuestions.org, a friendly and active Linux Community.

You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!

Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.

Are you new to LinuxQuestions.org? Visit the following links:
Site Howto | Site FAQ | Sitemap | Register Now

If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.

Having a problem logging in? Please visit this page to clear all LQ-related cookies.

Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.

Exclusive for LQ members, get up to 45% off per month. Click here for more info.

Reply

Search this Thread

Old

09-03-2013, 07:13 AM

#1

sumeet inani

Member

Registered: Oct 2008

Posts: 908

Blog Entries: 26

Rep:

Reputation: 49

wget mirroring

Hi ,
I ran

Code:

wget -mk -w 20 -np http://www.ign.com/wikis/saints-row-the-third

hoping that all files in hierarchy below 'www.ign.com/wikis/saints-row-the-third' will get downloaded
But my tree says

Quote:

.
|-- delete-these.txt
|-- wget-log
`-- www.ign.com
|-- robots.txt
`-- wikis
|-- content-ratings
| |-- allpages
| |-- longpages
| |-- shortpages
| `-- wantedpages
|-- how-to
| |-- upload
| | |-- File:A_solid.png
| | |-- File:A_solid_small.png
| | |-- File:Arrow_down-left_solid.png
| | |-- File:Arrow_down-right_solid.png
| | |-- File:Arrow_down_solid.png
| | |-- File:Arrow_left_solid.png
| | |-- File:Arrow_right_solid.png
| | |-- File:Arrow_up-left_solid.png
| | |-- File:Arrow_up-right_solid.png
| | |-- File:Arrow_up_solid.png
| | |-- File:B_solid.png
| | |-- File:B_solid_small.png
| | |-- File:Full_circle.png
| | |-- File:Guard_solid.png
| | |-- File:Guard_solid_small.png
| | |-- File:Kick_solid.png
| | |-- File:Kick_solid_small.png
| | `-- File:Neutral.png
| `-- upload.1
`-- saints-row-the-third
|-- 3_Count_Beat_Down
|-- A_Better_Person
|-- A_Remote_Chance
|-- Abilities_Upgrades
|-- Achievements
|-- Achievements_
| |-- Character_Gallery
| |-- Official_Soundtrack
| `-- _Trophies
|-- Achievements_&_Trophies
|-- Achievements_or_Trophies
|-- Act_1_Missions
|-- Act_2_Missions
|-- Act_3_Missions
|-- Activities
|-- Air_Steelport
|-- Ambulance
|-- Anchor
|-- And_Boom_Goes_the_Dynamite
|-- Angel's_Missions
|-- Apocalypse_Genki
|-- Assassinations
|-- Attrazione
|-- Bo-Duke-En
|-- Bonus_Upgrades
|-- Bonuses_Upgrades
|-- Bootlegger
|-- Bright_Lights,_Big_City
|-- C-C-C-Combo_Breaker
|-- Campaign_Walkthrough
|-- Cat_on_a_Hot_Tin_Roof
|-- Challenger
|-- Challenges
|-- Character_Gallery
|-- Cheats
|-- Combat_Upgrades
|-- Commander
|-- Commander_Cyrus
|-- Condor
|-- Convoy_Decoy
|-- Cooked_To_Perfection
|-- Cowboy_Up
|-- Cyber_Blazing
|-- Cyber_Blazing_(Mission)
|-- DLC
|-- Damage_Upgrades
|-- Dead_Presidents
|-- Deckers.Die
|-- Double_Dose_of_Pimping
|-- Eagle
|-- Easter_Eggs
|-- Easter_Eggs_and_Secrets
|-- End_of_Act_2_Missions
|-- Enter_the_Dominatrix
|-- Escort
|-- Escort_(Mission)
|-- Escort_(Mission)_&_Painting_a_Picture
|-- Estrada
|-- Everything_Is_Permitted
|-- Eye_of_the_Tiger
|-- Eye_of_the_Tiger_&_Face_Your_Fear
|-- Face_Your_Fear
|-- Faster,_More_Intense
|-- Feeding_Time
|-- Fence_Killa_2011
|-- File:Sr3-090.jpg
|-- File:Sr3-091.jpg
|-- File:Sr3-092.jpg
|-- File:Sr3-093.jpg
|-- File:Sr3-094.jpg
|-- File:Sr3-095.jpg
|-- File:Sr3-096.jpg
|-- File:Sr3-097.jpg
|-- File:Sr3-098.jpg
|-- File_talk:Sr3-090.jpg
|-- File_talk:Sr3-091.jpg
|-- File_talk:Sr3-092.jpg
|-- File_talk:Sr3-093.jpg
|-- File_talk:Sr3-094.jpg
|-- File_talk:Sr3-095.jpg
|-- File_talk:Sr3-096.jpg
|-- File_talk:Sr3-097.jpg
|-- File_talk:Sr3-098.jpg
|-- Final_Mission:_Three_Way
|-- Flame_On
|-- Flash_the_Pan
|-- Gang_Abilities_Upgrades
|-- Gangstas..._In_Space!
|-- Gangstas_In_Space
|-- Gangstas_in_Space
|-- Gangstas_in_Space_(DLC_Pack)
|-- Gatmobile
|-- Gellin'_Like_Magellan
|-- Gender_Equality
|-- Genki_Bowl_7
|-- Genki_Bowl_Champ
|-- Genki_Bowl_VII
|-- Genkibowl_7
|-- Genkibowl_VII
|-- Get_Off_My_Back
|-- Getting_the_Goods
|-- Glitches
|-- Go_Into_the_Light
|-- Gotta_Break_Em_In
|-- Guardian_Angel
|-- Guardian_Angel_(Kinzie)
|-- Guardian_Angel_(Mission)
|-- Guardian_Angel_(Pierce)
|-- Hack_the_Planet
|-- Hangar_18_1
| |-- 2
| |-- Faster,_More_Intense
| `-- That's_Not_in_the_Script
|-- Hanging_With_Mr._Pierce
|-- Haters_Gonna_Hate
|-- Have_a_Reality_Climax
|-- Health_Upgrades
|-- Heli_Assault
|-- Heli_Assault_(Mission)
|-- Heli_Assault_(Mission)_&_Phone_Phreak
|-- Hi-Jack_It
|-- Ho_Traffic
|-- Ho_Traffic_(Mission)
|-- I'm_Free_-_Free_Falling
|-- I_Heart_Nyte_Blayde
|-- Ign:Navigation
|-- Ign_talk:Navigation
|-- Insurance_Fraud
|-- Insurance_Fraud_(Mission)
|-- Jumped_In
|-- Kanada
|-- Kenshin
|-- Kill-deckers.exe
|-- Killbane
|-- Killing_Killbane
|-- Kingpin
|-- Kinzie's_Missions
|-- Knoxville
|-- Korbra
|-- Krukov
|-- Kuh,_Boom.
|-- Learning_Computer
|-- Life_of_the_Party
|-- Live_With_Killbane
|-- Love
| `-- Hate_Relationship
|-- Main_Page
|-- Matt_Miller
|-- Mayhem
|-- Mayhem_(Mission)
|-- Miami
|-- Mission_1
|-- Mission_2
|-- Mission_3
|-- Mourning_Stars
|-- Mr._Fury_Would_Be_Proud
|-- Municipal
|-- MurderBrawl_XXXI
|-- Murder_in_the_Jungle
|-- Murderbrawl_31
|-- My_Name_Is_Cyrus_Temple
|-- Nforcer
|-- Nyte_Blade's_Return
|-- Nyte_Blade_Pack
|-- Official_Soundtrack
|-- Oleg's_Missions
|-- Once_Bitten..._Braaaaaaains
|-- Opulence,_You_Has_It
|-- Ouch.
|-- Ow_My_Balls!
|-- Painting_a_Picture
|-- Party_Time
|-- Peacemaker
|-- Phoenix
|-- Phone_Phreak
|-- Pierce's_Act_2_Missions
|-- Pierce's_Missions
|-- Pimped_Out_Pad
|-- Pimps_Up,_Hos_Down
|-- Porkchop_Sandwiches
|-- Prof._Genki
|-- Professor_Genki's_S.E.R.C.
|-- Professor_Genki's_S.E.R.C._(Mission)
|-- Professor_Genki's_Super_Ethical_Reality_Climax
|-- Prologue
|-- RPG
|-- Reaper
|-- Repaircar
|-- Return_to_Steelport
|-- SAINTS_ROW_DLC
|-- STAG_Film
|-- STAG_Party
|-- STAG_Party_&_Stilwater_Blues
|-- Sad_Panda_Skyblazing
|-- Saints_Book
|-- Saints_Row_The_Third_Easter_Eggs
|-- Saintsbook
|-- Sandstorm
|-- Satchel_Charge
|-- Saving_Shaundi
|-- Saving_Shaundi_and_Viola
|-- Secrets
|-- Send_in_the_Clones
|-- Sexy_Kitten_Yarngasm
|-- Shake_and_Bake
|-- Shark
|-- Shaundi
|-- Shaundi's_Missions
|-- Sheperd
|-- Skies_Over_Steelport
|-- Snatch
|-- Snatch_(Mission)
|-- Spectre
|-- Squasar
|-- Status_Quo
|-- Stay_Classy_Steelport
|-- Steelport
|-- Steelport_Here_I_Am
|-- Stick_the_Landing
|-- Stilwater_Blues
|-- Stop_All_the_Downloading
|-- Storm_the_Yarn
|-- Super_Ethical_PR_Opportunity
|-- Takeover_the_City
|-- Talk:Achievements_
| `-- Official_Soundtrack
|-- Talk:Assassinations
|-- Talk:Bo-Duke-En
|-- Talk:Challenges
|-- Talk:Cheats
|-- Talk:Cowboy_Up
|-- Talk

LC
|-- Talk

amage_Upgrades
|-- Talk:Easter_Eggs
|-- Talk:Easter_Eggs_and_Secrets
|-- Talk:Gangstas_in_Space
|-- Talk:Gangstas_in_Space_(DLC_Pack)
|-- Talk:GenkiBowl_VII
|-- Talk:Genki_Bowl_7
|-- Talk:Genki_Bowl_VII
|-- Talk:Genkibowl_7
|-- Talk:Genkibowl_VII
|-- Talk:Glitches
|-- Talk:Guardian_Angel_(Pierce)
|-- Talk:I'm_Free_-_Free_Falling
|-- Talk:Official_Soundtrack
|-- Talk:Ow_My_Balls!
|-- Talk:Party_Time
|-- Talk:Prof._Genki
|-- Talk:Saints_Row_The_Third_Easter_Eggs
|-- Talk:Secrets
|-- Talk:Send_in_the_Clones
|-- Talk:Steelport_Here_I_Am
|-- Talk:Tour_de_Farce
|-- Talk:VTOL
|-- Talk:Vehicle_Thefts
|-- Talk:Weapons,_Outfits,_and_Vehicles
|-- Talk:Weird_Science
|-- Talk:When_Good_Heists_Go_Bad
|-- Tank_Mayhem
|-- Tank_Mayhem_(Mission)
|-- Tank_You_Very_Much
|-- Taxi
|-- That's_Not_in_the_Script
|-- Thats_not_in_the_Script
|-- The_American_Dream
|-- The_Belgian_Problem
|-- The_Ho_Boat
|-- The_Trouble_With_Clones
|-- The_Welcome_Wagon
|-- Third_and_30
|-- Three_Way
|-- Tiger_Escort
|-- Titan
|-- Titanic_Effort
|-- Toad
|-- Tornado
|-- Tour_de_Farce
|-- Tower_Defense
|-- Trafficking
|-- Trafficking_(Mission)
|-- Trail_Blazing
|-- Trail_Blazing_(Mission)
|-- Trojan_Whores
|-- Trophies
|-- Tune_In,_Drop_Off
|-- Unlockables
|-- Upgrades
|-- VTOL
|-- Vehicle_Thefts
|-- Vehicles_Upgrade
|-- Vehicles_Upgrades
|-- Vehicles_and_Weapons
|-- Vortex
|-- Vulture
|-- We're_Going_To_Need_Guns
|-- We're_Takin'_Over
|-- We've_Only_Just_Begun
|-- We've_Only_Just_Begun_&_Hit_the_Powder_Room
|-- Weapons,_Outfits,_and_Vehicles
|-- Weapons,_Outfits_and_Vehicles
|-- Weapons_and_Vehicles
|-- Weird_Science
|-- When_Good_Heists_Go_Bad
|-- Who_Loves_Ya_Baby
|-- Whore_gang_customization
|-- Widowmaker
|-- Woodpecker
|-- You're_My_Hero!
|-- You're_the_Best...
|-- Your_Backseat_Smells_Funny
|-- Zimos's_Missions
|-- Zombie_Attack
|-- allpages
|-- deadendpages
|-- edit
| |-- 1
| | |-- Achievements_
| | | `-- _Trophies
| | |-- DLC
| | `-- Send_in_the_Clones
| |-- 2
| | `-- DLC
| `-- Three Way
|-- empty
| |-- Commander_Cyrus
| |-- Hangar_18_1
| | `-- That's_Not_in_the_Script
| |-- Matt_Miller
| |-- Shaundi
| |-- Steelport
| |-- Talk:Gangstas_in_Space_(DLC_Pack)
| |-- Talk:Guardian_Angel_(Pierce)
| |-- Talk:I'm_Free_-_Free_Falling
| |-- Traill_Blazing
| `-- You're_the_Best...
|-- flag
| |-- Achievements_
| | `-- Official_Soundtrack
| |-- Cowboy_Up
| |-- DLC
| |-- Easter_Eggs_and_Secrets
| |-- Gangstas_in_Space
| |-- Gangstas_in_Space_(DLC_Pack)
| |-- Genki_Bowl_7
| |-- Genki_Bowl_VII
| |-- Genkibowl_7
| |-- Genkibowl_VII
| |-- Glitches
| |-- Guardian_Angel_(Pierce)
| |-- I'm_Free_-_Free_Falling
| |-- Ign:Navigation
| |-- Party_Time
| |-- Secrets
| |-- Steelport_Here_I_Am
| |-- VTOL
| |-- Vehicle_Thefts
| |-- Weapons,_Outfits,_and_Vehicles
| `-- When_Good_Heists_Go_Bad
|-- history
| |-- Achievements_
| | `-- Official_Soundtrack
| |-- Assassinations
| |-- Bo-Duke-En
| |-- Challenges
| |-- Cheats
| |-- Cowboy_Up
| |-- DLC
| |-- Damage_Upgrades
| |-- Easter_Eggs
| |-- Easter_Eggs_and_Secrets
| |-- File:Sr3-090.jpg
| |-- File:Sr3-091.jpg
| |-- File:Sr3-092.jpg
| |-- File:Sr3-093.jpg
| |-- File:Sr3-094.jpg
| |-- File:Sr3-095.jpg
| |-- File:Sr3-096.jpg
| |-- File:Sr3-097.jpg
| |-- File:Sr3-098.jpg
| |-- Gangstas_in_Space
| |-- Gangstas_in_Space_(DLC_Pack)
| |-- Genki_Bowl_7
| |-- Genki_Bowl_VII
| |-- Genkibowl_7
| |-- Genkibowl_VII
| |-- Glitches
| |-- Guardian_Angel_(Pierce)
| |-- I'm_Free_-_Free_Falling
| |-- Ign:Navigation
| |-- Official_Soundtrack
| |-- Ow_My_Balls!
| |-- Party_Time
| |-- Prof._Genki
| |-- Saints_Row_The_Third_Easter_Eggs
| |-- Secrets
| |-- Send_in_the_Clones
| |-- Steelport_Here_I_Am
| |-- Tour_de_Farce
| |-- VTOL
| |-- Vehicle_Thefts
| |-- Weapons,_Outfits,_and_Vehicles
| |-- Weird_Science
| `-- When_Good_Heists_Go_Bad
|-- longpages
|-- orphanedpages
|-- recentchanges
|-- shortpages
|-- topcontributors
|-- wantedpages
`-- wikitools

20 directories, 423 files

Also I do not want wget to download the file again if it changes during process I got lots of these like ./www.ign.com/wikis/saints-row-the-third/Easter_Eggs_and_Secrets?revision=899

Old

09-03-2013, 07:17 AM

#2

sumeet inani

Member

Registered: Oct 2008

Posts: 908

Original Poster

Blog Entries: 26

Rep:

Reputation: 49

Quote:

-m
--mirror
Turn on options suitable for mirroring. This option turns on
recursion and time-stamping, sets infinite recursion depth and
keeps FTP directory listings. It is currently equivalent to -r -N
-l inf --no-remove-listing.

must be the problem

09-03-2013, 08:24 AM	#3
sumeet inani Member Registered: Oct 2008 Posts: 908 Original Poster Blog Entries: 26 Rep:	This time I tried wget -k -r -l inf -np http://www.ign.com/wikis/saints-row-the-third Still i get many folders inside www.ign.com/wikis/ other than saints-row-the-third Anybody there ? Last edited by sumeet inani; 09-05-2013 at 09:23 AM.

Reply

Posting Rules
You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are On [IMG] code is Off HTML code is Off Forum Rules

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
How to resume an interrupted wget using wget.log?	misterJ	Linux - Software	2	06-19-2011 01:21 PM
wget not working! but "man wget is" ???	wrapster	Solaris / OpenSolaris	5	07-30-2008 03:00 AM
Problems mirroring an asp website with wget	el_pajaro!	Linux - General	1	02-16-2008 11:19 PM
wget mirroring issue - GET variables	Freezy	Linux - Networking	4	02-17-2006 06:31 PM
wget problem with mirroring	Tinku	Linux - General	3	01-15-2006 08:18 AM

All times are GMT -5. The time now is 09:17 PM.

Main Menu

Advertisement

My LQ

Write for LQ

LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.

Main Menu

Syndicate

Latest Threads

LQ News

Twitter: @linuxquestions

Open Source Consulting | Domain Registration