LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware
User Name
Password
Slackware This Forum is for the discussion of Slackware Linux.

Notices


Reply
  Search this Thread
Old 12-30-2009, 04:41 PM   #1
jong357
Senior Member
 
Registered: May 2003
Location: Columbus, OH
Distribution: DIYSlackware
Posts: 1,914

Rep: Reputation: 52
Need some wget help


Greets. Been messing around with this for quite some time. I've man'ed it, read the online manual and just plain googled it but I can't get wget to do what I need it to do...

Need to stick with wget is the thing. I know whatever-ftp would be way better but I'm trying to get wget to do it. Perhaps someone who knows wget better than I do can throw me a bone.

All I want is the last directory listed below. It recursively works backwards which I find really odd. That leads you to mirroring the entire slackware.cs.utah.edu site. All I want is the slackware64 directory.

Any clues? Thanks. I've tried a whole host of options and haven't found the right combination yet...

Code:
wget -e robots=off --wait 1 -r --reject "index.html*" --no-host-directories http://slackware.cs.utah.edu/pub/slackware/slackware64-13.0/slackware64
 
Old 12-30-2009, 04:44 PM   #2
Didier Spaier
LQ Addict
 
Registered: Nov 2008
Location: Paris, France
Distribution: Slint64-15.0
Posts: 11,057

Rep: Reputation: Disabled
-np will do.

Or --no-parent if you prefer the verbose form.

Last edited by Didier Spaier; 12-30-2009 at 04:46 PM.
 
Old 12-30-2009, 04:45 PM   #3
rweaver
Senior Member
 
Registered: Dec 2008
Location: Louisville, OH
Distribution: Debian, CentOS, Slackware, RHEL, Gentoo
Posts: 1,833

Rep: Reputation: 167Reputation: 167
You want the depth option "-l 1" (or is it 0?)

edit:
Code:
-l depth
       --level=depth
           Specify recursion maximum depth level depth.  The default maximum
           depth is 5.

Last edited by rweaver; 12-30-2009 at 04:46 PM.
 
Old 12-30-2009, 04:51 PM   #4
jong357
Senior Member
 
Registered: May 2003
Location: Columbus, OH
Distribution: DIYSlackware
Posts: 1,914

Original Poster
Rep: Reputation: 52
I've tried both those before....

It'll start working backwards and get everything in:

http://slackware.cs.utah.edu/pub/sla...ckware64-13.0/

as well. Once it gets all that, it will move back another directory and start downloading every slackware version made....

Been fighting with this for a couple hours off and on.
 
Old 12-30-2009, 04:52 PM   #5
Didier Spaier
LQ Addict
 
Registered: Nov 2008
Location: Paris, France
Distribution: Slint64-15.0
Posts: 11,057

Rep: Reputation: Disabled
I don't think -l will help as the OP don't want what is *above* slackware64 in the tree.

BTW to keep the wole distribution in sync I do this :
rsync -avz --del rsync://rsync.slackware.at/slackware/slackware64-13.0 /some/local/directory/
 
Old 12-30-2009, 04:58 PM   #6
jong357
Senior Member
 
Registered: May 2003
Location: Columbus, OH
Distribution: DIYSlackware
Posts: 1,914

Original Poster
Rep: Reputation: 52
Code:
wget -e robots=off --wait 1 -r --reject "index.html*" -np -l1 http://slackware.cs.utah.edu/pub/slackware/slackware64-13.0/slackware64
That doesn't recursively work backwards but it also doesn't grab any of the packages. It just makes the directory tree of all the package sets and then stops....

Arggg.....

Believe me, rsync would be my first choice but it's for a script and I need to keep the deps down to basic linux commands, one of which I consider wget to be. That and I already use wget in the script for a few other things.

So frustrating....
 
Old 12-30-2009, 05:08 PM   #7
Didier Spaier
LQ Addict
 
Registered: Nov 2008
Location: Paris, France
Distribution: Slint64-15.0
Posts: 11,057

Rep: Reputation: Disabled
IMHO wget is no more a basic Linux command than rsync. Or you could use lftp

Code:
lftp -c "open http://slackware.cs.utah.edu/pub/slackware/slackware64-13.0/; mirror slackware64"

Last edited by Didier Spaier; 12-30-2009 at 05:19 PM. Reason: Wrong command
 
Old 12-30-2009, 05:33 PM   #8
allend
LQ 5k Club
 
Registered: Oct 2003
Location: Melbourne
Distribution: Slackware64-15.0
Posts: 6,371

Rep: Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749
This command can be run to automatically download updates to a local mirror.
Note- Any outdated packages are still kept.

'wget --no-host-directories --cut-dirs=2 --mirror \
--directory-prefix=<your local mirror directory> \
--output-file=<path and name of output log file> \
ftp://<url of your mirror>/<path to slackware>/'

Last edited by allend; 12-30-2009 at 05:39 PM.
 
Old 12-30-2009, 05:34 PM   #9
egag
Senior Member
 
Registered: Jul 2004
Location: Netherlands
Distribution: Slackware
Posts: 2,721

Rep: Reputation: 53
Quote:
Originally Posted by jong357 View Post
Code:
wget -e robots=off --wait 1 -r --reject "index.html*" -np -l1 http://slackware.cs.utah.edu/pub/slackware/slackware64-13.0/slackware64
That doesn't recursively work backwards but it also doesn't grab any of the packages. It just makes the directory tree of all the package sets and then stops....


So frustrating....
Hi, jong357

You're almost there i guess.
Just remove the "-l1" option ( or use "-l3" or so... )

Some more info here : http://www.editcorp.com/Personal/Lar...v1/wget_7.html

cheers,
egag
 
Old 12-30-2009, 05:37 PM   #10
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983
Quote:
Originally Posted by jong357 View Post
Any clues? Thanks. I've tried a whole host of options and haven't found the right combination yet...
Use -np as suggested and put a slash at the end of the URL to make wget understand who the parent is.
Quote:
...can throw me a bone.
Really nice expression... I've never heard it. Funny!
 
Old 12-30-2009, 07:43 PM   #11
jong357
Senior Member
 
Registered: May 2003
Location: Columbus, OH
Distribution: DIYSlackware
Posts: 1,914

Original Poster
Rep: Reputation: 52
Sure wget is a basic command. Any system that doesn't have it installed isn't worth chicken scratch.

I however do not keep lftp or rsync on any of my systems. Anyhoo....

Removing the depth switch -l and especially adding a trailing slash at the end seems to have done it. Thank you both for your input.

Actually I won't know for sure until 30 minutes or so, but the behavior is different already. It went straight to slackware64 and is staying there and diligently downloading everything in that directory.

If I don't post back with reports of it downloading other crap it's not supposed to then it worked. Thanks guys!

Code:
wget -e robots=off --wait 1 -r --reject "index.html*" -np http://slackware.cs.utah.edu/pub/slackware/slackware64-13.0/slackware64/
Pretty sure I had to use "-e robots=off" because their robots.txt file was denying me access. The --wait is out of courtesy just for bypassing their robots file....
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
wget and '..' jackd1000 Programming 3 12-23-2008 09:00 AM
wget not working! but "man wget is" ??? wrapster Solaris / OpenSolaris 5 07-30-2008 03:00 AM
wget noir911 Linux - Newbie 8 07-30-2005 08:57 AM
wget noir911 Linux - General 1 06-12-2005 02:10 AM
wget toastermaker Linux - Software 4 11-13-2004 10:59 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware

All times are GMT -5. The time now is 05:34 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration