LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 11-11-2017, 06:47 AM   #1
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware & Android
Posts: 10,989

Rep: Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250
wget mirror options question


I am trying to sort something out in my head. If I want a local mirror of some directory, I usually do not want the lead-in. Take, for example my local slackware (and everything) mirror, which is

ftp://ftp.heanet.ie/pub/mirrors/slac...re64-<version>

I would get a directory starting ftp.heanet.ie/ with an unpredictable number of unwanted branches. What I would like is one directory slackware64-<version> with it's files.

can I get that from wget?
 
Old 11-11-2017, 08:51 AM   #2
wpeckham
Senior Member
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, Fedora, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, Vsido, tinycore, Q4OS
Posts: 3,189

Rep: Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380
The answer is "kinda yes", but there may be better tools for that job.

Is it that you want to replicate that tree ONCE, or keep a copy in sync? What is your real purpose or goal here?

And, naturally, did you bother to read the MAN page on wget? I have, and it looks to me as if some of the clearly documented options would help you.
 
Old 11-11-2017, 09:21 AM   #3
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware & Android
Posts: 10,989

Original Poster
Rep: Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250
Thanks for the reply.

I read the first 20 screenfuls of the man page and glazed over. I imagine rsync would do it. No, I don't want a synced copy, I just want a download. wget is not into syncing copies, and it's research, not porn, so I didn't was the encrypt --> decrypt that goes on with some copy schemes, because it slows matters.
 
Old 11-11-2017, 08:19 PM   #4
wpeckham
Senior Member
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, Fedora, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, Vsido, tinycore, Q4OS
Posts: 3,189

Rep: Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380Reputation: 1380
Quote:
Originally Posted by business_kid View Post
Thanks for the reply.

I read the first 20 screenfuls of the man page and glazed over. I imagine rsync would do it. No, I don't want a synced copy, I just want a download. wget is not into syncing copies, and it's research, not porn, so I didn't was the encrypt --> decrypt that goes on with some copy schemes, because it slows matters.
Well, if the information was right in front of you but "made your eyes glaze over" we may not be of much help.
But
I get the impression that a clone of the path is not what you want.
What it appears that you want will be a cone (snapshot) of the folder that is your source. You do NOT want "sync" so your original word choice may have been misleading.
Rsync is surprisingly good at that, and encryption is not what slows you down on modern processors. Lftp is also good, and both have specific options that make them excellent tools for this purpose. Also, there are excellent tutorials that describe options for almost exactly your use case!

To use ANY tool properly you will have to learn to read the documentation and understand at least some of how it works. If you want to make wget work for this, you can. You are going to have to find some of the options that might work, run some test transfers, and find your solution. Once you actually make a start, let us know what you did and what happened so we can advise you.
 
Old 11-11-2017, 08:32 PM   #5
braveranger
LQ Newbie
 
Registered: Nov 2017
Posts: 5

Rep: Reputation: Disabled
Here's how to do it with LFTP

i created a .lftprc file in home dir which contained the following:

open user: password@mysite.com (no spaces)
set ssl:verify-certificate no
cd
lcd ~/local/path/to/dir/
cd remote/path/to/dir
mput * (or) mget *
bye

Last edited by braveranger; 11-11-2017 at 08:53 PM.
 
Old 11-12-2017, 04:50 AM   #6
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware & Android
Posts: 10,989

Original Poster
Rep: Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250Reputation: 1250
You're right, lftp would be a superior was to go. It's not always the same site.

I certainly don't want a clone of a website, more a mirror of one particular small branch (sometimes with subdirs, sometimes not). Browsers are also effective at it. I got used to operating under fierce stress and pressure - now gone - and I find my threshold for bilge has deteriorated. If you lit a fire under my backside, I suppose I could assimilate and process bilge as well as formerly; It's rarely you hear someone complain about a lack of pressure :-P.
 
Old 11-14-2017, 12:44 AM   #7
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,511

Rep: Reputation: 1007Reputation: 1007Reputation: 1007Reputation: 1007Reputation: 1007Reputation: 1007Reputation: 1007Reputation: 1007
Wget is good if you have to use http, like on web sites that don't intend to be mirrored by the general public. Rsync is probably the best, if the site permits rsync.

Wget has a switch to save the download in whatever directory name you want, not necessarily the domain name. But rsync can exclude whatever you want with the --exclude switch.

Last edited by AwesomeMachine; 11-14-2017 at 12:51 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Downloading 14.2 from mirror with wget? colinh2 Slackware 4 07-04-2016 12:00 PM
I would like to mirror some websites using wget and need a little help linuxus3r Linux - Software 6 05-24-2016 09:57 AM
wget --mirror https Jorris Linux - Newbie 1 08-23-2011 02:01 AM
wget (or other util): how do I mirror parts of this site? exscape Linux - Software 2 08-04-2010 01:03 PM
wget and mirror scottrell Linux - General 1 05-30-2003 05:54 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 09:44 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration