LinuxQuestions.org
Did you know LQ has a Linux Hardware Compatibility List?
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 03-20-2009, 11:32 PM   #1
JoeBleaux
LQ Newbie
 
Registered: Mar 2009
Distribution: Slamd64
Posts: 26

Rep: Reputation: 15
Bash script to download a directory structure and a file...


Any Bash guys out there can tell me how to do this...

I want to read an ftp site (ftp://ftp.slackware.com/pub/slackwar...12.2/slackware)
...and download the sub-directory structure and ONLY the tagfiles to the local directory.

FROM THIS:

Code:
slackware
     +--a
     |  |\aaa_base-12.2.0-noarch-1.tgz
     |  |\aaa_base-12.2.0-noarch-1.tgz.asc
     |  ...
     |  |\tagfile
     |  |\tar-1.16.1-i486-1.tgz
     |  ...
     |
     +--ap
     |  |\a2ps-4.13b-i386-2.tgz
     |  |\a2ps-4.13b-i386-2.tgz.asc
     |  ...
     |  |\tagfile
     |  |\texinfo-4.13a-i486-2.tgz
     |  ...
     ...
TO THIS:

Code:
slackware
     +--a
     |   \tagfile
     |
     |
     +--ap
     |   \tagfile
     |
     |
     ...


I tried wget + grep but that didn't work too well.
 
Old 03-21-2009, 12:02 AM   #2
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,695
Blog Entries: 5

Rep: Reputation: 240Reputation: 240Reputation: 240
If you can use Python
Code:
#!/usr/bin/python
import sys,urllib2,urllib,os
url="ftp://ftp.slackware.com/pub/slackware/slackware-12.2/slackware"
page=urllib2.urlopen(url)
result=page.readlines()
for lines in result:
    lines=lines.split()
    if lines[0].startswith("d"):
        #if its a directory, get the directory name
        getdir=lines[-1] # get directory name
        if not os.path.exists(getdir): os.mkdir(getdir,777) #create local directory
        URL="%s/%s" % (url,getdir) #url to get tagfile
        subpages = urllib.urlretrieve(URL,"./%s/%s" %(getdir,"tagfile"))
you can also use wget to follow directories and get only tagfile. check the wget man page for how to do it
 
Old 03-21-2009, 11:21 AM   #3
H_TeXMeX_H
Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269
If you have lftp installed, you could also do a 1 liner:

Code:
lftp -c 'open -e "mget -d */tagfile" ftp://ftp.slackware.com/pub/slackware/slackware-12.2/slackware/'
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Script to find file differences in two directory trees (bash) Syqers Linux - Newbie 3 10-01-2008 11:55 PM
how many different ways we can remove the file from a directory structure. dina3e Linux - General 8 10-01-2008 03:06 AM
bash - copy with a directory structure mkiler Linux - General 6 06-03-2008 08:45 AM
[SOLVED] A bash script to automatically copy all documents with directory structure thelordmule Programming 8 11-30-2007 02:30 AM
file or directory? bash script efus Programming 3 04-26-2007 06:11 PM


All times are GMT -5. The time now is 02:21 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration