Bash script to download a directory structure and a file...

JoeBleaux · 03-20-2009, 11:32 PM

Any Bash guys out there can tell me how to do this...

I want to read an ftp site (ftp://ftp.slackware.com/pub/slackwar...12.2/slackware)
...and download the sub-directory structure and ONLY the tagfiles to the local directory.

FROM THIS:

Code:

slackware
     +--a
     |  |\aaa_base-12.2.0-noarch-1.tgz
     |  |\aaa_base-12.2.0-noarch-1.tgz.asc
     |  ...
     |  |\tagfile
     |  |\tar-1.16.1-i486-1.tgz
     |  ...
     |
     +--ap
     |  |\a2ps-4.13b-i386-2.tgz
     |  |\a2ps-4.13b-i386-2.tgz.asc
     |  ...
     |  |\tagfile
     |  |\texinfo-4.13a-i486-2.tgz
     |  ...
     ...

TO THIS:

Code:

slackware
     +--a
     |   \tagfile
     |
     |
     +--ap
     |   \tagfile
     |
     |
     ...

I tried wget + grep but that didn't work too well.

ghostdog74 · 03-21-2009, 12:02 AM

If you can use Python

Code:

#!/usr/bin/python
import sys,urllib2,urllib,os
url="ftp://ftp.slackware.com/pub/slackware/slackware-12.2/slackware"
page=urllib2.urlopen(url)
result=page.readlines()
for lines in result:
    lines=lines.split()
    if lines[0].startswith("d"):
        #if its a directory, get the directory name
        getdir=lines[-1] # get directory name
        if not os.path.exists(getdir): os.mkdir(getdir,777) #create local directory
        URL="%s/%s" % (url,getdir) #url to get tagfile
        subpages = urllib.urlretrieve(URL,"./%s/%s" %(getdir,"tagfile"))

you can also use wget to follow directories and get only tagfile. check the wget man page for how to do it

H_TeXMeX_H · 03-21-2009, 11:21 AM

If you have lftp installed, you could also do a 1 liner:

Code:

lftp -c 'open -e "mget -d */tagfile" ftp://ftp.slackware.com/pub/slackware/slackware-12.2/slackware/'