download all the files of Version A3(1.0) on the web page

powah · 05-22-2009, 03:19 PM

How to write a script to download all the files of Version A3(1.0) on the web page
ftp://ftp-sj.cisco.com/pub/mibs/supp...portlist.html?

Alien_Hominid · 05-24-2009, 01:36 AM

Code:

lynx -dump "ftp://ftp-sj.cisco.com/pub/mibs/supportlists/ace-appliance/ace-appliance-supportlist.html?" | grep -o "*.my" >file.txt

You should get all file names. Now you could use wget.

ghostdog74 · 05-24-2009, 03:07 AM

if you have Python

Code:

#!/usr/bin/env python
import urllib2
url="ftp://ftp-sj.cisco.com/pub/mibs/supportlists/ace-appliance/ace-appliance-supportlist.html"
page=urllib2.urlopen(url)
f=0
links=[]
for item in data:
    if "</table>" in item: f=0
    if "Version" in item and "A3" in item and "1.0" in item: f=1
    if f and "href" in item:        
        item=item.replace('href="',"").strip()
        ind=item.index('">')
        links.append(item[:ind]) #grab all ftp links 
# download all links
for link in links:
    filename=link.split("/")[-1]
    print "downloading ... " + filename
    u=urllib2.urlopen(link)
    p=u.read()
    open(filename,"w").write(p)

powah · 05-25-2009, 09:54 AM

Quote:

Originally Posted by Alien_Hominid

Code:

lynx -dump "ftp://ftp-sj.cisco.com/pub/mibs/supportlists/ace-appliance/ace-appliance-supportlist.html?" | grep -o "*.my" >file.txt

You should get all file names. Now you could use wget.

grep -o "*.my"
will create an empty file.txt.
I do this:
lynx -dump "ftp://ftp-sj.cisco.com/pub/mibs/supportlists/ace-appliance/ace-appliance-supportlist.html?" | grep ".my" >file.txt

file.txt is:
href="ftp://ftp.cisco.com/pub/mibs/v2/CISCO-AAA-SERVER-EXT-MIB.my">CISCO-
class=SpellE>MIB.my </A><A
href="ftp://ftp.cisco.com/pub/mibs/v2/CISCO-AAA-SERVER-MIB.my">CISCO-AAA-
class=SpellE>MIB.my</A> <A
href="ftp://ftp.cisco.com/pub/mibs/v2/CISCO-ENHANCED-SLB-MIB.my">CISCO-EN
class=SpellE>MIB.my</A> <A
href="ftp://ftp.cisco.com/pub/mibs/v2/CISCO-ENTITY-VENDORTYPE-OID-MIB.my"
class=SpellE>MIB.my</A> <A
href="ftp://ftp.cisco.com/pub/mibs/v2/CISCO-IF-EXTENSION-MIB.my">CISCO-IF
class=SpellE>MIB.my</A> <A
href="ftp://ftp.cisco.com/pub/mibs/v2/CISCO-IP-PROTOCOL-FILTER-MIB.my">CI
class=SpellE>MIB.my</A> <A
...

powah · 05-25-2009, 10:03 AM

Quote:

Originally Posted by ghostdog74

if you have Python

Code:

#!/usr/bin/env python
import urllib2
url="ftp://ftp-sj.cisco.com/pub/mibs/supportlists/ace-appliance/ace-appliance-supportlist.html"
page=urllib2.urlopen(url)
f=0
links=[]
for item in data:
    if "</table>" in item: f=0
    if "Version" in item and "A3" in item and "1.0" in item: f=1
    if f and "href" in item:        
        item=item.replace('href="',"").strip()
        ind=item.index('">')
        links.append(item[:ind]) #grab all ftp links 
# download all links
for link in links:
    filename=link.split("/")[-1]
    print "downloading ... " + filename
    u=urllib2.urlopen(link)
    p=u.read()
    open(filename,"w").write(p)

On my FC6 linux computer:
$ python
Python 2.4.4 (#1, Oct 23 2006, 13:58:00)
[GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

I put your script in a file and run it:
$ ~/python/downloadFile.py
Traceback (most recent call last):
File "/home/powah/python/downloadFile.py", line 7, in ?
for item in data:
NameError: name 'data' is not defined

ghostdog74 · 05-25-2009, 07:13 PM

Quote:

Originally Posted by powah

On my FC6 linux computer:
$ python
Python 2.4.4 (#1, Oct 23 2006, 13:58:00)
[GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

I put your script in a file and run it:
$ ~/python/downloadFile.py
Traceback (most recent call last):
File "/home/powah/python/downloadFile.py", line 7, in ?
for item in data:
NameError: name 'data' is not defined

Code:

.....
links=[]
data=page.read().split("\n") <<---- insert this line
for item in data:
........

powah · 05-25-2009, 10:49 PM

Quote:

Originally Posted by ghostdog74

Code:

.....
links=[]
data=page.read().split("\n") <<---- insert this line
for item in data:
........

It works.
Thanks!

powah · 05-25-2009, 10:55 PM

I want to download all files from the web page
ftp://ftp-sj.cisco.com/pub/mibs/supp...pportlist.html.

I modify the script:
#!/usr/bin/env python
import urllib2
url="ftp://ftp-sj.cisco.com/pub/mibs/supportlists/vpn3000/vpn3000-supportlist.html"
page=urllib2.urlopen(url)
f=0
links=[]
data=page.read().split("\n")
for item in data:
if "href" in item:
item=item.replace('href="',"").strip()
ind=item.index('">')
links.append(item[:ind]) #grab all ftp links
# download all links
for link in links:
filename=link.split("/")[-1]
print "downloading ... " + filename
u=urllib2.urlopen(link)
p=u.read()
open(filename,"w").write(p)

Running the script has the following error. Please help. Thanks.
$ ~/python/downloadFile2.py
downloading ... v2
downloading ... ADMIN-AUTH-STATS-MIB.my
downloading ... ALTIGA-ADDRESS-STATS-MIB.my
downloading ... ALTIGA-BMGT-STATS-MIB.my
downloading ... ALTIGA-CAP.my
Traceback (most recent call last):
File "/home/powah/python/downloadFile2.py", line 20, in ?
u=urllib2.urlopen(link)
File "/usr/lib/python2.4/urllib2.py", line 130, in urlopen
return _opener.open(url, data)
File "/usr/lib/python2.4/urllib2.py", line 358, in open
response = self._open(req, data)
File "/usr/lib/python2.4/urllib2.py", line 381, in _open
'unknown_open', req)
File "/usr/lib/python2.4/urllib2.py", line 337, in _call_chain
result = func(*args)
File "/usr/lib/python2.4/urllib2.py", line 1053, in unknown_open
raise URLError('unknown url type: %s' % type)
urllib2.URLError: <urlopen error unknown url type: <dd><a ftp>

ghostdog74 · 05-25-2009, 11:59 PM

Quote:

Originally Posted by powah

I want to download all files from the web page
ftp://ftp-sj.cisco.com/pub/mibs/supp...pportlist.html.

I modify the script:

Code:

#!/usr/bin/env python
import urllib2
url="ftp://ftp-sj.cisco.com/pub/mibs/supportlists/vpn3000/vpn3000-supportlist.html"
page=urllib2.urlopen(url)
f=0
links=[]
data=page.read().split("\n")
for item in data:
    if "href" in item:     
        item=item.replace('href="',"").strip()
        ind=item.index('">')
        links.append(item[:ind]) #grab all ftp links 
# download all links
for link in links:
    filename=link.split("/")[-1]
    print "downloading ... " + filename
    u=urllib2.urlopen(link)
    p=u.read()
    open(filename,"w").write(p)

put your code in code tags next time

Code:

import urllib2,os,urlparse
url="ftp://ftp-sj.cisco.com/pub/mibs/supportlists/vpn3000/vpn3000-supportlist.html"
page=urllib2.urlopen(url)
f=0
links=[]
data=page.read().split("\n")
for item in data:
    if "href" in item:        
        ftpind=item.index("ftp://")
        item=item[ftpind:]
        ind=item.index('">')
        links.append(item[:ind]) #grab all links 
# download all links
for link in links:
    filename=link.split("/")[-1]
    print "downloading ... " + filename
    u=urllib2.urlopen(link)
    p=u.read()
    open(filename,"w").write(p)

to troubleshoot your code, put print statements. also, please read up on Python if you want to use it. See my sig.