LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Programming (https://www.linuxquestions.org/questions/programming-9/)
-   -   download all the files of Version A3(1.0) on the web page (https://www.linuxquestions.org/questions/programming-9/download-all-the-files-of-version-a3-1-0-on-the-web-page-727869/)

powah 05-22-2009 03:19 PM
download all the files of Version A3(1.0) on the web page
 
How to write a script to download all the files of Version A3(1.0) on the web page
ftp://ftp-sj.cisco.com/pub/mibs/supp...portlist.html?

Alien_Hominid 05-24-2009 01:36 AM
Code:
lynx -dump "ftp://ftp-sj.cisco.com/pub/mibs/supportlists/ace-appliance/ace-appliance-supportlist.html?" | grep -o "*.my" >file.txt
You should get all file names. Now you could use wget.

ghostdog74 05-24-2009 03:07 AM
if you have Python
Code:
#!/usr/bin/env python
import urllib2
url="ftp://ftp-sj.cisco.com/pub/mibs/supportlists/ace-appliance/ace-appliance-supportlist.html"
page=urllib2.urlopen(url)
f=0
links=[]
for item in data:
    if "</table>" in item: f=0
    if "Version" in item and "A3" in item and "1.0" in item: f=1
    if f and "href" in item:       
        item=item.replace('href="',"").strip()
        ind=item.index('">')
        links.append(item[:ind]) #grab all ftp links
# download all links
for link in links:
    filename=link.split("/")[-1]
    print "downloading ... " + filename
    u=urllib2.urlopen(link)
    p=u.read()
    open(filename,"w").write(p)

powah 05-25-2009 09:54 AM
Quote:
Originally Posted by Alien_Hominid (Post 3550813)
Code:
lynx -dump "ftp://ftp-sj.cisco.com/pub/mibs/supportlists/ace-appliance/ace-appliance-supportlist.html?" | grep -o "*.my" >file.txt
You should get all file names. Now you could use wget.

grep -o "*.my"
will create an empty file.txt.
I do this:
lynx -dump "ftp://ftp-sj.cisco.com/pub/mibs/supportlists/ace-appliance/ace-appliance-supportlist.html?" | grep ".my" >file.txt

file.txt is:
href="ftp://ftp.cisco.com/pub/mibs/v2/CISCO-AAA-SERVER-EXT-MIB.my">CISCO-
class=SpellE>MIB.my</SPAN><BR></A><A
href="ftp://ftp.cisco.com/pub/mibs/v2/CISCO-AAA-SERVER-MIB.my">CISCO-AAA-
class=SpellE>MIB.my</SPAN></A><BR><A
href="ftp://ftp.cisco.com/pub/mibs/v2/CISCO-ENHANCED-SLB-MIB.my">CISCO-EN
class=SpellE>MIB.my</SPAN></A><BR><A
href="ftp://ftp.cisco.com/pub/mibs/v2/CISCO-ENTITY-VENDORTYPE-OID-MIB.my"
class=SpellE>MIB.my</SPAN></A><BR><A
href="ftp://ftp.cisco.com/pub/mibs/v2/CISCO-IF-EXTENSION-MIB.my">CISCO-IF
class=SpellE>MIB.my</SPAN></A><BR><A
href="ftp://ftp.cisco.com/pub/mibs/v2/CISCO-IP-PROTOCOL-FILTER-MIB.my">CI
class=SpellE>MIB.my</SPAN></A><BR><A
...

powah 05-25-2009 10:03 AM
Quote:
Originally Posted by ghostdog74 (Post 3550846)
if you have Python
Code:
#!/usr/bin/env python
import urllib2
url="ftp://ftp-sj.cisco.com/pub/mibs/supportlists/ace-appliance/ace-appliance-supportlist.html"
page=urllib2.urlopen(url)
f=0
links=[]
for item in data:
    if "</table>" in item: f=0
    if "Version" in item and "A3" in item and "1.0" in item: f=1
    if f and "href" in item:       
        item=item.replace('href="',"").strip()
        ind=item.index('">')
        links.append(item[:ind]) #grab all ftp links
# download all links
for link in links:
    filename=link.split("/")[-1]
    print "downloading ... " + filename
    u=urllib2.urlopen(link)
    p=u.read()
    open(filename,"w").write(p)
On my FC6 linux computer:
$ python
Python 2.4.4 (#1, Oct 23 2006, 13:58:00)
[GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

I put your script in a file and run it:
$ ~/python/downloadFile.py
Traceback (most recent call last):
File "/home/powah/python/downloadFile.py", line 7, in ?
for item in data:
NameError: name 'data' is not defined

ghostdog74 05-25-2009 07:13 PM
Quote:
Originally Posted by powah (Post 3552109)
On my FC6 linux computer:
$ python
Python 2.4.4 (#1, Oct 23 2006, 13:58:00)
[GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

I put your script in a file and run it:
$ ~/python/downloadFile.py
Traceback (most recent call last):
File "/home/powah/python/downloadFile.py", line 7, in ?
for item in data:
NameError: name 'data' is not defined
Code:
.....
links=[]
data=page.read().split("\n") <<---- insert this line
for item in data:
........

powah 05-25-2009 10:49 PM
Quote:
Originally Posted by ghostdog74 (Post 3552510)
Code:
.....
links=[]
data=page.read().split("\n") <<---- insert this line
for item in data:
........
It works.
Thanks!

powah 05-25-2009 10:55 PM
download all files from the web page
 
I want to download all files from the web page
ftp://ftp-sj.cisco.com/pub/mibs/supp...pportlist.html.

I modify the script:
#!/usr/bin/env python
import urllib2
url="ftp://ftp-sj.cisco.com/pub/mibs/supportlists/vpn3000/vpn3000-supportlist.html"
page=urllib2.urlopen(url)
f=0
links=[]
data=page.read().split("\n")
for item in data:
if "href" in item:
item=item.replace('href="',"").strip()
ind=item.index('">')
links.append(item[:ind]) #grab all ftp links
# download all links
for link in links:
filename=link.split("/")[-1]
print "downloading ... " + filename
u=urllib2.urlopen(link)
p=u.read()
open(filename,"w").write(p)






Running the script has the following error. Please help. Thanks.
$ ~/python/downloadFile2.py
downloading ... v2
downloading ... ADMIN-AUTH-STATS-MIB.my
downloading ... ALTIGA-ADDRESS-STATS-MIB.my
downloading ... ALTIGA-BMGT-STATS-MIB.my
downloading ... ALTIGA-CAP.my
Traceback (most recent call last):
File "/home/powah/python/downloadFile2.py", line 20, in ?
u=urllib2.urlopen(link)
File "/usr/lib/python2.4/urllib2.py", line 130, in urlopen
return _opener.open(url, data)
File "/usr/lib/python2.4/urllib2.py", line 358, in open
response = self._open(req, data)
File "/usr/lib/python2.4/urllib2.py", line 381, in _open
'unknown_open', req)
File "/usr/lib/python2.4/urllib2.py", line 337, in _call_chain
result = func(*args)
File "/usr/lib/python2.4/urllib2.py", line 1053, in unknown_open
raise URLError('unknown url type: %s' % type)
urllib2.URLError: <urlopen error unknown url type: <dd><a ftp>

ghostdog74 05-25-2009 11:59 PM
Quote:
Originally Posted by powah (Post 3552636)
I want to download all files from the web page
ftp://ftp-sj.cisco.com/pub/mibs/supp...pportlist.html.

I modify the script:
Code:
#!/usr/bin/env python
import urllib2
url="ftp://ftp-sj.cisco.com/pub/mibs/supportlists/vpn3000/vpn3000-supportlist.html"
page=urllib2.urlopen(url)
f=0
links=[]
data=page.read().split("\n")
for item in data:
    if "href" in item:   
        item=item.replace('href="',"").strip()
        ind=item.index('">')
        links.append(item[:ind]) #grab all ftp links
# download all links
for link in links:
    filename=link.split("/")[-1]
    print "downloading ... " + filename
    u=urllib2.urlopen(link)
    p=u.read()
    open(filename,"w").write(p)
put your code in code tags next time

Code:
import urllib2,os,urlparse
url="ftp://ftp-sj.cisco.com/pub/mibs/supportlists/vpn3000/vpn3000-supportlist.html"
page=urllib2.urlopen(url)
f=0
links=[]
data=page.read().split("\n")
for item in data:
    if "href" in item:       
        ftpind=item.index("ftp://")
        item=item[ftpind:]
        ind=item.index('">')
        links.append(item[:ind]) #grab all links
# download all links
for link in links:
    filename=link.split("/")[-1]
    print "downloading ... " + filename
    u=urllib2.urlopen(link)
    p=u.read()
    open(filename,"w").write(p)
to troubleshoot your code, put print statements. also, please read up on Python if you want to use it. See my sig.


All times are GMT -5. The time now is 07:06 AM.