Python web page parser

snowman81 · 01-10-2009, 06:14 PM

I found a guide online for writing a simple program for parsing a webpage. The example outputs the full path of all the images in a particular webpage. I want to change it so that it will parse a webpage for all instances of an IP address/port combo. For instance, say a page has a listing like this

Code:

 
                               this one is 127.0.0.1:80
                               this one is 127.0.1.1:80
                               this one is 127.0.2.1:80

It will only output the IP address/port combo.

Code:

                               127.0.0.1:80
                               127.0.1.1:80
                               127.0.2.1:80

Anyway, the code opens a page and does error checking, I think I have that figured out. The code that I think needs to be different is this:

Code:

matches = sre.findall('<img .*src="(.*?)"', website_text)

for match in matches:
        if match[:7] != "http://":
                if match[0] == "/":
                        slash = ""
                else:
                        slash = "/"
                match_set.add(dir + slash + match)
        else:
                match_set.add(match)

match_set = list(match_set)
match_set.sort()

for item in match_set:
        print item

If you need me to I can post the whole thing. Since I do not need it to be in the form of a web address I figured i could strip out most of the above code and maybe do something like

Code:

 \b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\{1,4}\b

but it didn't really work when I tried it. Any suggestions?

maroonbaboon · 01-11-2009, 06:26 AM

Quote:

Originally Posted by snowman81

I figured i could strip out most of the above code and maybe do something like

Code:

 \b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\{1,4}\b

but it didn't really work when I tried it. Any suggestions?

There is a typo in your regex. Missing 'd' in front of {1,4}

This seems to work:

Code:

import re
foo = re.compile("\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,4})\s")
bar = "looking for 100.10.20.30:40 in this string."
ip = foo.search(bar)
print ip.groups()

If you are only interested in links the python HTML parser module will find all the links for you.

snowman81 · 01-11-2009, 11:01 AM

Yep, that looks exactly like what I need. I can't believe I missed something simple like that. Thanks.

Is

Code:

print ip.groups()

the only way to output the results? I tried just

Code:

print ip

but it wouldn't work. It seems when you print it in groups it adds a comma and a bracket type thing. I need it to be just the IP/port combo.

bgeddy · 01-11-2009, 12:25 PM

I've made a slight change to your code to find mutiple matches even next to each other. This does not assume a IP string should be surrounded by whitespace so beware.

Code:

import re
foo = re.compile("(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,4})")
bar = "looking for 100.10.20.30:40 10.1.20.2:33 in this string."
ip = foo.findall(bar)
for finds in ip:
    print finds

maroonbaboon · 01-11-2009, 06:03 PM

Quote:

Originally Posted by snowman81

Is

Code:

print ip.groups()

the only way to output the results? I tried just

Code:

print ip

but it wouldn't work.

It's a while since I did any serious python programming. I was delighted to see that the library documentation has taken a quantum leap upwards in quality.

http://docs.python.org/library/re.html

should tell you everything you need to know. Or just use the suggestion in the post above.

snowman81 · 01-11-2009, 07:18 PM

Ok, thanks for all the help, the program runs as designed now.