Hi
for those who know what web bots are I was wondering if anyone knows where to find books on how to program web spiders in Python or C?? and that it is updated, because I stumbled on the book by O'reilly "Spidering Hacks" which sort of throw bits to you about how to program web bots using Perl, but I don't want to learn another computer lanuage....and plus the book was written a few years ago....any new updated books that are published??, THANKS
Chris

Like a web spider?

I just googled for my research,as I wanted to do one in C more as a learning exercise than anything else.


If you go the C route,you may want to look at the libxml library at xmlsoft.org.

It has libs for parsing html handling uri's and they also have a HTTP implementation,(although I didn't like it so I wrote my own

You will want to research the robots exclusion policy and robot etiquette and also consider how you are going to sort and manage all that data.
A robot soon amasses thousands of links,so you will want to be clear in your own mind how your robot will process manage and store the data of interest.Something I am still trying to fine tune in my project.

best of luck.

Do you have any suggestions on what book/tutorial I should read up before I begin, cause I'm very new with the idea of web robots.
Thanks

Sorry,I just googled'google'd then google'd some more.

I found these helpful on my merry way.
http://en.wikipedia.org/wiki/Web_crawler
xmlsoft.org(start with the dom parser,but then look at the sax interface;much faster)
http://www.robotstxt.org/wc/robots.html
http://www.garshol.priv.no/download/text/http-tut.html
http://www.phy.duke.edu/~rgb/General...g_example.html

sorry I couldn't be of more help,but If you have any specific questions I will try my best to help.