what would be the best way to extract info out of an html page

mia_tech · 03-27-2014, 01:01 AM

let's say I'm monitoring a webpage which is constantly updating its information like: for example, an ebay page reflecting current bids, and I want to check on the latest bid every 10 min when there's only one hour left. Would it be better to use the source code of the page or dumping the actual page with lynx or other text base browser, and then parse the info I'm interested in.

dugan · 03-27-2014, 01:17 AM

I'd look into this:

http://phantomjs.org/

bonnydeal · 03-27-2014, 11:30 AM

I would look into THIS
http://developer.ebay.com/common/api/

szboardstretcher · 03-27-2014, 11:32 AM

+1 for phantomjs. It really is the fastest way to accomplish this without starting from scratch.

dugan · 03-27-2014, 11:42 AM

There's also this, which also let's you easily get information out of an HTML page:

http://docs.seleniumhq.org/

So: plenty of good solutions that don't require you to scrape the HTML.

Definitely try the API first though.

schneidz · 03-27-2014, 11:53 AM

i've made a few site scrapers for xbmc using python:
http://hyper.dyndns-home.com/xbmc/

if you know the url to the item you can do something like this

Code:

[schneidz@hyper ebay]$ wget -q -O - 'http://www.ebay.com/itm/Google-Nexus-5-16GB-Black-Unlocked-Smartphone-/111310254176?pt=Cell_Phones&hash=item19ea9bb060' | egrep -o "(\"timeLeftInMins\":[0-9]*,|itemprop=\"price\".*[0-9])"
itemprop="price">US $255.00
"timeLeftInMins":7444,