Extract Data using CURL

ludo33 · 11-24-2009, 11:28 AM

Hello folks,

I have a list of terms in a mysql database, I need to extract the page contents from wikipedia for each term and write the result to the mysql table. I've been informed that CURL is the way to go about this, this is all new to me, can anybody suggest any resources to help with this?

Thanks in advance.

centosboy · 11-25-2009, 03:59 AM

Quote:

Originally Posted by ludo33

Hello folks,

I have a list of terms in a mysql database, I need to extract the page contents from wikipedia for each term and write the result to the mysql table. I've been informed that CURL is the way to go about this, this is all new to me, can anybody suggest any resources to help with this?

Thanks in advance.

because the wiki page search does a redirect on terms searched in certain case, you would need something like

Code:

curl -L http://en.wikipedia.org/wiki/<string>

this would output the contents of this page to the terminal, so you would need to direct it somewhere.

ludo33 · 11-25-2009, 06:11 AM

Quote:

Originally Posted by centosboy

this would output the contents of this page to the terminal, so you would need to direct it somewhere.

This is the bit that I am stuck at, how to write the result to a mysql table?

centosboy · 11-25-2009, 09:27 AM

Quote:

Originally Posted by ludo33

This is the bit that I am stuck at, how to write the result to a mysql table?

made a slight mistake here...

what i should have said is something like this:

use mysqlimport to import the data back into a database.

if for example the file dumped from curl is called madonna.txt, then mysqlimport would import it into a table (assuming it is already created) called madonna.

the madonna text file would need some sort of parsing before hand (tab separation).

assuming this has all been done, then this could work.

Code:

curl -L http://en.wikipedia.org/wiki/madonna > madonna.txt
some file parsing code here.............(tab separated columns)
mysqlimport -h host -pxxx dbname --local madonna.txt

ludo33 · 11-29-2009, 02:17 AM

Thanks for your help, all seems to be working fine.

Again thanks for yout informed reply!