Grabbing wiki code using wget

jgombos · 08-02-2010, 02:38 AM

I would like to grab wiki code from a wiki page using wget. Running this grabs HTML:

wget -O wikihtml.html http://en.wikipedia.org/wiki/Lyman_Enos_Knapp

The first attempt at getting wiki code was to pretend to edit, and run:

wget -O wikiedit.html http://en.wikipedia.org/w/index.php?...pp&action=edit

but of course that grabs GUI HTML. I thought perhaps the text inside the text box would be in tact, but HTML is througout. Any ideas how to get just the raw wiki code?

Artanicus · 08-02-2010, 06:02 AM

Will get you pretty close, need just a bit more parsing:

Code:

wget -O - "http://en.wikipedia.org/w/index.php?title=Lyman_Enos_Knapp&action=edit" | awk '/textarea/,/<\/textarea>/'

edit:
And to get rid of the entities, just pipe it onwards to elinks / lynx.

jgombos · 08-02-2010, 06:53 AM

Quote:

Originally Posted by Artanicus

Will get you pretty close, need just a bit more parsing:

Code:

wget -O - "http://en.wikipedia.org/w/index.php?title=Lyman_Enos_Knapp&action=edit" | awk '/textarea/,/<\/textarea>/'

edit:
And to get rid of the entities, just pipe it onwards to elinks / lynx.

Thanks! I ended up doing something similar:

Code:

wget -O - "http://en.wikipedia.org/w/index.php?title=Lyman_Enos_Knapp&action=edit" | sed -ne '/textarea/,/textarea/p'

In case anyone goes down this path for twiki (not wikipedia), there's an attribute for getting the wiki code without editing (still needs some trimming though):

Code:

wget -q --no-proxy -O - http://cds.u-strasbg.fr/twikiDCA/bin/view/EuroVODCA/DCASchedule?raw=on | sed -ne '/textarea/,/textarea/{;1,2d;p;}'