dump HTML that contains JavaScript?

thomasjunier · 09-02-2008, 01:43 AM

Hi,

I'm trying to dump some web page into a text file. Normally, I do it like this:

Code:

$ wget -q -O - <URL> > file.html

The problem is, part of the page's contents are downloaded through a bit of JavaScript, and wget does not seem to support it, so I end up with only part of the page. I also tried with lynx and w3m: no luck.

So basically what I need is a browser that supports JavaScript and that can dump HTML pages into files from the command line.

Any ideas?

Thanks,

Thomas

keefaz · 09-03-2008, 10:41 AM

Have you a link to the page you want to download ?

thomasjunier · 09-05-2008, 01:59 AM

Quote:

Originally Posted by keefaz

Have you a link to the page you want to download ?

Well, for example this one:

http://img.jgi.doe.gov/cgi-bin/pub/m..._oid=638154501

The data I'm interested in is a list of genes which gets downloaded through a XMLHttpRequest, as far as I understand. This works with Firefox and other Javascript-enabled browsers, but not with (e.g.) wget.

TIA,

Thomas

blackhole54 · 09-05-2008, 03:00 AM

I searched on the terms/phrase firefox, "command line," and automate. A cursory examination led me to think that other people are trying to do similar things. In particular I wondered if this page is useful to you.

I am thinking (but not certain) that what you are trying to do is just not what wget was designed for.

theYinYeti · 09-05-2008, 04:10 AM

You could try links -source or links -dump.

Yves.

thomasjunier · 09-08-2008, 02:51 PM

Quote:

Originally Posted by theYinYeti

You could try links -source or links -dump.
Yves.

I tried this, however, and it does not work either (also tried w3m -dump, etc).

Thanks anyway,

Thomas

thomasjunier · 09-08-2008, 02:58 PM

Quote:

Originally Posted by blackhole54

I searched on the terms/phrase firefox, "command line," and automate. A cursory examination led me to think that other people are trying to do similar things. In particular I wondered if this page is useful to you.

I am thinking (but not certain) that what you are trying to do is just not what wget was designed for.

This sure looks interesting. I'll give it a try and post a follow-up with the results.

Regarding wget, yes, it seems that it was not intended that way, although there are talks of adding JavaScript capabilities as a plugin, apparently.

Thanks,

Thomas