LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 09-02-2008, 02:43 AM   #1
thomasjunier
LQ Newbie
 
Registered: Sep 2008
Distribution: Ubuntu, SuSE, Fedora, CentOS
Posts: 6

Rep: Reputation: 0
dump HTML that contains JavaScript?


Hi,

I'm trying to dump some web page into a text file. Normally, I do it like this:

Code:
$ wget -q -O - <URL> > file.html
The problem is, part of the page's contents are downloaded through a bit of JavaScript, and wget does not seem to support it, so I end up with only part of the page. I also tried with lynx and w3m: no luck.

So basically what I need is a browser that supports JavaScript and that can dump HTML pages into files from the command line.

Any ideas?

Thanks,

Thomas
 
Old 09-03-2008, 11:41 AM   #2
keefaz
LQ Guru
 
Registered: Mar 2004
Distribution: Slackware
Posts: 6,230

Rep: Reputation: 724Reputation: 724Reputation: 724Reputation: 724Reputation: 724Reputation: 724Reputation: 724
Have you a link to the page you want to download ?
 
Old 09-05-2008, 02:59 AM   #3
thomasjunier
LQ Newbie
 
Registered: Sep 2008
Distribution: Ubuntu, SuSE, Fedora, CentOS
Posts: 6

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by keefaz View Post
Have you a link to the page you want to download ?
Well, for example this one:

http://img.jgi.doe.gov/cgi-bin/pub/m..._oid=638154501

The data I'm interested in is a list of genes which gets downloaded through a XMLHttpRequest, as far as I understand. This works with Firefox and other Javascript-enabled browsers, but not with (e.g.) wget.

TIA,

Thomas
 
Old 09-05-2008, 04:00 AM   #4
blackhole54
Senior Member
 
Registered: Mar 2006
Posts: 1,896

Rep: Reputation: 61
I searched on the terms/phrase firefox, "command line," and automate. A cursory examination led me to think that other people are trying to do similar things. In particular I wondered if this page is useful to you.

I am thinking (but not certain) that what you are trying to do is just not what wget was designed for.

Last edited by blackhole54; 09-05-2008 at 04:02 AM.
 
Old 09-05-2008, 05:10 AM   #5
theYinYeti
Senior Member
 
Registered: Jul 2004
Location: France
Distribution: Arch Linux
Posts: 1,897

Rep: Reputation: 66
You could try links -source or links -dump.

Yves.
 
Old 09-08-2008, 03:51 PM   #6
thomasjunier
LQ Newbie
 
Registered: Sep 2008
Distribution: Ubuntu, SuSE, Fedora, CentOS
Posts: 6

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by theYinYeti View Post
You could try links -source or links -dump.
Yves.
I tried this, however, and it does not work either (also tried w3m -dump, etc).

Thanks anyway,

Thomas
 
Old 09-08-2008, 03:58 PM   #7
thomasjunier
LQ Newbie
 
Registered: Sep 2008
Distribution: Ubuntu, SuSE, Fedora, CentOS
Posts: 6

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by blackhole54 View Post
I searched on the terms/phrase firefox, "command line," and automate. A cursory examination led me to think that other people are trying to do similar things. In particular I wondered if this page is useful to you.

I am thinking (but not certain) that what you are trying to do is just not what wget was designed for.
This sure looks interesting. I'll give it a try and post a follow-up with the results.

Regarding wget, yes, it seems that it was not intended that way, although there are talks of adding JavaScript capabilities as a plugin, apparently.

Thanks,

Thomas
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
tshark dump HTML page adityaj123 Linux - Networking 0 05-10-2008 02:23 PM
javascript html select sudif Programming 1 04-22-2008 01:18 AM
Javascript & HTML Tux-O-Matic Programming 1 12-07-2006 06:32 PM
Javascript / HTML <select> tag djgerbavore Programming 3 04-23-2005 11:51 AM
Html Javascript help apt Programming 3 03-21-2005 12:46 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 03:27 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration