LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 11-28-2016, 12:27 PM   #1
StorageDon
LQ Newbie
 
Registered: May 2013
Posts: 10

Rep: Reputation: Disabled
retrieve html output in bash


Greetings.
If I post the URL below in a web browser, I get a one page html output.
What is the best way for me to use bash to retrieve the html page?
I want to submit this several times while changing one of the variables.

Code:
http://weather.uwyo.edu/cgi-bin/balloon_traj?TIME=2016112806&FCST=0&POINT=none&LAT=34.65&LON=-117.6&TOP=6000&CALCDROP=on&MASS=1.2&DIAM=2&Cd=0.7&OUTPUT=list&Submit=Submit&.cgifields=POINT&.cgifields=FCST&.cgifields=CALCDROP&.cgifields=TIME&.cgifields=OUTPUT
Thanks in advance

Last edited by StorageDon; 11-28-2016 at 12:28 PM.
 
Old 11-28-2016, 12:34 PM   #2
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,573

Rep: Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142Reputation: 2142
wget would likely be the simplest option. Just put your URL in quotes and stick "wget" in front of it. By default it will write a file with the same name as the page (in this case a long and complicated mess), but you can use the -O option to name the output file how you like, or you can use "-O -" to spit to stdout so you can read it into a variable, eg:
Code:
val=$(wget -O - "http://weather.uwyo.edu/cgi-bin/balloon_traj?TIME=2016112806&FCST=0&POINT=none&LAT=34.65&LON=-117.6&TOP=6000&CALCDROP=on&MASS=1.2&DIAM=2&Cd=0.7&OUTPUT=list&Submit=Submit&.cgifields=POINT&.cgifields=FCST&.cgifields=CALCDROP&.cgifields=TIME&.cgifields=OUTPUT")
You could also use the text-based web browser lynx if you like:
Code:
val=$(lynx --dump "http://weather.uwyo.edu/cgi-bin/balloon_traj?TIME=2016112806&FCST=0&POINT=none&LAT=34.65&LON=-117.6&TOP=6000&CALCDROP=on&MASS=1.2&DIAM=2&Cd=0.7&OUTPUT=list&Submit=Submit&.cgifields=POINT&.cgifields=FCST&.cgifields=CALCDROP&.cgifields=TIME&.cgifields=OUTPUT")
Which will format the output similar to how you'd see it on a web browser.

Last edited by suicidaleggroll; 11-28-2016 at 12:36 PM.
 
Old 11-28-2016, 12:36 PM   #3
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 7,419
Blog Entries: 3

Rep: Reputation: 3788Reputation: 3788Reputation: 3788Reputation: 3788Reputation: 3788Reputation: 3788Reputation: 3788Reputation: 3788Reputation: 3788Reputation: 3788Reputation: 3788
You'd want to fetch the web page with 'wget' or 'curl' and enclose the target URL with double quotes. See the manual pages for each for the many options.

But then you have the issue of what information you want extracted from that web page and how to extract it. There are a lot of ways.
 
Old 11-28-2016, 12:37 PM   #4
szboardstretcher
Senior Member
 
Registered: Aug 2006
Location: Detroit, MI
Distribution: GNU/Linux systemd
Posts: 4,278

Rep: Reputation: 1694Reputation: 1694Reputation: 1694Reputation: 1694Reputation: 1694Reputation: 1694Reputation: 1694Reputation: 1694Reputation: 1694Reputation: 1694Reputation: 1694
Or use the script friendly --data way:

Code:
curl -G -v "http://weather.uwyo.edu/cgi-bin/balloon_traj" --data "TIME=2016112806" --data "FCST=0" --data "POINT=none" --data "LAT=34.65" --data "LON=-117.6" --data "TOP=6000" --data "CALCDROP=on" --data "MASS=1.2" --data "DIAM=2" --data "Cd=0.7" --data "OUTPUT=list" --data "Submit=Submit" --data ".cgifields=POINT" --data ".cgifields=FCST" --data ".cgifields=CALCDROP" --data ".cgifields=TIME" --data ".cgifields=OUTPUT"
More easily readable as:
Code:
curl -G -v "http://weather.uwyo.edu/cgi-bin/balloon_traj" \
--data "TIME=2016112806" \
--data "FCST=0" \
--data "POINT=none" \
--data "LAT=34.65" \
--data "LON=-117.6" \
--data "TOP=6000" \
--data "CALCDROP=on" \
--data "MASS=1.2" \
--data "DIAM=2" \
--data "Cd=0.7" \
--data "OUTPUT=list" \
--data "Submit=Submit" \
--data ".cgifields=POINT" \
--data ".cgifields=FCST" \
--data ".cgifields=CALCDROP" \
--data ".cgifields=TIME" \
--data ".cgifields=OUTPUT"

Last edited by szboardstretcher; 11-28-2016 at 12:42 PM. Reason: readable-ity
 
1 members found this post helpful.
Old 11-28-2016, 01:06 PM   #5
StorageDon
LQ Newbie
 
Registered: May 2013
Posts: 10

Original Poster
Rep: Reputation: Disabled
This works.

Quote:
Originally Posted by szboardstretcher View Post
Or use the script friendly --data way:

Code:
curl -G -v "http://weather.uwyo.edu/cgi-bin/balloon_traj" --data "TIME=2016112806" --data "FCST=0" --data "POINT=none" --data "LAT=34.65" --data "LON=-117.6" --data "TOP=6000" --data "CALCDROP=on" --data "MASS=1.2" --data "DIAM=2" --data "Cd=0.7" --data "OUTPUT=list" --data "Submit=Submit" --data ".cgifields=POINT" --data ".cgifields=FCST" --data ".cgifields=CALCDROP" --data ".cgifields=TIME" --data ".cgifields=OUTPUT"
This works the best for me. I'll post a new thread if i need help parsing the data, but I think I have that.

Thanks all for your super fast response!
 
  


Reply

Tags
html, http, post


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Bash; awk or sed output to variable: how keep newline at end of each output line porphyry5 Programming 3 06-10-2011 05:50 PM
c++ retrieve output of whoami Endarur Programming 3 03-08-2011 08:38 AM
[SOLVED] storing grep output to feed awk to retrieve entire records matching variable chargaff Programming 8 08-13-2010 06:10 AM
[SOLVED] Getting xdebug 2.0.5 switch text output to html output lhorace Linux - Server 1 11-01-2009 05:23 PM
Bash CGI / HTML output on one line sir-lancealot Programming 1 01-07-2009 03:16 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 07:17 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration