[SOLVED] retrieve html output in bash

StorageDon · 11-28-2016, 12:27 PM

Greetings.
If I post the URL below in a web browser, I get a one page html output.
What is the best way for me to use bash to retrieve the html page?
I want to submit this several times while changing one of the variables.

Code:

http://weather.uwyo.edu/cgi-bin/balloon_traj?TIME=2016112806&FCST=0&POINT=none&LAT=34.65&LON=-117.6&TOP=6000&CALCDROP=on&MASS=1.2&DIAM=2&Cd=0.7&OUTPUT=list&Submit=Submit&.cgifields=POINT&.cgifields=FCST&.cgifields=CALCDROP&.cgifields=TIME&.cgifields=OUTPUT

Thanks in advance

suicidaleggroll · 11-28-2016, 12:34 PM

wget would likely be the simplest option. Just put your URL in quotes and stick "wget" in front of it. By default it will write a file with the same name as the page (in this case a long and complicated mess), but you can use the -O option to name the output file how you like, or you can use "-O -" to spit to stdout so you can read it into a variable, eg:

Code:

val=$(wget -O - "http://weather.uwyo.edu/cgi-bin/balloon_traj?TIME=2016112806&FCST=0&POINT=none&LAT=34.65&LON=-117.6&TOP=6000&CALCDROP=on&MASS=1.2&DIAM=2&Cd=0.7&OUTPUT=list&Submit=Submit&.cgifields=POINT&.cgifields=FCST&.cgifields=CALCDROP&.cgifields=TIME&.cgifields=OUTPUT")

You could also use the text-based web browser lynx if you like:

Code:

val=$(lynx --dump "http://weather.uwyo.edu/cgi-bin/balloon_traj?TIME=2016112806&FCST=0&POINT=none&LAT=34.65&LON=-117.6&TOP=6000&CALCDROP=on&MASS=1.2&DIAM=2&Cd=0.7&OUTPUT=list&Submit=Submit&.cgifields=POINT&.cgifields=FCST&.cgifields=CALCDROP&.cgifields=TIME&.cgifields=OUTPUT")

Which will format the output similar to how you'd see it on a web browser.

Turbocapitalist · 11-28-2016, 12:36 PM

You'd want to fetch the web page with 'wget' or 'curl' and enclose the target URL with double quotes. See the manual pages for each for the many options.

But then you have the issue of what information you want extracted from that web page and how to extract it. There are a lot of ways.

szboardstretcher · 11-28-2016, 12:37 PM

Or use the script friendly --data way:

Code:

curl -G -v "http://weather.uwyo.edu/cgi-bin/balloon_traj" --data "TIME=2016112806" --data "FCST=0" --data "POINT=none" --data "LAT=34.65" --data "LON=-117.6" --data "TOP=6000" --data "CALCDROP=on" --data "MASS=1.2" --data "DIAM=2" --data "Cd=0.7" --data "OUTPUT=list" --data "Submit=Submit" --data ".cgifields=POINT" --data ".cgifields=FCST" --data ".cgifields=CALCDROP" --data ".cgifields=TIME" --data ".cgifields=OUTPUT"

More easily readable as:

Code:

curl -G -v "http://weather.uwyo.edu/cgi-bin/balloon_traj" \
--data "TIME=2016112806" \
--data "FCST=0" \
--data "POINT=none" \
--data "LAT=34.65" \
--data "LON=-117.6" \
--data "TOP=6000" \
--data "CALCDROP=on" \
--data "MASS=1.2" \
--data "DIAM=2" \
--data "Cd=0.7" \
--data "OUTPUT=list" \
--data "Submit=Submit" \
--data ".cgifields=POINT" \
--data ".cgifields=FCST" \
--data ".cgifields=CALCDROP" \
--data ".cgifields=TIME" \
--data ".cgifields=OUTPUT"

StorageDon · 11-28-2016, 01:06 PM

Quote:

Originally Posted by szboardstretcher

Or use the script friendly --data way:

Code:

curl -G -v "http://weather.uwyo.edu/cgi-bin/balloon_traj" --data "TIME=2016112806" --data "FCST=0" --data "POINT=none" --data "LAT=34.65" --data "LON=-117.6" --data "TOP=6000" --data "CALCDROP=on" --data "MASS=1.2" --data "DIAM=2" --data "Cd=0.7" --data "OUTPUT=list" --data "Submit=Submit" --data ".cgifields=POINT" --data ".cgifields=FCST" --data ".cgifields=CALCDROP" --data ".cgifields=TIME" --data ".cgifields=OUTPUT"

This works the best for me. I'll post a new thread if i need help parsing the data, but I think I have that.

Thanks all for your super fast response!