[SOLVED] Help with Text manipulation

gilesaj001 · 04-04-2023, 02:01 AM

Quote:

Originally Posted by pan64

Code:

awk '/[[]37]/,/[[]39]/'

is almost perfect, just you need to cut a few lines

So I am 73 and I don't want to appear as a dumbass but I have no clue how to run that command against the weather-data.txt file ??

gilesaj001 · 04-04-2023, 02:06 AM

Sorry double post

pan64 · 04-04-2023, 02:11 AM

there are two ways:

Code:

lynx ... | awk '/[[]37]/,/[[]39]/' | head --lines=-2 | | tail --lines=+3
# or
awk '/[[]37]/,/[[]39]/' filename | ...

gilesaj001 · 04-04-2023, 09:16 AM

OK still no luck but I found a program to convert the https://weather.gc.ca/city/pages/on-118_metric_e.html into JSON which I might have a chance of parcing ??

This is a link to the github page but not sure how to install in Ubuntu or how to run it. https://github.com/jschnurr/ec-weather

Installation

Code:

npm install ec-weather

API
The module is called with a single options object.

Code:

const ecweather = require('ec-weather');

var options = {
  lang: 'en',
  city: 'on-118',
};

ecweather(options)

Any help appresiated.

Turbocapitalist · 04-04-2023, 09:38 AM

I thought it would be just a matter of looking that up on their extensive pages, but if they have that information visible it is buried in all the noise there. I suspect it is a matter of just finding the right query string and then parsing the resulting JSON.

So I would definitely contact them for a short example on what HTTP GET or POST query to use to fetch the data you are looking for in JSON format. https://weather.gc.ca/mainmenu/contact_us_e.html

teckk · 04-04-2023, 10:40 AM

Couple of basic examples:

Code:

agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0"

url="https://weather.gc.ca/city/pages/on-118_metric_e.html"

curl -A "$agent" "$url" -o - | html2text - | less

Or:

Code:

#!/usr/bin/python

from urllib import request, error
from html2text import html2text, HTML2Text
from pydoc import pager

agent = ('Mozilla/5.0 (iPhone; CPU iPhone OS 16_2 like Mac OS X) '
        'AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/108.0.5359.112 '
        'Mobile/15E148 Safari/604.1')
           
header = {
        'User-Agent': agent,
        'Accept': 'text/html,application/xhtml+xml,'
        'image/avif,image/webp,image/apng,'
        'application/signed-exchange;v=b3;q=0.9'  
        'application/xml;q=0.9,*/*;q=0.8',
        'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
        'Accept-Language': 'en-US,en;q=0.8',
        }
        
url = 'https://weather.gc.ca/city/pages/on-118_metric_e.html'
#url = 'file:///path/to/weather.html'

try:
    req = request.Request(url, data=None, headers=header)
    page = request.urlopen(req)

except error.HTTPError as e:
    print("Http error")
except error.URLError as e:
    print('Url Error')
except TypeError as e:
    print('Type Error')
except ValueError as e:
    print("Value error")

try:
    html = page.read().decode('utf-8', 'ignore')

except NameError:
    print('Name error')

noLinks = HTML2Text()
noLinks.ignore_links = False
txt = noLinks.handle(html)

with open('weather.txt', 'a') as f:
    f.write(txt)

#print(txt)
pager(txt)

You can further parse that to get just what you want.

gilesaj001 · 04-04-2023, 08:40 PM

I found a program on github that converts the https://weather.gc.ca/city/pages/on-118_metric_e.html page to JSON format. With this I can change the existing script and point it to the new file and get the correct weather.

The program is at https://github.com/jschnurr/ec-weather but I don't know how to install on Ubuntu 20.04 and run it in ?

Code:

Installation
npm install ec-weather
API
The module is called with a single options object.

const ecweather = require('ec-weather');

var options = {
  lang: 'en',
  city: 'on-118',
};

Any help appresiated.

Turbocapitalist · 04-04-2023, 10:08 PM

Quote:

Originally Posted by gilesaj001

I found a program on github that converts the https://weather.gc.ca/city/pages/on-118_metric_e.html page to JSON format. With this I can change the existing script and point it to the new file and get the correct weather.

That program scrapes the HTML to attempt to produce JSON instead of using the weather service's own JSON. Somewhere the weather service has a link which you can query to get the JSON directly and thus avoid the brittleness which comes from trying to scrape. There's a feedback form at the bottom of the page you link to, if nothing else try contacting the weather service to let them know that their API is more or less hidden.

Edit: if you can get the API link from the weather service and post it here any number of us can provide an example of how to use the JSON it provides

gilesaj001 · 04-04-2023, 11:16 PM

@Turbocapitalist Thanks for the reply and I have sent a message to the weather.gc.ca site for information on a JSON lor API link.

@Treckk I tried your scripts and got these errors

I tried the first option and ran with sh and bash and got this error:

Code:

(END)  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  3  225k    3  7699    0     0  12417      0  0:00:18 --:--:--  0:00:18 12417
curl: (23) Failed writing body (0 != 7699)

I tried the second option and get an error if I run as bash or sh:

Code:

from: can't read /var/mail/urllib
from: can't read /var/mail/html2text
from: can't read /var/mail/pydoc
strip_weather.sh: 7: Syntax error: "(" unexpected

MadeInGermany · 04-05-2023, 02:58 AM

Perhaps a little shell script to pull a certain item from a text file?

Code:

cat witem

Code:

#!/bin/bash
while read line
do
 line=${line%$'\r'} # Remove a WinDOS CR
 case $line in ( "$1:" )
   read line2
   line2=${line2%$'\r'}
   echo "$line2"
   break # Stop at first match
 esac
done < weather-data.txt

Run as

Code:

bash witem Visibility

Or make it executable and run it like a program:

Code:

./witem Tendency

gilesaj001 · 04-05-2023, 05:49 AM

@MadeInGermany

That is exactly what I was looking for and it works great. It works for all the fields except Wind Chill because there is also a field called Wind. I tried to put "Wind Chill" in brackets and also tried Wind+Chill but Wind was still found.

If you have an answer to that you will be my champion.

https://dingo-den.com/index.php?nav=cam1

MadeInGermany · 04-05-2023, 09:35 AM

You can search a keyword with spaces in the usual way:

Code:

./witem "Observed at"

Maybe the reason why it does not find "Wind chill" is that, unlike the other keywords, there is no colon after it.
And without the colon it will find "Wind chill" 3 times...

gilesaj001 · 04-05-2023, 10:56 AM

Quote:

Originally Posted by MadeInGermany

You can search a keyword with spaces in the usual way:

Code:

./witem "Observed at"

Maybe the reason why it does not find "Wind chill" is that, unlike the other keywords, there is no colon after it.
And without the colon it will find "Wind chill" 3 times...

I tried to run

Code:

./witem "Wind Chill"

again but it did not return anything ?

The first wind chill in the file at the moment is [38]Wind Chill:
-10

so I ran

Code:

./witem "[38]Wind Chill"

and it works so I hope that number does not change.

Well that didn't work the number goes up it is now 40 so I will have to leave wind chill off for now

Thanks for your help it has been a struggle for me but I have almost everything now. I would like to be able to capture the warnings but there doesn't seem to be anything to search for. Sometimes they are there other times they are not.

It's 01:50 here time for bed.

pan64 · 04-05-2023, 11:05 AM

in that case you need to use regex, that will make it a bit different:

Code:

#!/bin/bash
while read line
do
 line=${line%$'\r'} # Remove a WinDOS CR
 [[ $line =~ "[]][0-9]*[[]$1:" ]] || continue
 read line2
 line2=${line2%$'\r'}
 echo "$line2"
done < weather-data.txt

(not tested)

Turbocapitalist · 04-05-2023, 01:46 PM

It turns out I was mistaken about JSON, they have very skillfully done XML instead. It'll save a lot of trouble and brittleness.

Lookup your station:

https://dd.meteo.gc.ca/citypage_weather/xml/

https://dd.meteo.gc.ca/citypage_weat...l/siteList.xml

Then go to the right subdirectory and find the XML file to parse using Python or Perl. For example,

https://dd.meteo.gc.ca/citypage_weat...s0000430_e.xml

for current conditions at that location in English.