[SOLVED] Help with Text manipulation

gilesaj001 · 04-05-2023, 09:26 PM

@pan64 I tried your code but it returned nothing even with the normal ones.

@Turbocapitalist Yes I sent a request on the weather.gc.ca website and got an email back today specifying exactly what you have posted. I have all the fields except Wind Chill worked out as per "MadeInGermany" post so I will just try and parse the Wind chill from the https://dd.meteo.gc.ca/citypage_weat...s0000430_e.xml file for now and maybe change it over time.

I have to look at how to do that but google is my friend

Thank you all for your help. I will let you know how I get on, but first I have to go shopping as it is Easter tomorrow and I need food and medicines. Happy Easter if you celebrate it, not sure what the conventions are these days on religious holidays.

Turbocapitalist · 04-05-2023, 10:59 PM

Quote:

Originally Posted by gilesaj001

@Turbocapitalist Yes I sent a request on the weather.gc.ca website and got an email back today specifying exactly what you have posted. I have all the fields except Wind Chill worked out as per "MadeInGermany" post so I will just try and parse the Wind chill from the https://dd.meteo.gc.ca/citypage_weat...s0000430_e.xml file for now and maybe change it over time.

Their data is well-structured and clear, so it is easy to use with an XML parser in Python or Perl.

Code:

#!/usr/bin/perl

use XML::XPath;
use XML::XPath::XMLParser;
                                                                                 
use strict;                                                                      
use warnings;

# use wget or curl to fetch the file, see the system() function
my $xml = XML::XPath->new(filename => 's0000430_e.xml');

# list of XML elements                         
my @weatherInfo = (
    'condition',
    'temperature',
    'relativeHumidity',
    'wind/speed',
    'wind/direction',
    'windChill',
    'pressure',
    );

foreach my $observation (@weatherInfo) {
    print "$observation\n";

    foreach my $measurement ($xml->findnodes("//currentConditions/$observation")) {
        print "\t", $measurement->string_value;
        my $units = $measurement->getAttribute('units');
        if ($units) {
            print " $units";
        }
        my $tendency = $measurement->getAttribute('tendency');
        if ($tendency) {
            print ", $tendency";
        }
        print "\n";
    }
}

exit(0);

Note: the above code was blocked by LQ until javascript was turned on.

gilesaj001 · 04-06-2023, 01:10 AM

Hi Turbocapitalist

Thanks for the script in perl. After a bit of searching google I had to install libxml-parser-perl and libxml-xpath-perl as I got errors but it runs now.

I setup a cron job to wget the xml file and saved it to s0000430_e.xml I put your code into a file called weather-get-data.pl and added Visibility to the list.

I ran the script and it gives the following

Code:

>perl weather-get-data.pl

condition
        Light Drizzle and Fog
temperature
        0.0 C
relativeHumidity
        100 %
wind/speed
        8 km/h
wind/direction
        NE
windChill
        -3
pressure
        101.4 kPa, falling
visibility
        0.8 km

so how do I use it? I tried to echo $visibility and nothing returned. How do I use it to put on the web site ?

I feel like a dumb ass but after four days of trying to parse from html then parse from JSON and now xml and perl my mind is a bit numb. 73 year old has hit oldtimers

Turbocapitalist · 04-06-2023, 01:37 AM

Quote:

Originally Posted by gilesaj001

so how do I use it? I tried to echo $visibility and nothing returned. How do I use it to put on the web site ?

It would depend on how you have your web site set up. Is it a static site?

One way would be to have the perl script create a full web page and dump that to a file as HTML. Or you could use Server-Side Includes to import just the weather table into your main document. But there are an infinite number of other ways too.

Please describe more about how you have your web site set up, or would like it set up, and it is likely that there are some easy methods to incorporate script output into it.

gilesaj001 · 04-06-2023, 02:14 AM

well I am not a programmer but I currently gat all the data into variables and then just display those variables on the image like this.

Code:

convert -font /usr/share/fonts/truetype/ttf-bitstream-vera/Vera.ttf -fill "$myco
lor" -pointsize 35 -draw "text 30,60 'http://dingo-den.com'"  -pointsize 25 -dra
w "text 30,100'$DATE'"  -draw "text 1100,80 'Current Temperature :     $temp
$description'"  -draw "text 1100,200 'Pressure :                        $pressur
e'" -draw "text 1100,140 'Humidity :                       $humidity'" -draw "te
xt 1100,170 'Wind :                             $windspeed'"   -draw "text 1100,
110 'Wind Chill  :                    $windchill'" -draw "text 1100,230 'Visibil
ity :                        $visibility'" CamImg.tmp2.jpg CamImg.tmp2.jpg

So what is need to do is get the values from the xml and set them as variables so they are displayed on the image.

gilesaj001 · 04-06-2023, 11:39 PM

I have been scouring google and the internet and could not find a way to save the results of "weather-get-data.pl" to variables like the ones used on my website weather image https://dingo-den.com below..

Any help comments appreciated.

Turbocapitalist · 04-07-2023, 01:07 AM

Quote:

Originally Posted by gilesaj001

I have been scouring google and the internet and could not find a way to save the results of "weather-get-data.pl" to variables like the ones used on my website weather image https://dingo-den.com below..

Any help comments appreciated.

Since you're using ImageMagick, you could add that to the Perl script via the normal API for it:

Code:

#!/usr/bin/perl -T

use XML::XPath;
use XML::XPath::XMLParser;
use Image::Magick;

use strict;
use warnings;

# use wget or curl to fetch the file
# https://dd.meteo.gc.ca/citypage_weather/xml/ON/s0000430_e.xml
my $xml = XML::XPath->new(filename => 's0000430_e.xml');

# list of XML elements from dd.meteo.gc.ca
my @weatherInfo = (
    'condition',
    'temperature',
    'relativeHumidity',
    'wind/speed',
    'wind/direction',
    'windChill',
    'pressure',
    'visibility',
    );

# pass the XML elements and the XML tree to the search subroutine
my $condition = &reading('condition', $xml);
my $visibility = &reading('visibility', $xml);
my $pressure = &reading('pressure', $xml);

# See https://imagemagick.org/script/perl-magick.php
my $graphic = new Image::Magick;

# load image to work on
$graphic->Read("CamImg.tmp2.jpg");

# set defaults for the annotations,
# can be overridden by any specific annotation below if desired
$graphic->Set(
    stroke=>'black',
    fill=>'red',
    pointsize=>28,
    font=>'LiberationSerif-Regular.ttf',
    );

$graphic->Annotate(
    text=>"Condition: $condition",
    x=>1130, y=>512,
    );

$graphic->Annotate(
    text=>"Visibility: $visibility",
    x=>1130, y=>542,
    );

$graphic->Annotate(
    text=>"Pressure: $pressure",
    x=>1130, y=>572,
    fill=>'green',
    );

# etc

# jpeg is not suitable for rendering text
$graphic->Write("/tmp/y.png");

exit(0);

sub reading {
    my ($condition, $xml) = (@_);

    # a list is the expected result from the XPath search
    my ($measurement) = $xml->findnodes("//currentConditions/$condition");

    my $reading = $measurement->string_value;
    my $units = $measurement->getAttribute('units');
    if ($units) {
	$reading = $reading . "$units";
    }
    my $tendency = $measurement->getAttribute('tendency');
    if ($tendency) {
	$reading = $reading . ", $tendency";
    }

    return($reading);
}

Perl is quite easy once you figure it out. The simultaneous advantage and disadvantage is that it is flexible and expressive. It is also very fast to write, so it becomes especially important to make an effort towards readability.

gilesaj001 · 04-07-2023, 02:37 AM

Thanks for the reply.

I installed apt install libimage-magick-perl then I created another file weather-data-test.pl and made it executable.

I copied my script for the web page to another file and took out all the variable info and added this just before the image "convert".

None of the variables came over'

Code:

perl -T weather-data-test.pl
echo "This is the Tendency" $tendency
DATE2=$(date +"%H%M%S")
if [ $DATE2 -gt 065500 ] && [ $DATE2 -lt 191000 ] ; then
     mycolor="Black"
else
     mycolor="White"
fi
echo "Text colour is " $mycolor
echo "The time is *********" $DATE2
echo "The colour is ********" $mycolor
# FOR IMAGE 1924X1084
cd /home/www/localhost/htdocs/Jpeg2
convert -font /usr/share/fonts/truetype/ttf-bitstream-vera/Vera.ttf -fill "$mycolor" -pointsize 35 -draw "text 30,60 'http://dingo-den.com'"  -pointsize 25 -draw "text 30,100'$DATE'"  -draw "text 1100,80 'Current Temperature :     $temp    $description'"  -draw "text 1100,200 'Pressure :                        $pressure    $tendency'" -draw "text 1100,140 'Humidity :                       $humidity'" -draw "text 1100,170 'Wind :                             $windspeed'"   -draw "text 1100,110 'Wind Chill  :                    $windchill'" -draw "text 1100,230 'Visibility :                        $visibility'" CamImg.tmp2.jpg CamImg.tmp2.jpg

Is there a way to pass them over or add the perl script into the bash script ??

Turbocapitalist · 04-07-2023, 08:44 PM

The script in post #37 above reads the weather service's XML file and extracts weather observations. It then reads in an image file and overlays the weather observations onto the image file before saving the modified image in a second location. The Perl script is self-contained except for the fetching of the XML data and of the image file to be modified. However, those activities could be added via the system() function.

gilesaj001 · 04-07-2023, 09:31 PM

I took all the code I had to add data to the image out and ran my script again and there was no data on the image. I am running your script inside a bash script as the bash script creates the file that is used by your script. I run your script inside the bash script with

Code:

perl -T weather-data-test.pl

but the image is blank as per attached image file

Turbocapitalist · 04-07-2023, 09:39 PM

Quote:

Originally Posted by gilesaj001

I took all the code I had to add data to the image out and ran my script again and there was no data on the image. I am running your script inside a bash script as the bash script creates the file that is used by your script. I run your script inside the bash script with

Code:

perl -T weather-data-test.pl

but the image is blank as per attached image file

If the Perl script is named weather-data-test.pl and marked executable and in /home/gilesaj001/bin, then it only needs to be called by name and you can use a wrapper script:

Code:

#!/bin/bash

PATH=/home/gilesaj001/bin:/usr/local/bin:/usr/bin:/bin

pushd /tmp/

wget https://dd.meteo.gc.ca/citypage_weather/xml/ON/s0000430_e.xml

weather-data-test.pl

popd

exit 0

Then the modified image with the weather data will be in /tmp/y.png or whatever you changed the location to in the Perl script.

gilesaj001 · 04-07-2023, 10:20 PM

I created a wrapper file call wrapper-weather.sh and put your code in it just changing the path to the scripts and made it executable

This script goes to the web site every time it is run which is every minute. I have a cron job that runs the wget at intervals so the weather site is not hit every minute. The xml is saved to a file in the Path called "s0000430_e.xml" can I use that file instead of using the wget. If so how do I use the file instead of the wget.

I ran the script and there is are errors "wrapper-weather.sh: 5: pushd: not found" and " popd: not found" ?

There is a /tmp folder

Code:

sh wrapper-weather.sh
wrapper-weather.sh: 5: pushd: not found
--2023-04-07 22:54:54--  https://dd.meteo.gc.ca/citypage_weather/xml/ON/s0000430_e.xml
Resolving dd.meteo.gc.ca (dd.meteo.gc.ca)... 205.189.10.47
Connecting to dd.meteo.gc.ca (dd.meteo.gc.ca)|205.189.10.47|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 41029 (40K) [application/xml]
Saving to: ‘s0000430_e.xml.3’

s0000430_e.xml.3     100%[======================>]  40.07K  --.-KB/s    in 0.1s

2023-04-07 22:54:54 (367 KB/s) - ‘s0000430_e.xml.3’ saved [41029/41029]

wrapper-weather.sh: 11: popd: not found

You must be losing your patience with me sorry.

Turbocapitalist · 04-07-2023, 10:25 PM

The scripts already start with a shebang, #!/bin/bash for the one, #!/usr/bin/perl for the other, so overriding that with an interpreter on the command line is not desirable.

So instead of the following,

Code:

sh wrapper-weather.sh

Try,

Code:

./wrapper-weather.sh

Try also section four of The Linux Command Line by William E Shotts.

You can disable wget by removing that line from the script or by inserting a pound sign # as the very first character.

gilesaj001 · 04-07-2023, 10:44 PM

I changed sh to ./ on the command line ( the error disappeared) and in parliament-test.sh but when I run parliament-test.sh the image is still blank.

I can attach all the scripts if you think that is safe and you can have a look.

Turbocapitalist · 04-08-2023, 12:40 AM

This line might be of interest,

Code:

$graphic->Write("/tmp/y.png");

Is there any new file named y.png in the /tmp/ directory?