LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 06-09-2021, 07:09 PM   #1
Johng
Member
 
Registered: Feb 2002
Location: NZ
Distribution: Mint Suse
Posts: 356

Rep: Reputation: 30
Using sed to modify the date/time in a gpx file


I have some gpx files that I want to modify the date/time. Typically the lines look like:
Code:
    <time>2001-10-23T21:02:56.000Z</time>
    <time>2001-10-23T21:26:27.000Z</time>
    <time>2001-10-23T22:57:21.000Z</time>
    <time>2001-10-23T23:00:10.000Z</time>
    <time>2001-10-24T00:49:47.000Z</time>
    <time>2001-10-24T01:56:57.000Z</time>
    <time>2001-10-24T02:39:59.000Z</time>
I have created a script to modify the dates in the file, by adding 7168 days to the date:
Code:
#!/bin/bash
echo GPX files:  `ls -1 *.gpx`
echo
echo -n "Enter filename: "
read -e FILENAMEX

FILENAME=$FILENAMEX
echo

Day1=$(grep -m1 'time' $FILENAME)
Day2=$(grep time  $FILENAME | tail -1)

echo "Day1: "  "$Day1"
echo "Day2: "  "$Day2"
echo

MYVAR="$Day1"
Date1=${MYVAR%T*}
echo "Date1: ""$Date1"

MYVAR="$Date1"
Date1x=${MYVAR##*>}

MYVAR="$Day2"
Date2=${MYVAR%T*}
echo "Date2: ""$Date2"

MYVAR="$Date2"
Date2x=${MYVAR##*>}

echo "$Date1x   $Date2x"

TrueDate1=$(date -d "$Date1x 7168 days"  +%F)

TrueDate2=$(date -d "$Date2x 7168 days"  +%F)

echo "$TrueDate1   $TrueDate2"

sed  "s/$Date1x/$TrueDate1/g; s/$Date2x/$TrueDate2/g" "$FILENAME" > "$TrueDate2".gpx

FILENAMEz="$TrueDate2".gpx

Day1t=$(grep -m1 'time' $FILENAMEz)
Day2t=$(grep time  $FILENAMEz | tail -1)

echo "Day1t: "  "$Day1t"
echo "Day2t: "  "$Day2t"
Now I need help to change the two digit value after the "T" in each line by adding 12 if the hour is less than 12, or minus 12 if the hour is greater than 12
 
Old 06-10-2021, 12:21 AM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 19,705

Rep: Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548
I admire anyone who insists on fighting away with bash rather than use a real language for this sort of thing. I appreciate there is new functionality constantly being added, but it always looks like something just tacked on. Anyway, enuff of that ...

Were it me, I'd simply pull out the hours, add 12 and use modulo 24 to get the remainder. Much simpler and bash has all those operators.
KISS.
 
Old 06-10-2021, 01:11 AM   #3
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 17,237
Blog Entries: 10

Rep: Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160
Your file is XML.
Use an XML tool to modify it.
I have a fairly similar date-changing situation here:
Code:
recentfile=~/.local/share/recently-used.xbel
item=file:///path/to/some.file
TZ=UTC printf -v time "%(%FT%R:%S.499999Z)T" # bash doesn't deal with fractions of seconds, they have been replaced with 499999
xmlstarlet ed -u "/xbel/bookmark[@href=\"$item\"]/@visited" -v "$time" "$recentfile" # possibly need to append: 2>/dev/null
It's a little confusing, but believe me: you want xmlstarlet for this if you are doing it inside a shell script.
If you have ~/.local/share/recently-used.xbel on your system you can check exactly how it works.
Also see: http://xmlstar.sourceforge.net/doc/UG/ch04.html
 
1 members found this post helpful.
Old 06-10-2021, 02:27 AM   #4
shruggy
Senior Member
 
Registered: Mar 2020
Posts: 2,321

Rep: Reputation: Disabled
My take on it.
Code:
#!/bin/bash

# dateutils have non-standard names on Debian-based systems
. /etc/os-release
[[ $ID_LIKE =~ debian ]] && dateadd=dateutils.dadd || dateadd=dateadd

for f in *.gpx
do
  echo "== $f =="
  # xmlstarlet is picky about namespaces
  #   - WP example contains xmlns attributes: use default NS
  #   - OSM example doesn't: don't use NS
  grep -q 'xmlns=' "$f" && ns=_: || ns=

  timestamps=(
    $(xmlstarlet sel -t -v //${ns}trkpt/${ns}time -n "$f" | $dateadd 7168d)
  )
  args=
  for k in "${!timestamps[@]}"
  do
    args+=" -u //${ns}trkpt[$((k+1))]/${ns}time"
    args+=" -v ${timestamps[$k]}.000Z"
  done
  xmlstarlet ed $args "$f"
  echo =============
done
 
Old 06-10-2021, 02:39 AM   #5
Johng
Member
 
Registered: Feb 2002
Location: NZ
Distribution: Mint Suse
Posts: 356

Original Poster
Rep: Reputation: 30
Hi syg00
Thank you, I agree that all I need to do is simply pull out the hours, add 12, and put it back. That's where I have hit the wall, looking at grep and sed, and something like IF(T1>12,T2=12,T1) etc

Thank you ondoho
I appreciate that it is an XML file. The script I quoted above is an extract from a larger script, created to do several other operations. It's a coincidence that my question happens to refer to a KML file this time.
 
Old 06-10-2021, 03:01 AM   #6
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 16,490

Rep: Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532
Quote:
Originally Posted by Johng View Post
Hi syg00
Thank you, I agree that all I need to do is simply pull out the hours, add 12, and put it back. That's where I have hit the wall, looking at grep and sed, and something like IF(T1>12,T2=12,T1) etc
As syg00 wrote it is not really implementable [easily] in shell, using a huge amount of different sed/grep/date/shell commands is not the right way to handle complex structured files (like xml). Additionally parsing date/time fields is not an easy case. I would rather try to convert them to epoch time (with date) and calculate what you want (and format at the end). But xmlstarlet is still a much better choice.
Quote:
Originally Posted by Johng View Post
Thank you ondoho
I appreciate that it is an XML file. The script I quoted above is an extract from a larger script, created to do several other operations. It's a coincidence that my question happens to refer to a KML file this time.
That is more or less irrelevant, if you want to handle those values you need to implement a function (or new script) to do that. Other parts of that larger script is now out of scope.
 
Old 06-10-2021, 03:59 AM   #7
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 19,705

Rep: Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548Reputation: 3548
Quote:
Originally Posted by pan64 View Post
As syg00 wrote it is not really implementable [easily] in shell,
Allow me to demur.
The data appear well structured, so substringing to extract the hour value is trivial given the OPs grasp of bash demonstrated so far. Basic arithmetic to adjust and sed to plug it back in.
Seems to fit with the requirements even if I (we ?) wouldn't necessarily do it that way.
 
Old 06-10-2021, 04:44 AM   #8
Johng
Member
 
Registered: Feb 2002
Location: NZ
Distribution: Mint Suse
Posts: 356

Original Poster
Rep: Reputation: 30
Thank you syg00
As you have noted, the exercise is not how to modify a KML file.
I have chosen to ignore that the two digits have anything to do with time. All I am asking is how do I change the value of a changing two digit numeral in repeating lines by adding or subtracting 12 depending on whether the two digits represent a value greater than or less than 12.

Thank you shruggy and pan64, I will study your contributions in the morning
 
Old 06-10-2021, 05:03 AM   #9
shruggy
Senior Member
 
Registered: Mar 2020
Posts: 2,321

Rep: Reputation: Disabled
Wait, is it a KML or a GPX file? Because they are different.
Quote:
Originally Posted by Johng View Post
how do I change the value of a changing two digit numeral in repeating lines by adding or subtracting 12 depending on whether the two digits represent a value greater than or less than 12.
Did you pay attention to what syg00 wrote above?
Quote:
Originally Posted by syg00 View Post
I'd simply pull out the hours, add 12 and use modulo 24 to get the remainder.
Code:
#!/bin/bash
for f in *.gpx
do
  echo "== $f =="
  hours=(
    $(sed -En 's/.*T([0-2][0-9]):.*/\1/p' "$f")
  )
  for h in "${hours[@]}"
  do echo $h : $(( (10#$h+12)%24 ))
  done
  echo
done
Instead of sed the command could be e.g.
Code:
grep -Po '\d\dT\K\d\d(?=:)' "$f"
but sed is probably more efficient.

Last edited by shruggy; 06-10-2021 at 05:42 AM.
 
Old 06-10-2021, 01:38 PM   #10
MadeInGermany
Senior Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 1,741

Rep: Reputation: 790Reputation: 790Reputation: 790Reputation: 790Reputation: 790Reputation: 790Reputation: 790
Well, I think the .* and substitution overhead make sed slower than grep.

For these things perl is good:
Code:
#!/bin/bash
for f in *.gpx
do
  echo "== $f =="
  perl -ne 'm/\d\dT(\d\d):/ and printf "%02d : %02d\n", $1, ($1+12)%24' "$f"
  echo
done
The 1st parenthesis lands in $1, and so forth.
The m operator allows delimiters other than /
(just like the s operator in perl and sed)!
Without the m it must be /
You see the many bridges for migrating shell scripts to perl.

Of course you could run perl directly on the input files...
 
Old 06-10-2021, 06:40 PM   #11
Johng
Member
 
Registered: Feb 2002
Location: NZ
Distribution: Mint Suse
Posts: 356

Original Poster
Rep: Reputation: 30
Quote:
Originally Posted by shruggy
Wait, is it a KML or a GPX file? Because they are different.
Why does it matter? For the purpose of my inquiry I am asking how to modify a file that has lines with a particular pattern; something I understood was a function suited to sed.

The shruggy bash script produces an output like
Quote:
02 : 14
03 : 15
Where the last two digits are the values that have been increased/decreased by 12. The grep script extracts only the uncorrected value.

MadeInGermany's perl script produces an output like shruggy's.

What I am trying to achieve is command I can add to the script in my first post that says something like:

If there are lines in the file that looks like "<time>2021-05-23T20:55:13.000Z</time>" and the value after the "T" is greater than 12 subtract 12, if the value after the "T" is less than 12 add 12.

So lines like:
Quote:
<time>2021-05-23T20:55:13.000Z</time>
<time>2021-05-23T03:55:13.000Z</time>
are transformed to:
Quote:
<time>2021-05-23T08:55:13.000Z</time>
<time>2021-05-23T15:55:13.000Z</time>
Thank you for your interest to date. I hope this clarifies the question.
 
Old 06-11-2021, 01:03 AM   #12
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 17,237
Blog Entries: 10

Rep: Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160Reputation: 5160
Quote:
Originally Posted by Johng View Post
I appreciate that it is an XML file. The script I quoted above is an extract from a larger script, created to do several other operations. It's a coincidence that my question happens to refer to a KML file this time.
Like I said, if you want to do this inside a shell script, you probably want xmlstarlet.

The file's extension is irrelevant, and various subsets of XML exist (like the aforementioned .xbel). You could show us a complete example file, we will tell you. Actually strike that, the first few lines of the file will tell you:
Code:
$> head -4 ~/.local/share/recently-used.xbel
<?xml version="1.0" encoding="UTF-8"?>
<xbel version="1.0"
      xmlns:bookmark="http://www.freedesktop.org/standards/desktop-bookmarks"
      xmlns:mime="http://www.freedesktop.org/standards/shared-mime-info"

Quote:
Originally Posted by Johng View Post
Why does it matter? For the purpose of my inquiry I am asking how to modify a file that has lines with a particular pattern; something I understood was a function suited to sed.
You were told that you most probably have a screw there, and should therefore use a screwdriver to screw it in.
Are you now telling us that it works well enough with a hammer, too?

That's fine, I don't really care about your workflow, I'm just telling you from my personal experience: evtl. I recognized that an initial bump in learning a new tool will save me a lot of work down the way, and make my scripts more robust. Because, while you have all values neatly in lines right now, XML doesn't require that at all.
 
Old 06-11-2021, 01:54 AM   #13
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 16,490

Rep: Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532Reputation: 5532
Quote:
Originally Posted by Johng View Post
Why does it matter? For the purpose of my inquiry I am asking how to modify a file that has lines with a particular pattern; something I understood was a function suited to sed.
To understand it better: you can modify Txx to Tyy, but you need to ensure it will only happen in between <time> and </time> and not elsewhere. But [obviously] you can ignore it.
Also sed is not the right tool to make decision based calculations (like if xx>12 than use xx-12 else use xx+12). Also sed it not really useful when you want to parse datetime fields. But [obviously] you can try that.

I see you checked the suggestions you got. Did you try to put it in your script?
 
Old 06-11-2021, 02:11 AM   #14
shruggy
Senior Member
 
Registered: Mar 2020
Posts: 2,321

Rep: Reputation: Disabled
@OP. Have you tried my script in #4? I'm asking because I suspect the XY problem here. That is, I don't see the point in adjusting the time at all if the script correctly adds 7168 days to the date. Consider the difference in the output of
Code:
date -d '2001-10-23T23:00:10.000Z 7168 days' +%F
and
Code:
TZ=UTC0 date -d '2001-10-23T23:00:10.000Z 7168 days' +%F
From the GNU Coreutils manual:
Quote:
Also, take care when manipulating dates around clock changes such as daylight saving leaps. In a few cases these have added or subtracted as much as 24 hours from the clock, so it is often wise to adopt universal time by setting the TZ environment variable to ‘UTC0’ before embarking on calendrical calculations.
 
Old 06-11-2021, 04:37 AM   #15
MadeInGermany
Senior Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 1,741

Rep: Reputation: 790Reputation: 790Reputation: 790Reputation: 790Reputation: 790Reputation: 790Reputation: 790
With perl's /e modifier you can substitute by an expression
Code:
perl -pe 's/(?<=T)\d\d/sprintf("%02d", ($&+12)%24)/e' input.gpx
Substitute the two digits right from the T by a 0-padded computed number.
 
  


Reply

Tags
xml


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
GPSbabel convert from gpx file to csv file Can't get all elements of waypoints pizzipie Linux - General 2 11-23-2015 10:10 AM
.gpx (GPS) analysis software enine Linux - Software 0 04-12-2011 07:21 AM
Problem copy GPX files from cell phone dagbj Linux - Software 2 07-13-2010 08:35 AM
TFTP and Grandstream GPX 2000 Update metallica1973 Linux - Server 0 06-18-2008 01:10 AM
Start Date + Time Duration = End Date/Time calculator? ToBe Linux - General 3 09-26-2005 10:17 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 12:36 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration