[SOLVED] Need some script help scrubbing tarball list for directory name greater than

Hello, all

Figure some here are script ninjas and will know of a way to do this!

I have to pull a tarball from a remote site on a nightly basis, unpack it and run an installer from within the created directory. I only want to extract these tarballs if the contents are newer than a previous version. Otherwise, the tarball will get removed. The tarballs are not always dished out guaranteed every day ergo the need to verify the version within before using it. The version is in the directory name(s) within the tarball.

Unfortunately, the version number is not presented on the tarball following download. There is a wildcard in the download URL to direct to whatever the latest build is, resulting in the downloaded tarball version hidden behind the parenthesis with no version number. Forcing to scrub the contents.

i.e.

# software14.2-(version).tar.gz
#tar tvf software14.2-(version).tar.gz 
...
drwxr--r-- 0/0 11555 1898-01-07 05:06 software14.2-58224.443-bin/webapp/
...

The bolded version number in the dir name is what I would like to read during the script run such that only if the version is greater than say *58224*, then keep going in the script.

The script so far, without checking the tar contents

#!/bin/bash

cd /usr/local/src

wget http://domain.com/software14.2-%7Bbuild.number%7D.tar.gz
RTN=$?
if [ $RTN -ne 0 ] ; then
  tail -n10 /tmp/dwnld.log | mail -s "Download Error Report" -r "root@vm.local" "skagnola@domain.com"
else
  find *.tar.gz -ctime -1 -exec tar zxpf {} \; \
  && find *.tar.gz -ctime -1 -exec rm -rf {} \; \
  && cd $(ls -1t | head -1) \
  && ./upgrade.sh --noPrompt
RTN=$?
if [ $RTN -ne 0 ] ; then
  tail -n10 /tmp/dwnld.log | mail -s "Download Error Report" -r "root@vm.local" "skagnola@domain.com"
  fi
fi

I'm not very familiar with using awk but have seen some use this to filter the output from viewing the contents of tar? Or if there is another way to accomplish this?

Halp! heh

scrub that idea....

A way:

old=58224
new=$(tar tvf software14.2-(version).tar.gz | grep -Po -m1 '[^-]+-\K\d+(?=\.\d+-bin)')

if (( new > old )); then
  # continue with update
fi

Originally Posted by BW-userx scrub that idea....

Ha! No kidding!

Originally Posted by keefaz A way: old=58224 new=$(tar tvf software14.2-(version).tar.gz | grep -Po -m1 '[^-]+-\K\d+(?=\.\d+-bin)') if (( new > old )); then # continue with update fi

I will give this a shot and report back! Thanks, keefaz

No need of the decimal portion of the number for the comparison ?. What if only that portion increments (insufficient data presented to say).

Originally Posted by syg00 No need of the decimal portion of the number for the comparison ?. What if only that portion increments (insufficient data presented to say).

Yeah, I thought of that as well. Looked at the way the vendor is incrementing their version numbers and the longer portion is a more reliable number to base from.


So I am testing with this, a newer file version than the 58224; 58277 is within

old=58224
new=$(tar tvf software14.2-(version).tar.gz | grep -Po -m1 '[^-]+-\K\d+(?=\.\d+-bin)')
if (( new > old )); then
  find *.tar.gz -ctime -1 -exec tar zxpf {} \; && find *.tar.gz -ctime -1 -exec rm -rf {} \; && cd $(ls -1t | head -1)
else
echo "Old file"
fi

But when running it against the archive, it is echoing back "Old file". What can be added to echo out what the regex is finding, for the sake of verifying?

btw, that regex is some really cool stuff! I need to brush up on regex skills. It does really cool things!

Ahh pewp.

Apologies. The dir names inside the archive look like so, so it may affect the regex...

software14.2-SNAPSHOT-58277.489-blabla-bin/

Yes regexp needs to be retailored. Is this line model consistant across all tarballs?
Maybe you could extract the whole number then

oldMajor=58224
oldMinor=487

new=$(tar tvf software14.2-(version).tar.gz | grep -Po -m1 '\-\K\d+\.\d+(?=.*bin/)')
newMajor="${new%.*}"
newMinor="${new#*.}"

if (( newMajor > oldMajor || newMajor == oldMajor && newMinor > oldMinor ))
then
    # continue with update
fi

Bash keeps improving, but you still need to work around its shortcomings - nice keefaz.

Originally Posted by skagnola btw, that regex is some really cool stuff! I need to brush up on regex skills. It does really cool things!

Not a day goes by without me using regex. Couldn't live without it.
However ... the perlre that keefaz used is amongst the most difficult - to learn and get correct. It can frustrate as well as enthuse - I tend to use it only when "normal" regex fails me.

It's not that difficult, perlre is just another approach. I tend to use it because it feels more natural to me
eg instead of

sed -nr 's/.*SNAPSHOT-([^-]+).*bin\/$/\1/p'

(and I use -r options to avoid evil backslashes...)

keefaz, that code is amazing. It works like a charm. I'm still in awe of that stuff. From an untrained eye it looks like jibberish, but it all has a purpose!


But now, the more I'm thinking about the way this vendor tarball pull is, the script comparing versions before doing anything, I am wondering if it would be more 'fool-proof' to determine a particular file inside has a time stamp within the last 24hrs? If it does, then proceed.

There is always a file...

-rw-r--r-- 0/0  14 2018-04-06 05:58 software14.2-SNAPSHOT-58277.489-blabla-bin/VERSION

... stamped with the date it was completed. Since the vendor has had days when they did not upload a tarball - trying to figure the best way to let the script run without downloading / unpacking a version already worked with.

"Well what if they don't deploy for over 24hrs? Say over a weekend they do nothing? How would the date check work?"

I can run this only during the M-F week. From what I see in their version list history, the most they have missed are single days during the work week.

Can't thank you guys enough for the help so far!

I don't know the whole tar content, but say if you have just one file with name that contains "SNAPSHOT " into it

# extract date from filename
dateTarbal=$(tar tvf software14.2-(version).tar.gz |  awk '/SNAPSHOT/{print $4" "$5}')

# create today timestamp (seconds since Unix time)
today=$(date +%s)

# create snapshot timestamp
snapshot=$(date -d "$dateTarbal" +%s)

# now just substract the timestamps and convert result in hours
hours=$(( (today - snapshot) /60/60 ))

if (( hours > 24 )); then
  # continue with update
fi

But I think it would be better to rely on software versioning to check for update

More so if you don't know the timezone(s). Since we're talking awk, how about something like

awk -v pre=$oldver '/SNAPSHOT/ {match($0, /(.*SNAPSHOT-)([[:digit:].]+)(.*)/, a) ; if (a[2] > pre) {print "new version: "}}'

Presumes previous version in bash variable oldver - could also be passed in directly.