Suggestion: script for tracking upstream updates

Luiz Ramos · 11-28-2013, 07:52 PM

Hello all,

I am a long time user of Slackware, and some things makes me think a little about things that may be lacking. One of them is how to keep the system up and running with the latest upstream version, mainly because of security concerns.

Of course Slackware security group delivers new versions when security issues hit packages Slackware packages. However, the system I typically use has some packages which come from other sources, mainly slackbuilds.org. And there are some packages which I managed to create my own slackbuild scripts, which don't have security updates, or a service which provides them in a timely manner. The bottom line is that these packages represent a hole in the whole system in terms of security.

The idea I could suggest is to build scripts which could be run for each of these packages, showing the currently installed version, the latest stable version delivered by upstream and, if possible, the URL with the latest upstream source package. Running these scripts for some or all the packages, and checking one version against the other could deliver to the system administrator a picture of what is going on in the system.

The trick is that these scripts should hint the latest version automatically, based on the upstream home page, or download page, or whatever. I tried to test this concept making such scripts for two packages I have installed in my machine (wireshark and gparted), and it seems they worked quite well for current versions. They download the "downloads" page for each project, and use 'sed' to extract the version from the contents (see below).

Now, I noticed that these scripts may be interesting also for folks which build packages for the distribution. They may help them to track upstream versions and to provide the latest packages.

I called such scripts "SlackTrackUpstream", borrowing the concept of the "SlackBuild" scripts which are coupled with (almost) every Slackware package. As the SlackBuilds, I called them <package>.SlackTrackUpstream, and assumed they should give outputs like these:

Code:

[user@machine:/home/user/gparted-0.16.2/slackbuild] $ ./gparted.SlackTrackUpstream
Local version               : 0.16.2
Upstream latest version     : 0.16.2
Upstream latest source      : http://sourceforge.net/projects/gparted/files/gparted/gparted-0.16.2/gparted-0.16.2.tar.bz2

It's quite easy to a "supervisor" script call the SlackTrackUpdate for each package of a list and check the local version to the upstream latest version for each of them. If both differ, a yellow light could be switched on, and this specific package may be subject to upgrade.

Below I provide the two scripts I built. Hope it may help somebody.

Also, if someone finds it useful, and know something about legalese, please could advise me about licences, etc.

Thanks,

Luiz Ramos
lramos dot prof at yahoo dot com dot br
São Paulo - Brazil

=== wireshark.SlackTrackUpstream =====================

Code:

#!/bin/sh

# Slackware upstream tracking script for wireshark

# Copyright 2013  Luiz Ramos, Brazil
# Originally written by Luiz Ramos (lramos.prof@yahoo.com.br)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

PRGNAM=wireshark

SRC=""
VRS=""
VARIANT[x64]=""

get_latest_src_package() {
	local _tmpfile=/tmp/slack-upstream.tmp
	local _url="http://www.wireshark.org/download.html"
	wget -q -O ${_tmpfile} ${_url}
	[ $? != 0 ] && return
	local _vrs=$(cat ${_tmpfile} | \
		sed -n -e '/The current stable release of Wireshark is/s|.\+is \([1234567890\.]\+\)\.|\1|gp')
	_pkg="http://www.wireshark.org/download/src/wireshark-${_vrs}.tar.bz2"
	echo "${_pkg}"
	rm -f ${_tmpfile}
	echo "${_vrs}"
}

get_local_vrs() {
	local LOCVRS=$(ls -1 /var/log/packages/$PRGNAM-* | \
		sed -n -e 's/.\+wireshark-\([1234567890\.]\+\)-.*/\1/g;p')
	echo $LOCVRS
}

print_output() {
	echo "Local version               : $(get_local_vrs)"

	echo "Upstream latest version     : $VRS"
	if [ -z "$VARIANT[x64]" ]; then
		echo "Upstream latest variant(x64): $VARIANT[x64]"
	fi
	echo "Upstream latest source      : $SRC"
}



# main

#set -x

set -- $(get_latest_src_package)
SRC="$1"
VRS="$2"

print_output

=== gparted.SlackTrackUpstream =====================

Code:

#!/bin/sh

# Slackware upstream tracking script for gparted

# Copyright 2013  Luiz Ramos, Brazil
# Originally written by Luiz Ramos (lramos.prof@yahoo.com.br)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

PRGNAM=gparted

SRC=""
VRS=""
VARIANT[x64]=""

get_latest_src_package() {
	local _tmpfile=/tmp/slack-upstream.tmp
	local _url="http://gparted.org/"
	wget -q -O ${_tmpfile} ${_url}
	[ $? != 0 ] && return
	local _vrs=$(cat ${_tmpfile} | \
		sed -n -e '/Stable release:/{n;s|.\+GParted \([1234567890\.]\+\).\+|\1|gp;q}')
	local _pkg=$(cat ${_tmpfile} | \
		sed -n -e '/Stable release:/{n;s|.\+href="\(.\+\)gparted-\([1234567890\.]\+\)/".*|\1gparted-\2/gparted-\2.tar.bz2|gp;q}')
	echo "${_pkg}"
	rm -f ${_tmpfile}
	echo "${_vrs}"
}

get_local_vrs() {
	local LOCVRS=$(ls -1 /var/log/packages/$PRGNAM-* | \
		sed -n -e 's/.\+gparted-\([1234567890\.]\+\)-.*/\1/g;p')
	echo $LOCVRS
}

print_output() {
	echo "Local version               : $(get_local_vrs)"

	echo "Upstream latest version     : $VRS"
	if [ -z "$VARIANT[x64]" ]; then
		echo "Upstream latest variant(x64): $VARIANT[x64]"
	fi
	echo "Upstream latest source      : $SRC"
}



# main

#set -x

set -- $(get_latest_src_package)
SRC="$1"
VRS="$2"

print_output

nobodino · 11-29-2013, 12:19 AM

I tried to experiment your script on the first file in the a series "acl" but can't get anything interesting (no last version): see acl.SlackTrackUpstream.
I'm not as experimented as you, so the reason for the failure?

Code:

#!/bin/sh

# Slackware upstream tracking script for wireshark

# Copyright 2013  Luiz Ramos, Brazil
# Originally written by Luiz Ramos (lramos.prof@yahoo.com.br)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

PRGNAM=acl

SRC=""
VRS=""
VARIANT[x64]=""

get_latest_src_package() {
	local _tmpfile=/tmp/slack-upstream.tmp
	local _url="http://download.savannah.gnu.org/releases/acl/"
	wget -q -O ${_tmpfile} ${_url}
	[ $? != 0 ] && return
	local _vrs=$(cat ${_tmpfile} | \
		sed -n -e '/The current stable release of acl is/s|.\+is \([1234567890\.]\+\)\.|\1|gp')
	_pkg="http://download.savannah.gnu.org/releases/acl/acl-${_vrs}.tar.bz2"
	echo "${_pkg}"
#	rm -f ${_tmpfile}
	echo "${_vrs}"
}

get_local_vrs() {
	local LOCVRS=$(ls -1 /var/log/packages/$PRGNAM-* | \
		sed -n -e 's/.\+acl-\([1234567890\.]\+\)-.*/\1/g;p')
	echo $LOCVRS
}

print_output() {
	echo "Local version               : $(get_local_vrs)"

	echo "Upstream latest version     : $VRS"
	if [ -z "$VARIANT[x64]" ]; then
		echo "Upstream latest variant(x64): $VARIANT[x64]"
	fi
	echo "Upstream latest source      : $SRC"
}



# main

#set -x

set -- $(get_latest_src_package)
SRC="$1"
VRS="$2"

print_output

Luiz Ramos · 11-30-2013, 07:32 AM

Quote:

I tried to experiment your script on the first file in the a series "acl" but can't get anything interesting (no last version): see acl.SlackTrackUpstream.

Yes, it's not supposed to work automatically with every site/URL/package. The tricky thing is to code some heuristics when evaluating _vrs and _pkg.

I managed to tweak your script, and it seems to work now. See below:

Code:

#!/bin/sh

# Slackware upstream tracking script for wireshark

# Copyright 2013  Luiz Ramos, Brazil
# Originally written by Luiz Ramos (lramos.prof@yahoo.com.br)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

PRGNAM=acl

SRC=""
VRS=""
VARIANT[x64]=""

get_latest_src_package() {
	local _tmpfile=/tmp/slack-upstream.tmp
	local _url="http://download.savannah.gnu.org/releases/acl/"
	wget -q -O ${_tmpfile} ${_url}
	[ $? != 0 ] && return
	local _vrs_list=$(cat ${_tmpfile} | \
		sed -n -e '/href\="acl.*.tar.gz"/s|.\+href\="acl-\([1234567890\.]\+\)\.src\+.*\([1234567890]\{2\}-...-[1234567890]\{4\}\ [1234567890:]\{5\}\).*|\1|gp')
	local _pkg_list=$(cat ${_tmpfile} | \
		sed -n -e '/href\="acl.*.tar.gz"/s|.\+href\="\(acl-[1234567890\.]\+\.src.tar.gz\)\+.*\([1234567890]\{2\}-...-[1234567890]\{4\}\ [1234567890:]\{5\}\).*|\1|gp')
	local _pkg=$(echo "$_pkg_list" | tail -1)
	local _vrs=$(echo "$_vrs_list" | tail -1)
	_pkg="${_url}${_pkg}"
	echo "${_pkg}"
	rm -f ${_tmpfile}
	echo "${_vrs}"
}

get_local_vrs() {
	local LOCVRS=$(ls -1 /var/log/packages/$PRGNAM-* | \
		sed -n -e 's/.\+acl-\([1234567890\.]\+\)-.*/\1/g;p')
	echo $LOCVRS
}

print_output() {
	echo "Local version               : $(get_local_vrs)"

	echo "Upstream latest version     : $VRS"
	if [ -z "$VARIANT[x64]" ]; then
		echo "Upstream latest variant(x64): $VARIANT[x64]"
	fi
	echo "Upstream latest source      : $SRC"
}



# main

#set -x

set -- $(get_latest_src_package)
SRC="$1"
VRS="$2"

print_output

I made two assumptions regarding the heuristics of finding the latest version: one is that the most accurate main page of acl is the one you've pointed out. So, there is no page where the upstream developers state explictly what the latest stable version is. I used pages like that in wireshark and gparted. Second, I assumed the latest stable version is the one which is at the bottom of the list. This may not be the case in all cases.

Both assumptions may not hold forever, but until forever don't come, we'll save some time trying to catch up with the latest developments...

And one last thing: I think this case you brought to discuss seems to be a specific instance of the more general case of a project which puts all versions in a directory accessible by http (or even ftp). The script above seem to be proper for a number of other packages.

Thanks, and hopes it works fine.

Luiz

nobodino · 12-01-2013, 12:59 PM

Thanks for debugging my script.
I tried with one other package from sourceforge (acpid2), where the latest is at the top and not at the bottom, the result is an empty line (for the latest), and packages from sourceforge are not so rare.

Luiz Ramos · 12-03-2013, 10:39 AM

Hello, nobodino.

Quote:

Originally Posted by nobodino

Thanks for debugging my script.
I tried with one other package from sourceforge (acpid2), where the latest is at the top and not at the bottom, the result is an empty line (for the latest), and packages from sourceforge are not so rare.

Yes, sourceforge packages are in a significant number. Checking directories is a strategy which should apply to this case, and to other as well (see, for instance, ffmpeg). But I still don't have a good solution. We may select the files which has a version information inside its name (excluding "-snapshot", "-current", and so on), and take the one whose number is higher. That's one approach. One which would be better IMHO, would be selecting the same files, but sorting them by date, and taking the most recent.

For acpid2, I managed to do one other trick as a good workaround. If you check this URL: http://sourceforge.net/projects/acpid2/files, there is a phrase telling explictly what the latest version is. I changed your script to get into this page and extract information from that phrase. It worked!

However, doing so, the more general case pictured above will be left for the next days...

Code:

#!/bin/sh

# Slackware upstream tracking script for acpid2

# Copyright 2013  Luiz Ramos, Brazil
# Originally written by Luiz Ramos (lramos.prof@yahoo.com.br)
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

PRGNAM=acpid2

SRC=""
VRS=""
VARIANT[x64]=""

get_latest_src_package() {
	local _tmpfile=/tmp/slack-upstream.tmp
	local _url="http://sourceforge.net/projects/acpid2/files/"
	wget -q -O ${_tmpfile} ${_url}
	[ $? != 0 ] && return
	local _vrs=$(cat ${_tmpfile} | \
		sed -n -e '/Looking for the latest version\?/{n;s|.\+acpid-\([1234567890\.]\+\)\..\+|\1|gp;q}')
	local _pkg=$(cat ${_tmpfile} | \
		sed -n -e '/Looking for the latest version\?/{n;s|.\+href=".\+\(acpid\)-\([1234567890\.]\+\)\.\(tar.xz\): .*|http://sourceforge.net/projects/acpid2/files/\1-\2.\3|gp;q}')
	echo "${_pkg}"
	rm -f ${_tmpfile}
	echo "${_vrs}"
}

get_local_vrs() {
	local LOCVRS=$(ls -1 /var/log/packages/$PRGNAM-* | \
		sed -n -e 's/.\+gparted-\([1234567890\.]\+\)-.*/\1/g;p')
	echo $LOCVRS
}

print_output() {
	echo "Local version               : $(get_local_vrs)"

	echo "Upstream latest version     : $VRS"
	if [ -z "$VARIANT[x64]" ]; then
		echo "Upstream latest variant(x64): $VARIANT[x64]"
	fi
	echo "Upstream latest source      : $SRC"
}



# main

#set -x

set -- $(get_latest_src_package)
SRC="$1"
VRS="$2"

print_output

Thanks,

Luiz