How do you guys maintain packages from Robby, Eric, and others?

agi93 · 06-29-2010, 07:33 PM

I recently started to get my large packages (like openoffice, texlive, and the VLC SlackBuild) from Robby and Eric's webpages. Is there an easy way to keep these packages up to date? The only way I know of is manually checking their websites periodically to see if there has been a change in the package version, but I would much rather be notified in some way when there is an update available.

Does anyone have a script or similar system to do this? I ask here because I want to know if other Slackers have already figured out a better method than what I would be able to think of.

Thanks for any help.

kingbeowulf · 06-29-2010, 10:02 PM

If it ain't on Slackbuilds.org I pretty much ignore it, if possible. SBo has gotten a good selection so meets 95% of my needs - and I only bother with about 5% of what there!. Although I run VLC in my Win boxes, I stick with mplayer + gnome-mplayer + gecko-mediaplayer on Slackware since the dependency headaches are less. Openoffice is also on SBo for both x68 and x86_64. It is easy enough to adjust the scripts for your own preferences (configure options etc).

Even if SBo runs a version (or minor version) behind, their scripts usually don't take too much tweaking to allow compile/packaging of the newer version.

IIRC, there is a script that assists in tracking the SBo content and updates, but at the moment the name eludes me.

damgar · 06-29-2010, 10:07 PM

sbopkg can be used to track and upgrade the packages and you built/installed from SBo using sbopkg if I'm not mistaken. With Robby and Eric's packages I dunno. I've been following Eric's KDE packages, but I do that manually.

BILLYKANE · 06-29-2010, 10:50 PM

I use slapt-get/gslapt to manager packages from slacky.eu,So maybe you can use it to manager.

slackass · 06-29-2010, 11:23 PM

I use local rsync mirrors for that stuff. I can see the changes when I update my mirrors.
http://slackware.org.uk/

Additionally I use rss feeds:

Eric has an rss feed at:
http://connie.slackware.com/~alien/

Robby has one at:
http://rlworkman.net/pkgs/

hope this helps.

Alien Bob · 06-30-2010, 03:44 AM

Definitely use the RSS feed. By the way, my package feed is manyfold:

http://slackware.com/~alien/slackbuilds/ChangeLog.rss is the RSS feed for my main package repository.

http://slackware.org.uk/people/alien.../ChangeLog.rss is the RSS feed for packages that have patent issues and can not be hosted in the US.

http://alien.slackbook.org/ktown/ChangeLog.rss is the RSS feed exclusively for my KDE packages.

Eric

niels.horn · 06-30-2010, 10:46 AM

This thread caught my attention in the LQ community bulletin

I have a very ugly script that checks installed packages (/var/log/packages), filters the non-standard ones and then checks for updates for some "known" (that's known *by me*) packagers.

I check for SBo (built from SlackBuilds on SBo w/o modifications), rlw (OpenOffice and some Xfce plugins), alien (vlc basically) and nhh (my own).

For each extension there is a little subscript to check for updates, basically reading the ChangeLogs I download daily.

It is really too ugly (and a few days ago even gave an error on a special package name) to make it public. But the idea of reading rss feeds may give me some inspiration to improve it

agi93 · 06-30-2010, 08:43 PM

Thanks for all the useful posts! I have subscribed to Robby and Eric's RSS feeds. Also, I found this little gem of a script on Eric's site that allows one to locally mirror his repository. It's under http://connie.slackware.com/~alien/tools/ and is called mirror-alien-repository.sh. I wonder if a few modifications would allow me to mirror Robby's packages as well. I wouldn't mind having all of them in /home/ftp/alien and /home/ftp/rlw (now I finally found a use for /home/ftp!), since a few lost gigs of disk space is no harm to me.

The script goes like this:

Code:

#!/bin/sh
# $Id: mirror-alien-repository.sh,v 1.1 2008/03/15 15:15:05 root Exp root $
# Written 2008 Eric Hameleers <alien@slackware.com>, Eindhoven, Netherlands
#
# Mirror the http://www.slackware.com/~alien/slackbuilds repository.
# Since the repository allows only http transfers, use lftp instead of rsync.
# Don't use 'wget' because that does not delete remote files when they
# disappear on the remote server.
#
#  The script works by downloading a copy of the remote ChangeLog.txt and then
#  comparing it to your local version of this file. If no differences
#  are found, the script will stop right there.
#  If differences are found, the script will show the 'diff' output and then
#  continue to mirror the remote repository. By using lftp, this process will
#  be relatively efficient.
#  When you want to run this script in a cron job, be sure to filter out STDERR
#  so that you will only see emails from cron when there is actually a change.
#  Example cron entry if you put the script in /usr/local/bin :
#  30 6 * * *      /usr/local/bin/mirror-alien-repository.sh 2>/dev/null
#
# ---------------------------------------------------------------------------

# Do we want more messages? A '1' means 'yes', a '0' means 'no'.
DEBUG=${DEBUG:-0}

# We prevent the mirror script from running more than one instance:
PIDFILE=/var/tmp/$(basename $0 .sh).pid

# Make sure the PID file is removed when we kill the process
trap 'rm -f $PIDFILE; exit 1' TERM INT

if [ -e $PIDFILE ]; then
  echo "Another instance (`cat $PIDFILE`) still running?"
  echo "If you are sure that no other instance is running, delete the lockfile"
  echo "'${PIDFILE}' and re-start this script."
  echo "Aborting now..."
  exit 1
else
  echo $$ > $PIDFILE
fi

# Our local mirror:
LOCALTREE=${LOCALTREE:-"/tmp/alien_repository"}
# The URL for the remote repository:
REMOTEURL=${REMOTEURL:-"http://www.slackware.com/~alien/slackbuilds"}
# Where we store temporary files if needed:
TMP=${TMP:-"/tmp"}
# Tools:
LFTP=${LFTP:-"/usr/bin/lftp"}

# Sanity checks:
if [ ! -d $TMP  ]; then
  echo "Temp directory '$TMP' does not exist yet, creating directory now..."
  mkdir -p $TMP
fi

if [ ! -w $TMP  ]; then
  echo "Temp directory '$TMP' is not writable! Quitting now..."
  rm -f $PIDFILE
  exit 1
fi

if [ ! -d $LOCALTREE  ]; then
  echo "Local tree '$LOCALTREE' does not exist yet, creating directory now..."
  mkdir -p $LOCALTREE
fi

if [ ! -w $LOCALTREE  ]; then
  echo "Local tree '$LOCALTREE' is not writable! Aborting..."
  rm -f $PIDFILE
  exit 1
fi

# Check for an updated ChangeLog.txt:
[ $DEBUG -eq 1 ] && echo "$(date) [$$]: Getting ChangeLog.txt..."
rm -f $TMP/alien_ChangeLog.txt
cd $TMP
# Direct all output to stderr so that you can filter it out in a cron job:
$LFTP -e "get ChangeLog.txt -o alien_ChangeLog.txt && quit" $REMOTEURL 1>&2
cd -
if [ ! -s $TMP/alien_ChangeLog.txt ]; then
  echo "$(date) [$$]: Could not retrieve ChangeLog.txt! Aborting..."
  rm -f $PIDFILE
  exit 1
fi

# If the ChangeLog.txt on our local mirror doesn't exist, it might mean that
# this is a first-time mirror. To prevent the script from aborting, we
# create an empty ChangeLog.txt file...
if [ ! -e ${LOCALTREE}/ChangeLog.txt ]; then
  touch ${LOCALTREE}/ChangeLog.txt
fi

diff -b ${LOCALTREE}/ChangeLog.txt $TMP/alien_ChangeLog.txt
STATUS="$?"
if [ "$STATUS" == "2" ]; then
  echo "$(date) [$$]: Trouble when running diff, aborting..."
  rm -f $PIDFILE
  exit 1
elif [ "$STATUS" == 0 ]; then
  [ $DEBUG == 1 ] && echo "$(date) [$$]: No difference found, quitting now..."
  rm -f $PIDFILE
  exit 0
else
  echo "$(date) [$$]: ChangeLog.txt has been updated."
fi

# Starting the mirror:
echo "$(date) [$$]: Starting the mirroring process."
# Direct all output to stderr so that you can filter it out in a cron job:
lftp -c "open $REMOTEURL ; mirror --verbose=0 --delete --continue . $LOCALTREE/" 1>&2
echo "$(date) [$$]: Finished the mirroring process."

rm -f $PIDFILE

Would I just have to modify the $REMOTEURL, the name of the ChangeLog, the $LOCALTREE, and maybe add "x86_64" after "mirror" in the lftp command to only get the appropriate packages? Or is there something else I need to worry about? I ask you because I'm a newbie to shell scripting, and I don't want to mess something up or be left clueless if trouble arises. I can't just try it now because I don't have access to my Slackware box right now

Better yet, instead of having two separate scripts to run, I could have a single script that updates both mirrors. Or maybe I could have these two scripts with different PID files and have another, simple script that calls those two. Oh, the possibilities with Slackware...

I don't really understand what's going on in the line that defines PIDFILE. What is $(basename $0 .sh).pid, and will I need to change that if I modify a copy of the script for Robby's repo.

I could just have the simple script that calls the other two one at a time. Yes, that should work. Again, is there anything else I need to change in Eric's script to work with Robby's repo?

slakmagik · 06-30-2010, 10:12 PM

Quote:

Originally Posted by agi93

I don't really understand what's going on in the line that defines PIDFILE. What is $(basename $0 .sh).pid, and will I need to change that if I modify a copy of the script for Robby's repo.

I don't have time right now to read the whole script but to answer that specific question, it's using basename to strip the path and the '.sh' from the script and appends '.pid' to the string. So, in this case, since it's named 'mirror-alien-repository.sh' by default, your pid file will be 'mirror-alien-repository.pid'. And if you change it to 'my-variant-to-mirror-robbys-stuff.sh' it'd have a pid file of 'my-variant-to-mirror-robbys-stuff.pid'.

agi93 · 07-01-2010, 11:44 AM

Quote:

Originally Posted by slakmagik

I don't have time right now to read the whole script but to answer that specific question, it's using basename to strip the path and the '.sh' from the script and appends '.pid' to the string. So, in this case, since it's named 'mirror-alien-repository.sh' by default, your pid file will be 'mirror-alien-repository.pid'. And if you change it to 'my-variant-to-mirror-robbys-stuff.sh' it'd have a pid file of 'my-variant-to-mirror-robbys-stuff.pid'.

Beautiful. That's a great system, too. I'm now back on my Slackware box so I'll mess around with the scripts.

agi93 · 07-01-2010, 02:55 PM

Alright everyone I got it working. I am now mirroring Eric's repository in /home/ftp/alien and Robby's in /home/ftp/rlw. I had to change the ownership and permissions of ftp; I made it owned by nobody:ftp instead of root:root and I added write permissions for the group. I also added my normal user (agi) to the ftp group. It allows me to run the scripts without root permission. Is this particularly bad? I just did it for convenience, but I hope I didn't introduce some sort of security or stability risk!

Here is the script I use for Eric's repo:

Code:

#!/bin/sh
# $Id: mirror-alien-repository.sh,v 1.1 2008/03/15 15:15:05 root Exp root $
# Written 2008 Eric Hameleers <alien@slackware.com>, Eindhoven, Netherlands
#
# Mirror the http://www.slackware.com/~alien/slackbuilds repository.
# Since the repository allows only http transfers, use lftp instead of rsync.
# Don't use 'wget' because that does not delete remote files when they
# disappear on the remote server.
#
#  The script works by downloading a copy of the remote ChangeLog.txt and then
#  comparing it to your local version of this file. If no differences
#  are found, the script will stop right there.
#  If differences are found, the script will show the 'diff' output and then
#  continue to mirror the remote repository. By using lftp, this process will
#  be relatively efficient.
#  When you want to run this script in a cron job, be sure to filter out STDERR
#  so that you will only see emails from cron when there is actually a change.
#  Example cron entry if you put the script in /usr/local/bin :
#  30 6 * * *      /usr/local/bin/mirror-alien-repository.sh 2>/dev/null
#
# ---------------------------------------------------------------------------

# Do we want more messages? A '1' means 'yes', a '0' means 'no'.
DEBUG=${DEBUG:-0}

# We prevent the mirror script from running more than one instance:
PIDFILE=/var/tmp/$(basename $0 .sh).pid

# Make sure the PID file is removed when we kill the process
trap 'rm -f $PIDFILE; exit 1' TERM INT

if [ -e $PIDFILE ]; then
  echo "Another instance (`cat $PIDFILE`) still running?"
  echo "If you are sure that no other instance is running, delete the lockfile"
  echo "'${PIDFILE}' and re-start this script."
  echo "Aborting now..."
  exit 1
else
  echo $$ > $PIDFILE
fi

# Our local mirror:
#LOCALTREE=${LOCALTREE:-"/tmp/alien_repository"}
LOCALTREE=${LOCALTREE:-"/home/ftp/alien"}
# The URL for the remote repository:
REMOTEURL=${REMOTEURL:-"http://www.slackware.com/~alien/slackbuilds"}
# Where we store temporary files if needed:
TMP=${TMP:-"/tmp"}
# Tools:
LFTP=${LFTP:-"/usr/bin/lftp"}

# Sanity checks:
if [ ! -d $TMP  ]; then
  echo "Temp directory '$TMP' does not exist yet, creating directory now..."
  mkdir -p $TMP
fi

if [ ! -w $TMP  ]; then
  echo "Temp directory '$TMP' is not writable! Quitting now..."
  rm -f $PIDFILE
  exit 1
fi

if [ ! -d $LOCALTREE  ]; then
  echo "Local tree '$LOCALTREE' does not exist yet, creating directory now..."
  mkdir -p $LOCALTREE
fi

if [ ! -w $LOCALTREE  ]; then
  echo "Local tree '$LOCALTREE' is not writable! Aborting..."
  rm -f $PIDFILE
  exit 1
fi

# Check for an updated ChangeLog.txt:
[ $DEBUG -eq 1 ] && echo "$(date) [$$]: Getting ChangeLog.txt..."
rm -f $TMP/alien_ChangeLog.txt
cd $TMP
# Direct all output to stderr so that you can filter it out in a cron job:
$LFTP -e "get ChangeLog.txt -o alien_ChangeLog.txt && quit" $REMOTEURL 1>&2
cd -
if [ ! -s $TMP/alien_ChangeLog.txt ]; then
  echo "$(date) [$$]: Could not retrieve ChangeLog.txt! Aborting..."
  rm -f $PIDFILE
  exit 1
fi

# If the ChangeLog.txt on our local mirror doesn't exist, it might mean that
# this is a first-time mirror. To prevent the script from aborting, we
# create an empty ChangeLog.txt file...
if [ ! -e ${LOCALTREE}/ChangeLog.txt ]; then
  touch ${LOCALTREE}/ChangeLog.txt
fi

diff -b ${LOCALTREE}/ChangeLog.txt $TMP/alien_ChangeLog.txt
STATUS="$?"
if [ "$STATUS" == "2" ]; then
  echo "$(date) [$$]: Trouble when running diff, aborting..."
  rm -f $PIDFILE
  exit 1
elif [ "$STATUS" == 0 ]; then
  [ $DEBUG == 1 ] && echo "$(date) [$$]: No difference found, quitting now..."
  rm -f $PIDFILE
  exit 0
else
  echo "$(date) [$$]: ChangeLog.txt has been updated."
fi

# Starting the mirror:
echo "$(date) [$$]: Starting the mirroring process."
# Direct all output to stderr so that you can filter it out in a cron job:
lftp -c "open $REMOTEURL ; mirror --verbose=0 --delete --continue . $LOCALTREE/" 1>&2
echo "$(date) [$$]: Finished the mirroring process."

rm -f $PIDFILE

And here is what I use for Robby's repo. I made some minor modifications and set it to only mirror x86_64 by changing the lftp line at the end:

Code:

#!/bin/sh
# mirror-rlw-repository.sh
#
# Modified from Eric Hameleer's mirror-alien-repository.sh script.
#
# Mirror the http://rlworkman.net/pkgs/13.1/ repository.
# Since the repository allows only http transfers, use lftp instead of rsync.
# Don't use 'wget' because that does not delete remote files when they
# disappear on the remote server.
#
#  The script works by downloading a copy of the remote ChangeLog.txt and then
#  comparing it to your local version of this file. If no differences
#  are found, the script will stop right there.
#  If differences are found, the script will show the 'diff' output and then
#  continue to mirror the remote repository. By using lftp, this process will
#  be relatively efficient.
#  When you want to run this script in a cron job, be sure to filter out STDERR
#  so that you will only see emails from cron when there is actually a change.
#  Example cron entry if you put the script in /usr/local/bin :
#  30 6 * * *      /usr/local/bin/mirror-alien-repository.sh 2>/dev/null
#
# ---------------------------------------------------------------------------

# Do we want more messages? A '1' means 'yes', a '0' means 'no'.
DEBUG=${DEBUG:-0}

# We prevent the mirror script from running more than one instance:
PIDFILE=/var/tmp/$(basename $0 .sh).pid

# Make sure the PID file is removed when we kill the process
trap 'rm -f $PIDFILE; exit 1' TERM INT

if [ -e $PIDFILE ]; then
  echo "Another instance (`cat $PIDFILE`) still running?"
  echo "If you are sure that no other instance is running, delete the lockfile"
  echo "'${PIDFILE}' and re-start this script."
  echo "Aborting now..."
  exit 1
else
  echo $$ > $PIDFILE
fi

# Our local mirror:
#LOCALTREE=${LOCALTREE:-"/tmp/alien_repository"}
LOCALTREE=${LOCALTREE:-"/home/ftp/rlw"}
# The URL for the remote repository:
REMOTEURL=${REMOTEURL:-"http://rlworkman.net/pkgs/13.1/"}
# Where we store temporary files if needed:
TMP=${TMP:-"/tmp"}
# Tools:
LFTP=${LFTP:-"/usr/bin/lftp"}

# Sanity checks:
if [ ! -d $TMP  ]; then
  echo "Temp directory '$TMP' does not exist yet, creating directory now..."
  mkdir -p $TMP
fi

if [ ! -w $TMP  ]; then
  echo "Temp directory '$TMP' is not writable! Quitting now..."
  rm -f $PIDFILE
  exit 1
fi

if [ ! -d $LOCALTREE  ]; then
  echo "Local tree '$LOCALTREE' does not exist yet, creating directory now..."
  mkdir -p $LOCALTREE
fi

if [ ! -w $LOCALTREE  ]; then
  echo "Local tree '$LOCALTREE' is not writable! Aborting..."
  rm -f $PIDFILE
  exit 1
fi

# Check for an updated ChangeLog.txt:
[ $DEBUG -eq 1 ] && echo "$(date) [$$]: Getting ChangeLog.txt..."
rm -f $TMP/rlw_ChangeLog.txt
cd $TMP
# Direct all output to stderr so that you can filter it out in a cron job:
$LFTP -e "get ChangeLog.txt -o rlw_ChangeLog.txt && quit" $REMOTEURL 1>&2
cd -
if [ ! -s $TMP/rlw_ChangeLog.txt ]; then
  echo "$(date) [$$]: Could not retrieve ChangeLog.txt! Aborting..."
  rm -f $PIDFILE
  exit 1
fi

# If the ChangeLog.txt on our local mirror doesn't exist, it might mean that
# this is a first-time mirror. To prevent the script from aborting, we
# create an empty ChangeLog.txt file...
if [ ! -e ${LOCALTREE}/ChangeLog.txt ]; then
  touch ${LOCALTREE}/ChangeLog.txt
fi

diff -b ${LOCALTREE}/ChangeLog.txt $TMP/rlw_ChangeLog.txt
STATUS="$?"
if [ "$STATUS" == "2" ]; then
  echo "$(date) [$$]: Trouble when running diff, aborting..."
  rm -f $PIDFILE
  exit 1
elif [ "$STATUS" == 0 ]; then
  [ $DEBUG == 1 ] && echo "$(date) [$$]: No difference found, quitting now..."
  rm -f $PIDFILE
  exit 0
else
  echo "$(date) [$$]: ChangeLog.txt has been updated."
fi

# Starting the mirror:
echo "$(date) [$$]: Starting the mirroring process."
# Direct all output to stderr so that you can filter it out in a cron job:
#lftp -c "open $REMOTEURL ; mirror --verbose=0 --delete --continue . $LOCALTREE/" 1>&2
lftp -c "open $REMOTEURL ; mirror --verbose=0 --delete --continue x86_64/ $LOCALTREE/" 1>&2
echo "$(date) [$$]: Finished the mirroring process."

rm -f $PIDFILE

Thank you so much, Eric, for this amazing script!

agi93 · 07-01-2010, 04:21 PM

One thing I've noticed is that alien's repository is taking quite a long time to download and quite a bit of space. This seems to be because I'm getting all sorts of packages for different versions of Slackware and different architectures. Is there a way to only mirror the "build" and "pkg64" directories for each package? I can't seem to figure this out since Eric's repository is organized by package rather than architecture (viz. the highest directory is the name of each package, which each has a build/, pkg/ and pkg64/ directory). I'll keep experimenting, but please let me know if you know how to do this!

I tried to change the lftp line near the end of the script:

Code:

lftp -c "open $REMOTEURL ; mirror --verbose=0 --delete --continue . $LOCALTREE/" 1>&2

To this:

Code:

lftp -c "open $REMOTEURL ; mirror --verbose=0 --delete --continue */pkg64/13.1/ $LOCALTREE/" 1>&2
]

with the hopes that I could mirror the pkg64 directories of every package as long as it had a 13.1 version. Unfortunately, this resulted in a 404 not found error.

What else can I do?