Comparing installed packages with the changelog...

Lufbery · 12-19-2007, 10:01 AM

Hi all,

I'm looking for a command, or even a script, that compares my list of installed packages with a Slackware changelog.

Does anyone have any suggestions on how to get started.

I don't want to use slackpkg or similar tools. I simply want to quickly generate a list of new or changed packages when the changelog is compared to the list of installed packages.

Thanks,

-Drew

rworkman · 12-19-2007, 10:08 AM

Sure. Use the functions from slackpkg, as that's their intended purpose

It's all written in shell language, so there's no reason not to re-use the code; it's known to work.
Of course, if you're going to do that, then you may as well just use slackpkg.
:-)

Lufbery · 12-19-2007, 10:28 AM

Yeah yeah yeah, Robby, I get where you're coming from. I'll take a look at their code.

What I'm mostly interested in is how to parse the changelog and compare it to the log of installed files. I suspect that it's probably as simple as using grep to grab the lines starting with patches/ or extra/ and then using diff on the log of isntalled files.

I used slackpkg a year or so ago, but actually found it simpler to read the changelog, download the updates, and use upgradepkg manually.

And anyway, the is Slackware -- there's nothing wrong with learning how to do it myself.

Regards,

-Drew

acummings · 12-19-2007, 12:08 PM

http://www.darklinux.net/slackupdate/

http://www.darklinux.net/slackupdate...pdate_0.7.1.sh

next, copied from the above sh script:

# Check to see if there is a package on the server that is newer than one on this machine
match_packages() { <snip> }

Does that function (match_packages) discern when/if pkg (patches) updates are available.

I've not learned sh or bash scripting except for the simplest of things.
--

Essentially, it (your task) should be: 1. get the output of ls on the /var/log/packages

2. get the listing of patches/packages (or parse the changelog) from the patches/update server

3. compare items that exist in both 1 and 2 whereas any version number difference(s) when compared means "not (exactly) the same" translates to "an update is available"

All of the above does not account for an "added" (very rare for stable) pkg. And, ditto, for any "blacklisted (not Slack official pkg)"

slackupdate has capability as far as blacklist goes -- evidently slackpkg does too.

Perl, Python could do this compare (just to name a few other than sh or bash script)

--
Alan.

Lufbery · 12-20-2007, 09:57 AM

Alan,

Thanks for the breakdown of tasks and the resources you posted.

Quote:

Originally Posted by acummings

Perl, Python could do this compare (just to name a few other than sh or bash script)

It's funny you should mention other languages. I'm actually thinking of processing the two lists (changelog and packages list) using Emacs Lisp from within Emacs.

Regards,

-Drew

Lufbery · 12-24-2007, 08:38 AM

Hi all,

This is turning out to be a little harder than I expected. It was relatively easy to manually parse the changelog, extract the name of the packages, and sort them alphabetically. It was also easy to pipe the contents of /var/log/packages to a text file.

But, doing a comparison with diff didn't work too well. The packages names are seemingly treated as a single word, and since the whole package names are different the output of the diff command simply shows both files as being completely different. I was hoping for a finer-grained comparison where I could see the differences in version numbers.

I'll have to dig deeper into the diff command...

Regards,

-Drew

gnashley · 12-24-2007, 08:58 AM

Maybe you should cut the version loose from the package name when you parse the changelog and ls /var/log/packages output and write them to a file with a <tab> between them. Then, after sorting well, you can compare the lists more easily by matching the first field with the package name and then comparing the version from the second field of the two lists.
You might also investigate using the 'comm' command for comparing the lists.

Code:

cat list-a |sort -u |uniq -u > list-a.tmp
rm list-a
cat list-b |sort -u |uniq -u > list-b.tmp
rm list-b

for line in $(cat list-a.tmp) ; do
name=$(echo $line | rev |cut f-4- -d'-' |rev)
version=$(echo $line | rev |cut f-1-3 -d'-' |rev)
echo -n "$name\t" >> list-a
echo $version >> list-a
done
rm list-a.tmp
# repeat for list-b

for line in $(cat list-a) ; do

done

gnashley · 12-24-2007, 09:13 AM

Maybe you should cut the version loose from the package name when you parse the changelog and ls /var/log/packages output and write them to a file with an arbitrary character between them like ':'. Then, after sorting well, you can compare the lists more easily by matching the first field with the package name and then comparing the version from the second field of the two lists.
You might also investigate using the 'comm' command for comparing the lists.

Here's a rough idea that doesn't need any sed, awk or perl:

Code:

cat list-a |sort -u |uniq -u > list-a.tmp
rm list-a
cat list-b |sort -u |uniq -u > list-b.tmp
rm list-b

for line in $(cat list-a.tmp) ; do
name=$(echo $line | rev |cut f-4- -d'-' |rev)
version=$(echo $line | rev |cut f-1-3 -d'-' |rev)
echo -n "$name:" >> list-a
echo $version >> list-a
done
rm list-a.tmp
# repeat for list-b

for line in $(cat list-a) ; do
  a-name=$(echo $line |cut -f 1 -d':')
  a-version=%(echo $line |cut -f 2 -d':')
  if [[ $(grep $name list-b) ]] ; then
   b-name=$(grep $name list-b |cut -f 1 -d':')
   b-version=$(grep $name list-b |cut -f 2 -d':')
  fi
  if [[ $b-version >= $a-version ]] ; then
   # do what you need here to download and install the new version
  fi
  
done

You may need or want to compare the versions a little differently, but I think this way might effectively do the trick so that a package with the same program version but a newer build number would be detected. I haven't tested this code, but it might give you some ideas about one way to approach this. I'm sure there are guys here on the forum(or in the Programming forum) who can do the whole thing with an awk one-liner
but I rather dislike obscure syntax (at least until I master it, then maybe I think it's the coolest thing since sliced bread). I used code very similar to this to construct complex name parsers for src2pkg. It's very clumsy looking, but fairly easier to understand, debug and modify than fancy regular expressions.

Lufbery · 12-24-2007, 10:04 AM

Quote:

Originally Posted by gnashley

Maybe you should cut the version loose from the package name when you parse the changelog and ls /var/log/packages output and write them to a file with an arbitrary character between them like ':'.

I was thinking the same thing. Even just a space between the names and version numbers might help. I'm already dropping the path from the line in the changelog with the package name.

So,

Code:

patches/packages/mysql-5.0.51-i486-1_slack12.0.tgz

becomes simply

Code:

mysql-5.0.51-i486-1_slack12.0.tgz

After you script, it would be

Code:

mysql:5.0.51:i486:1_slack12.0.tgz

Right?

I'm thinking of making it

Code:

mysql 5.0.51 i486 1_slack12.0.tgz

I'll need to work on this iteratively. After I've figured out how to compare the changelog with my installed packages, I want to re-attach the path to the package name so I know where to get the files -- not all of the files come from /patches. Sometimes they also come from /extra.

Thanks for the code snippet. I'll try it out and let you know what happens. I'll try to figure out what I need to do to get it to work.

I agree with you about obscure syntax. I think regular expressions are pretty neat, but they're not intuitive and I have a lot to learn about using them. For that matter, I have a lot to learn about bash scripting, and I can't shake the conviction that this kind of exercise would work well in Lisp -- either in Emacs Lisp (I have Emacs open most of the time anyway), or Common Lisp (which comes with Slackware). Either way, I need to learn how to write it.

Regards, and happy holidays,

-Drew

gnashley · 12-24-2007, 12:35 PM

My poor example code ignored removing the .tgz suffix from the package names. Of course the listing you get from /var/log/packages doesn't include the suffix. For the list-b (from the changelog) you'd just need to add a step to strip off the last bit and also handle the path separately.

Just use more code like above -flip the whole path+name over with rev, strip off the .tgz and skip to the first / and flip it back over to get the package name. Doing the same but skipping over everything up to the first slash (right to left) will get the path

fullpath="extras/some-fine-pkg-0.1-i486-1.tgz"

pkgname=$(echo $fullpath |rev cut -f 2- -d'.' |rev)
pkgpath=$(echo $fullpath |rev cut -f 2- -d'/' |rev)
Then do the tricks as above to separate name, version and build and compare
Then cat it all back together as needed.
You could use shell substitution to make things shorter, but it's harder to read. rev & cut are easy to understand and master. pkgtools uses this method to handle package names as well.
You can easily experiment or check the results on the command line also.
paste in terminal:

Code:

fullpath="extras/some-fine-pkg-0.1-i486-1.tgz" ; echo $fullpath |rev | cut -f 2- -d'.' |rev

Happy Holidays Drew!

acummings · 12-26-2007, 05:04 AM

Hi,

"On the contrary" (regex)

I had learnt some Perl and some of Perl's regex

And, couple that with Perl's various different ways to say what you want to say when using if, unless, etc.

And (intermediate--newbie me) gets working Perl script.

1st list printed to screen be the updates available from the www.

2nd list printed to screen is pkgs from local machine sys.

Script is close. Works. But somehow snags a few too many gimp* pkgs from local mach sys.

All it does is compare and print to screen.

I suspect the minor problem is at either # here1 or at # here2 in the code.

Code:

#!/usr/bin/perl

use warnings;
use strict;
# use diagnostics;

my $url = 'ftp://slackware.mirrors.tds.net/pub/slackware/slackware-12.0/patches/packages';

my $pid;
my $upd_pkgs;
$pid = open $upd_pkgs, "-|", "lynx", "-dump", "$url" or die "cant fork: $!\n";
print "pid is: $pid\n";
my @patches;
while (<$upd_pkgs>) {
next unless /slack.+patches$/i .. /eof/; # range operator
s#^.+slackware-12.0/patches/packages/##;
s#\.tgz##;
# print unless /\d.+patches$|\.asc|\.txt/;
push @patches, $_ unless /\d.+patches$|\.asc|\.txt/;
}

# begin of here1
my $apend;
my @dashes;
my @dashes2;
foreach ( @patches ) {
@dashes = split(/-/, $_);
$apend .= $dashes[0];
$apend .= '-';
$apend .= $dashes[1];
if ( $apend =~ s/-\d.+$// ) {
push @dashes2, $apend;
# print $apend, "hi1\n";
$apend = '';
} else {
$apend .= '-';
$apend .= $dashes[2];
# print $apend, "hi2\n";
push @dashes2, $apend;
$apend = '';
}
}
# end of here1

# print @dashes2, "\n";
print @patches;
print "\n---\n\n";

my $pid2;
my $sys_pkgs;
$pid2 = open $sys_pkgs, "-|", "find", "/var/log/packages" or die "cant fork: $!\n";
print "pid is: $pid2\n";
my @sys_pkg_list;
while (<$sys_pkgs>) {
next if /packages$/;
s#/var/log/packages/##;
push @sys_pkg_list, $_;
}
@sys_pkg_list = sort { $a cmp $b } @sys_pkg_list;
# print @sys_pkg_list, "\n";

foreach my $x (@sys_pkg_list)
{
    foreach (@dashes2)
    {
        print $x if $x =~ /^$_/; # here2
    }
}

--
Alan.

Lufbery · 12-26-2007, 08:20 AM

Alan,

Thanks for the Perl script! I'll try it out soon. Like I said to Gilbert, I think regular expressions are neat, I'm just have a lot to learn about using them.

I find a couple of things interesting about your script. If I'm reading it correctly, you're comparing the contents of the /patches directory with the contents of your /var/log/packages directory. That's an approach I hadn't though of. I think it would help to also check against the contents (and sub-directories) of the /extra directory too. It's rare, but Pat sometimes updates packages in /extra.

I'm not sure what's happening in here1 (I don't know much Perl). Is that where you're separating the package name from its version information?

Thanks Alan and Gilbert for your help. I'm pretty much a rank newbie when it comes to scripting in Linux.

Regards,

-Drew

acummings · 12-27-2007, 12:45 AM

Gotcha on the /extra

Will incorporate into a later version. May parse changelog then instead of list mirror server's dir contents. Hey, right now, I run it on my machine -- it's fast -- only a second or two of run time and it's done.

Why does Pat keep the changelog so big? It's still got Slack 11 on it too. Or, is it for all of the "stable" and currently security update supported versions of Slackware?

(changelog) It's a huge file to be pumping across the www every time someone wants to check on Slack 12 patches (reason to join the security updates emailing list, I guess).

Quote:

I'm not sure what's happening in here1 (I don't know much Perl). Is that where you're separating the package name from its version information?

Yes. It had a bug in that section too. But not now; I fixed it.

More help comments included now. Re labled/named some section(s). Hopefully more clarity exist now of what is happening when.

Code:

#!/usr/bin/perl

use warnings;
use strict;
# use diagnostics;

# 12-26-2007 unfinished. See at bottom of code for yet to do
# reports whether or not any new (not yet installed) Slackware
# patches/updates are available
# compares a mirror server's updates with /var/log/packages

my $url = 'ftp://slackware.mirrors.tds.net/pub/slackware/slackware-12.0/patches/packages';

my $pid;
my $upd_pkgs;
$pid = open $upd_pkgs, "-|", "lynx", "-dump", "$url" or die "cant fork: $!\n";
print "pid is: $pid\n";
my @patches;
while (<$upd_pkgs>) {
next unless /slack.+patches$/i .. /eof/; # range operator
s#^.+slackware-12.0/patches/packages/##;
s#\.tgz##;
# print unless /\d.+patches$|\.asc|\.txt/;
push @patches, $_ unless /\d.+patches$|\.asc|\.txt/;
} # got patches file names in the @patches array

# next section strips the numeric part of patch file names
# which leaves us with only the leading  alpha and dashes
# up to but not including any of the digits

# begin of alpha_only (number, numeric not allowed)
my $apend;
my @dashes;
my @dashes2;
foreach ( @patches ) {
@dashes = split(/-/, $_);
$apend .= $dashes[0];
$apend .= '-';
$apend .= $dashes[1];
if ( $apend =~ s/-\d.+$// ) {
push @dashes2, $apend;
# print $apend, "hi1\n"; # pkg name leading alpha and dashes
$apend = '';
} else {
$apend .= '-';
$apend .= $dashes[2];
$apend =~ s/-\d.+$//;
# print $apend, "hi2\n"; # pkg name leading alpha and dashes
push @dashes2, $apend;
$apend = '';
}
} # got alpha_only and dashes in the @dashes2 array
# end of alpha_only

# print @dashes2, "\n";
print "Patches on mirror server:\n";
print @patches; # mirror server's patches print to screen
print "\n---\n\n";

my $pid2;
my $sys_pkgs;
$pid2 = open $sys_pkgs, "-|", "find", "/var/log/packages" or die "cant fork: $!\n";
print "pid is: $pid2\n";
my @sys_pkg_list;
while (<$sys_pkgs>) {
next if /packages$/;
s#/var/log/packages/##;
push @sys_pkg_list, $_;
} # got /var/log/packages into the @sys_pkg_list array
@sys_pkg_list = sort { $a cmp $b } @sys_pkg_list;
# and we now have a sorted or ordered list

# next, using the alpha_only and dashes (@dashes2 array) we
# compare each with each that's in the @sys_pkg_list array
# @alpha_matches array gets items from /var/log/packages
# that have a match on the mirror server
# though the digit portion may or may
# not match at this point in the game
# thus the array name of matches via alpha or alpha_matches
my @alpha_matches;
foreach my $x (@sys_pkg_list) {
    foreach (@dashes2) {
#        print $x if $x =~ /^$_-\d/;
# make sure is followed by a dash digit not a dash alpha
# thus the -\d (dash digit) in the next regex
        push @alpha_matches, $x if $x =~ /^$_-\d/;
    }
} # @alpha_matches tells us, irrespective of version number,
# which pkgs exist on both our sys and on the mirror server
# IOW @patches and @alpha_matches should each now be holding
# the same pkg name contents though version #'s may not match

# next, the need to discern and report about yes/no version
# differences between sys pkgs and mirror server patches
# (hint -- we'll likely use a hash since hash keys must be
#  unique)  IOW version matches will be gone and the only
# remaining will be an "either -- or" that is either yes
# new updates exist or no all our pkgs currently up to date
# if yes, it will list which pkgs updates but if no then it
# will say "Your sys pkgs are currently 100% up to date there
# are no new updates for your sys on the mirror server".

# until we resume coding, for now, we'll just print @alpha_matches
# matches via alpha which exist on our sys print to screen
print "From sys the pkg matches via alpha to patches:\n";
print @alpha_matches, "\n";
# end

--
Alan

acummings · 12-27-2007, 01:10 AM

Oh, BTW the line with split in it splits on dashes into the @dashes array. Then does some appending of strings (array indices ie [0], [1], etc) coupled with some if and some if else control statements/blocks coupled with some regex substitution. Tie (the integration of) all of that together and we got some power there.

Shell script has power too, but is different (than Perl), shell script uses all the various proggies ie awk sed cut and more unless it's less instead :-)

With a Perl guru, Perl is even more Perlish than how I write it (a more idiomatic Perl than what I write). But that's more advanced too. I have fun with Perl.

Lisp (the Emacs operating system). Anyways, lisp surely can do this. Lisp is functional programming.

The book entitled "Higher Order Perl" has an excellent portrayal of how to do (and how well Perl does) functional programming with Perl.

http://gnosis.cx/publish/programming..._python_13.txt

I see there that Python can be "charmed" into doing functional programming.

http://www.engin.umd.umich.edu/CIS/c...lisp/lisp.html

http://en.wikipedia.org/wiki/Functional_programming

--
Alan.

acummings · 12-27-2007, 04:22 PM

Outside the fun been having with Perl . . .

It just dawned on me that this wheel has long ago already been invented. As in:

Code:

al@AB60R:~$ slackupdate -l 0 -s ftp://slackware.mirrors.tds.net/pub/slackware

Run slackupdate sh script using the run level zero switch

Code:

# RunLevels:	0 - Only Check for Updates
		# 		1 - Check for and Download Updates [Recomended]
		#		2 - Automaticly Install Updates (Be carefull with this one)

It also is blacklist capable (my Perl is not).

http://www.darklinux.net/slackupdate/

alias in .bashrc

Code:

alias slupdck='slackupdate -l 0 -s ftp://slackware.mirrors.tds.net/pub/slackware'

Thus I'm steering you to slackupdate if you want a way to ocasionally check for if your sys is fully updated or not. Slackupdate has already long since been tested and is ready right now for your specific use in this case.

At a slow pace I may poke my head back in here a few times just in order to share any further Perl tinkerings such as the hash, etc. with which to finish off this Perl script.

--
Alan.