pkgtools enhancement request: Log WARNINGs

drumz · 04-12-2022, 10:56 AM

Do you ever see these WARNING messages flash by the console when doing removepkg or upgradepkg?

Code:

  --> Deleting empty directory /usr/lib64/java/
  --> Deleting empty directory /usr/doc/openjdk-8u312_b07/icedtea/
  --> Deleting empty directory /usr/doc/openjdk-8u312_b07/
WARNING: Unique directory /etc/java/ contains new files

Only to see them scroll away, forever lost? Don't you wish you could go back and review them? Particularly when upgrading Slackware versions, when you are upgrading over a thousand packages?

I made 2 different modifications to log what pkgtools is doing. Either can be used separately, but they also work together in tandem.

The first is to create a simple wrapper script for installpkg, removepkg, and upgradepkg, and placed in /usr/local/sbin. The script is identical for all 3, so you only need to create 1 file and then create 2 symlinks:

Code:

$ ls -l /usr/local/sbin/
-rwxr-xr-x 1 root root  270 Feb 17 15:52 installpkg*
lrwxrwxrwx 1 root root   10 Feb 17 15:52 removepkg -> installpkg*
lrwxrwxrwx 1 root root   10 Feb 17 15:52 upgradepkg -> installpkg*

And here is /usr/local/sbin/installpkg:

Code:

#!/bin/bash
# Wrapper around /sbin/{installpkg,removepkg,upgradepkg} to log all output

LOGDIR=/var/log/pkgtools/log
LOGFILE="$LOGDIR/$(basename "$0")_$(date '+%F_%T').log"

echo "$(basename "$0") $*" > "$LOGFILE"

"/sbin/$(basename "$0")" "$@" 2>&1 | tee -a "$LOGFILE"

You must create the directory /var/log/pkgtools/log before using - the script itself doesn't do that just to avoid the overhead of checking every time it is run.

Here is an example log file, /var/log/pkgtools/log/removepkg_2022-04-11_14:00:08.log:

Code:

removepkg brscan4
Removing package: brscan4-0.4.11-x86_64-6_SBo
Removing files:
  --> Deleting symlink /etc/opt/brother/scanner/brscan4/Brsane4.ini
  --> Deleting symlink /etc/opt/brother/scanner/brscan4/brsanenetdevice4.cfg
  --> Deleting symlink /etc/opt/brother/scanner/brscan4/models4
  --> Deleting symlink /usr/bin/brsaneconfig4
  --> Deleting symlink /usr/lib64/sane/libsane-brother4.so
  --> Deleting symlink /usr/lib64/sane/libsane-brother4.so.1
  --> Deleting /etc/sane.d/dll.d/brscan4.conf
  --> Deleting /lib/udev/rules.d/80-brother-brscan4-libsane-type1.rules
  --> Deleting /opt/brother/scanner/brscan4/Brsane4.ini
  --> Deleting /opt/brother/scanner/brscan4/brsaneconfig4
  --> Deleting /opt/brother/scanner/brscan4/brsanenetdevice4.cfg.new
  --> Deleting /opt/brother/scanner/brscan4/brscan_cnetconfig
  --> Deleting /opt/brother/scanner/brscan4/brscan_gnetconfig
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_1.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_10.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_11.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_12.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_13.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_14.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_15.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_16.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_17.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_18.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_19.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_2.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_20.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_21.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_22.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_23.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_24.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_25.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_26.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_27.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_3.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_4.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_5.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_6.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_7.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_8.ini
  --> Deleting /opt/brother/scanner/brscan4/models4/ext_9.ini
  --> Deleting /usr/doc/brscan4-0.4.11/brscan4.SlackBuild
  --> Deleting /usr/doc/brscan4-0.4.11/readme.txt
  --> Deleting /usr/lib64/sane/libsane-brother4.so.1.0.7
  --> Deleting empty directory /usr/doc/brscan4-0.4.11/
  --> Deleting empty directory /opt/brother/scanner/brscan4/models4/
WARNING: Unique directory /opt/brother/scanner/brscan4/ contains new files
WARNING: Unique directory /opt/brother/scanner/ contains new files
WARNING: Unique directory /opt/brother/ contains new files
  --> Deleting empty directory /etc/opt/brother/scanner/brscan4/
  --> Deleting empty directory /etc/opt/brother/scanner/
  --> Deleting empty directory /etc/opt/brother/

And here is a sample log file where multiple packages were installed at once (/var/log/pkgtools/log/installpkg_2022-04-11_13:40:18.log):

Code:

installpkg python/cppy/cppy-1.2.0-x86_64-1_SBo.txz python/python3-aiorpcX/python3-aiorpcX-0.22.1-x86_64-1_SBo.txz
Verifying package cppy-1.2.0-x86_64-1_SBo.txz.
Installing package cppy-1.2.0-x86_64-1_SBo.txz:
PACKAGE DESCRIPTION:
# cppy (C++ header library)
#
# cppy is a small C++ header library which makes it easier to write 
# Python extension modules.
#
Package cppy-1.2.0-x86_64-1_SBo.txz installed.
Verifying package python3-aiorpcX-0.22.1-x86_64-1_SBo.txz.
Installing package python3-aiorpcX-0.22.1-x86_64-1_SBo.txz:
PACKAGE DESCRIPTION:
# python3-aiorpcX (Generic async RPC implementation)
#
# aiorpcX is a generic asyncio library implementation of RPC suitable
# for an application that is a client, server or both.
#
# https://aiorpcx.readthedocs.io/en/stable/
#
Package python3-aiorpcX-0.22.1-x86_64-1_SBo.txz installed.

So as you can see, you can simply grep for "WARNING" among all files in /var/log/pkgtools/log/ to go back and review the warnings.

There is quite a bit of duplicated information in /var/log/pkgtools/log/* with /var/lib/pkgtools/packages/* and /var/log/pkgtools/removed_packages/*, but they serve slightly different purposes. Notably, the native database files do not contain the WARNING messages.

The second change I made is a modification to /sbin/removepkg and logs all WARNING messages to /var/log/pkgtools/warnings:

Code:

diff --git a/removepkg.orig b/removepkg
index cb7e911..7f76eea 100755
--- a/removepkg.orig
+++ b/removepkg
@@ -191,6 +191,18 @@ extract_links() {
  sed -n 's,^[ ]*( [ ]*cd[ ]* \(.*\) [ ]*; [ ]*rm [ ]*-rf[ ]* \(.*\) [ ]*)[ ]*$,\1/\2,p'
 }
 
+echo_warning() {
+ if [ ! "$WARN" = "true" ]; then
+  if ! [ $TERSE ]; then
+   echo "WARNING: $1" | tee -a $LOG_DIR/warnings
+  else
+   echo "WARNING: $1" >> $LOG_DIR/warnings
+  fi
+ else
+  echo "WARNING: $1"
+ fi
+}
+
 preserve_file() {
  if [ "$PRESERVE" = "true" ]; then
   F="$(basename "$1")"
@@ -220,7 +232,7 @@ keep_files() {
     preserve_file "$FILE"
    else
     if [ "$(echo $FILE | cut -b1-8)" != "install/" ]; then
-     ! [ $TERSE ] && echo "WARNING: Nonexistent $ROOT/$FILE was found in another package. Skipping."
+     echo_warning "Nonexistent $ROOT/$FILE was found in another package. Skipping."
     fi
    fi
   else
@@ -234,7 +246,7 @@ keep_links() {
   if [ -L "$ROOT/$LINK" ]; then
    ! [ $TERSE ] && echo "  --> $ROOT/$LINK (symlink) was found in another package. Skipping."
   else
-   ! [ $TERSE ] && echo "WARNING: Nonexistent $ROOT/$LINK (symlink) was found in another package. Skipping."
+   echo_warning "Nonexistent $ROOT/$LINK (symlink) was found in another package. Skipping."
   fi
  done
 }
@@ -246,7 +258,7 @@ delete_files() {
   if [ ! -d "$ROOT/$FILE" ]; then
    if [ -r "$ROOT/$FILE" ]; then
     if [ "$ROOT/$FILE" -nt "$ADM_DIR/packages/$PKGNAME" ]; then
-     ! [ $TERSE ] && echo "WARNING: $ROOT/$FILE changed after package installation."
+     echo_warning "$ROOT/$FILE changed after package installation."
     fi
     if [ ! "$WARN" = "true" ]; then
      ! [ $TERSE ] && echo "  --> Deleting $ROOT/$FILE"
@@ -290,7 +302,7 @@ delete_dirs() {
         ! [ $TERSE ] && echo "  --> Deleting empty directory $ROOT/$DIR"
         rmdir "$ROOT/$DIR" 2> /dev/null # Using 2> /dev/null to prevent noise from upgradepkg when a directory changes to a symlink.
       else
-        ! [ $TERSE ] && echo "WARNING: Unique directory $ROOT/$DIR contains new files"
+        echo_warning "Unique directory $ROOT/$DIR contains new files"
       fi
     else
      ! [ $TERSE ] && echo "  --> $ROOT/$DIR (dir) would be deleted if empty"

I've been running with these modifications for over a month now, and here is my /var/log/pkgtools/warnings file:

Code:

WARNING: Unique directory /etc/java/ contains new files
WARNING: Nonexistent /etc/fonts/conf.d/35-freefont.conf (symlink) was found in another package. Skipping.
WARNING: Nonexistent /usr/man/man8/tsig-keygen.8.gz (symlink) was found in another package. Skipping.
WARNING: Nonexistent /usr/sbin/tsig-keygen (symlink) was found in another package. Skipping.
WARNING: Nonexistent /etc/localtime localtime-copied-from (symlink) was found in another package. Skipping.
WARNING: Nonexistent /etc/localtime-copied-from (symlink) was found in another package. Skipping.
WARNING: /opt/brother/scanner/brscan4/brsanenetdevice4.cfg changed after package installation.
WARNING: /usr/doc/FlightGear-data-2020.3.11/FlightGear-data.SlackBuild changed after package installation.
WARNING: Unique directory /opt/brother/scanner/brscan4/ contains new files
WARNING: Unique directory /opt/brother/scanner/ contains new files
WARNING: Unique directory /opt/brother/ contains new files

Using only this second modification (and not the wrapper scripts), you can still search for filename patterns in /var/lib/pkgtools/packages/* and/or /var/log/pkgtools/remove_packages/* to see why the WARNING messages were generated.

With these logs in place, a system administrator can periodically review and manually remove files/directories that are no longer needed.

chrisretusn · 04-13-2022, 02:16 AM

This has been discussed in the past. Quite a while ago.

I created a patch that logs the WARNINGS I want to see.

Code:

--- removepkg.old	2020-12-30 04:12:09.000000000 +0800
+++ removepkg	2021-04-02 23:15:02.318938772 +0800
@@ -247,6 +247,7 @@
    if [ -r "$ROOT/$FILE" ]; then
     if [ "$ROOT/$FILE" -nt "$ADM_DIR/packages/$PKGNAME" ]; then
      ! [ $TERSE ] && echo "WARNING: $ROOT/$FILE changed after package installation."
+     echo "WARNING: $ROOT/$FILE changed after package installation." >> /root/removepkg-WARNING.log
     fi
     if [ ! "$WARN" = "true" ]; then
      ! [ $TERSE ] && echo "  --> Deleting $ROOT/$FILE"
@@ -291,6 +292,7 @@
         rmdir "$ROOT/$DIR" 2> /dev/null # Using 2> /dev/null to prevent noise from upgradepkg when a directory changes to a symlink.
       else
         ! [ $TERSE ] && echo "WARNING: Unique directory $ROOT/$DIR contains new files"
+        ! [ $TERSE ] && echo "WARNING: Unique directory $ROOT/$DIR contains new files" >> /root/removepkg-WARNING.log
       fi
     else
      ! [ $TERSE ] && echo "  --> $ROOT/$DIR (dir) would be deleted if empty"