Anyone recently played with adding support for further compression formats into pkgtools?

ruario · 04-11-2018, 02:42 AM

To see how fast lbzip2 is at creating a similarly sized (actually 2M smaller) file, here is a comparison with gzip, pigz and bzip2.

Code:

$ time gzip -9c < skypeforlinux-8.18.0.6-x86_64-1ro.tar > skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz

real	0m40.449s
user	0m40.361s
sys	0m0.084s
$ ls -lh skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz
-rw-r--r-- 1 ruario users 98M Apr 11 09:01 skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz
$ time gzip -cd < skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz > /dev/null 

real	0m2.206s
user	0m2.174s
sys	0m0.032s
$ time pigz -9c < skypeforlinux-8.18.0.6-x86_64-1ro.tar > skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz

real	0m12.723s
user	0m47.323s
sys	0m0.184s
$ ls -lh skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz
-rw-r--r-- 1 ruario users 98M Apr 11 09:02 skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz
$ time pigz -cd < skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz > /dev/null 

real	0m1.171s
user	0m1.426s
sys	0m0.068s
$ time bzip2 -1c < skypeforlinux-8.18.0.6-x86_64-1ro.tar > skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2

real	0m28.074s
user	0m27.947s
sys	0m0.124s
$ ls -lh skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2
-rw-r--r-- 1 ruario users 96M Apr 11 09:04 skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2
$ time bzip2 -cd < skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2 > /dev/null

real	0m10.266s
user	0m10.219s
sys	0m0.047s
$ time lbzip2 -1c < skypeforlinux-8.18.0.6-x86_64-1ro.tar > skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2

real	0m7.962s
user	0m30.513s
sys	0m0.513s
$ ls -lh skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2
-rw-r--r-- 1 ruario users 96M Apr 11 09:07 skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2
$ time lbzip2 -cd < skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2 > /dev/null

real	0m2.380s
user	0m9.143s
sys	0m0.300s

Pigz is slightly faster at decompression (with this example and my setup) but is worse on file size and compression. Also worse if you consider both compression and decompression, which would be the case for Slackbuild, creation followed by install.

In fact lbzip2 is so fast at compressing that it still beats pigz, even with -9 and knocks another 5M off the file size

Code:

$ time lbzip2 -9c < skypeforlinux-8.18.0.6-x86_64-1ro.tar > skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2

real	0m7.440s
user	0m29.320s
sys	0m0.164s
$ time pigz -9c < skypeforlinux-8.18.0.6-x86_64-1ro.tar > skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz

real	0m11.681s
user	0m46.082s
sys	0m0.163s
$ ls -lh skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2 skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz
-rw-r--r-- 1 ruario users 91M Apr 11 09:29 skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2
-rw-r--r-- 1 ruario users 98M Apr 11 09:30 skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz

Using -9 with lbzip2 does cause the result file to be slower to decompress however:

Code:

$ time pigz -cd < skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz > /dev/null

real	0m1.176s
user	0m1.439s
sys	0m0.064s
$ time lbzip2 -cd < skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2 > /dev/null

real	0m3.896s
user	0m15.216s
sys	0m0.251s

But the total compression and decompression time is still less. Plus you saved 7M! In summary lbzip2 is a really good compressor and a great choice as a default for self created packages (including SlackBuilds from SBo).

ruario · 04-11-2018, 06:03 AM

If anyone else wants to see the potential benefits of lbzip2. Here are patches you could apply to Pkgtools from 14.2. You will need to apply these, rebuild Pkgtools and upgrade to the patched version. I would also suggest you “export PKGTYPE=tbz” somewhere appropriate for your root user (since you build with root, right‽). That will cause any SlackBuilds that follow the SBo templates to output .tbz packages instead of .tgz.

Building should be faster for many (most?) people (especially for repacks) and will result in smaller packages but of course the speed depends on your hardware. Oh and don't forget to install lbzip2 first or you will only make things slower!

Code:

--- a/scripts/explodepkg
+++ b/scripts/explodepkg
@@ -69,10 +69,18 @@
     packagecompression=gzip
     ;;
   'tbz' )
-    packagecompression=bzip2
+    if which lbzip2 1> /dev/null 2> /dev/null ; then
+      packagecompression=lbzip2
+    else
+      packagecompression=bzip2
+    fi
     ;;
   'bz2' )
-    packagecompression=bzip2
+    if which lbzip2 1> /dev/null 2> /dev/null ; then
+      packagecompression=lbzip2
+    else
+      packagecompression=bzip2
+    fi
     ;;
   'tlz' )
     packagecompression=lzma
--- a/scripts/installpkg
+++ b/scripts/installpkg
@@ -231,7 +231,11 @@
       packagecompression=gzip
       ;;
     'tbz' )
-      packagecompression=bzip2
+      if which lbzip2 1> /dev/null 2> /dev/null ; then
+        packagecompression=lbzip2
+      else
+        packagecompression=bzip2
+      fi
       ;;
     'tlz' )
       packagecompression=lzma
@@ -304,7 +308,11 @@
     packagecompression=gzip
     ;;
   'tbz' )
-    packagecompression=bzip2
+    if which lbzip2 1> /dev/null 2> /dev/null ; then
+      packagecompression=lbzip2
+    else
+      packagecompression=bzip2
+    fi
     ;;
   'tlz' )
     packagecompression=lzma
--- a/scripts/makepkg
+++ b/scripts/makepkg
@@ -312,10 +312,18 @@
   fi
   ;;
 'tbz' | 'tar.bz2' )
-  $TAR cvf - . | bzip2 -9c > ${TARGET_NAME}/${TAR_NAME}.${EXTENSION}
-  ERRCODE=$?
-  if [ ! $ERRCODE = 0 ]; then
-    echo "ERROR:  bzip2 returned error code $ERRCODE -- makepkg failed."
+  if which lbzip2 1> /dev/null 2> /dev/null ; then
+    $TAR cvf - . | lbzip2 -9c > ${TARGET_NAME}/${TAR_NAME}.${EXTENSION}
+    ERRCODE=$?
+    if [ ! $ERRCODE = 0 ]; then
+      echo "ERROR:  lbzip2 returned error code $ERRCODE -- makepkg failed."
+    fi
+  else
+    $TAR cvf - . | bzip2 -9c > ${TARGET_NAME}/${TAR_NAME}.${EXTENSION}
+    ERRCODE=$?
+    if [ ! $ERRCODE = 0 ]; then
+      echo "ERROR:  bzip2 returned error code $ERRCODE -- makepkg failed."
+    fi
   fi
   ;;
 'tlz' | 'tar.lzma' )

I also tried applying these patches to -curent Pkgtools but the changes to makepkg did not apply because of the recent updates. So here is a different set of patches that should work for -current. Beware however, I do not have a -current install to hand, so have not tested these changes on -current yet myself.

Code:

--- a/scripts/explodepkg
+++ b/scripts/explodepkg
@@ -56,10 +56,18 @@
     packagecompression=gzip
     ;;
   'tbz' )
-    packagecompression=bzip2
+    if which lbzip2 1> /dev/null 2> /dev/null ; then
+      packagecompression=lbzip2
+    else
+      packagecompression=bzip2
+    fi
     ;;
   'bz2' )
-    packagecompression=bzip2
+    if which lbzip2 1> /dev/null 2> /dev/null ; then
+      packagecompression=lbzip2
+    else
+      packagecompression=bzip2
+    fi
     ;;
   'tlz' )
     if which plzip 1> /dev/null 2> /dev/null ; then
--- a/scripts/installpkg
+++ b/scripts/installpkg
@@ -233,7 +233,11 @@
       packagecompression=gzip
       ;;
     'tbz' )
-      packagecompression=bzip2
+      if which lbzip2 1> /dev/null 2> /dev/null ; then
+        packagecompression=lbzip2
+      else
+        packagecompression=bzip2
+      fi
       ;;
     'tlz' )
       if which plzip 1> /dev/null 2> /dev/null ; then
@@ -317,7 +321,11 @@
     packagecompression=gzip
     ;;
   'tbz' )
-    packagecompression=bzip2
+    if which lbzip2 1> /dev/null 2> /dev/null ; then
+      packagecompression=lbzip2
+    else
+      packagecompression=bzip2
+    fi
     ;;
   'tlz' )
     if which plzip 1> /dev/null 2> /dev/null ; then
--- a/scripts/makepkg
+++ b/scripts/makepkg
@@ -175,17 +175,27 @@
   fi
 elif [ ! "$(basename $PACKAGE_NAME .tbz)" = "$PACKAGE_NAME" ]; then
   EXTENSION="tbz"
-  COMPRESSOR="bzip2 -9 -c"
-  if ! which bzip2 1> /dev/null 2> /dev/null ; then
-    echo "ERROR:  bzip2 compression utility not found in \$PATH."
-    exit 3
+  if which lbzip2 1> /dev/null 2> /dev/null ; then
+    COMPRESSOR="lbzip2 -9 -c"
+  else
+    if which bzip2 1> /dev/null 2> /dev/null ; then
+      COMPRESSOR="bzip2 -9 -c"
+    else
+      echo "ERROR:  bzip2 compression utility not found in \$PATH."
+      exit 3
+    fi
   fi
 elif [ ! "$(basename $PACKAGE_NAME .tar.bz2)" = "$PACKAGE_NAME" ]; then
   EXTENSION="tar.bz2"
-  COMPRESSOR="bzip2 -9 -c"
-  if ! which bzip2 1> /dev/null 2> /dev/null ; then
-    echo "ERROR:  bzip2 compression utility not found in \$PATH."
-    exit 3
+  if which lbzip2 1> /dev/null 2> /dev/null ; then
+    COMPRESSOR="lbzip2 -9 -c"
+  else
+    if which bzip2 1> /dev/null 2> /dev/null ; then
+      COMPRESSOR="bzip2 -9 -c"
+    else
+      echo "ERROR:  bzip2 compression utility not found in \$PATH."
+      exit 3
+    fi
   fi
 elif [ ! "$(basename $PACKAGE_NAME .tlz)" = "$PACKAGE_NAME" ]; then
   EXTENSION="tlz"

Also a final warning. Be careful testing these changes. I have to tried to keep the alterations minimal and in line with pkgtools conventions and have tested them on 14.2 (but not -current yet). Nonetheless, tweaking pkgtools is making a change to a very central part of Slackware. If something goes horribly wrong I disclaim any responsiblity. Maybe test in a VM or chroot first!

Didier Spaier · 04-11-2018, 06:35 AM

Quote:

Originally Posted by ruario

(since you build with root, right‽)

I always use fakeroot (and as an aside set TMP to $(pwd)), especially when trying my own scripts. I have suffered too much from my mistakes to do otherwise

ruario · 04-11-2018, 06:53 AM

To be 100% honest, so do I but I wrote it like this because I suspect most people use root and I didn't want them to set the variable in the wrong place and then wonder why it didn't work. If people are building/packaging as their user like you and I, then they should set the variable for their user

ruario · 04-11-2018, 09:22 AM

I can make further patches to provide support for brotli, lz4 and lzop if people want. I thought I would do lbzip2 on its own first as this is the one I figure is most likely to be accepted upstream (if any). In addition it might be that I am the only one who has any interest in the others. I do a lot of repacks locally, so really fast compressors are of special interest to me (since this is most of the time in packaging repacks). And brotli is mainly interesting because it is somewhat unique with its small packages and fairly fast decompress. Though I am realistic that its compression time is so long, perhaps it is just too much.

ruario · 04-11-2018, 10:56 PM

Well, I just saw the updated -current Changelog and it had a nice surprise for me! It looks like on -current you won't need my patch to test this any more. Just “export PKGTYPE=tbz” and give it a try.

Thank you @volkerdi!

solarfields · 04-11-2018, 11:30 PM

very interesting thread, ruario
thanks!

ruario · 04-12-2018, 10:16 AM

Ok, I lost my motivation for brotli and lz4 for now at least. The compress times for brotli are so high. With regards to lz4, while a little better than lzop it is no where near as established. Lzop has been around for decades, hence direct support from the likes of GNU tar. Anyway, for those who want to try lzop support, here is a patch for Slackware 14.2 to add it to Pkgtools.

(Note: If you want lbzip2 support as well, apply my earlier patch for 14.2 first)

Code:

--- a/scripts/explodepkg
+++ b/scripts/explodepkg
@@ -43,7 +43,7 @@ if [ $# = 0 ]; then
 Usage: explodepkg package_name [package_name2, ...]
 
 Explodes a Slackware compatible software package
-(or any tar+{gzip,bzip2,lzma,xz archive) in the current directory.
+(or any tar+{gzip,bzip2,lzma,lzop,xz} archive) in the current directory.
 Equivalent to (for each package listed):
 
   ( umask 000 ; cat package_name | COMPRESSOR -dc | tar xvf package_name )
@@ -86,6 +86,22 @@ for PKG in $* ; do
   'xz' )
     packagecompression=xz
     ;;
+  'tzo' )
+    if which lzop 1> /dev/null 2> /dev/null ; then
+      packagecompression=lzop
+    else
+      echo "ERROR:  lzop compression utility not found in \$PATH."
+      exit 3
+    fi
+    ;;
+  'lzo' )
+    if which lzop 1> /dev/null 2> /dev/null ; then
+      packagecompression=lzop
+    else
+      echo "ERROR:  lzop compression utility not found in \$PATH."
+      exit 3
+    fi
+    ;;
   esac
   ( umask 000 ; cat $PKG | $packagecompression -dc | $TAR xvf - 2> /dev/null )
   if [ -r install/doinst.sh ]; then
--- a/scripts/installpkg
+++ b/scripts/installpkg
@@ -68,8 +68,8 @@ export LANG
 # Return a package name that has been stripped of the dirname portion
 # and any of the valid extensions (only):
 pkgbase() {
-  # basename + strip extensions .tbz, .tgz, .tlz and .txz
-  echo "$1" | sed 's?.*/??;s/\.t[bglx]z$//'
+  # basename + strip extensions .tbz, .tgz, .tlz, .txz and .tzo
+  echo "$1" | sed 's?.*/??;s/\.t\([bglx]z\|zo\)$//'
 }
 
 # If installpkg encounters a problem, it will return a non-zero error code.
@@ -109,8 +109,8 @@ usage() {
  cat << EOF
 Usage: installpkg [options] <package_filename>
 
-Installpkg is used to install a .t{gz,bz,lz,xz} package like this:
-   installpkg slackware-package-1.0.0-i486-1.tgz (or .tbz, .tlz, .txz)
+Installpkg is used to install a .t{gz,bz,lz,xz,zo} package like this:
+   installpkg slackware-package-1.0.0-i486-1.tgz (or .tbz, .tlz, .txz, .tzo)
 
 options:      --warn (warn if files will be overwritten, but do not install)
               --root /mnt (install someplace else, like /mnt)
@@ -239,6 +239,14 @@ if [ "$MODE" = "warn" ]; then
     'txz' )
       packagecompression=xz
       ;;
+    'tzo' )
+      if which lzop 1> /dev/null 2> /dev/null ; then
+        packagecompression=lzop
+      else
+        echo "ERROR:  lzop compression utility not found in \$PATH."
+        exit 3
+      fi
+      ;;
     esac
     ( cd $TMP/scan$$ ; $packagecompression -dc | $TAR xf - install ) < $1 2> /dev/null 
     if [ -r $TMP/scan$$/install/doinst.sh ]; then
@@ -280,17 +288,17 @@ for package in $* ; do
   fi
 
   # "shortname" isn't really THAT short...
-  # it's just the full name without ".t{gz,bz,lz,xz}"
+  # it's just the full name without ".t{gz,bz,lz,xz,zo}"
   shortname="$(pkgbase $package)"
   packagedir="$(dirname $package)"
   # This is the base package name, used for grepping tagfiles and descriptions:
   packagebase="$(package_name $shortname)"
 
-  # Reject package if it does not end in '.t{gz,bz,lz,xz}':
+  # Reject package if it does not end in '.t{gz,bz,lz,xz,zo}':
   if [ "$shortname" = "$(basename $package)" ]; then
     EXITSTATUS=3
     if [ "$MODE" = "install" ]; then
-      echo "Cannot install $package:  file does not end in .tgz, .tbz, .tlz, or .txz"
+      echo "Cannot install $package:  file does not end in .tgz, .tbz, .tlz, .txz or .tzo"
     fi
     continue;
   fi
@@ -312,6 +320,14 @@ for package in $* ; do
   'txz' )
     packagecompression=xz
     ;;
+  'tzo' )
+    if which lzop 1> /dev/null 2> /dev/null ; then
+      packagecompression=lzop
+    else
+      echo "ERROR:  lzop compression utility not found in \$PATH."
+      exit 3
+    fi
+    ;;
   esac
 
   # Test presence of external compression utility:
--- a/scripts/makepkg
+++ b/scripts/makepkg
@@ -76,8 +76,7 @@ make_install_script() {
 usage() {
   cat << EOF
 
-Usage: makepkg package_name.tgz
-       (or: package_name.tbz, package_name.tlz, package_name.txz)
+Usage: makepkg package_name.tgz (or: .tbz, .tlz, .txz, tzo)
 
 Makes a Slackware compatible package containing the contents of the current
 and all subdirectories.  If symbolic links exist, they will be removed and
@@ -156,6 +155,10 @@ elif [ ! "$(basename $PACKAGE_NAME .txz)" = "$PACKAGE_NAME" ]; then
   EXTENSION="txz"
 elif [ ! "$(basename $PACKAGE_NAME .tar.xz)" = "$PACKAGE_NAME" ]; then
   EXTENSION="tar.xz"
+elif [ ! "$(basename $PACKAGE_NAME .tzo)" = "$PACKAGE_NAME" ]; then
+  EXTENSION="tzo"
+elif [ ! "$(basename $PACKAGE_NAME .tar.lzo)" = "$PACKAGE_NAME" ]; then
+  EXTENSION="tar.lzo"
 else
   EXTENSION="$(echo $PACKAGE_NAME | rev | cut -f 1 -d . | rev)"
   echo "ERROR:  Package extension .$EXTENSION is not supported."
@@ -196,6 +199,12 @@ case $EXTENSION in
     exit 3
   fi
   ;;
+'tzo' | 'tar.lzo' )
+  if ! which lzop 1> /dev/null 2> /dev/null ; then
+    echo "ERROR:  lzop compression utility not found in \$PATH."
+    exit 3
+  fi
+  ;;
 esac
 
 echo
@@ -263,6 +272,24 @@ if [ ! "$(cat $INST)" = "" ]; then
       echo "Creating your new ./install/doinst.sh..."
     fi
   fi
+elif [ ! "$(basename $PACKAGE_NAME .tzo)" = "$PACKAGE_NAME" ]; then
+  EXTENSION="tzo"
+  # Since lzop is unlikely be selected for high compression, use defaults for
+  # speed. (i.e. no -9)
+  COMPRESSOR="lzop -c"
+  if ! which lzop 1> /dev/null 2> /dev/null ; then
+    echo "ERROR:  lzop compression utility not found in \$PATH."
+    exit 3
+  fi
+elif [ ! "$(basename $PACKAGE_NAME .tar.lzo)" = "$PACKAGE_NAME" ]; then
+  EXTENSION="tar.lzo"
+  # Since lzop is unlikely be selected for high compression, use defaults for
+  # speed. (i.e. no -9)
+  COMPRESSOR="lzop -c"
+  if ! which lzop 1> /dev/null 2> /dev/null ; then
+    echo "ERROR:  lzop compression utility not found in \$PATH."
+    exit 3
+  fi
 else
   echo "No symbolic links were found, so we won't make an installation script."
   echo "You can make your own later in ./install/doinst.sh and rebuild the"
@@ -332,6 +359,13 @@ case $EXTENSION in
     echo "ERROR:  xz returned error code $ERRCODE -- makepkg failed."
   fi
   ;;
+'tzo' | 'tar.lzo' )
+  $TAR cvf - . | lzop -c > ${TARGET_NAME}/${TAR_NAME}.${EXTENSION}
+  ERRCODE=$?
+  if [ ! $ERRCODE = 0 ]; then
+    echo "ERROR:  lzop returned error code $ERRCODE -- makepkg failed."
+  fi
+  ;;
 esac
 
 # Warn of zero-length files:
--- a/scripts/pkgtool
+++ b/scripts/pkgtool
@@ -69,6 +69,9 @@ pkgbase() {
   'txz' )
     PKGRETURN=$(basename $1 .txz)
     ;;
+  'tzo' )
+    PKGRETURN=$(basename $1 .tzo)
+    ;;
   *)
     PKGRETURN=$(basename $1)
     ;;
@@ -493,11 +496,11 @@ install_disk() {
    PACKAGE_DIR=$PACKAGE_DIR/$1
  fi
 
- # If this directory is missing or contains no *.t?z files, bail.
+ # If this directory is missing or contains no *.t?z and *.tzo files, bail.
  if [ ! -d $PACKAGE_DIR ]; then
   return 1
  fi
- if ! ls $PACKAGE_DIR/*.t?z 1> /dev/null 2> /dev/null ; then
+ if ! ls $PACKAGE_DIR/ 2> /dev/null | grep -q '\.t\([bglx]z\|zo\)$' ; then
   return 1
  fi
 
@@ -581,7 +584,7 @@ install_disk() {
     # First we check for missing packages...
     for PKGTEST in $(grep "^CONTENTS:" $PACKAGE_DIR/$CATALOG_FILE | cut -f2- -d : 2> /dev/null) ; do
      # This is not a perfect test.  (say emacs is missing but emacs-nox is not)
-     if ls $PACKAGE_DIR/$PKGTEST*.t?z 1> /dev/null 2> /dev/null ; then # found something like it
+     if ls $PACKAGE_DIR/ 2> /dev/null | grep -q '\.t\([bglx]z\|zo\)$' ; then # found something like it
       true
      else
       cat << EOF > $TMP/tmpmsg
@@ -591,7 +594,7 @@ WARNING!!!
 While looking through your index file ($CATALOG_FILE),
 I noticed that you might be missing a package:
 
-$PKGTEST-\*-\*-\*.t?z
+$PKGTEST-\*-\*-\*.t??
 
 that is supposed to be on this disk (disk $1). You may go
 on with the installation if you wish, but if this is a 
@@ -605,7 +608,7 @@ EOF
     done # checking for missing packages
     # Now we test for extra packages:
     ALLOWED="$(grep CONTENTS: $PACKAGE_DIR/$CATALOG_FILE | cut -b10- 2> /dev/null)" 
-    for PACKAGE_FILENAME in $PACKAGE_DIR/*.t?z; do
+    for PACKAGE_FILENAME in $(echo $PACKAGE_DIR/* | grep '\.t\([bglx]z\|zo\)$'); do
      BASE=$(pkgbase $PACKAGE_FILENAME)
      BASE="$(package_name $BASE)"
      if echo $ALLOWED | grep $BASE 1> /dev/null 2> /dev/null ; then
@@ -618,7 +621,7 @@ WARNING!!!
 While looking through your index file ($CATALOG_FILE),
 I noticed that you have this extra package:
 
-($BASE.t?z) 
+($BASE.t??) 
 
 that I don't recognize. Please be sure this package is
 really supposed to be here, and is not left over from an
@@ -636,8 +639,8 @@ EOF
  fi # check for missing/extra packages
 
  # Install the packages:
- for PACKAGE_FILENAME in $PACKAGE_DIR/*.t?z; do
-  if [ "$PACKAGE_FILENAME" = "$PACKAGE_DIR/*.t?z" ]; then
+ for PACKAGE_FILENAME in $(echo $PACKAGE_DIR/* | grep '\.t\([bglx]z\|zo\)$'); do
+  if [ "$PACKAGE_FILENAME" = "$(echo $PACKAGE_DIR/* | grep '\.t\([bglx]z\|zo\)$')" ]; then
    continue;
   fi
   if [ "$ASK" = "never" ]; then # install the package
--- a/scripts/upgradepkg
+++ b/scripts/upgradepkg
@@ -45,7 +45,7 @@
 # and any of the valid extensions (only):
 pkgbase() {
   PKGRETURN=${1##*/}
-  case "$PKGRETURN" in *.t[gblx]z)
+  case "$PKGRETURN" in *.t[gblx]z|*.tzo)
     PKGRETURN=${PKGRETURN%.*}
   esac
   echo "$PKGRETURN"
@@ -57,10 +57,10 @@ usage() {
 Usage: upgradepkg newpackage [newpackage2 ... ]
        upgradepkg oldpackage%newpackage [oldpackage2%newpackage2 ... ]
 
-Upgradepkg upgrades a Slackware package (.tgz, .tbz, .tlz, .txz) from an
-older version to a newer one.  It does this by INSTALLING the new package
-onto the system, and then REMOVING any files from the old package that
-aren't in the new package.  If the old and new packages have the same
+Upgradepkg upgrades a Slackware package (.tgz, .tbz, .tlz, .txz, .tzo)
+from an older version to a newer one.  It does this by INSTALLING the new
+package onto the system, and then REMOVING any files from the old package
+that aren't in the new package.  If the old and new packages have the same
 name, a single argument is all that is required.  If the packages have
 different names, supply the name of the old package followed by a percent
 symbol (%), then the name of the new package.  Do not add any extra
@@ -73,7 +73,7 @@ to be installed, remove the old ones manually prior to running upgradepkg.
 
 To upgrade in a directory other than / (such as /mnt):  
 
-   ROOT=/mnt upgradepkg package.tgz (or .tbz, .tlz, .txz)
+   ROOT=/mnt upgradepkg package.tgz (or .tbz, .tlz, .txz, .tzo)
 
 EOF
 }

If you run -current, then try this (I even did the english man pages for this one).

(Note: -current has lbzip2 already. If your one does not, update)

Code:

--- a/manpages/explodepkg.8
+++ b/manpages/explodepkg.8
@@ -21,7 +21,7 @@ software package) in the current directory.
 .BI ]
 .SH DESCRIPTION
 .B explodepkg
-uncompresses and untars Slackware *.tgz (or .tbz, .tlz, .txz) packages (or any archive 
+uncompresses and untars Slackware *.tgz (or .tbz, .tlz, .txz, .tzo) packages (or any archive 
 that was created by
 compressing a tarfile with one of the supported compression utilities) in the current directory. It is not usually
 used to install packages, since it doesn't execute the installation scripts 
--- a/manpages/installpkg.8
+++ b/manpages/installpkg.8
@@ -48,8 +48,8 @@ installpkg \- install Slackware packages.
 ]
 .SH DESCRIPTION
 .B installpkg
-installs single or multiple *.tgz (or .tbz, .tlz, .txz) binary packages designed
-for use with the Slackware Linux distribution onto your system.
+installs single or multiple *.tgz (or .tbz, .tlz, .txz, .tzo) binary packages
+designed for use with the Slackware Linux distribution onto your system.
 .SH OPTIONS
 .TP
 .B \--warn packagename
--- a/manpages/makepkg.8
+++ b/manpages/makepkg.8
@@ -40,8 +40,8 @@ appended to the primary installation script
 The package will be written out to the file
 .BI packagename
 which should be the full name, including the extension.  This is usually .txz,
-but .tgz, .tbz, and .tlz are also accepted.  The proper compression utility
-(xz, gzip, bzip2, or lzip) needs to be installed on the machine.
+but .tgz, .tbz, .tlz and .tzo are also accepted.  The proper compression utility
+(xz, gzip, bzip2, lzip or lzop) needs to be installed on the machine.
 .SH OPTIONS
 .TP
 .B \-l, --linkadd y|n
--- a/manpages/pkgdiff.8
+++ b/manpages/pkgdiff.8
@@ -26,8 +26,8 @@ pkgdiff \- compare the file contents of two packages.
 .SH DESCRIPTION
 .B pkgdiff
 Displays the differences in the files contained in two packages. This works with
-plain .tar, Slackware packages (.tgz, .tbz, .tlz, .txz), .rpm, and .deb. By default,
-the results will be displayed in a tree-style unified diff format.
+plain .tar, Slackware packages (.tgz, .tbz, .tlz, .txz, tzo), .rpm, and .deb. By
+default, the results will be displayed in a tree-style unified diff format.
 .SH OPTIONS
 .TP
 .B \--help
--- a/manpages/removepkg.8
+++ b/manpages/removepkg.8
@@ -39,7 +39,7 @@ full package name (as you'd see listed in /var/log/packages/), or by the
 base package name.  For example, the package foo-1.0-i486-1.tgz may be removed
 with any of the following commands:
 
-removepkg foo-1.0-i486-1.tgz (also recognized: .tbz, .tlz, .txz)
+removepkg foo-1.0-i486-1.tgz (also recognized: .tbz, .tlz, .txz, .tzo)
 
 removepkg foo-1.0-i486-1
 
--- a/manpages/upgradepkg.8
+++ b/manpages/upgradepkg.8
@@ -60,7 +60,7 @@ name, it will remove them all.
 .TP
 To upgrade in a directory other than / (such as /mnt):
 .TP
-ROOT=/mnt upgradepkg package.tgz (or .tbz, .tlz, .txz)
+ROOT=/mnt upgradepkg package.tgz (or .tbz, .tlz, .txz, .tzo)
 .SH OPTIONS
 .TP
 .B \--dry-run
--- a/scripts/explodepkg
+++ b/scripts/explodepkg
@@ -26,7 +26,7 @@ if [ $# = 0 ]; then
 Usage: explodepkg package_name [package_name2, ...]
 
 Explodes a Slackware compatible software package
-(or any tar+{gzip,bzip2,lz,xz archive) in the current directory.
+(or any tar+{gzip,bzip2,lz,lzop,xz} archive) in the current directory.
 Equivalent to (for each package listed):
 
   ( umask 000 ; cat package_name | COMPRESSOR -dc | tar xpvf package_name )
@@ -98,6 +98,22 @@ for PKG in $* ; do
   'xz' )
     packagecompression="xz --threads=${THREADS}"
     ;;
+  'tzo' )
+    if which lzop 1> /dev/null 2> /dev/null ; then
+      packagecompression=lzop
+    else
+      echo "ERROR:  lzop compression utility not found in \$PATH."
+      exit 3
+    fi
+    ;;
+  'lzo' )
+    if which lzop 1> /dev/null 2> /dev/null ; then
+      packagecompression=lzop
+    else
+      echo "ERROR:  lzop compression utility not found in \$PATH."
+      exit 3
+    fi
+    ;;
   esac
   ( umask 000 ; cat $PKG | $packagecompression -dc | tar --xattrs --xattrs-include='*' --keep-directory-symlink -xpvf - 2> /dev/null )
   if [ -r install/doinst.sh ]; then
--- a/scripts/installpkg
+++ b/scripts/installpkg
@@ -70,8 +70,8 @@
 # Return a package name that has been stripped of the dirname portion
 # and any of the valid extensions (only):
 pkgbase() {
-  # basename + strip extensions .tbz, .tgz, .tlz and .txz
-  echo "$1" | sed 's?.*/??;s/\.t[bglx]z$//'
+  # basename + strip extensions .tbz, .tgz, .tlz, .txz, .tzo
+  echo "$1" | sed 's?.*/??;s/\.t\([bglx]z\|zo\)$//'
 }
 
 # If installpkg encounters a problem, it will return a non-zero error code.
@@ -95,8 +95,8 @@ usage() {
  cat << EOF
 Usage: installpkg [options] <package_filename>
 
-Installpkg is used to install a .t{gz,bz,lz,xz} package like this:
-   installpkg slackware-package-1.0.0-i486-1.tgz (or .tbz, .tlz, .txz)
+Installpkg is used to install a .t{gz,bz,lz,xz,zo} package like this:
+   installpkg slackware-package-1.0.0-i486-1.tgz (or .tbz, .tlz, .txz, .tzo)
 
 options:      --warn (warn if files will be overwritten, but do not install)
               --root /mnt (install someplace else, like /mnt)
@@ -256,6 +256,14 @@ if [ "$MODE" = "warn" ]; then
         packagecompression="xz --threads=2"
       fi
       ;;
+    'tzo' )
+      if which lzop 1> /dev/null 2> /dev/null ; then
+        packagecompression=lzop
+      else
+        echo "ERROR:  lzop compression utility not found in \$PATH."
+        exit 3
+      fi
+      ;;
     esac
     ( cd $TMP/scan$$ ; $packagecompression -dc | tar xf - install ) < $1 2> /dev/null 
     if [ -r $TMP/scan$$/install/doinst.sh ]; then
@@ -297,17 +305,17 @@ for package in $* ; do
   fi
 
   # "shortname" isn't really THAT short...
-  # it's just the full name without ".t{gz,bz,lz,xz}"
+  # it's just the full name without ".t{gz,bz,lz,xz,zo}"
   shortname="$(pkgbase $package)"
   packagedir="$(dirname $package)"
   # This is the base package name, used for grepping tagfiles and descriptions:
   packagebase="$(package_name $shortname)"
 
-  # Reject package if it does not end in '.t{gz,bz,lz,xz}':
+  # Reject package if it does not end in '.t{gz,bz,lz,xz,zo}':
   if [ "$shortname" = "$(basename $package)" ]; then
     EXITSTATUS=3
     if [ "$MODE" = "install" ]; then
-      echo "Cannot install $package:  file does not end in .tgz, .tbz, .tlz, or .txz"
+      echo "Cannot install $package:  file does not end in .tgz, .tbz, .tlz, .txz or .tzo"
     fi
     continue;
   fi
@@ -344,6 +352,14 @@ for package in $* ; do
       packagecompression="xz --threads=2"
     fi
     ;;
+  'tzo' )
+    if which lzop 1> /dev/null 2> /dev/null ; then
+      packagecompression=lzop
+    else
+      echo "ERROR:  lzop compression utility not found in \$PATH."
+      exit 3
+    fi
+    ;;
   esac
 
   # Test presence of external compression utility:
--- a/scripts/makepkg
+++ b/scripts/makepkg
@@ -66,8 +66,7 @@ make_install_script() {
 usage() {
   cat << EOF
 
-Usage: makepkg package_name.tgz
-       (or: package_name.tbz, package_name.tlz, package_name.txz)
+Usage: makepkg package_name.tgz (or: .tbz, .tlz, .txz, tzo)
 
 Makes a Slackware compatible package containing the contents of the current
 and all subdirectories.  If symbolic links exist, they will be removed and
@@ -260,6 +259,24 @@ elif [ ! "$(basename $PACKAGE_NAME .tar.xz)" = "$PACKAGE_NAME" ]; then
     echo "ERROR:  xz compression utility not found in \$PATH."
     exit 3
   fi
+elif [ ! "$(basename $PACKAGE_NAME .tzo)" = "$PACKAGE_NAME" ]; then
+  EXTENSION="tzo"
+  # Since lzop is unlikely be selected for high compression, use defaults for
+  # speed. (i.e. no -9)
+  COMPRESSOR="lzop -c"
+  if ! which lzop 1> /dev/null 2> /dev/null ; then
+    echo "ERROR:  lzop compression utility not found in \$PATH."
+    exit 3
+  fi
+elif [ ! "$(basename $PACKAGE_NAME .tar.lzo)" = "$PACKAGE_NAME" ]; then
+  EXTENSION="tar.lzo"
+  # Since lzop is unlikely be selected for high compression, use defaults for
+  # speed. (i.e. no -9)
+  COMPRESSOR="lzop -c"
+  if ! which lzop 1> /dev/null 2> /dev/null ; then
+    echo "ERROR:  lzop compression utility not found in \$PATH."
+    exit 3
+  fi
 else
   EXTENSION="$(echo $PACKAGE_NAME | rev | cut -f 1 -d . | rev)"
   echo "ERROR:  Package extension .$EXTENSION is not supported."
--- a/scripts/pkgdiff.orig
+++ b/scripts/pkgdiff
@@ -36,7 +36,7 @@ helpme() {
 Usage: pkgdiff [OPTION] FILE1 FILE2
 Show which files are new and which are removed between two tar archives.
 The tar archives may be uncompressed, or compressed with gzip, bzip2,
-xz, or lzip. Also works with .rpm and .deb.
+xz, lzip or lzop. Also works with .rpm and .deb.
 
   --help           display this help and exit
   -c               use ANSI color with default tree mode
--- a/scripts/pkgtool
+++ b/scripts/pkgtool
@@ -60,6 +60,9 @@ pkgbase() {
   'txz' )
     PKGRETURN=$(basename $1 .txz)
     ;;
+  'tzo' )
+    PKGRETURN=$(basename $1 .tzo)
+    ;;
   *)
     PKGRETURN=$(basename $1)
     ;;
@@ -450,11 +453,11 @@ install_disk() {
    PACKAGE_DIR=$PACKAGE_DIR/$1
  fi
 
- # If this directory is missing or contains no *.t?z files, bail.
+ # If this directory is missing or contains no *.t?z and *.tzo files, bail.
  if [ ! -d $PACKAGE_DIR ]; then
   return 1
  fi
- if ! ls $PACKAGE_DIR/*.t?z 1> /dev/null 2> /dev/null ; then
+ if ! ls $PACKAGE_DIR/ 2> /dev/null | grep -q '\.t\([bglx]z\|zo\)$' ; then
   return 1
  fi
 
@@ -538,7 +541,7 @@ install_disk() {
     # First we check for missing packages...
     for PKGTEST in $(grep "^CONTENTS:" $PACKAGE_DIR/$CATALOG_FILE | cut -f2- -d : 2> /dev/null) ; do
      # This is not a perfect test.  (say emacs is missing but emacs-nox is not)
-     if ls $PACKAGE_DIR/$PKGTEST*.t?z 1> /dev/null 2> /dev/null ; then # found something like it
+     if ls $PACKAGE_DIR/ 2> /dev/null | grep -q '\.t\([bglx]z\|zo\)$' ; then # found something like it
       true
      else
       cat << EOF > $TMP/tmpmsg
@@ -548,7 +551,7 @@ WARNING!!!
 While looking through your index file ($CATALOG_FILE),
 I noticed that you might be missing a package:
 
-$PKGTEST-\*-\*-\*.t?z
+$PKGTEST-\*-\*-\*.t??
 
 that is supposed to be on this disk (disk $1). You may go
 on with the installation if you wish, but if this is a 
@@ -562,7 +565,7 @@ EOF
     done # checking for missing packages
     # Now we test for extra packages:
     ALLOWED="$(grep CONTENTS: $PACKAGE_DIR/$CATALOG_FILE | cut -b10- 2> /dev/null)" 
-    for PACKAGE_FILENAME in $PACKAGE_DIR/*.t?z; do
+    for PACKAGE_FILENAME in $(echo $PACKAGE_DIR/* | grep '\.t\([bglx]z\|zo\)$'); do
      BASE=$(pkgbase $PACKAGE_FILENAME)
      BASE="$(package_name $BASE)"
      if echo $ALLOWED | grep $BASE 1> /dev/null 2> /dev/null ; then
@@ -575,7 +578,7 @@ WARNING!!!
 While looking through your index file ($CATALOG_FILE),
 I noticed that you have this extra package:
 
-($BASE.t?z) 
+($BASE.t??) 
 
 that I don't recognize. Please be sure this package is
 really supposed to be here, and is not left over from an
@@ -593,8 +596,8 @@ EOF
  fi # check for missing/extra packages
 
  # Install the packages:
- for PACKAGE_FILENAME in $PACKAGE_DIR/*.t?z; do
-  if [ "$PACKAGE_FILENAME" = "$PACKAGE_DIR/*.t?z" ]; then
+ for PACKAGE_FILENAME in $(echo $PACKAGE_DIR/* | grep '\.t\([bglx]z\|zo\)$'); do
+  if [ "$PACKAGE_FILENAME" = "$(echo $PACKAGE_DIR/* | grep '\.t\([bglx]z\|zo\)$')" ]; then
    continue;
   fi
   if [ "$ASK" = "never" ]; then # install the package
--- a/scripts/removepkg
+++ b/scripts/removepkg
@@ -119,8 +119,8 @@ shopt -s extglob
 # Return a package name that has been stripped of the dirname portion
 # and any of the valid extensions (only):
 pkgbase() {
-  # basename + strip extensions .tbz, .tgz, .tlz and .txz
-  echo "$1" | sed 's?.*/??;s/\.t[bglx]z$//'
+  # basename + strip extensions .tbz, .tgz, .tlz, .txz, tzo
+  echo "$1" | sed 's?.*/??;s/\.t\([bglx]z\|zo\)$//'
 }
 
 # This makes "sort" run much faster:
--- a/scripts/upgradepkg
+++ b/scripts/upgradepkg
@@ -45,7 +45,7 @@
 # and any of the valid extensions (only):
 pkgbase() {
   PKGRETURN=${1##*/}
-  case "$PKGRETURN" in *.t[gblx]z)
+  case "$PKGRETURN" in *.t[gblx]z|*.tzo)
     PKGRETURN=${PKGRETURN%.*}
   esac
   echo "$PKGRETURN"
@@ -57,10 +57,10 @@ usage() {
 Usage: upgradepkg newpackage [newpackage2 ... ]
        upgradepkg oldpackage%newpackage [oldpackage2%newpackage2 ... ]
 
-Upgradepkg upgrades a Slackware package (.tgz, .tbz, .tlz, .txz) from an
-older version to a newer one.  It does this by INSTALLING the new package
-onto the system, and then REMOVING any files from the old package that
-aren't in the new package.  If the old and new packages have the same
+Upgradepkg upgrades a Slackware package (.tgz, .tbz, .tlz, .txz, .tzo)
+from an older version to a newer one.  It does this by INSTALLING the new
+package onto the system, and then REMOVING any files from the old package
+that aren't in the new package.  If the old and new packages have the same
 name, a single argument is all that is required.  If the packages have
 different names, supply the name of the old package followed by a percent
 symbol (%), then the name of the new package.  Do not add any extra
@@ -73,7 +73,7 @@ to be installed, remove the old ones manually prior to running upgradepkg.
 
 To upgrade in a directory other than / (such as /mnt):  
 
-   ROOT=/mnt upgradepkg package.tgz (or .tbz, .tlz, .txz)
+   ROOT=/mnt upgradepkg package.tgz (or .tbz, .tlz, .txz, .tzo)
 
 EOF
 }

WARNING: I have only done basic testing! Once again, if something goes horribly wrong I disclaim any responsiblity. Maybe test in a VM or chroot first!

franzen · 04-13-2018, 07:58 AM

Hi ruario,

maybe zstd is also worth it for your project.

Johannes

ruario · 04-13-2018, 09:48 AM

I am aware of it and thought about it but have't actually tried it yet. As stated earlier one of the things I do a lot is repackage internal builds of Vivaldi (we have around 10 master builds a day, plus branch builds). I now repackage many of them into slackware format. I do not install every one but I can unpack (and run) or install any I need for regression testing. Given the are similarity in source data between builds, the training/dictionary parts of zstd are interesting for my own personal use case. However that dictionary would be of little use for others. I also couldn't provide public copies of even offical builds compressed with a custom dictionary because they would not be decompressable without this dictionary.

Still it is an interesting project and I am sure there are other benefits, so I could give it a go. If I think it has value I can share a patch set. My gut feeling from the get go is that this is too new to end up in Slackware any time soon. I was actually (plesently) surprised that @volkerdi took my lbzip2 change and that is by far the least invasive of all of these compression options. As you can see from the length of lzop patches, you have to make quite a lot of changes because of extension changes alone, particularly when they do not pattern match .t?z.

I provided the lzop patches in case someone wanted to try and or give feedback or so that they could use this for their own collection of self-built packages but I am a realist and hence (again) suspect that they won't end up in Slackware. However, if @volkerdi wants to give me another suprise, I obviously won't complain!

orbea · 04-13-2018, 11:37 AM

Not all that important, but I just want to suggest to not use 'which', `command -v` is now included in the POSIX spec and should be preferred while 'which' is legacy from c shells.

You can see these links for more info.
https://unix.stackexchange.com/quest...en/85250#85250
http://pubs.opengroup.org/onlinepubs...s/command.html

Note that older versions of the POSIX spec did not require `command -v` and not all shells have caught up or even implement it.

In those cases this function should be portable to anything short of true bourne shells (See heirloom-sh at SBo).

Code:

exists () {
  v=1
  while [ "$#" -gt 0 ]; do
    arg="$1"; shift
    case "$arg" in ''|*/) continue ;; esac
    x="${arg##*/}" z="${arg%/*}"
    [ ! -f "$z/$x" ] || [ ! -x "$z/$x" ] && [ "$z/$x" = "$arg" ] && continue
    [ "$x" = "$z" ] && [ -x "$z/$x" ] && [ ! -f "$arg" ] && z=
    p=":$z:$PATH"
    while [ "$p" != "${p#*:}" ]; do
      p="${p#*:}"; d="${p%%:*}"; b="$d/$x"
      { [ -f "$b" ] && [ -x "$b" ] && { printf %s\\n "$b"; v=0; break; }; } || :
    done
  done
  return "$v"
}

bormant · 04-14-2018, 10:49 AM

It seems .tzo is not good suffix as it breaks package-*.t?z unified mask...

ruario · 04-14-2018, 03:00 PM

The thing is, it IS the short suffix for an lzop compressed tar. It is not just something I arbitrarily selected. I would have made my life easier otherwise. TZO is mentioned in lzop documentation and directly supported by GNU tar (i.e. tar recognizes that it is lzop compressed with that extension). In any case, I believe I accounted for it being different in my patch set (e.g. my changes like "sed 's?.*/??;s/\.t$[bglx]z\|zo$$//'". Are you saying I missed something?

chrisretusn · 04-15-2018, 04:17 AM

Just ran my skypeforlinux.SlackBuild using the new compression. All my SlackBuild scripts have a package type variable as part of the package variables. Set PKGTYPE=${PKGTYPE:-tbz} and ran it. When building with package type txz, packaging slowed at "usr/bin/skypeforlinux". When building with package type tbz, there was no noticeable slowdown. Quite impressed. That said, there was a huge difference in package size with the txz package weighing in at 68M and the tbz package at 91M. I I'd rather see a slowdown and a smaller package than a bigger one. The time it takes to package skypeforliunx is not that long with txz, I can live with that.

ruario · 04-15-2018, 06:44 AM

Not in front of a linux machine at the moment but on my wife’s Mac (2015 Macbook pro) a single threaded XZ compressor (libarchive/bsdtar) takes roughly 2 mins to compress skypeforlinux. A multithreaded compressor (7zip outputting as XZ) takes 45 seconds and a multithreaded compressor (7zip outputting as bzip2) takes 20 seconds.

Assuming similar results under Slackware, this machine should be less than half the time in the compression stage (under something like -current) and six times as quick (under something like 14.2, where a single thread is used for XZ).

Like you, I see a 13M difference in file size but really, what is 13M on modern machines? In any case the uncompressed size of the tar is 229M, so you still saved 138M!

I guess what matters depends on hardware and what the person packaging cares about more, speed or final size. So it is nice to have options.

P.S. You do not need to edit the SlackBuilds (or latest-skye). You can just export PKGTYPE=tbz before running them.