Anyone recently played with adding support for further compression formats into pkgtools?
 

There are four that interest me: brotli, lzop, lz4 and using lbzip2 to handle tbz instead of bzip2.

First, let me state that brotli is absurdly slow to compress… BUT it creates packages that are around the size of XZ and decompresses much faster than gzip. This makes it interesting for large package sets (like a distro), where one individual has to take the brunt of slow compression but all the users benefit from a very fast install when multiple packages are installed at the same time.

lz4 and lzop are compressors that don't make particularly small files and hence don't make a lot of sense for distro provided packages but they are both really quick in compression and decompression. They are great for self made packages (e.g. SlackBuilds), since you get a “reasonable” space saving, while package creation and installation is as fast as can be.

Lz4 is certainly the fastest decompressor but lzop has been around longer, thus its more likely to be stable and has wider support (e.g. GNU tar will handle tar.lzo/tzo files “automagicly” if lzop is in the path).

It would also be great if lbzip2 could be supported by pkgtools (if present). This makes bzip2 support suddendly more interesting because is has really good and fast multithreaded decompression support (even with files made by a different bzip2 encoder). With the right machine (and/or a -1 setting) it can leave gzip behind in terms of compression speed, size and decompression speed.

P.S. If anyone wants patches to pkgtools, I can knock something up. I already have support for this kind of stuff in my own patched pkgtools but only the minimum (installpkg and upgradepkg) to get things working (I use my own makepkg implementation) and I haven't done it in the best possible way. Plus I use 14.2 and see that there are quite a few changes in current pkgtools. But I am willing to clean things up and provide patches for both 14.2 and -current pkgtools, if there is interest.

I did a quick test on the machine I have to hand (a Lenovo Yoga 2 Pro). Comparing “gzip -9” compression (the default setting for makepkg) with “lbzip2 -1”.

Not a super scientific test since I have other stuff running in the background and I am testing a single package (an uncompressed version of the skype package). Feel free to repeat your own tests along these lines.

Roughly 4 times faster at creation, a smaller package and approximately the same speed to decompress. Pretty nice for those personal SlackBuilds IMHO

Here is the same machine and same file testing with lz4 and lzop:

lz4 is 10 times the speed at decompression, compared with "gzip -9" and "lzip2 -1"!

Dear @ruario, I really doubt that Slackware needs a faster decompression for its packages... ;)

That because, at last theoretically, the packages are installed one-time only, then any brave Slackwarian knows that the additional packages are built from source via the Holly SBO, where of course the major members of community had their shares of scripts copyright. ;)

BUT, if we want to talk about a better compression ratio, then smaller packages, that's another story...

Because I see that the Slackware distribution grows and grows in size, and I bet in several releases will reach the need for blu-ray disks for shipping... :D

I am not sure you are thinking this through…

1. Upgrades
2. User making and installing their own packages

The making/installing step speeds things up if you have a smaller fast compressor/decompressor and people do this all the time with SlackBuilds. This can be significant in the case of repacks (like Skype, Vivaldi, Opera, etc.) where compression and uncompression are the bulk of the time taken, since there is no compile step.

Faster decompression is also important when installing a lot of packages in one go. Consider the the number of packages unpackaged when you install or upgrade the distro. The decompression time is a very large part of the total install time. If you can have packages that are around the size of XZ compressed packages but decompress them faster than gzip, this speeds up the Slackware installation process.

However, recompressing the entire distro with brotli (at least with its default compression settings [which are maximum]) would take a crazy long time. The --quality setting would probably need to be tweaked to find a happy compromise.

Of course, my words has no weight in the Slackware development, then what I say is purely a theoretical approach.

BUT, today the Slackware current has around 12GB installed, and it still does not adopted yet the Plasma5. So, I bet on Slackware 15 reaching around 15GB installed.

Long story short, I guess that the Slackware installation kit today enters in a DVD at limits, and more likely that standard DVD in the future will be too small to host a Slackware release. Food for your troughs. ;)

To give you some idea about brotli. Here it is (same setup as before) compared with XZ.

Now as you can see it is slow as hell with that maximum compression and at that speed it would likely take days to recompress all the Slackware packages. However, the compressed file size is close to XZ and in the decompression department it blows away not only XZ but it is also around half the time of gzip and lbzip2!

P.S. There might be a better compromise with another --quality setting

Personally, I don't really care much about install size as 15Gb is nothing in modern terms.

I explained the context, in the next sentence. ;)

Again it is a personal thing I suppose but I care less about the total Slackware install disk size than I do about package creation and installation times. My views might be clouded because I repackage a lot of binaries. I regularly recompress internal Vivaldi builds into Slackware format to upgrade my primary install.

In any case, even if brotli, lzop and lz4 are of little interest to others I still think the possiblity of using "lbzip2 -1" breaths new life into .tbz and might cause it to be more widely used for self created packages. It has no real downsides over "gzip -9c" (for my setup at least, though probably for others as well).

Dear @ruario, please take my words with a bit of salt...

BUT, I believe that if you find a way for our BDFL to still ship Slackware 15 in ONE DVD, and not in TWO, will be a great achievement. And for that, I think is needed a better compression ratio. ;)

Since SBO builds packages (which are then installed), I don't see why improved compression/decompression would not be welcome there.