Anyone recently played with adding support for further compression formats into pkgtools?
SlackwareThis Forum is for the discussion of Slackware Linux.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Anyone recently played with adding support for further compression formats into pkgtools?
There are four that interest me: brotli, lzop, lz4 and using lbzip2 to handle tbz instead of bzip2.
First, let me state that brotli is absurdly slow to compress… BUT it creates packages that are around the size of XZ and decompresses much faster than gzip. This makes it interesting for large package sets (like a distro), where one individual has to take the brunt of slow compression but all the users benefit from a very fast install when multiple packages are installed at the same time.
lz4 and lzop are compressors that don't make particularly small files and hence don't make a lot of sense for distro provided packages but they are both really quick in compression and decompression. They are great for self made packages (e.g. SlackBuilds), since you get a “reasonable” space saving, while package creation and installation is as fast as can be.
Lz4 is certainly the fastest decompressor but lzop has been around longer, thus its more likely to be stable and has wider support (e.g. GNU tar will handle tar.lzo/tzo files “automagicly” if lzop is in the path).
It would also be great if lbzip2 could be supported by pkgtools (if present). This makes bzip2 support suddendly more interesting because is has really good and fast multithreaded decompression support (even with files made by a different bzip2 encoder). With the right machine (and/or a -1 setting) it can leave gzip behind in terms of compression speed, size and decompression speed.
P.S. If anyone wants patches to pkgtools, I can knock something up. I already have support for this kind of stuff in my own patched pkgtools but only the minimum (installpkg and upgradepkg) to get things working (I use my own makepkg implementation) and I haven't done it in the best possible way. Plus I use 14.2 and see that there are quite a few changes in current pkgtools. But I am willing to clean things up and provide patches for both 14.2 and -current pkgtools, if there is interest.
It would also be great if lbzip2 could be supported by pkgtools (if present). This makes bzip2 support suddendly more interesting because is has really good and fast multithreaded decompression support (even with files made by a different bzip2 encoder). With the right machine (and/or a -1 setting) it can leave gzip behind in terms of compression speed, size and decompression speed.
I did a quick test on the machine I have to hand (a Lenovo Yoga 2 Pro). Comparing “gzip -9” compression (the default setting for makepkg) with “lbzip2 -1”.
Not a super scientific test since I have other stuff running in the background and I am testing a single package (an uncompressed version of the skype package). Feel free to repeat your own tests along these lines.
Code:
$ time gzip -9c < skypeforlinux-8.18.0.6-x86_64-1ro.tar > skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz
real 0m40.257s
user 0m40.149s
sys 0m0.103s
$ time lbzip2 -1c < skypeforlinux-8.18.0.6-x86_64-1ro.tar > skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2
real 0m10.141s
user 0m39.582s
sys 0m0.474s
$ ls -lh skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2
-rw-r--r-- 1 ruario users 96M Apr 10 16:25 skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2
-rw-r--r-- 1 ruario users 98M Apr 10 16:24 skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz
$ time gzip -dc < skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz > /dev/null
real 0m2.202s
user 0m2.190s
sys 0m0.012s
$ time lbzip2 -dc < skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2 > /dev/null
real 0m2.373s
user 0m9.130s
sys 0m0.252s
Roughly 4 times faster at creation, a smaller package and approximately the same speed to decompress. Pretty nice for those personal SlackBuilds IMHO
Dear @ruario, I really doubt that Slackware needs a faster decompression for its packages...
That because, at last theoretically, the packages are installed one-time only, then any brave Slackwarian knows that the additional packages are built from source via the Holly SBO, where of course the major members of community had their shares of scripts copyright.
BUT, if we want to talk about a better compression ratio, then smaller packages, that's another story...
Because I see that the Slackware distribution grows and grows in size, and I bet in several releases will reach the need for blu-ray disks for shipping...
Last edited by Darth Vader; 04-10-2018 at 09:58 AM.
That because at last theoretically, the packages are installed one-time only
I am not sure you are thinking this through…
Upgrades
User making and installing their own packages
The making/installing step speeds things up if you have a smaller fast compressor/decompressor and people do this all the time with SlackBuilds. This can be significant in the case of repacks (like Skype, Vivaldi, Opera, etc.) where compression and uncompression are the bulk of the time taken, since there is no compile step.
Faster decompression is also important when installing a lot of packages in one go. Consider the the number of packages unpackaged when you install or upgrade the distro. The decompression time is a very large part of the total install time. If you can have packages that are around the size of XZ compressed packages but decompress them faster than gzip, this speeds up the Slackware installation process.
However, recompressing the entire distro with brotli (at least with its default compression settings [which are maximum]) would take a crazy long time. The --quality setting would probably need to be tweaked to find a happy compromise.
Of course, my words has no weight in the Slackware development, then what I say is purely a theoretical approach.
BUT, today the Slackware current has around 12GB installed, and it still does not adopted yet the Plasma5. So, I bet on Slackware 15 reaching around 15GB installed.
Long story short, I guess that the Slackware installation kit today enters in a DVD at limits, and more likely that standard DVD in the future will be too small to host a Slackware release. Food for your troughs.
Last edited by Darth Vader; 04-10-2018 at 10:19 AM.
To give you some idea about brotli. Here it is (same setup as before) compared with XZ.
Code:
$ time xz -T0 -9kc skypeforlinux-8.18.0.6-x86_64-1ro.tar > skypeforlinux-8.18.0.6-x86_64-1ro.tar.xz
real 1m59.830s
user 2m22.119s
sys 0m1.732s
$ time brotli -c < skypeforlinux-8.18.0.6-x86_64-1ro.tar > skypeforlinux-8.18.0.6-x86_64-1ro.tar.br
real 16m36.540s
user 16m34.046s
sys 0m0.258s
$ ls -lh skypeforlinux-8.18.0.6-x86_64-1ro.tar.xz skypeforlinux-8.18.0.6-x86_64-1ro.tar.br
-rw-r--r-- 1 ruario users 70M Apr 10 17:02 skypeforlinux-8.18.0.6-x86_64-1ro.tar.br
-rw-r--r-- 1 ruario users 68M Apr 10 16:45 skypeforlinux-8.18.0.6-x86_64-1ro.tar.xz
$ time xz -dc < skypeforlinux-8.18.0.6-x86_64-1ro.tar.xz > /dev/null
real 0m5.681s
user 0m5.637s
sys 0m0.041s
$ time brotli -dc < skypeforlinux-8.18.0.6-x86_64-1ro.tar.br > /dev/null
real 0m1.229s
user 0m1.210s
sys 0m0.019s
Now as you can see it is slow as hell with that maximum compression and at that speed it would likely take days to recompress all the Slackware packages. However, the compressed file size is close to XZ and in the decompression department it blows away not only XZ but it is also around half the time of gzip and lbzip2!
P.S. There might be a better compromise with another --quality setting
BUT, today the Slackware current has around 12GB installed, and it still does not adopted yet the Plasma5. So, I bet on Slackware 15 reaching around 15GB installed.
Personally, I don't really care much about install size as 15Gb is nothing in modern terms.
Long story short, I guess that the Slackware installation kit today enters in a DVD at limits, and more likely that standard DVD in the future will be too small to host a Slackware release. Food for your troughs.
Last edited by Darth Vader; 04-10-2018 at 10:22 AM.
Again it is a personal thing I suppose but I care less about the total Slackware install disk size than I do about package creation and installation times. My views might be clouded because I repackage a lot of binaries. I regularly recompress internal Vivaldi builds into Slackware format to upgrade my primary install.
In any case, even if brotli, lzop and lz4 are of little interest to others I still think the possiblity of using "lbzip2 -1" breaths new life into .tbz and might cause it to be more widely used for self created packages. It has no real downsides over "gzip -9c" (for my setup at least, though probably for others as well).
Quote:
Originally Posted by ruario
[compared to “gzip -9”, it is] Roughly 4 times faster at creation, a smaller package and approximately the same speed to decompress.
Dear @ruario, please take my words with a bit of salt...
BUT, I believe that if you find a way for our BDFL to still ship Slackware 15 in ONE DVD, and not in TWO, will be a great achievement. And for that, I think is needed a better compression ratio.
That because, at last theoretically, the packages are installed one-time only, then any brave Slackwarian knows that the additional packages are built from source via the Holly SBO, where of course the major members of community had their shares of scripts copyright.
Since SBO builds packages (which are then installed), I don't see why improved compression/decompression would not be welcome there.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
