LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware
User Name
Password
Slackware This Forum is for the discussion of Slackware Linux.

Notices


Reply
  Search this Thread
Old 04-10-2018, 10:01 AM   #1
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 2,379

Rep: Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429
Anyone recently played with adding support for further compression formats into pkgtools?


There are four that interest me: brotli, lzop, lz4 and using lbzip2 to handle tbz instead of bzip2.

First, let me state that brotli is absurdly slow to compress… BUT it creates packages that are around the size of XZ and decompresses much faster than gzip. This makes it interesting for large package sets (like a distro), where one individual has to take the brunt of slow compression but all the users benefit from a very fast install when multiple packages are installed at the same time.

lz4 and lzop are compressors that don't make particularly small files and hence don't make a lot of sense for distro provided packages but they are both really quick in compression and decompression. They are great for self made packages (e.g. SlackBuilds), since you get a “reasonable” space saving, while package creation and installation is as fast as can be.

Lz4 is certainly the fastest decompressor but lzop has been around longer, thus its more likely to be stable and has wider support (e.g. GNU tar will handle tar.lzo/tzo files “automagicly” if lzop is in the path).

It would also be great if lbzip2 could be supported by pkgtools (if present). This makes bzip2 support suddendly more interesting because is has really good and fast multithreaded decompression support (even with files made by a different bzip2 encoder). With the right machine (and/or a -1 setting) it can leave gzip behind in terms of compression speed, size and decompression speed.

P.S. If anyone wants patches to pkgtools, I can knock something up. I already have support for this kind of stuff in my own patched pkgtools but only the minimum (installpkg and upgradepkg) to get things working (I use my own makepkg implementation) and I haven't done it in the best possible way. Plus I use 14.2 and see that there are quite a few changes in current pkgtools. But I am willing to clean things up and provide patches for both 14.2 and -current pkgtools, if there is interest.
 
Old 04-10-2018, 10:38 AM   #2
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 2,379

Original Poster
Rep: Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429
Quote:
Originally Posted by ruario View Post
It would also be great if lbzip2 could be supported by pkgtools (if present). This makes bzip2 support suddendly more interesting because is has really good and fast multithreaded decompression support (even with files made by a different bzip2 encoder). With the right machine (and/or a -1 setting) it can leave gzip behind in terms of compression speed, size and decompression speed.
I did a quick test on the machine I have to hand (a Lenovo Yoga 2 Pro). Comparing “gzip -9” compression (the default setting for makepkg) with “lbzip2 -1”.

Not a super scientific test since I have other stuff running in the background and I am testing a single package (an uncompressed version of the skype package). Feel free to repeat your own tests along these lines.

Code:
$ time gzip -9c < skypeforlinux-8.18.0.6-x86_64-1ro.tar > skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz

real	0m40.257s
user	0m40.149s
sys	0m0.103s
$ time lbzip2 -1c < skypeforlinux-8.18.0.6-x86_64-1ro.tar > skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2

real	0m10.141s
user	0m39.582s
sys	0m0.474s
$ ls -lh skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2
-rw-r--r-- 1 ruario users 96M Apr 10 16:25 skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2
-rw-r--r-- 1 ruario users 98M Apr 10 16:24 skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz
$ time gzip -dc < skypeforlinux-8.18.0.6-x86_64-1ro.tar.gz > /dev/null 

real	0m2.202s
user	0m2.190s
sys	0m0.012s
$ time lbzip2 -dc < skypeforlinux-8.18.0.6-x86_64-1ro.tar.bz2 > /dev/null

real	0m2.373s
user	0m9.130s
sys	0m0.252s
Roughly 4 times faster at creation, a smaller package and approximately the same speed to decompress. Pretty nice for those personal SlackBuilds IMHO
 
Old 04-10-2018, 10:42 AM   #3
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 2,379

Original Poster
Rep: Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429
Here is the same machine and same file testing with lz4 and lzop:

Code:
$ time lz4 -c < skypeforlinux-8.18.0.6-x86_64-1ro.tar > skypeforlinux-8.18.0.6-x86_64-1ro.tar.lz4

real	0m0.761s
user	0m0.634s
sys	0m0.125s
$ time lzop -c < skypeforlinux-8.18.0.6-x86_64-1ro.tar > skypeforlinux-8.18.0.6-x86_64-1ro.tar.lzo

real	0m2.544s
user	0m0.763s
sys	0m0.091s
$ ls -lh skypeforlinux-8.18.0.6-x86_64-1ro.tar.lz4 skypeforlinux-8.18.0.6-x86_64-1ro.tar.lzo
-rw-r--r-- 1 ruario users 132M Apr 10 16:37 skypeforlinux-8.18.0.6-x86_64-1ro.tar.lz4
-rw-r--r-- 1 ruario users 131M Apr 10 16:37 skypeforlinux-8.18.0.6-x86_64-1ro.tar.lzo
$ time lz4 -dc < skypeforlinux-8.18.0.6-x86_64-1ro.tar.lz4 > /dev/null 

real	0m0.264s
user	0m0.226s
sys	0m0.039s
$ time lzop -dc < skypeforlinux-8.18.0.6-x86_64-1ro.tar.lzo > /dev/null 

real	0m0.645s
user	0m0.619s
sys	0m0.026s
lz4 is 10 times the speed at decompression, compared with "gzip -9" and "lzip2 -1"!

Last edited by ruario; 04-10-2018 at 10:44 AM.
 
Old 04-10-2018, 10:50 AM   #4
Darth Vader
Senior Member
 
Registered: May 2008
Location: Romania
Distribution: DARKSTAR Linux 2008.1
Posts: 2,727

Rep: Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231
Dear @ruario, I really doubt that Slackware needs a faster decompression for its packages...

That because, at last theoretically, the packages are installed one-time only, then any brave Slackwarian knows that the additional packages are built from source via the Holly SBO, where of course the major members of community had their shares of scripts copyright.

BUT, if we want to talk about a better compression ratio, then smaller packages, that's another story...

Because I see that the Slackware distribution grows and grows in size, and I bet in several releases will reach the need for blu-ray disks for shipping...

Last edited by Darth Vader; 04-10-2018 at 10:58 AM.
 
Old 04-10-2018, 10:56 AM   #5
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 2,379

Original Poster
Rep: Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429
Quote:
Originally Posted by Darth Vader View Post
That because at last theoretically, the packages are installed one-time only
I am not sure you are thinking this through…
  1. Upgrades
  2. User making and installing their own packages

The making/installing step speeds things up if you have a smaller fast compressor/decompressor and people do this all the time with SlackBuilds. This can be significant in the case of repacks (like Skype, Vivaldi, Opera, etc.) where compression and uncompression are the bulk of the time taken, since there is no compile step.
 
Old 04-10-2018, 11:00 AM   #6
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 2,379

Original Poster
Rep: Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429
Faster decompression is also important when installing a lot of packages in one go. Consider the the number of packages unpackaged when you install or upgrade the distro. The decompression time is a very large part of the total install time. If you can have packages that are around the size of XZ compressed packages but decompress them faster than gzip, this speeds up the Slackware installation process.
 
Old 04-10-2018, 11:02 AM   #7
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 2,379

Original Poster
Rep: Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429
However, recompressing the entire distro with brotli (at least with its default compression settings [which are maximum]) would take a crazy long time. The --quality setting would probably need to be tweaked to find a happy compromise.
 
Old 04-10-2018, 11:11 AM   #8
Darth Vader
Senior Member
 
Registered: May 2008
Location: Romania
Distribution: DARKSTAR Linux 2008.1
Posts: 2,727

Rep: Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231
Of course, my words has no weight in the Slackware development, then what I say is purely a theoretical approach.

BUT, today the Slackware current has around 12GB installed, and it still does not adopted yet the Plasma5. So, I bet on Slackware 15 reaching around 15GB installed.

Long story short, I guess that the Slackware installation kit today enters in a DVD at limits, and more likely that standard DVD in the future will be too small to host a Slackware release. Food for your troughs.

Last edited by Darth Vader; 04-10-2018 at 11:19 AM.
 
Old 04-10-2018, 11:15 AM   #9
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 2,379

Original Poster
Rep: Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429
To give you some idea about brotli. Here it is (same setup as before) compared with XZ.

Code:
$ time xz -T0 -9kc skypeforlinux-8.18.0.6-x86_64-1ro.tar > skypeforlinux-8.18.0.6-x86_64-1ro.tar.xz

real	1m59.830s
user	2m22.119s
sys	0m1.732s
$ time brotli -c < skypeforlinux-8.18.0.6-x86_64-1ro.tar > skypeforlinux-8.18.0.6-x86_64-1ro.tar.br

real	16m36.540s
user	16m34.046s
sys	0m0.258s
$ ls -lh skypeforlinux-8.18.0.6-x86_64-1ro.tar.xz skypeforlinux-8.18.0.6-x86_64-1ro.tar.br
-rw-r--r-- 1 ruario users 70M Apr 10 17:02 skypeforlinux-8.18.0.6-x86_64-1ro.tar.br
-rw-r--r-- 1 ruario users 68M Apr 10 16:45 skypeforlinux-8.18.0.6-x86_64-1ro.tar.xz
$ time xz -dc < skypeforlinux-8.18.0.6-x86_64-1ro.tar.xz > /dev/null 

real	0m5.681s
user	0m5.637s
sys	0m0.041s
$ time brotli -dc < skypeforlinux-8.18.0.6-x86_64-1ro.tar.br > /dev/null 

real	0m1.229s
user	0m1.210s
sys	0m0.019s
Now as you can see it is slow as hell with that maximum compression and at that speed it would likely take days to recompress all the Slackware packages. However, the compressed file size is close to XZ and in the decompression department it blows away not only XZ but it is also around half the time of gzip and lbzip2!

P.S. There might be a better compromise with another --quality setting

Last edited by ruario; 04-10-2018 at 11:19 AM.
 
Old 04-10-2018, 11:19 AM   #10
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 2,379

Original Poster
Rep: Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429
Quote:
Originally Posted by Darth Vader View Post
BUT, today the Slackware current has around 12GB installed, and it still does not adopted yet the Plasma5. So, I bet on Slackware 15 reaching around 15GB installed.
Personally, I don't really care much about install size as 15Gb is nothing in modern terms.
 
Old 04-10-2018, 11:21 AM   #11
Darth Vader
Senior Member
 
Registered: May 2008
Location: Romania
Distribution: DARKSTAR Linux 2008.1
Posts: 2,727

Rep: Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231
I explained the context, in the next sentence.

Quote:
Originally Posted by Darth Vader View Post
Long story short, I guess that the Slackware installation kit today enters in a DVD at limits, and more likely that standard DVD in the future will be too small to host a Slackware release. Food for your troughs.

Last edited by Darth Vader; 04-10-2018 at 11:22 AM.
 
Old 04-10-2018, 11:25 AM   #12
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 2,379

Original Poster
Rep: Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429
Quote:
Originally Posted by Darth Vader View Post
I explained the context, in the next sentence.
Again it is a personal thing I suppose but I care less about the total Slackware install disk size than I do about package creation and installation times. My views might be clouded because I repackage a lot of binaries. I regularly recompress internal Vivaldi builds into Slackware format to upgrade my primary install.
 
Old 04-10-2018, 11:28 AM   #13
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 2,379

Original Poster
Rep: Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429Reputation: 1429
In any case, even if brotli, lzop and lz4 are of little interest to others I still think the possiblity of using "lbzip2 -1" breaths new life into .tbz and might cause it to be more widely used for self created packages. It has no real downsides over "gzip -9c" (for my setup at least, though probably for others as well).

Quote:
Originally Posted by ruario View Post
[compared to “gzip -9”, it is] Roughly 4 times faster at creation, a smaller package and approximately the same speed to decompress.
 
Old 04-10-2018, 12:09 PM   #14
Darth Vader
Senior Member
 
Registered: May 2008
Location: Romania
Distribution: DARKSTAR Linux 2008.1
Posts: 2,727

Rep: Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231Reputation: 1231
Dear @ruario, please take my words with a bit of salt...

BUT, I believe that if you find a way for our BDFL to still ship Slackware 15 in ONE DVD, and not in TWO, will be a great achievement. And for that, I think is needed a better compression ratio.
 
Old 04-10-2018, 10:13 PM   #15
Richard Cranium
Senior Member
 
Registered: Apr 2009
Location: Carrollton, Texas
Distribution: Slackware64 14.2
Posts: 3,092

Rep: Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469Reputation: 1469
Quote:
Originally Posted by Darth Vader View Post
That because, at last theoretically, the packages are installed one-time only, then any brave Slackwarian knows that the additional packages are built from source via the Holly SBO, where of course the major members of community had their shares of scripts copyright.
Since SBO builds packages (which are then installed), I don't see why improved compression/decompression would not be welcome there.
 
2 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Adding gzip compression to a netcat proxy haggismn Linux - Networking 3 03-04-2012 05:31 AM
Recommend GOOD video games that you've recently played H_TeXMeX_H General 7 09-17-2010 12:34 AM
adding different file formats to kaffeine player hbob49 Linux - Software 3 10-26-2007 03:21 PM
kaffeine recently played history rafc Linux - Software 3 07-04-2004 05:06 AM
Came a-cross some linux compression formats but not sure how to open them in Windows maximalred Linux - Distributions 1 06-09-2004 07:13 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware

All times are GMT -5. The time now is 11:30 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration