Changing package compression from gzip to LZMA (txz)

drmozes · 03-26-2014, 03:27 AM

Hello

Slackware switched to using LZMA to compress packages in order to solve space issues on the CDs. Since this isn't an issue for ARM, and coupled with my understanding (from xz(1)) that there would be issues decompressing content on machines with limited RAM (128MB, 256MB), it made no sense to change.

However, I'm not sure if there really are issues decompressing LZMA packed packages on machines with small amounts of RAM, so I am thinking of changing the compression. I would assume that the decompression would definitley be slower, however.

Can any of you with real hardware (there's no point testing QEMU - it's not representative of real hardware speed) that 128 or 256MB RAM test these assumptions and provide some timing info please?

Thanks

louigi600 · 03-26-2014, 07:27 AM

I downloadad the kernel source package from slackware x86 (it's one of the beefy packages).

Firs I timed how lon it takes to do a "tar tf":

Code:

real    0m50.771s
user    0m37.350s
sys     0m7.170s

Then I tryed to install it:

Code:

bash-4.2# time installpkg kernel-source-3.10.17_smp-noarch-3.txz 
Verifying package kernel-source-3.10.17_smp-noarch-3.txz.
Installing package kernel-source-3.10.17_smp-noarch-3.txz:
PACKAGE DESCRIPTION:
# kernel-source (Linux kernel source)
#
# Source code for Linus Torvalds' Linux kernel.
#
# This is the complete and unmodified source code for the Linux kernel.
#
Executing install script for kernel-source-3.10.17_smp-noarch-3.txz.
Package kernel-source-3.10.17_smp-noarch-3.txz installed.


real    4m7.048s
user    1m58.150s
sys     0m59.510s
bash-4.2#

After having installed it I still have 91Mb free memory (and I've no swap):

Code:

bash-4.2# free
             total       used       free     shared    buffers     cached
Mem:        122996     120484       2512          0      36840      51692
-/+ buffers/cache:      31952      91044
Swap:            0          0          0
bash-4.2#

I did notice though that the cpu usage mas much more intensive then memory usage ... top was telling me that xz was using over 90% cpu and only about 10% ram (but I did not sit the whole 4 minutes watching it).

justwantin · 03-27-2014, 03:26 PM

I have installed gcc-4.8.2-arm-1.tgz on a raspberrypi with 256mb ram and gpu_mem set at 32mb. Install was off an external usb drive which is where I have my /Slackware directory when I actually install slackware arm on an sd card in a pi. Pi access was via ssh. I installed your tgz packaged first then used makepkg to create a txz package to install.

Code:

root@shed-2:/mnt/sda1# ROOT=/tmp/build time installpkg gcc-4.8.2-arm-1.tgz 
Verifying package gcc-4.8.2-arm-1.tgz.
Installing package gcc-4.8.2-arm-1.tgz:
<snipped>

12.36user 4.42system 0:21.89elapsed 76%CPU (0avgtext+0avgdata 6256maxresident)k
3264inputs+85888outputs (15major+45596minor)pagefaults 0swaps

Code:

root@shed-2:/mnt/sda1# ROOT=/tmp/build time installpkg gcc-4.8.2-arm-1.txz
Verifying package gcc-4.8.2-arm-1.txz.
Installing package gcc-4.8.2-arm-1.txz:
<snipped>

25.83user 6.20system 0:39.00elapsed 82%CPU (0avgtext+0avgdata 35872maxresident)k
152inputs+85936outputs (2major+51284minor)pagefaults 0swaps

I've done this several times installing both to a direrctory on the pi and the usb drive and time/cpu load wil vary a bit each time. Below are results from installs to the external drive.

I can do some more testing maybe this evening or tomorrow if you want me to do something specific.

Code:

11.95user 5.10system 0:20.08elapsed 84%CPU (0avgtext+0avgdata 6256maxresident)k
39136inputs+85896outputs (33major+40829minor)pagefaults 0swaps

25.99user 5.99system 0:36.53elapsed 87%CPU (0avgtext+0avgdata 35872maxresident)k
13288inputs+85944outputs (1major+51296minor)pagefaults 0swaps

drmozes · 03-28-2014, 11:35 AM

Hi guys

Thanks for that feedback. It's not far off what I was expecting really, and it's good to see that swap wasn't impacted.
I'll slowly start changing the package compression to LZMA then as the packages are updated.

guanx · 03-28-2014, 04:33 PM

Quote:

Originally Posted by louigi600

...
After having installed it I still have 91Mb free memory (and I've no swap):
...

So what? The package is not installed into memory.

guanx · 03-28-2014, 04:37 PM

Quote:

Originally Posted by drmozes

However, I'm not sure if there really are issues decompressing LZMA packed packages on machines with small amounts of RAM, so I am thinking of changing the compression.

If you have more than 65MB free memory on your ARM hardware, then use xz if you wish. The official distribution is intended to be generic, however, so it must stay with gzip.

drmozes · 03-29-2014, 03:52 AM

Quote:

Originally Posted by guanx

If you have more than 65MB free memory on your ARM hardware, then use xz if you wish. The official distribution is intended to be generic, however, so it must stay with gzip.

The official distribution? What do you mean?

guanx · 03-29-2014, 06:39 AM

Quote:

Originally Posted by drmozes

The official distribution? What do you mean?

The slackwarearm compiled and distributed by Stuart Winter.

drmozes · 03-29-2014, 07:17 AM

Quote:

Originally Posted by guanx

The slackwarearm compiled and distributed by Stuart Winter.

That's me.

guanx · 03-29-2014, 07:20 AM

Quote:

Originally Posted by drmozes

That's me.

Ok. The doctor advised me to take complete rest. Now I see he is right.

louigi600 · 03-29-2014, 03:09 PM

Quote:

Originally Posted by guanx

So what? The package is not installed into memory.

No it simply means that checking, uncompressing and installing of the package did not use much of the the 128Mb avalible on the system to do the job.

metaschima · 03-29-2014, 04:40 PM

I know there is a limit on RAM required to decompress a package compress with xz. This depends on the options used to compress it, see xz man page. The default -6 level requires 9 MiB when decompressing, while -9 requires 65 MiB.

As for CPU usage, it is definitely higher than with gzip.

guanx · 03-29-2014, 06:15 PM

Quote:

Originally Posted by louigi600

No it simply means that checking, uncompressing and installing of the package did not use much of the the 128Mb avalible on the system to do the job.

I guess that's simply because your memory gets reclaimed after use. Correct me if it does not.

louigi600 · 03-31-2014, 05:35 AM

Quote:

Originally Posted by guanx

I guess that's simply because your memory gets reclaimed after use. Correct me if it does not.

Slackware's package creation utility uses default compression level (-6), or at least that's what I could tell by examining the script, so I think that I never used that much memory anyway ... as someone said earlier for -6 compression level only 9 Mb are required for decompression and that's in line with what top was telling me about xz as the installation was going on.
I don't think that the kernel would free up 9 Mb when there are 90 Mb free out of 128 and there is not much running on my nas that would require memory anyway.

And BTW: Slackware for the Intel platform has been using xz package compression for quite some tine now so I guess that Stuart wants to make the official ARM port more standard exactly by using the standard Slackware package compression algorithm.

guanx · 03-31-2014, 07:30 AM

Quote:

Originally Posted by louigi600

Slackware's package creation utility uses default compression level (-6), or at least that's what I could tell by examining the script, so I think that I never used that much memory anyway ...

Thanks for the info! That looks good. Let's hope no one sets the XZ_DEFAULTS environment variable to change compression to -9.

Quote:

Originally Posted by louigi600

I don't think that the kernel would free up 9 Mb when there are 90 Mb free out of 128

That your kernel doesn't free up memory of dead processes really surprised me but since I am still under medical supervision and cannot use computer for long time, I won't make any comment on this.

Quote:

Originally Posted by louigi600

And BTW: Slackware for the Intel platform has been using xz package compression for quite some tine now so I guess that Stuart wants to make the official ARM port more standard exactly by using the standard Slackware package compression algorithm.

My opinion is that Stuart can make the decision on his own. Whatever his decision is, it won't affect me very much because I rarely install packages directly on small systems.