LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Slackware - ARM (https://www.linuxquestions.org/questions/slackware-arm-108/)
-   -   Changing package compression from gzip to LZMA (txz) (https://www.linuxquestions.org/questions/slackware-arm-108/changing-package-compression-from-gzip-to-lzma-txz-4175499505/)

drmozes 03-26-2014 03:27 AM

Changing package compression from gzip to LZMA (txz)
 
Hello

Slackware switched to using LZMA to compress packages in order to solve space issues on the CDs. Since this isn't an issue for ARM, and coupled with my understanding (from xz(1)) that there would be issues decompressing content on machines with limited RAM (128MB, 256MB), it made no sense to change.

However, I'm not sure if there really are issues decompressing LZMA packed packages on machines with small amounts of RAM, so I am thinking of changing the compression. I would assume that the decompression would definitley be slower, however.

Can any of you with real hardware (there's no point testing QEMU - it's not representative of real hardware speed) that 128 or 256MB RAM test these assumptions and provide some timing info please?

Thanks

louigi600 03-26-2014 07:27 AM

I downloadad the kernel source package from slackware x86 (it's one of the beefy packages).

Firs I timed how lon it takes to do a "tar tf":
Code:

real    0m50.771s
user    0m37.350s
sys    0m7.170s

Then I tryed to install it:
Code:

bash-4.2# time installpkg kernel-source-3.10.17_smp-noarch-3.txz
Verifying package kernel-source-3.10.17_smp-noarch-3.txz.
Installing package kernel-source-3.10.17_smp-noarch-3.txz:
PACKAGE DESCRIPTION:
# kernel-source (Linux kernel source)
#
# Source code for Linus Torvalds' Linux kernel.
#
# This is the complete and unmodified source code for the Linux kernel.
#
Executing install script for kernel-source-3.10.17_smp-noarch-3.txz.
Package kernel-source-3.10.17_smp-noarch-3.txz installed.


real    4m7.048s
user    1m58.150s
sys    0m59.510s
bash-4.2#

After having installed it I still have 91Mb free memory (and I've no swap):
Code:

bash-4.2# free
            total      used      free    shared    buffers    cached
Mem:        122996    120484      2512          0      36840      51692
-/+ buffers/cache:      31952      91044
Swap:            0          0          0
bash-4.2#

I did notice though that the cpu usage mas much more intensive then memory usage ... top was telling me that xz was using over 90% cpu and only about 10% ram (but I did not sit the whole 4 minutes watching it).

justwantin 03-27-2014 03:26 PM

I have installed gcc-4.8.2-arm-1.tgz on a raspberrypi with 256mb ram and gpu_mem set at 32mb. Install was off an external usb drive which is where I have my /Slackware directory when I actually install slackware arm on an sd card in a pi. Pi access was via ssh. I installed your tgz packaged first then used makepkg to create a txz package to install.
Code:

root@shed-2:/mnt/sda1# ROOT=/tmp/build time installpkg gcc-4.8.2-arm-1.tgz
Verifying package gcc-4.8.2-arm-1.tgz.
Installing package gcc-4.8.2-arm-1.tgz:
<snipped>

12.36user 4.42system 0:21.89elapsed 76%CPU (0avgtext+0avgdata 6256maxresident)k
3264inputs+85888outputs (15major+45596minor)pagefaults 0swaps

Code:

root@shed-2:/mnt/sda1# ROOT=/tmp/build time installpkg gcc-4.8.2-arm-1.txz
Verifying package gcc-4.8.2-arm-1.txz.
Installing package gcc-4.8.2-arm-1.txz:
<snipped>

25.83user 6.20system 0:39.00elapsed 82%CPU (0avgtext+0avgdata 35872maxresident)k
152inputs+85936outputs (2major+51284minor)pagefaults 0swaps

I've done this several times installing both to a direrctory on the pi and the usb drive and time/cpu load wil vary a bit each time. Below are results from installs to the external drive.

I can do some more testing maybe this evening or tomorrow if you want me to do something specific.
Code:

11.95user 5.10system 0:20.08elapsed 84%CPU (0avgtext+0avgdata 6256maxresident)k
39136inputs+85896outputs (33major+40829minor)pagefaults 0swaps

25.99user 5.99system 0:36.53elapsed 87%CPU (0avgtext+0avgdata 35872maxresident)k
13288inputs+85944outputs (1major+51296minor)pagefaults 0swaps


drmozes 03-28-2014 11:35 AM

Hi guys

Thanks for that feedback. It's not far off what I was expecting really, and it's good to see that swap wasn't impacted.
I'll slowly start changing the package compression to LZMA then as the packages are updated.

guanx 03-28-2014 04:33 PM

Quote:

Originally Posted by louigi600 (Post 5141489)
...
After having installed it I still have 91Mb free memory (and I've no swap):
...

So what? The package is not installed into memory.

guanx 03-28-2014 04:37 PM

Quote:

Originally Posted by drmozes (Post 5141398)
However, I'm not sure if there really are issues decompressing LZMA packed packages on machines with small amounts of RAM, so I am thinking of changing the compression.

If you have more than 65MB free memory on your ARM hardware, then use xz if you wish. The official distribution is intended to be generic, however, so it must stay with gzip.

drmozes 03-29-2014 03:52 AM

Quote:

Originally Posted by guanx (Post 5143027)
If you have more than 65MB free memory on your ARM hardware, then use xz if you wish. The official distribution is intended to be generic, however, so it must stay with gzip.

The official distribution? What do you mean?

guanx 03-29-2014 06:39 AM

Quote:

Originally Posted by drmozes (Post 5143190)
The official distribution? What do you mean?

The slackwarearm compiled and distributed by Stuart Winter.

drmozes 03-29-2014 07:17 AM

Quote:

Originally Posted by guanx (Post 5143223)
The slackwarearm compiled and distributed by Stuart Winter.

That's me.

guanx 03-29-2014 07:20 AM

Quote:

Originally Posted by drmozes (Post 5143229)
That's me.

Ok. The doctor advised me to take complete rest. Now I see he is right.

louigi600 03-29-2014 03:09 PM

Quote:

Originally Posted by guanx (Post 5143024)
So what? The package is not installed into memory.

No it simply means that checking, uncompressing and installing of the package did not use much of the the 128Mb avalible on the system to do the job.

metaschima 03-29-2014 04:40 PM

I know there is a limit on RAM required to decompress a package compress with xz. This depends on the options used to compress it, see xz man page. The default -6 level requires 9 MiB when decompressing, while -9 requires 65 MiB.

As for CPU usage, it is definitely higher than with gzip.

guanx 03-29-2014 06:15 PM

Quote:

Originally Posted by louigi600 (Post 5143394)
No it simply means that checking, uncompressing and installing of the package did not use much of the the 128Mb avalible on the system to do the job.

I guess that's simply because your memory gets reclaimed after use. Correct me if it does not.

louigi600 03-31-2014 05:35 AM

Quote:

Originally Posted by guanx (Post 5143445)
I guess that's simply because your memory gets reclaimed after use. Correct me if it does not.

Slackware's package creation utility uses default compression level (-6), or at least that's what I could tell by examining the script, so I think that I never used that much memory anyway ... as someone said earlier for -6 compression level only 9 Mb are required for decompression and that's in line with what top was telling me about xz as the installation was going on.
I don't think that the kernel would free up 9 Mb when there are 90 Mb free out of 128 and there is not much running on my nas that would require memory anyway.

And BTW: Slackware for the Intel platform has been using xz package compression for quite some tine now so I guess that Stuart wants to make the official ARM port more standard exactly by using the standard Slackware package compression algorithm.

guanx 03-31-2014 07:30 AM

Quote:

Originally Posted by louigi600 (Post 5144005)
Slackware's package creation utility uses default compression level (-6), or at least that's what I could tell by examining the script, so I think that I never used that much memory anyway ...

Thanks for the info! That looks good. Let's hope no one sets the XZ_DEFAULTS environment variable to change compression to -9.

Quote:

Originally Posted by louigi600 (Post 5144005)
I don't think that the kernel would free up 9 Mb when there are 90 Mb free out of 128

That your kernel doesn't free up memory of dead processes really surprised me but since I am still under medical supervision and cannot use computer for long time, I won't make any comment on this.

Quote:

Originally Posted by louigi600 (Post 5144005)
And BTW: Slackware for the Intel platform has been using xz package compression for quite some tine now so I guess that Stuart wants to make the official ARM port more standard exactly by using the standard Slackware package compression algorithm.

My opinion is that Stuart can make the decision on his own. Whatever his decision is, it won't affect me very much because I rarely install packages directly on small systems.


All times are GMT -5. The time now is 04:09 AM.