LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware > Slackware - ARM
User Name
Password
Slackware - ARM This forum is for the discussion of Slackware ARM.

Notices

Reply
 
Search this Thread
Old 03-26-2014, 03:27 AM   #1
drmozes
Slackware Contributor
 
Registered: Apr 2008
Location: Surrey, England
Distribution: Slackware
Posts: 144

Rep: Reputation: 114Reputation: 114
Changing package compression from gzip to LZMA (txz)


Hello

Slackware switched to using LZMA to compress packages in order to solve space issues on the CDs. Since this isn't an issue for ARM, and coupled with my understanding (from xz(1)) that there would be issues decompressing content on machines with limited RAM (128MB, 256MB), it made no sense to change.

However, I'm not sure if there really are issues decompressing LZMA packed packages on machines with small amounts of RAM, so I am thinking of changing the compression. I would assume that the decompression would definitley be slower, however.

Can any of you with real hardware (there's no point testing QEMU - it's not representative of real hardware speed) that 128 or 256MB RAM test these assumptions and provide some timing info please?

Thanks
 
Old 03-26-2014, 07:27 AM   #2
louigi600
Member
 
Registered: Dec 2013
Location: Italy
Distribution: Slackware
Posts: 149

Rep: Reputation: 9
I downloadad the kernel source package from slackware x86 (it's one of the beefy packages).

Firs I timed how lon it takes to do a "tar tf":
Code:
real    0m50.771s
user    0m37.350s
sys     0m7.170s
Then I tryed to install it:
Code:
bash-4.2# time installpkg kernel-source-3.10.17_smp-noarch-3.txz 
Verifying package kernel-source-3.10.17_smp-noarch-3.txz.
Installing package kernel-source-3.10.17_smp-noarch-3.txz:
PACKAGE DESCRIPTION:
# kernel-source (Linux kernel source)
#
# Source code for Linus Torvalds' Linux kernel.
#
# This is the complete and unmodified source code for the Linux kernel.
#
Executing install script for kernel-source-3.10.17_smp-noarch-3.txz.
Package kernel-source-3.10.17_smp-noarch-3.txz installed.


real    4m7.048s
user    1m58.150s
sys     0m59.510s
bash-4.2#
After having installed it I still have 91Mb free memory (and I've no swap):
Code:
bash-4.2# free
             total       used       free     shared    buffers     cached
Mem:        122996     120484       2512          0      36840      51692
-/+ buffers/cache:      31952      91044
Swap:            0          0          0
bash-4.2#
I did notice though that the cpu usage mas much more intensive then memory usage ... top was telling me that xz was using over 90% cpu and only about 10% ram (but I did not sit the whole 4 minutes watching it).

Last edited by louigi600; 03-27-2014 at 02:01 AM.
 
Old 03-27-2014, 03:26 PM   #3
justwantin
Member
 
Registered: Aug 2003
Location: Melbourne, Australia
Distribution: Slackware, Slackware ARM, Salix and Porteus
Posts: 551

Rep: Reputation: 43
I have installed gcc-4.8.2-arm-1.tgz on a raspberrypi with 256mb ram and gpu_mem set at 32mb. Install was off an external usb drive which is where I have my /Slackware directory when I actually install slackware arm on an sd card in a pi. Pi access was via ssh. I installed your tgz packaged first then used makepkg to create a txz package to install.
Code:
root@shed-2:/mnt/sda1# ROOT=/tmp/build time installpkg gcc-4.8.2-arm-1.tgz 
Verifying package gcc-4.8.2-arm-1.tgz.
Installing package gcc-4.8.2-arm-1.tgz:
<snipped>

12.36user 4.42system 0:21.89elapsed 76%CPU (0avgtext+0avgdata 6256maxresident)k
3264inputs+85888outputs (15major+45596minor)pagefaults 0swaps
Code:
root@shed-2:/mnt/sda1# ROOT=/tmp/build time installpkg gcc-4.8.2-arm-1.txz
Verifying package gcc-4.8.2-arm-1.txz.
Installing package gcc-4.8.2-arm-1.txz:
<snipped>

25.83user 6.20system 0:39.00elapsed 82%CPU (0avgtext+0avgdata 35872maxresident)k
152inputs+85936outputs (2major+51284minor)pagefaults 0swaps
I've done this several times installing both to a direrctory on the pi and the usb drive and time/cpu load wil vary a bit each time. Below are results from installs to the external drive.

I can do some more testing maybe this evening or tomorrow if you want me to do something specific.
Code:
11.95user 5.10system 0:20.08elapsed 84%CPU (0avgtext+0avgdata 6256maxresident)k
39136inputs+85896outputs (33major+40829minor)pagefaults 0swaps

25.99user 5.99system 0:36.53elapsed 87%CPU (0avgtext+0avgdata 35872maxresident)k
13288inputs+85944outputs (1major+51296minor)pagefaults 0swaps
 
Old 03-28-2014, 11:35 AM   #4
drmozes
Slackware Contributor
 
Registered: Apr 2008
Location: Surrey, England
Distribution: Slackware
Posts: 144

Original Poster
Rep: Reputation: 114Reputation: 114
Hi guys

Thanks for that feedback. It's not far off what I was expecting really, and it's good to see that swap wasn't impacted.
I'll slowly start changing the package compression to LZMA then as the packages are updated.
 
Old 03-28-2014, 04:33 PM   #5
guanx
Senior Member
 
Registered: Dec 2008
Posts: 1,014

Rep: Reputation: 146Reputation: 146
Quote:
Originally Posted by louigi600 View Post
...
After having installed it I still have 91Mb free memory (and I've no swap):
...
So what? The package is not installed into memory.
 
Old 03-28-2014, 04:37 PM   #6
guanx
Senior Member
 
Registered: Dec 2008
Posts: 1,014

Rep: Reputation: 146Reputation: 146
Quote:
Originally Posted by drmozes View Post
However, I'm not sure if there really are issues decompressing LZMA packed packages on machines with small amounts of RAM, so I am thinking of changing the compression.
If you have more than 65MB free memory on your ARM hardware, then use xz if you wish. The official distribution is intended to be generic, however, so it must stay with gzip.
 
Old 03-29-2014, 03:52 AM   #7
drmozes
Slackware Contributor
 
Registered: Apr 2008
Location: Surrey, England
Distribution: Slackware
Posts: 144

Original Poster
Rep: Reputation: 114Reputation: 114
Quote:
Originally Posted by guanx View Post
If you have more than 65MB free memory on your ARM hardware, then use xz if you wish. The official distribution is intended to be generic, however, so it must stay with gzip.
The official distribution? What do you mean?
 
Old 03-29-2014, 06:39 AM   #8
guanx
Senior Member
 
Registered: Dec 2008
Posts: 1,014

Rep: Reputation: 146Reputation: 146
Quote:
Originally Posted by drmozes View Post
The official distribution? What do you mean?
The slackwarearm compiled and distributed by Stuart Winter.
 
Old 03-29-2014, 07:17 AM   #9
drmozes
Slackware Contributor
 
Registered: Apr 2008
Location: Surrey, England
Distribution: Slackware
Posts: 144

Original Poster
Rep: Reputation: 114Reputation: 114
Quote:
Originally Posted by guanx View Post
The slackwarearm compiled and distributed by Stuart Winter.
That's me.
 
Old 03-29-2014, 07:20 AM   #10
guanx
Senior Member
 
Registered: Dec 2008
Posts: 1,014

Rep: Reputation: 146Reputation: 146
Quote:
Originally Posted by drmozes View Post
That's me.
Ok. The doctor advised me to take complete rest. Now I see he is right.
 
Old 03-29-2014, 03:09 PM   #11
louigi600
Member
 
Registered: Dec 2013
Location: Italy
Distribution: Slackware
Posts: 149

Rep: Reputation: 9
Quote:
Originally Posted by guanx View Post
So what? The package is not installed into memory.
No it simply means that checking, uncompressing and installing of the package did not use much of the the 128Mb avalible on the system to do the job.
 
Old 03-29-2014, 04:40 PM   #12
metaschima
Member
 
Registered: Dec 2013
Distribution: Slackware
Posts: 980

Rep: Reputation: Disabled
I know there is a limit on RAM required to decompress a package compress with xz. This depends on the options used to compress it, see xz man page. The default -6 level requires 9 MiB when decompressing, while -9 requires 65 MiB.

As for CPU usage, it is definitely higher than with gzip.
 
Old 03-29-2014, 06:15 PM   #13
guanx
Senior Member
 
Registered: Dec 2008
Posts: 1,014

Rep: Reputation: 146Reputation: 146
Quote:
Originally Posted by louigi600 View Post
No it simply means that checking, uncompressing and installing of the package did not use much of the the 128Mb avalible on the system to do the job.
I guess that's simply because your memory gets reclaimed after use. Correct me if it does not.
 
Old 03-31-2014, 05:35 AM   #14
louigi600
Member
 
Registered: Dec 2013
Location: Italy
Distribution: Slackware
Posts: 149

Rep: Reputation: 9
Quote:
Originally Posted by guanx View Post
I guess that's simply because your memory gets reclaimed after use. Correct me if it does not.
Slackware's package creation utility uses default compression level (-6), or at least that's what I could tell by examining the script, so I think that I never used that much memory anyway ... as someone said earlier for -6 compression level only 9 Mb are required for decompression and that's in line with what top was telling me about xz as the installation was going on.
I don't think that the kernel would free up 9 Mb when there are 90 Mb free out of 128 and there is not much running on my nas that would require memory anyway.

And BTW: Slackware for the Intel platform has been using xz package compression for quite some tine now so I guess that Stuart wants to make the official ARM port more standard exactly by using the standard Slackware package compression algorithm.
 
Old 03-31-2014, 07:30 AM   #15
guanx
Senior Member
 
Registered: Dec 2008
Posts: 1,014

Rep: Reputation: 146Reputation: 146
Quote:
Originally Posted by louigi600 View Post
Slackware's package creation utility uses default compression level (-6), or at least that's what I could tell by examining the script, so I think that I never used that much memory anyway ...
Thanks for the info! That looks good. Let's hope no one sets the XZ_DEFAULTS environment variable to change compression to -9.

Quote:
Originally Posted by louigi600 View Post
I don't think that the kernel would free up 9 Mb when there are 90 Mb free out of 128
That your kernel doesn't free up memory of dead processes really surprised me but since I am still under medical supervision and cannot use computer for long time, I won't make any comment on this.

Quote:
Originally Posted by louigi600 View Post
And BTW: Slackware for the Intel platform has been using xz package compression for quite some tine now so I guess that Stuart wants to make the official ARM port more standard exactly by using the standard Slackware package compression algorithm.
My opinion is that Stuart can make the decision on his own. Whatever his decision is, it won't affect me very much because I rarely install packages directly on small systems.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] What are the differences between the Gzip, Lzma, Xz and Lzop compression smag Linux - Kernel 4 08-29-2011 10:07 AM
Quiz: Why did gzip, bzip2 and lzma fail on this piece of data? What is it? irmin General 7 07-20-2010 02:06 PM
Determining the compression ratio when using -9 with gzip kaplan71 Linux - Software 2 04-08-2010 03:45 PM
Is gzip -c9 giving the highest compression? Thaidog Linux - General 2 04-23-2007 05:15 AM
gzip , general compression question ?? glenn69 Linux - Newbie 5 10-12-2004 12:20 AM


All times are GMT -5. The time now is 03:38 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration