LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 02-28-2013, 06:41 AM   #1
alaios
Senior Member
 
Registered: Jan 2003
Location: Aachen
Distribution: Opensuse 11.2 (nice and steady)
Posts: 2,203

Rep: Reputation: 45
Lzma Vs Bzip2 (Stability and performance)


Dear all,
I have been reading the following link, that is about lzma performance compred to bzip2.

Lzma Vs Bzip2 Better Compression than bzip2 on UNIX / Linux

I would like to ask you if you have the same feeling that the article claims that lzma can achieve a better of compressed ratio. Moreover, I would like to ask you if the lzma is well supported in linux (in terms that I can download it and start using it, without major and minor bugs floating around).I

I would like to thank you in advance for your reply

Regards
Alex
 
Old 02-28-2013, 08:24 AM   #2
kooru
Senior Member
 
Registered: Sep 2012
Posts: 1,385

Rep: Reputation: 274Reputation: 274Reputation: 274
I don't see the link into your post
 
Old 02-28-2013, 09:04 AM   #3
H_TeXMeX_H
LQ Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292
I recommend LZMA2 = xz. Yes it has been well tested and should be mostly bug-free. If you want the best compression algorithm available, then this is it.

On a daily basis, however, I still use gz, because it is much faster and I've got plenty of space on the disk.
 
Old 02-28-2013, 11:35 AM   #4
alaios
Senior Member
 
Registered: Jan 2003
Location: Aachen
Distribution: Opensuse 11.2 (nice and steady)
Posts: 2,203

Original Poster
Rep: Reputation: 45
Hi,
thanks for the answer.
In my opensuse I have actually almost all the tools
man lzma returns at header
Quote:
XZ(1) XZ Utils XZ(1)



NAME
xz, unxz, xzcat, lzma, unlzma, lzcat - Compress or decompress .xz and
.lzma files
1. Is that what you mean?
2. I just wonder if tar and the aforementioned algorithms were made with large files in mind. For example, I have a tar of 1.2 TB that should be largely compressible, file includes mostly sequences of 0s and 1s.... I have found though in the man lzma the following text

Quote:
Level xz LZMA Utils 4.32.x
-0 3 MiB N/A
-1 9 MiB 2 MiB
-2 17 MiB 12 MiB
-3 32 MiB 12 MiB
-4 48 MiB 16 MiB
-5 94 MiB 26 MiB
-6 94 MiB 45 MiB
-7 186 MiB 83 MiB
-8 370 MiB 159 MiB
-9 674 MiB 311 MiB

.....
Column descriptions:

· DictSize is the LZMA2 dictionary size. It is waste of memory to use a dictionary bigger than the size of the uncompressed file.
This is why it is good to avoid using the presets -7 ... -9 when there's no real need for them. At -6 and lower, the amount of
memory wasted is usually low enough to not matter.


In my case the -9 setting should be okay as my file is way larger that the dictionary (check bolds). I just wonder though if these compression schemes and tar were made for so large files
Alex
 
Old 02-28-2013, 11:45 AM   #5
designator
Member
 
Registered: Jun 2003
Location: California, USA
Distribution: OpenSUSE Tumbleweed
Posts: 219

Rep: Reputation: 37
If you have enough RAM and a fast processor, lzma2 is great. You can also play with dictionary size to get even better ratios especially since you say your files have a lot of repetitive sequences.

The easiest way to find out is to try both. In my experience with everyday files you want to compress (backups,) the difference between bzip2 and lzma2 is minimal and rarely justifies the extra time and memory lzma2 requires.
 
Old 02-28-2013, 03:52 PM   #6
jefro
Moderator
 
Registered: Mar 2008
Posts: 20,435

Rep: Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263
It would depend on data to be compressed as inferred by designator. My guess is that most people could depend on LZMA to provide better results in general data if newish system.

It is quite easy to use older tools and provide backward support for older systems. That may be a reason to use older means.

Also consider the ability of your apps to use smp. The newer tools tend to use multiple cpu's much better.

Last edited by jefro; 02-28-2013 at 03:58 PM.
 
Old 03-01-2013, 03:07 AM   #7
H_TeXMeX_H
LQ Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292
Wow, 1.2 TB ... that will probably take a long time to compress (days). Maybe try gzip first, because it is much faster.
 
Old 03-05-2013, 06:22 AM   #8
alaios
Senior Member
 
Registered: Jan 2003
Location: Aachen
Distribution: Opensuse 11.2 (nice and steady)
Posts: 2,203

Original Poster
Rep: Reputation: 45
Hi I have did with gzip -9 as it finishes in only 4 days...
 
Old 03-05-2013, 05:56 PM   #9
jefro
Moderator
 
Registered: Mar 2008
Posts: 20,435

Rep: Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263
Ouch! "gzip -9 as it finishes in only 4 days..."
 
1 members found this post helpful.
Old 03-06-2013, 02:51 AM   #10
alaios
Senior Member
 
Registered: Jan 2003
Location: Aachen
Distribution: Opensuse 11.2 (nice and steady)
Posts: 2,203

Original Poster
Rep: Reputation: 45
Indeed I had the xz also running at the first four days. It had only written 44GigaBytes and as the source was 1.2T I though this will takes months to finish
 
Old 03-06-2013, 08:09 AM   #11
Mr. Alex
Senior Member
 
Registered: May 2010
Distribution: No more Linux. Done with it.
Posts: 1,238

Rep: Reputation: Disabled
LZMA is definitely better than bzip2. But it's also a lot slower.

Actually, there are barely any tasks that require compression these days. Algorithms like gzip/bzip2/lzma are designed to compress text, not binary data. So it's good for, say, a lot of source code (Linux kernel). But when you try to compress images, videos, other binary stuff, you just waste your time because processing several GB of that data will take you a lot of time and you will only save maybe a couple of megabytes of space... And then you'll transfer it through Internet with speed like 3 MB/s. Thus it will save you about 1 second of transferring and take several hours to compress and then decompress.
 
Old 03-06-2013, 08:47 AM   #12
H_TeXMeX_H
LQ Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292Reputation: 1292
This benchmark is relevant:
http://stephane.lesimple.fr/blog/201...-reloaded.html
 
Old 03-06-2013, 04:46 PM   #13
jefro
Moderator
 
Registered: Mar 2008
Posts: 20,435

Rep: Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263
Those kind of numbers should make you look at using way lower compression settings. Seems your system is pretty much low on resources during all of this. All the compression tools rely on processor(s). Newer tools might rely on smp above 4 cores.

I use some very very (very) old computers to back up a qnx system. Even at 1.5G it gets done in 2 or 3 hours.

Last edited by jefro; 03-06-2013 at 04:48 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Release 4.8: Features, Stability, Performance LXer Syndicated Linux News 0 01-26-2012 03:11 AM
Quiz: Why did gzip, bzip2 and lzma fail on this piece of data? What is it? irmin General 7 07-20-2010 03:06 PM
Xorg performance / KDE stability work-around on Slackware64 13.0 GazL Slackware 0 09-08-2009 06:38 AM
LXer: KDE 4 rev 790000: Better stability and performance LXer Syndicated Linux News 0 03-30-2008 06:40 PM
Cinelerra - Performance, Picture Quality, and Stability Sir Isaac Newton Linux - Newbie 2 12-11-2006 04:52 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 08:32 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration