LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 07-28-2009, 03:58 PM   #1
JDska55
LQ Newbie
 
Registered: Jun 2009
Location: Iowa City, IA
Distribution: SuSE 11, ubuntu Hardy
Posts: 28

Rep: Reputation: 15
Best compression method for >2GB files?


Hey all-
I'm trying to compress a huge (2.3 GB) .raw file down as much as possible so that when I download it later it doesn't take absolutely forever. I tried using bzip2 as it seems to be more effective (if a bit slower) than gzip, but it only knocked the size down to 2.1 GB. Do you guys know of any tools that might work better than bzip2 for this? Speed of compression isn't a huge deal, as my main hangup at the moment is downloading a bunch of these files at the end. Thanks in advance
 
Old 07-28-2009, 04:06 PM   #2
GrapefruiTgirl
LQ Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551
LZMA !!

That would be the thing. I don't have a link handy, but Google is your friend!

Sasha

PS - IIRC, lzma is a part of the xz tool kit, available from tukaani. Google either LZMA, tukaani, or xz compression tool. You'll find it.

Cheers!

Last edited by GrapefruiTgirl; 07-28-2009 at 04:07 PM.
 
Old 07-28-2009, 05:03 PM   #3
lazlow
Senior Member
 
Registered: Jan 2006
Posts: 4,362

Rep: Reputation: 172Reputation: 172
If bzip only brought it down .2GB then you probably are not going to see a huge drop(say below 1.7GB) no matter what you do.
 
Old 07-28-2009, 05:16 PM   #4
johnsfine
LQ Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,286

Rep: Reputation: 1190Reputation: 1190Reputation: 1190Reputation: 1190Reputation: 1190Reputation: 1190Reputation: 1190Reputation: 1190Reputation: 1190
Quote:
Originally Posted by JDska55 View Post
trying to compress a huge (2.3 GB) .raw file
What is a .raw file?

Maybe I missed something, but the use of that extension isn't consistent enough for me to know.

Is it some format of digital video? If you are compressing digital video, you want to use a compression method that knows it is compressing digital video.
 
Old 07-28-2009, 05:27 PM   #5
karamarisan
Member
 
Registered: Jul 2009
Location: Illinois, US
Distribution: Fedora 11
Posts: 374

Rep: Reputation: 55
Additionally, if that file is already pretty random - the most obvious way for this to happen is if the data is already compressed by something else - you're not going to see good gains with anything.
 
Old 07-28-2009, 06:21 PM   #6
GrapefruiTgirl
LQ Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551
Please allow me to clarify, per the context of the OQ:

Generally speaking, *IF* anything is going to give you better compression than you are getting with bz2 or gz, it would be LZMA, with the exception perhaps being wrt what johnsfine wrote about digital video. I say 'perhaps' because I personally don't know either way about digital video compression.

Do let us know what you use in the end, and/or/if you use something besides bz2/gz, how it turns out. I'm curious now!

Sasha
 
Old 07-29-2009, 03:35 PM   #7
JDska55
LQ Newbie
 
Registered: Jun 2009
Location: Iowa City, IA
Distribution: SuSE 11, ubuntu Hardy
Posts: 28

Original Poster
Rep: Reputation: 15
Well!
Thanks for the tips guys. For the record, the .raw file is a displacement field generated from comparing 2 sets of lung CT images. It's essentially a giant data file that our software overlays onto the images.

gzip did basically nothing, maybe a 100MB reduction. bzip2 was only a little better, reducing the size by about 200MB. LZMA though... WOW! It reduced the files all the way down to 1.45GB. I'm currently toying around with the settings, though, since at the default quality setting, LZMA took over 3 hours to run!

And apparently I spoke too soon. I just ran LZMA at a reduced quality setting of -4 instead of -7, and it only took 10 minutes. Huzzah!

I'll update this thread if I decide to head down another road.

Thanks all!

Last edited by JDska55; 07-29-2009 at 03:36 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
What is the best compression method for huge folders? swalker23 Linux - Newbie 6 04-14-2009 02:34 PM
ReiserFS 2Gb maximum file size limit? Cannot copy files bigger than 2Gb ihtus SUSE / openSUSE 2 10-26-2007 10:21 AM
What compression method? pilatus666 Mandriva 4 10-11-2007 07:04 AM
Samba and 2GB + files TheRealDeal Linux - Networking 7 08-22-2005 09:24 PM
apache > 2GB files RDove Linux - Software 1 03-11-2005 11:53 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 02:48 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration