Best compression method for >2GB files?
 

Hey all-
I'm trying to compress a huge (2.3 GB) .raw file down as much as possible so that when I download it later it doesn't take absolutely forever. I tried using bzip2 as it seems to be more effective (if a bit slower) than gzip, but it only knocked the size down to 2.1 GB. Do you guys know of any tools that might work better than bzip2 for this? Speed of compression isn't a huge deal, as my main hangup at the moment is downloading a bunch of these files at the end. Thanks in advance :D

LZMA !!

That would be the thing. I don't have a link handy, but Google is your friend!

Sasha

PS - IIRC, lzma is a part of the xz tool kit, available from tukaani. Google either LZMA, tukaani, or xz compression tool. You'll find it.

Cheers!

If bzip only brought it down .2GB then you probably are not going to see a huge drop(say below 1.7GB) no matter what you do.

What is a .raw file?

Maybe I missed something, but the use of that extension isn't consistent enough for me to know.

Is it some format of digital video? If you are compressing digital video, you want to use a compression method that knows it is compressing digital video.

Additionally, if that file is already pretty random - the most obvious way for this to happen is if the data is already compressed by something else - you're not going to see good gains with anything.

Please allow me to clarify, per the context of the OQ:

Generally speaking, *IF* anything is going to give you better compression than you are getting with bz2 or gz, it would be LZMA, with the exception perhaps being wrt what johnsfine wrote about digital video. I say 'perhaps' because I personally don't know either way about digital video compression.

Do let us know what you use in the end, and/or/if you use something besides bz2/gz, how it turns out. I'm curious now!

Sasha

Well!
Thanks for the tips guys. For the record, the .raw file is a displacement field generated from comparing 2 sets of lung CT images. It's essentially a giant data file that our software overlays onto the images.

gzip did basically nothing, maybe a 100MB reduction. bzip2 was only a little better, reducing the size by about 200MB. LZMA though... WOW! It reduced the files all the way down to 1.45GB. I'm currently toying around with the settings, though, since at the default quality setting, LZMA took over 3 hours to run!

And apparently I spoke too soon. I just ran LZMA at a reduced quality setting of -4 instead of -7, and it only took 10 minutes. Huzzah!

I'll update this thread if I decide to head down another road.

Thanks all!