What is the best tool for compressing and un-compressing files?

dwhswebhosting · 03-23-2017, 05:14 PM

If a server crashes, we have to tar up about 200 gigs or data. We then transfer it to a new server and untar.

I was just wondering if anyone happens to know if there is a better program to tar and untar? (compress and un-compress)

When compressing millions of files into one it can take hours if not days and with customer sites possibly down, the fastest method can make a huge difference.

Thank you,
Charles

hydrurga · 03-23-2017, 05:43 PM

Do you have multiple processors/cores? If so, have a look at pigz (http://zlib.net/pigz/).

See here for the developer's comments on the fact that compression is much faster but decompression isn't a great advance over single core. I guess you can't have everything.

http://stackoverflow.com/questions/1...-decompression

dwhswebhosting · 03-23-2017, 08:31 PM

Perfect thanks, it does have a ton of processors.

r3sistance · 03-24-2017, 04:11 AM

pigz would probably be fastest but pxz might be preferable as it uses lzma compression and is the multi-threaded version of xz. while pxz might not be quiet as fast as pbzip (bzip) or pigz (gzip), it has a better compression ratio which will help when transferring the compressed data. Also remember that most these compression utilities have configurable options for compression ratio vs. speed, so check the manpages too.

Turbocapitalist · 03-24-2017, 04:17 AM

zpaq might be worth investigating.

The different compression tools work best with different types and quantities of data, so you'll probably have to test a few to see which is best for your situation.

rtmistler · 03-24-2017, 06:43 AM

Tar is not a tool for compression, it originally was tape archiver and wasn't intended for compression. In fact, it was designed for exactly what you are doing, backing up entire hard drives to secondary storage media.

Many of the other tools which other members have mentioned are typically for compressing, and tar can usually make use of them based on extensions or command line arguments.

If you are backing up data, I recommend still using tar, but also specifying that you wish the contents to be compressed.

sundialsvcs · 03-24-2017, 09:21 AM

A so-called .tar.gz file is a .tar file that has been compressed using the gzip algorithm.

File transfer tools such as rsync and scp have options to apply gzip compression to the transfer, so that the data is compressed on-the-fly before being put onto the wire, and decompressed as it is being taken off.

Compression algorithms are basically a compromise between speed and file-size. In addition to gzip, "(PK)zip" and "bzip2" are in common use.

Turbocapitalist · 03-24-2017, 10:25 AM

Quote:

Originally Posted by sundialsvcs

A so-called .tar.gz file is a .tar file that has been compressed using the gzip algorithm..

Yes, the old way to compress was to pipe to gzip

Code:

tar cf - ./*.odt | gzip > ~/documents.tar.gz

cd /tmp/

gzip -cd ~/documents.tar.gz | tar xf -

The same can still be done for compression methods that are not build into but that can send or receive via stdout and stdin.

jefro · 03-24-2017, 03:30 PM

There have been a number or web pages that give the various types of compression programs a review as to how well they handle files. Knowing both your system and the type of file and size of file may allow you to select the best program. Remember too that many offer levels like .gz does. It's default I think is 5 but can range from about 1 to 9 for speed versus size.