7zip not performing so good

Slick666 · 11-04-2008, 06:35 AM

What I'm trying to do is compress a 1GB image of an SD card. I've DD'ed the whole device and want to a file.

The Image is 978MBs in size. The zip (not 7zip but just plain old zip) compresses it down to only 683 MBs. When I used 7zip I didn't get much better 675MBs.

The reason I think that it can do much better is that the disk image has three partitions

Code:

 - Part1 -  2.3MBs used
 - Part2 - 63.4MBs used
 - Part3 -  0.0MBs used
 - Total - 65.7MBs

If I took just the data files out I would get a smaller size. I want to dd so I can easily and simply pull and push the data but I don't want to waste 609Mbs on empty space.

What could I be doing wrong?

H_TeXMeX_H · 11-04-2008, 07:00 AM

Compressing raw data using standard compression algorithms is not going to give good results. You're right, you'll get better compression if you compress the files themselves not the dd image of the drive.

Slick666 · 11-04-2008, 08:17 AM

It would be nice to keep the img format so I could dd the image right back to the device. Are there any algorithms that are specifically tailored for this operation?

estabroo · 11-04-2008, 09:11 AM

what you need to do is zero out all your unused sectors, this will allow it to compress much better. If it's ext2 or ext3 you can use a program call zerofree.

jiml8 · 11-04-2008, 09:22 AM

regardless of filesystem, you can use dd to zero out the unused sectors.

dd if=/dev/zero of=/mytargetpartition/myzerofile

This will run until the file myzerofile fills all the free space in the partition, at which time it will bomb dd.

Then you just rm the file myzerofile and you've zeroed all the blank space in your partition.

comm2k · 11-04-2008, 09:53 AM

Also if the content is already heavily compressed (MPEG audio/video for ex.) then there is little point in further trying to reduce its size by zip/7zip compression.

Slick666 · 11-04-2008, 09:59 AM

Quote:

Also if the content is already heavily compressed (MPEG audio/video for ex.) then there is little point in further trying to reduce its size by zip/7zip compression.

True, but the total spaced used by the files is only about 70MBs. I'm going to try

Quote:

dd if=/dev/zero of=/mytargetpartition/myzerofile

and see what I get. Here's the results so far.

Code:

LZMA     - 642M
Bzip2    - 671M
gZip     - 683M
Zip      - 683M
LZOP     - 704M
Original - 978M

Slick666 · 11-04-2008, 10:50 AM

Thanks Guys you Rock!

I implemented the suggestion by jiml8 and gziped compressed the file down to only 1.1MB! Thats 98% compression on the actual data and 99.888% compression total.

Thanks again to everyone LQ forums is awsome

Correction: My initial DD command wasn't quite correct because I on;y grabbed one partition. but even then it was reduced to 28 Megs.

estabroo · 11-04-2008, 11:17 AM

Just ignore this.