LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-26-2014, 03:50 AM   #1
nix.noob
LQ Newbie
 
Registered: Aug 2013
Posts: 25

Rep: Reputation: Disabled
adding new files to .tgz without decompressing


I have 100+ large files, each of which are zipped in a directory.
Currently, I have a nightly bash script which unzips them all at once and them tars+gzips them into a single .tgz file.

However, in the middle of this process all the files are decompressed and using up disk space. I feel I can be more space-efficient if add each large zipped file into some cumulative .tgz without decompressing it, but I don't know how to do that.
Is there a good way to do this?
 
Old 04-26-2014, 04:48 AM   #2
eklavya
Member
 
Registered: Mar 2013
Posts: 619

Rep: Reputation: 136Reputation: 136
You can make a compress file of already compressed file and new files.
or try this
http://stackoverflow.com/questions/2...compressed-tar

For zip files, it has been added without extracting the file. The command is

# cd /to/the/folder/where/zip/files/is/present
My zip file's name is test.zip and I want to add file abc.txt in it
# zip test.zip abc.txt
Suppose I want to add a directory which has several directory and files, the directory name is fol1
# zip -r test.zip fol1
Suppose you want to add multiple directories
# zip -r test.zip fol1 fol2
 
1 members found this post helpful.
Old 04-26-2014, 06:47 AM   #3
nix.noob
LQ Newbie
 
Registered: Aug 2013
Posts: 25

Original Poster
Rep: Reputation: Disabled
Thanks very much for this.
I'm now using this approach:

Code:
for file in $folder
do
        gunzip $file
        filename=$(basename "$file")
        newfile=${filename%.gz}
        echo $newfile
        zip -mq9 z.zip $newfile
done
I found that the compression here with zip does not seem to be as good as it was with gzip.
And I read that bzip2 gives even better compression

Is it possible to achieve the same result with better compression with one of these other utilities or with another approach? As far as I can see, bzip2 (for instance) doesn't appear to allow adding new files into an archive
 
Old 04-26-2014, 08:17 AM   #4
eklavya
Member
 
Registered: Mar 2013
Posts: 619

Rep: Reputation: 136Reputation: 136
bzip2 compresses files using the Burrows-Wheeler block sorting text compression algorithm, and Huffman coding.bzip2 expects a list of file names to accompany the command-line flags.Each file is replaced by a compressed version of itself, with the name "original_name.bz2". bzip2 and bunzip2 will by default not overwrite existing files.If you want this to happen, specify the -f flag. If no file names are specified, bzip2 compresses from standard input to standard output.In this case, bzip2 will decline to write compressed output to a terminal, as this would be entirely incomprehensible and therefore pointless. bunzip2 will correctly decompress a file which is the concatenation one two or more compressed files. The result is the concatenation of the corresponding uncompressed files. Integrity testing (-t) of concatenated compressed files is also supported.

Gzip reduces the size of the named files using Lempel-Ziv coding (LZ77).Whenever possible, each file is replaced by one with the extension .gz, while keeping the same ownership modes, access and modification times.(The default extension is -gz for VMS, z for MSDOS, OS/2 FAT, Windows NT FAT and Atari.) If the compressed file name is too long for its file system, gzip truncates it. Gzip attempts to truncate only the parts of the file name longer than 3 characters.(A part is delimited by dots.) If the name consists of small parts only, the longest parts are truncated. For example, if file names are limited to 14 characters, gzip.msdos.exe is compressed to gzi.msd.exe.gz. Names are not truncated on systems which do not have a limit on file name length.
gunzip can currently decompress files created by gzip, zip, compress, compress -H or pack. The detection of the input format is automatic. When using the first two formats, gunzip checks a 32 bit CRC. For pack, gunzip checks the uncompressed length. The standard compress format was not designed to allow consistency checks. How‐ever gunzip is sometimes able to detect a bad .Z file. If you get an error when uncompressing a .Z file, do not assume that the .Z file is correct simply because the standard uncompress does not complain. This generally means that the standard uncompress does not check its input, and happily generates garbage output. The SCO compress -H format (lzh compression method) does not include a CRC but also allows some consistency checks.

Take a look at here.
https://blogs.reucon.com/srt/compres...vs-7-zip-8296/
 
Old 04-26-2014, 10:48 AM   #5
nix.noob
LQ Newbie
 
Registered: Aug 2013
Posts: 25

Original Poster
Rep: Reputation: Disabled
Thanks for trying to add more to this thread. I must admit though, I didnt follow the relevance of what you've added here, especially regarding truncation of filenames, which I think is at best somewhat tangential. I hope you pasted this from somewhere else rather than writing all that out by hand.

I actually found that the answer to my question was easier to come by than I had realised:
zip takes a -Z argument and allows a compression method to be passed.

So I can use
Code:
zip -Z bzip2 ..
and it will use bzip2 for compression.

After a quick test, I found that in my case, bzip2 is giving me slightly less compression than the default zip, and that both give less compression than my original tar+gzip approach. But this new approach does at least mean that I don't need to have all files uncompressed at once, so I'll stick with it until something better turns up.
 
  


Reply

Tags
bash, gzip, tar, zip


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Problem on decompressing files satimis Fedora 2 08-23-2006 11:33 AM
when .tgz files are downloaded they turn into .tar files?!?! Flz Slackware 10 12-20-2005 03:32 PM
.tgz files mohapi Linux - General 2 05-03-2003 11:11 AM
tgz files moger Slackware 10 10-05-2002 02:29 AM
tgz files Chijtska Linux - General 1 02-07-2002 02:44 PM


All times are GMT -5. The time now is 08:59 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration