tar with parameter -T (or --files-from) does not compress?

Babelduo · 01-18-2011, 08:11 AM

Hi,

i have a root-server (debian 64bit lenny) with a few domains running on it. every sunday i run a full backup of all domain-files on the hard-disk.

but why is the "compressed" tar.bz2 file bigger than its content?

the compressed tar.bz2 file from my backup-script:

Code:

3.8G Jan 16 04:12 backup-20110116030001-full.tar.bz2

the extracted directory:

Code:

# du -sh backup-20110116030001-full
2.8G    backup-20110116030001-full

now i create a new tar.bz2 and archive the extracted folder again in a new archive. it is the same folder extracted from the 3.8G archive, but now its only 1.9GB!:

Code:

# tar -cjpf backup-20110116030001-full_NEW.tar.bz2 backup-20110116030001-full
1.9G Jan 18 10:30 backup-20110116030001-full_NEW.tar.bz2

the only difference is that my backup-script uses the -T (or --files-from) parameter to read all files from a list. so first i "find -mtime" all the files and then I create the archive.

does the parameter -T not compress? is there a alternative solution?

thx!
kim

jschiwal · 01-18-2011, 08:26 AM

The -T option won't effect compression. Are you certain that you don't have duplicates listed in in the filelist? If you do, they will be backed up twice.

Another possibility is that you are backing up the targets of links, instead of the links themselves.

colucix · 01-18-2011, 08:38 AM

To me there are duplicates in the file list. The find command will list the directory names and the files inside them one per line, e.g.

Code:

/path/to/dir
/path/to/dir/file1
/path/to/dir/file2
...

the tar command will archive the directory recursively, then the files inside it. Think at the tar archive as a stream of files: tar will archive every thing from the list without worrying about what has been archived previously (indeed the tar command was developed for tape archives and files were archived sequentially and the tape was never rewound in the process). When extracting the archive with duplicates, they are simply overwritten and the extracted directory tree preserves its original size.

To avoid this behaviour use the --no-recursion option of tar. In this way directories will be simply created (referenced) in the archive and not descended recursively.

Babelduo · 01-18-2011, 09:23 AM

thx for your responses! i think this could be the problem. i now use the following command after i created my whitelist/blacklist:

Code:

cat whitelist.lst | sort | uniq > whitelist.lst_uniq

the new "uniq" whitelist is about half the size of the original file (what matches excatly with the sizes i wrote in my first post: 1.9GB and 3.8GB)

i will try it out, i think it should work. but i'm wondering a little bit because i use "file -type f" to only find files and no directories. maybe i should look deeper into my backup-script to find the source of redundance!

thx

babel