I've been playing around with my backup script. It used to compress /home into a single large tar.bz2 file about 4G in size (tar -cjf /tmp/backups/home.tar.bz2 /home/* ).

I thought I'd try backing up individual directories instead. In the first cut, there are only two significant directories - one for shared folders and the other for samba profiles, etc. The Linux user homes are very small, since I'm the only one who uses a Linux shell. Most users don't have a Linux home directory.

The code for the archiving is:
cd /home
for dir in *; do
if [ -d $dir ]; then
tar -cjf /tmp/backups/$dir.tar.bz2 $dir
fi
done

On the test run, the two big directories compressed to more than 5G - about 30% larger than just having one archive and too large to fit on a DVD-RAM disk.

My question is why? My first guess would be that the users work on files on their local drives then copy the completed file to the network share. If bzip2 has all the files to deal with, it could be able to squash the near-duplicate drafts down to almost nothing, but when I split the share archive off from the profile archive, it can't do that.

This would be truly impressive since it would seem to require bzip2 to track patterns through the entire archive.

Is that what is happening? Or is there some other explanation for the increased size?

Most compression works on the concept of eliminating repeating patterns throughout the file in a consistent manner. Usually compressing more data together results in a better ratio because it finds more patterns to compress and you see less redundancy in the resulting file than if you compressed it into multiple archives.

Hello garydale

Was wondering whether you had tries squashfs, squashfs-lzma, or dir2lzm?

I usually use squashfs-lzma, but on big files it will take alot longer to compress vs squashfs/dir2lzm

Both bzip2 http://linux.die.net/man/1/bzip2 and gzip http://linux.die.net/man/1/gzip have options (values 1-9) to specify the level of compression. The higher/better the level, the longer it takes, but the smaller the output. Your choice.

It turns out that someone had actually increased the amount of space they were using rather dramatically. That's what led to the increased size - it was just a weird coincidence that it happened the same time I changed my script.

This leaves me with the problem of trying to fit the extra data into the DVD-RAM disk. I'm probably going to have to create an archive set and just back up the newer files from now on.

Thanks.

Found an easier solution. It turns out lzma is a lot better than bzip2 for compressing files. Switching to it saved about 20%, which gives me a comfortable margin for the next little while. The only changes I needed to make to my script were to change the "j" option to "a" (automatic) on the tar command line and rename the archive to end in .lzma.

Lzma is somewhat slower in compressing according to what I've read, but the better compression is worth it in my case. Also, it's apparently also faster in decompression, which should save me time when I need to recover files.

dir2lzm is about same as squashfs-lzma, I think

what version of squashfs-lzma are you using?

you can get latest squashfs-lzma and dir2lzm in any slax iso/tar.gz
in the slax/tools folder
that is squashfs-lzma 4.0

check this out too
http://www.squashfs-lzma.org/