LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 02-26-2007, 06:45 PM   #1
floog
Member
 
Registered: Oct 2004
Distribution: gentoo & slackware
Posts: 59

Rep: Reputation: 15
Bzip2 and File Size Limits


Hello Group,

I have been using the same bash script for archiving and backing up data for several years. I am getting concerned about possible data corruption in my .tar.bz2 backups because the size of them has grown immensely.

My bash script does the following:

1. makes a .tar.bz2 of each major data directory we have in the office.
2. then takes each .tar.bz2 and places it into an .iso file that can be mounted or burned to a DVD.

Each major directory holds a little over 5 gigs. of data.
When each is compressed into the .tar.bz2 file they are about 1.2-1.4 Gigs. in size.

I've done some googling on the subject and I'm not finding much empirical data about the limits of bzip2 files and potential data corruption or loss.

Does anyone have experiences with this issue?
And can you provide some good links for some worthwhile RTFM. :-)

Thank you for your time and insights.
 
Old 02-26-2007, 07:45 PM   #2
Matir
Moderator
 
Registered: Nov 2004
Location: San Jose, CA
Distribution: Ubuntu
Posts: 8,507

Rep: Reputation: 118Reputation: 118
I personally have not seen any data, but depending on what goes into the bzip2 file, you can get HUGE compression ratios. You can always test it... bunzip the compressed file and compare an md5sum.
 
Old 02-27-2007, 05:25 AM   #3
floog
Member
 
Registered: Oct 2004
Distribution: gentoo & slackware
Posts: 59

Original Poster
Rep: Reputation: 15
Thanks Matir.
Maybe bzip2 is so solid, perhaps there are no size limitations to what it can successfully compress/uncompress.
I just don't want to be the guy to find the limit, heh.
 
Old 02-27-2007, 10:14 AM   #4
Matir
Moderator
 
Registered: Nov 2004
Location: San Jose, CA
Distribution: Ubuntu
Posts: 8,507

Rep: Reputation: 118Reputation: 118
The only problem I could potentially see is if they use 32-bit file sizes in bzip2, but I find this highly unlikely. I've seen plenty of large files bz2 compressed and all seemed to have worked fine. You could always give it a test... dump some files together into one big file and bzip2 it while testing md5sums.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
partition size limits pedromdsantos Linux - General 7 07-01-2005 03:56 PM
Configure mailbox size limits email server...please help! locoiguana Linux - Software 7 05-11-2005 04:15 AM
Quotas and File Size Limits wenberg Slackware 1 12-02-2004 04:40 AM
HD size limits under slackware horndude Slackware 6 02-10-2004 07:34 PM
File Size limits ascii2k Linux - General 4 04-15-2002 10:25 AM


All times are GMT -5. The time now is 09:57 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration