LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-14-2009, 01:57 PM   #1
swalker23
Member
 
Registered: Aug 2008
Location: Virginia, USA
Distribution: Kubuntu 10.04 x64
Posts: 85

Rep: Reputation: 15
Question What is the best compression method for huge folders?


I need to format my backup storage HDD, it contains my library of anime, family movies, and other lil stuff I keep on it. The HDD is 1TB in size and I'm using roughly 700 gb. I need to compress some folders so I can store it on my linux HDD(640gb) while it is being formated. With linux I can use many of the compression methods but I don't know which one will give me the best results for huge folders. So which method would be the best for me?
 
Old 04-14-2009, 02:13 PM   #2
rweaver
Senior Member
 
Registered: Dec 2008
Location: Louisville, OH
Distribution: Debian, CentOS, Slackware, RHEL, Gentoo
Posts: 1,833

Rep: Reputation: 164Reputation: 164
Quote:
Originally Posted by swalker23 View Post
I need to format my backup storage HDD, it contains my library of anime, family movies, and other lil stuff I keep on it. The HDD is 1TB in size and I'm using roughly 700 gb. I need to compress some folders so I can store it on my linux HDD(640gb) while it is being formated. With linux I can use many of the compression methods but I don't know which one will give me the best results for huge folders. So which method would be the best for me?
It really depends, but if you're talking raw compression ratio bzip2 offers a high level of compression and works relatively fast. Are you going to be taring the directories up? Do you need to access individual files in the tarballs? If both answered yes then I would change my answer from bzip2 to perhaps zip running at highest compression ratio.

But for raw compression ratio, bzip2 is pretty good.
 
Old 04-14-2009, 02:15 PM   #3
linus72
LQ Guru
 
Registered: Jan 2009
Location: Gordonsville-AKA Mayberry-Virginia
Distribution: PocketWriter/MinimalX
Posts: 5,087

Rep: Reputation: 337Reputation: 337Reputation: 337Reputation: 337
Don't know if it helps but you could also compress it like a KNOPPIX image-my DSL goes from 200+MB to 48MB after compressing-is that comparable to bz2?
 
Old 04-14-2009, 02:18 PM   #4
swalker23
Member
 
Registered: Aug 2008
Location: Virginia, USA
Distribution: Kubuntu 10.04 x64
Posts: 85

Original Poster
Rep: Reputation: 15
I'm not going to be accessing the files through the compression. Just gonna compress them, move them to my linux HDD, format backup HDD, and move/uncompress them back to the backup HDD. Thanks for the reply I'll use the bzip2 method and will tar up the folder since I have directories in them.
 
Old 04-14-2009, 02:18 PM   #5
eco
Member
 
Registered: May 2006
Location: BE
Distribution: Debian/Gentoo
Posts: 412

Rep: Reputation: 48
well, if it's holding movies and pictures, you might get very minimal compression ratio as these are already compressed formats.

Still, have a look at bz2 or rar

Code:
$ tar jcvf myLargeZipFile.tar.bz2 /mnt/mydisk
or something like
Code:
$ rar a -m5 -v2048000b myLargeZipFile.rar /mnt/mydisk
-v will split the rar files to avoid one large file.
Check the syntax though
 
Old 04-14-2009, 02:28 PM   #6
taxtropel
Member
 
Registered: Mar 2005
Location: Cascade Mountains WA USA
Distribution: Linux From Scratch (LFS)
Posts: 149

Rep: Reputation: 16
Unless you have a lot of text files (very few binaries) then you are not going to fit 640GB + 700GB (compressed) on to a 1TB drive. Your best bet is to back up your files (/home) and your configuration files (/etc) and anything else you cannot replace on your linux drive. Then reinstall linux after the format.

That being said. Don't take my word for it; try some archiving programs out.

Basic archiving programs commonly used are:
  • Tar
  • Cpio
The 4 main compression types are:
  • Zip (plain'ole Zip files)
  • GZip
  • XZ-Utils (lzma)
  • BZip2

If you want to compress a folder with tar and bzip2 then use:
Quote:
tar -cfj my_archive_name.tar.bz2 /folder
 
Old 04-14-2009, 02:34 PM   #7
swalker23
Member
 
Registered: Aug 2008
Location: Virginia, USA
Distribution: Kubuntu 10.04 x64
Posts: 85

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by taxtropel View Post
Unless you have a lot of text files (very few binaries) then you are not going to fit 640GB + 700GB (compressed) on to a 1TB drive. Your best bet is to back up your files (/home) and your configuration files (/etc) and anything else you cannot replace on your linux drive. Then reinstall linux after the format.

That being said. Don't take my word for it; try some archiving programs out.

Basic archiving programs commonly used are:
  • Tar
  • Cpio
The 4 main compression types are:
  • Zip (plain'ole Zip files)
  • GZip
  • XZ-Utils (lzma)
  • BZip2

If you want to compress a folder with tar and bzip2 then use:
You must have miss understood me. My backup HDD is 1tb and I'm using 700 gb of that. I want to compress the 700gb so it will fit on my 640gb HDD which linux is installed on so I can format the 1tb HDD.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Method Not Allowed: The requested method POST is not allowed for the URL /writedhcp.p WiWa Linux - Networking 15 01-06-2011 02:20 PM
Huge amount of "PHP Notice:" -> huge logs Braynid Linux - Software 4 03-18-2008 07:35 AM
What compression method? pilatus666 Mandriva 4 10-11-2007 07:04 AM
Move huge amount of folders with bash alexweb Linux - Newbie 4 10-27-2006 05:06 AM
Huge Huge Problem With Forums!!! The_Insomniac Linux - General 1 06-07-2004 09:15 AM


All times are GMT -5. The time now is 06:34 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration