LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 05-30-2011, 04:06 PM   #1
xeon123
Member
 
Registered: Sep 2006
Posts: 374

Rep: Reputation: 16
split huge file into small parts and compress them.


Hi,

I've a file with a size of 6GB. I would like to compress this file and split them into smaller files. I was also thinking in use bzip2 to compress it, because if offers a good compression rate. How can I split this file into small ones to compress it?
 
Old 05-30-2011, 04:07 PM   #2
TobiSGD
Moderator
 
Registered: Dec 2009
Location: Germany
Distribution: Whatever fits the task best
Posts: 17,148
Blog Entries: 2

Rep: Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886Reputation: 4886
You should have a look at the split command, exactly what you need.
 
Old 05-30-2011, 07:55 PM   #3
MTK358
LQ 5k Club
 
Registered: Sep 2009
Posts: 6,443
Blog Entries: 3

Rep: Reputation: 723Reputation: 723Reputation: 723Reputation: 723Reputation: 723Reputation: 723Reputation: 723
Quote:
Originally Posted by xeon123 View Post
I was also thinking in use bzip2 to compress it, because if offers a good compression rate.
xz offers an even better compression rate and very fast decompression, but note that compresion is very slow.
 
Old 06-01-2011, 08:11 AM   #4
peter1234
Member
 
Registered: Apr 2009
Posts: 42

Rep: Reputation: 2
Hi xeon123,

You said...... first

Quote:
“I would like to compress this file and split them into smaller files.”
then you said …...

Quote:
“How can I split this file into small ones to compress it?”
Do you want to compress first or split first? What are you trying to do?

Split the 6gb file to x number of files (file size is not that important to you, you just want x number of parts)
or
you want to split file to be a specific size.. (like 700mb if you want to put it in cds)
in that case you should compress first and split later.

How you compress (gzip, bzip2) I'll leave it up to you. This is normally what I do......

1)make a checksum (md5, shaxxx) of your uncompressed file.
2)compress the file.
3)make a checksum of your compressed file.
4)split the file (by file size or number parts).
5)copy it, email it, or what ever you want to do with it.....
6)join the file parts.
7)check to see if the joined file checksum match the checksum @ step 3.
8)decompress the joined file
9)check to see if the decompressed (joined file) checksum match the checksum @ step 1.

You can skip steps 1,3,7&9 if you are not that concerned about few error(s) that MIGHT creep-in during the whole process.

If you are really concerned about error(s) creeping in you should make a checksum after step 4 and 5
also.

Like “TobiSGD” said split command is quite handy for a jobs like this (man split will give you more help)

I assume you will want to join the file parts …... for that use cat command

lets say your compressed split-files are :
myfile.bz.part1
myfile.bz.part2
myfile.bz.part3
myfile.bz.part4

run
Code:
cat  myfile.bz.part1  myfile.bz.part2  myfile.bz.part3  myfile.bz.part4  >myfile.bz
this will read myfile.bz.part1 to myfile.bz.part4 (in that order) and output to a file called “myfile.bz”

now you can decompress myfile.bz file.

I have used the above method for transferring a 10gb file via 4.7gb DVDrw.
Just make sure that you assemble the parts in correct order (checksums help here).

Hope you find this helpful

Peter.


.
 
Old 06-01-2011, 11:52 PM   #5
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,348

Rep: Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749
In re compression efficiency, both gzip & bzip2 have flags to specify compression efficiency (note as per xz ref above, this will take longer the more compression you ask for).
 
Old 06-02-2011, 04:17 AM   #6
xeon123
Member
 
Registered: Sep 2006
Posts: 374

Original Poster
Rep: Reputation: 16
Thanks, peter1234 for the reply. I haven't noticed about the contradiction.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Scripting ?, determine file type, compress, split and email edpatterson Linux - Newbie 2 01-17-2009 10:40 AM
Scripting ?, determine file type, compress, split and email edpatterson Linux - Newbie 1 01-17-2009 10:24 AM
bash: how to split a text file in two unequal parts Humbro Programming 7 11-23-2007 07:15 AM
Compress and split a big sized file into smaller files hicham007 Programming 3 07-28-2005 08:56 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 02:27 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration