LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 05-25-2006, 11:13 AM   #1
xushi
Senior Member
 
Registered: Jun 2003
Location: UK
Distribution: Gentoo
Posts: 1,288

Rep: Reputation: 45
Lightbulb rar or tar, or both for my situation?


Hi all,

I've written this in programming section coz.. well.. i like you all I hope it's the right section..

I have a folder with ~160k files in ~100k subfolders, totalling ~15GB, of which i'd like to backup onto DVDs. My first approach was to rar it all up, cutting the rar files every 100 or 200MB. This is the command i used
Code:
rar a -o+ -r -m5 -mm -v102400k compressed_file.rar folder/
(it has alot of images in it, and hence the optimisations above).

However, i recently realised that rar has a limit of 259 symbols for its pathname, and my paths stretch to over 280. So I need to either switch to tar completly, or use both.. What advice do you recommend? I also need to be able to uncompress it on (sigh...) windows too.

Can tar divide files?
Does tar have compression optimisations for images, etc.. ?
Finally, does tar have any recovery data? Like rar's -rr[N] or -rv[N], creates data about the overall compressed archive (making it slightly bigger in the end), which can be used to recover parts of it incase it gets corrupted.. I'll be burning this to DVDs so that's important..

So.. Should I tar it all into one big tar file, then use rar on that to divide it? Or are there better ways? And can i get any advice on the parameters I need to use?

Many Thanks.
 
Old 05-25-2006, 11:53 AM   #2
marozsas
Senior Member
 
Registered: Dec 2005
Location: Campinas/SP - Brazil
Distribution: SuSE, RHEL, Fedora, Ubuntu
Posts: 1,444
Blog Entries: 1

Rep: Reputation: 66
Well, I will take a different approach.

Considering media is cheap, and images does not compress well, why not to put images in DVD as is ? It is easy to browse then and selecting a few without uncompress the whole thing again. And is is cross plataform too ! Even more: If you create a vcd disk, you can play the disk in a standard DVD player to watch in your TV.

The hard part is to select which images should be moved to dvd, to maximaze the space. For this, I use a small file fitter program, named gaffiter (http://freshmeat.net/projects/gaffitter/) which selects the best set of images which will fit in my media.

think about, and sorry to not answer what you asked for, but I think this alternative is really cool.
 
Old 05-25-2006, 02:19 PM   #3
xushi
Senior Member
 
Registered: Jun 2003
Location: UK
Distribution: Gentoo
Posts: 1,288

Original Poster
Rep: Reputation: 45
Thanks for the reply. Ofcourse if i could burn it as is, i wouldn't mind.. it would save SO much trouble.. However, the problem is that the file/folder name length is way over the limit of Joilet. Joilet's Level 3 afaik is around ~120 characters. Even Level 4 (which is hard to find in linux, but can find in windows) is around 256 characters. But what i have is over 280 characters long.

Even when i compress it in rar, in linux that uncompresses fine, but in windows the uncompressor complains that it can't uncompress many of the files..
 
Old 05-25-2006, 03:02 PM   #4
marozsas
Senior Member
 
Registered: Dec 2005
Location: Campinas/SP - Brazil
Distribution: SuSE, RHEL, Fedora, Ubuntu
Posts: 1,444
Blog Entries: 1

Rep: Reputation: 66
Quote:
Originally Posted by xushi
However, the problem is that the file/folder name length is way over the limit of Joilet
I see...

Have you considered to truncate those long file names ? May be using some program to do that in a automatic way ? There is a few file name sanitizer out of there; pick one.
 
Old 05-25-2006, 03:09 PM   #5
xushi
Senior Member
 
Registered: Jun 2003
Location: UK
Distribution: Gentoo
Posts: 1,288

Original Poster
Rep: Reputation: 45
Hmm, would that cause any problems for a windows user when uncompressing ?

As far as i know, doesn't tar not have a limit for the length? (or atleast, it's over 300). And afaik windows' unzip or even winrar uncompresses tar files easely.
 
Old 05-25-2006, 06:36 PM   #6
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 7.7 (?), Centos 8.1
Posts: 17,767

Rep: Reputation: 2535Reputation: 2535Reputation: 2535Reputation: 2535Reputation: 2535Reputation: 2535Reputation: 2535Reputation: 2535Reputation: 2535Reputation: 2535Reputation: 2535
You could rename (ie mv) a few of the dir names to reduce the total length, or (better) script it to descend into a lower level and only do files below that level, so you end up with more backups with shorter names...
 
Old 05-25-2006, 08:43 PM   #7
aluser
Member
 
Registered: Mar 2004
Location: Massachusetts
Distribution: Debian
Posts: 557

Rep: Reputation: 43
It looks like tar does "multivolume archives" with the M and L options. I haven't tried it; check out the manual: http://www.gnu.org/software/tar/manu...47.html#SEC147
 
Old 05-26-2006, 02:03 AM   #8
xushi
Senior Member
 
Registered: Jun 2003
Location: UK
Distribution: Gentoo
Posts: 1,288

Original Poster
Rep: Reputation: 45
Quote:
Originally Posted by chrism01
You could rename (ie mv) a few of the dir names to reduce the total length, or (better) script it to descend into a lower level and only do files below that level, so you end up with more backups with shorter names...
Thanks, hmm, well it would be very difficult of me to modify the tree. I need the least editing possible, so that when extracted, it's immediately ready for use.

Quote:
It looks like tar does "multivolume archives" with the M and L options. I haven't tried it; check out the manual: http://www.gnu.org/software/tar/manu...47.html#SEC147
Not bad. But i see a few problems with this.

* Each volume will be ~4.4Gb (DVD), which is risky to have it that large, especially when i've yet to find recovery features in tar.

* I'm not sure how successful uncompressing in windows will be.

I think the best option so far is to leave recovery and multivolumes, as well as compression to rar, while putting the whole lot into a single container using tar. All i'm missing is any correct parameters or extra optimisations (for tar). Something like,

Code:
tar cvf compressed_file.tar folder/ 
rar a -o+ -r -m5 -rr[524288] -v204800k final.rar compressed_file.tar
What do you think? Btw, thanks for all the help so far!

edit: some info for rar
a = archive
-o+ = overwrite files
-r = recursive
-m5 = most compression (from 0 to 5)
-rr[N] = recovery containing up to 524288 recovery sectors
-v204800k = each volume is 200Mb

Last edited by xushi; 01-30-2009 at 06:19 PM.
 
Old 05-26-2006, 06:58 AM   #9
aluser
Member
 
Registered: Mar 2004
Location: Massachusetts
Distribution: Debian
Posts: 557

Rep: Reputation: 43
The -L option lets you give a "tape length" which I believe is the file size of the volumes. you would probably need gnu tar on the windows system though. you could do it with cygwin perhaps
 
Old 05-26-2006, 07:00 AM   #10
xushi
Senior Member
 
Registered: Jun 2003
Location: UK
Distribution: Gentoo
Posts: 1,288

Original Poster
Rep: Reputation: 45
Quote:
Originally Posted by aluser
The -L option lets you give a "tape length" which I believe is the file size of the volumes. you would probably need gnu tar on the windows system though. you could do it with cygwin perhaps
Hehe.. windows is the only thing that's making my life hard atm.. You don't think winrar from rarlabs can extract tar files with a -L parameter ?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
how can i decompress this tar.tar file? hmmm sounds new.. tar.tar.. help ;) kublador Linux - Software 14 10-25-2016 02:48 AM
tar tar cvf - . | (cd /root/; tar xvf -) ewt3y Linux - General 10 02-19-2014 10:55 AM
how to deal with .rar files in linux now that rar is no longer part of linux Lleb_KCir Linux - Software 14 11-22-2008 07:01 AM
extracting .rar with part01.rar? chime Linux - General 5 04-11-2006 06:59 PM
Diferance between rpm, tar, tar.gz, scr.tar, etc mobassir Linux - General 12 08-21-2003 06:30 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 11:24 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration