LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices



Reply
 
Search this Thread
Old 09-24-2009, 08:26 AM   #1
Thymox
Senior Member
 
Registered: Apr 2001
Location: Cornwall, England.
Distribution: Debian + Ubuntu
Posts: 4,345

Rep: Reputation: 57
Help creating multivolume tar archive


OK, so I'm still struggling to create a nice, clean backup script. Previous LQ's here and here.

So far, I have this:
Code:
  tar \
    --files-from=$strINCLUDE \
    --exclude-from=$strEXCLUDE \
    --recursion \
    --create \
    --file $strBACKUPLOCATION/backup \
    --multi-volume \
    --tape-length $strSIZE \
    --format=gnu \
    --new-volume-script=$strBACKUPLOCATION/post_tar_script.sh
In one of the other threads it was discovered that the GNU format the multi-volume header (still) cannot store filenames greater than 100 chars in length - which is causing big problems, but the POSIX format (which apparently does nicely support very long filenames) seems to break everything complaining of incompatible features.

So, what features from the above tar command are incompatible with the POSIX tar spec? I can't really afford to remove any of them unless there is a suitable replacement - I need to be able to specify what files to include and exclude, I need it to rucurse, I need to be able to split the tar file and I need to run some action on the split tar files (in this case, FTP them somewhere else).

Incidentally, creating a single tar file and subsequently splitting it is not an option. Splitting the tar file in-situ is not an option unless I can make it FTP those split files to another location whilst the backup is still occurring - we have ~60GB to backup and only ~40GB of space to backup into... hence the current script running "post_tar_script.sh" (which FTPs the just-completed part of the tar archive to our backup server) at every $strSIZE bytes.
 
Old 09-24-2009, 08:33 PM   #2
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.6, Centos 5.10
Posts: 16,324

Rep: Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041
I wouldn't create a tar file that big in the first place. Anything goes wrong with the file and you lose the entire backup.
Can't you split it (ie multiple tar cmds/files) logically eg by dir?
Also, does the -z (gzip option) help?
 
Old 09-25-2009, 05:10 AM   #3
Thymox
Senior Member
 
Registered: Apr 2001
Location: Cornwall, England.
Distribution: Debian + Ubuntu
Posts: 4,345

Original Poster
Rep: Reputation: 57
With the "multi-volume" thing, each individual file is a complete tar file in its own right - it's not necessary to concatenate all the files together to do an extract. Any files that are bigger than the span-size ($strSIZE) get put across multiple files, but they're quite rare in this case.

Unfortunately, it would seem that g/bzipping them with the tar file (ie -z or -j) doesn't work with multivolume. As it happens, part of the post_tar_script.sh we bzip the individual file prior to FTPing it off anyway.

The directories being included are (complete with current sizes):
/home - 3.3GB
/var - 84 GB
/root - 129MB
/usr/local - 794MB
I don't want to have to split the backups into this-part-of-var and that-part-of-var, etc, etc. It will only make it more complex to retrieve data from should we need to.
 
Old 09-25-2009, 08:52 AM   #4
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.6, Centos 5.10
Posts: 16,324

Rep: Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041
Given you have more data than space to create a local backup in (as I understand you), why not think laterally and just rsync it to the backup server instead?
Unless you are willing to break up the /var backup, you are going to run out of disk space according to
Quote:
and only ~40GB of space to backup into..
anyway.
 
Old 09-25-2009, 10:44 AM   #5
Thymox
Senior Member
 
Registered: Apr 2001
Location: Cornwall, England.
Distribution: Debian + Ubuntu
Posts: 4,345

Original Poster
Rep: Reputation: 57
Been there, done that. The destination computer in question is only accessible to us via FTP. It's a "1&1 Hosting thing" - and no, there's no option of changing provider!
 
Old 09-28-2009, 04:25 AM   #6
zhjim
Senior Member
 
Registered: Oct 2004
Distribution: Debian Squeeze x86_64
Posts: 1,467
Blog Entries: 11

Rep: Reputation: 184Reputation: 184
Also this does not solve the tar issue but easies the space problem. Mount the ftp directory and get things going there.
Some nice thing for this is http://ftpfs.sourceforge.net/

I also found some howtos in german to set things up. Do a quick "mount ftp" in google.
 
Old 09-29-2009, 01:53 AM   #7
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.6, Centos 5.10
Posts: 16,324

Rep: Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041
Actually, I think dar with -c, -s, -p options should do it.
You can create slices and tell dar to pause in between slices. Primarily designed for spreading large backups over eg multiple Cds, but no reason in principle you can't ftp each slice as it pauses.
It can do compression as well.
 
Old 09-29-2009, 03:07 AM   #8
jschiwal
Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 655Reputation: 655Reputation: 655Reputation: 655Reputation: 655Reputation: 655
Look in Section 4.6 of the tar info manual.

Quote:
tar -C sourcedir -cf - . | tar -C targetdir -xf -
Code:
  tar \
    --files-from=$strINCLUDE \
    --exclude-from=$strEXCLUDE \
    --recursion \
    --create \
    --file - \
    --format=gnu                    | ssh user@host cat >$str
would stream the entire archive to the storage server instead of breaking it up to a multi-volume set. This won't create a temporary tar file locally to be moved to the remote server.

AFAIK, this won't make it easier to backup, but it might be easier to restore. So maybe using the same backup script but using ssh to restore through a pipe could be what you are looking for. I have restored from a single backup this way, but not a multi-volume backup. I don't know if you can cat the volumes of a multi-volume tar backup ( at the storage server ) and use tar to restore files on the local side. I think it will work.

Using pipes this way may be more flexible if you use cpio instead of tar. I haven't tried it using dar.

You would need to use pubkey authentication. Locally, the script needs to be run as a backup user or root, but on the storage server, a normal user account would do.

I would also recommend using the -g option with tar for incremental backups. This will reduce the size of backups in between weekly/bi-weekly/monthly backups.

My favorite that I have played with is something like:
tar -C / -g podcasts/.snar -cf - <directory list> | tee /mnt/ndas/backupfile.tar | ssh -C / -xvf - >backuplog
I used this one-liner to replicate new podcasts from one computer to another, while simultaneously creating an incremental backup on a mounted NAS share. This is from memory. IIRC I also used the -v option and
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Creating multivolume archives using tar sharathksin Programming 6 03-06-2006 06:00 AM
Creating a spanning archive with tar Phyrexicaid Linux - Software 4 01-27-2005 09:04 PM
Problem Creating Archive With Tar In Redhat!!! hamidrh Linux - Newbie 4 09-16-2004 01:37 PM
Tar gives error when creating a tar file archive davidas Linux - Newbie 10 04-13-2004 01:35 AM
Problem with properly creating a tar archive FransE Linux - Software 5 02-28-2004 04:15 PM


All times are GMT -5. The time now is 03:56 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration