LinuxQuestions.org
Help answer threads with 0 replies.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 10-02-2013, 01:23 PM   #1
devUnix
Member
 
Registered: Oct 2010
Posts: 606

Rep: Reputation: 59
tar and deleting source files side by side


Hi,

I have to use tar with the --remove-files option because the partition on which the files to be archived are found may run short if archives are placed and source files are not removed side by side.

The director of the project has raised a question:

Will the files be lost for good if tar fails for whatever reason?

What do you think? I did a test and found that tar fials with error code 2 if diska space is not sufficient but the archive is available for use.If I use --remove-files option then can I still make sure that archive will be there and files not backed up will not be removed either?
 
Old 10-02-2013, 01:46 PM   #2
colucix
LQ Guru
 
Registered: Sep 2003
Location: Bologna
Distribution: CentOS 6.5 OpenSuSE 12.3
Posts: 10,509

Rep: Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983Reputation: 1983
The tar manual explains that files are removed after adding them to the archive therefore they should be preserved if not archived. Anyway, you can avoid problems if you remove them explicitly after the tar command, maybe by checking the actual content of the archive and remove the files only if they appear inside the archive. Just my
 
Old 10-02-2013, 04:37 PM   #3
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
...additionally you could tar files in stages. For example in alphabet order:
Code:
echo {a..z}|while read CHAR; do
 tar --remove-files -cjf /path/to/archive_${CHAR}.tbz2 `find /search/path -type f -iname "${CHAR}*"`
done
...but obviously the best way would be to not run out of disk space in the first place or at least use a disk that has enough free space. You could rsync / FTP / NFS mount and copy the tree to another server, then delete the tree on the source machine and then archive on the destination machine you know...
 
Old 10-03-2013, 03:37 AM   #4
devUnix
Member
 
Registered: Oct 2010
Posts: 606

Original Poster
Rep: Reputation: 59
Hi,


This is what I am actually doing:

Code:
tar -vzcf "/var/app-archive/app-docs-`date +%Y%m%d`.tar.gz" --remove-files `find /var/app-docs/ -type f -mtime +90 -print`

The files returned by the find command are of 1 GB in total and the free disk space is 0.75 GB hence I included the --remove-files options. I have not used that option for testing purpose on the live server.


colucix:- You have invalidated my belief that tar would add a file and delete it side by side and then add another and delete it and so will it go on with the rest of the files returned by the find command so as not to run out of the disk space. I was wrong.


Yup, the man page of tar clearly says:

Quote:
--remove-files
remove files after adding them to the archive

To avoid the problem of running out of disk space: I had written this script earlier:

Code:
for DIRECTORY in app-docs app-other-docs; do
        find /var/${DIRECTORY}/ -type f -mtime +90 > /var/tmp/files-older-than-90-days.list
        if [ -s /var/tmp/files-older-than-90-days.list ]; then                          # Proceed only if the list is Not Empty
#
        while read FILE; do
                mkdir -vp "/var/tmp/${DIRECTORY}-Backup`dirname "$FILE"`"                 # Cutting the File Name Off. Creating Directory Hierarcy.
                mv -v "$FILE" "/var/tmp/${DIRECTORY}-Backup`dirname "$FILE"`/"            # Moving the File.
        done </var/tmp/files-older-than-90-days.list
#
        rm -f /var/tmp/files-older-than-90-days.list
#
        cd /var/tmp/${DIRECTORY}-Backup/
        tar -czvf /var/app-archive/${DIRECTORY}-`date +%Y%m%d`.tar.gz *
        rm -rf /var/tmp/${DIRECTORY}-Backup                                     # Removing the Direcotry.
        fi
done
It went fine and moved hundreds of files in the temporary location so the disk space remained the same and then tarred the temporary directory and removed it.

However, because of special symbols including white spaces and (), -, - and other symbols not known in advance the script broke after moving and tarring hundreds of files and then overwrote the tar file with only a couple of empty directories. All data were lost!

I quoted the FILE variable above so that mkdir would not fail if there were spaces or special symbols in the directory names. Still somehow the script broke and loop got re-executed for the same first directory: app-docs resulting in moving the original files and packing only directory names by re-writing the tar file and then deleting the original files.

I am loaded with escalations!

What should I do?

I saved the contents of the "find" command because the complete structure or hierarchy of every file is to be maintained.

for example:

/var/app/dir1/dir1a/file1

/var/app/dir2/dir2a/file1

moving the second file will overwrite the first file in the destination folder: archive

if the parent directories were not created.

I changed the logic and use the find command along with the tar command as posted earlier.

I think I have explained my problem here.
 
Old 10-05-2013, 02:15 AM   #5
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by devUnix View Post
What should I do?
- don't hold back details when making your original post,
- learn that scripts should have proper error handling,
- test scripts in a safe environment before having them validated,
- have your scripts validated by a senior before using them in a production environment,
- and finally: ensure backups exist.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Generate SPECIAL alphanumeric WORDLIST - no repeating characters side-by-side Stanley_212 Programming 33 10-13-2022 06:06 PM
Side-by-side install of multiple versions of packages from source into /opt deskt0plinux Linux - Newbie 2 09-07-2012 01:50 AM
Mounting multiple NFS or Samba shares: client-side or server-side? mariogiov Linux - Server 4 04-03-2012 08:11 AM
"catting" files in one file side by side, not among each other indyn00b Programming 5 05-10-2007 01:05 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 03:18 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration