LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 08-29-2014, 02:52 AM   #1
_mz
Member
 
Registered: Jul 2013
Posts: 33

Rep: Reputation: Disabled
Bash script to archive files


Hi script experts,

# What to backup
/interfaces/data/trace/in/*
/interfaces/data/trace/out/*
/interfaces/data/test/in/*
/interfaces/data/test/out/*
/interfaces/data/error/in/*
/interfaces/data/error/out/*

# Where to backup to.
/interfaces/data/archive

I need to tar those files/folders in those directories that older than 30days with file name as /interfaces/data/archive/$dir1_$dir2_$date.tar.gz

where:
dir1 = 3rd directory like /trace, /test, /error;
dir2 = in or out;
date = 20140829 format

example of file: /interfaces/data/archive/trace_in_20140829.tar.gz

Thank you guys.
 
Old 08-29-2014, 03:32 AM   #2
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 9,439

Rep: Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770
and what is your problem? http://www.linuxquestions.org/linux/...Ask_a_Question
 
Old 08-29-2014, 03:56 AM   #3
_mz
Member
 
Registered: Jul 2013
Posts: 33

Original Poster
Rep: Reputation: Disabled
Quote:

I need to tar those files/folders in those directories that older than 30days with file name as /interfaces/data/archive/$dir1_$dir2_$date.tar.gz
This is my problem. How to get $dir, $dir so that I can create file like /interfaces/data/archive/$dir1_$dir2_$date.tar.gz
 
Old 08-29-2014, 04:25 AM   #4
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 9,439

Rep: Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770
can you post your script? You need to construct the target directory from your variables using parameter substitution.
 
Old 08-29-2014, 05:01 AM   #5
_mz
Member
 
Registered: Jul 2013
Posts: 33

Original Poster
Rep: Reputation: Disabled
Well, I am very very new to scripting and maybe I need to learn more about for loop, if statement and etc. I just started to write the script on my own and there are a lot of tries and errors. Here is my script.

--------
#!/bin/bash

# What to backup
src1="/interfaces/data/trace/in/"
src2="/interfaces/data/trace/out/"
src3="/interfaces/data/test/in/"
src4="/interfaces/data/test/out/"
src5="/interfaces/data/error/in/"
src6="/interfaces/data/error/out/"

# Where to backup to.
dest="/interfaces/data/archive"

# Retention period
max_days=30

# Create archive filename.
date=$(date '+%Y''%m''%d')
archive_file="$dir1_$dir2_$date.tar.gz"

# Log file
LOG_FILE="/interfaces/data/script/archive.$date.log"

# Print start status message.
echo "Backing up $src5 to $dest"
date
echo

# Backup the files using tar.
#cd $src5
#find . -mtime +$max_days | xargs tar cvzf $dest/error_in_$date.tar.gz

# ----------

#for f in $src1 $src2 $src3 $src4 $src5 $src6
#do
#ls -l $f
# tar cvzf $dest/trace_in_$date.tar.gz $src1
#sleep 1
#done

# -----------

#cd /interfaces/data/
#for i in *
#do
#[ -d "$i" ] && zip -r "$dest/$i.zip" "$i"
#done

# --------

#cd /interfaces/data/
for i in /interfaces/data/*
do
[ -d == "trace" ]

#zip -r "$dest/$i.zip" "$i"
#if [ "${f}" == "trace" && "${f}" == "test" && "${f}" == "error"]
#if [ "$string" == "trace" ]
#then
#echo "I am in"
ls -lh $i
#zip -r "$dest/$string.zip" "$string"

done

# Print end status message.
echo
echo "Backup finished"
date
echo

# Long listing of files in $dest to check file sizes.
echo "Checking the file sizes in $dest"
ls -lh $dest
-------


The easy and simplest way that I could think of is go to each directory and tar it with hard coded .../error_in_$date.tar.gz as below:
#cd $src5
#find . -mtime +$max_days | xargs tar cvzf $dest/error_in_$date.tar.gz

Again, I am a beginner and my way of scripting is in beginner level as well. Maybe there is more flexible way that I am looking for now.

Please advise guru
 
Old 08-29-2014, 05:07 AM   #6
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 9,439

Rep: Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770
as I told you you need to use parameter substitution. You can check it on the man page of bash.
Code:
#insert set -xv at the beginning to check what's happening
set -xv
....
for f in $src1 $src2 $src3 $src4 $src5 $src6
do
ls -l $f
dname=${f%/*}
dir2=${f##*/}
dir1=${dname##*/}
#and now you can use $dir1 and $dir2:
archive_file="$dir1_$dir2_$date.tar.gz"
...
sleep 1
done
 
Old 08-29-2014, 05:25 AM   #7
_mz
Member
 
Registered: Jul 2013
Posts: 33

Original Poster
Rep: Reputation: Disabled
new for me. Where should I insert full tar command? Could you please insert for me so that I can test it and hence see how your code manipulates the files.. This way I will learn your code.
 
Old 08-29-2014, 05:44 AM   #8
_mz
Member
 
Registered: Jul 2013
Posts: 33

Original Poster
Rep: Reputation: Disabled
I tried insert this code above "sleep 1" line:

find $f -mtime +$max_days | xargs tar cvzf $dest/$archive_file $f

Am I doing correct way?


However, the file created as 20140829.tar.gz. I untar the file, Here you go:

# ll interfaces/
total 4
drwxrwxr-x 3 root root 4096 Aug 29 18:38 data

# ll interfaces/data/
total 4
drwxrwxr-x 3 root root 4096 Aug 29 18:38 error

# ll interfaces/data/error/
total 4
drwxrwxr-x 2 2037 sapsys 4096 Jul 25 08:48 out

# ll interfaces/data/error/out/
total 224
-rw-rw---- 1 2031 sapsys 369 Jun 23 12:05 NDC900APF.NDC900APF_20140623_120526
-rw-rw---- 1 2031 sapsys 1107 Jun 25 09:20 NDC900APF.NDC900APF_20140625_092033
-rw-rw---- 1 2031 sapsys 1107 Jun 25 09:26 NDC900APF.NDC900APF_20140625_092658
-rw-rw---- 1 2031 sapsys 1107 Jun 25 10:14 NDC900APF.NDC900APF_20140625_101453
-rw-rw---- 1 2031 sapsys 1107 Jun 25 10:19 NDC900APF.NDC900APF_20140625_101954
 
Old 08-29-2014, 06:51 AM   #9
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 9,439

Rep: Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770
you can try:
find $f -mtime +$max_days | tar cvzf $dest/$archive_file -T -
 
Old 09-01-2014, 09:27 PM   #10
_mz
Member
 
Registered: Jul 2013
Posts: 33

Original Poster
Rep: Reputation: Disabled
Hi pan64,

I tried execute the script and this is what happened:

+ for f in '$src1' '$src2' '$src3' '$src4' '$src5' '$src6'
+ ls -l /interfaces/data/error/out/
total 224
...
files are here
...
...
+ dir2=
+ dir1=out
+ archive_file=20140902.tar.gz
+ find /interfaces/data/error/out/ -mtime +30
+ tar cvzf /interfaces/data/archive/20140902.tar.gz -T -

The loop for all the source directories are working fine. However, it looks like $dir2 could not retrieve the value. Maybe this is the reason why is the file created as 20140902.tar.gz. Actually what I need is:

dir1=error
dir2=out # but your code dir1=out
archive_file=$dir1_$dir2_$date.tar.gz

Some more only files in /interfaces/data/error/out are tar'ed. The rest of file are not despite they are more than 30 days old.
 
Old 09-02-2014, 12:19 AM   #11
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 9,439

Rep: Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770
actually you have not posted your script so I have no idea what is the reason. As I wrote you can put set -xv at the beginning of the script and you will see what's happening inside.
I can only guess the problem is that you put a final / at the end of the directories. You can remove it by:
Code:
for f in $src1 $src2 $src3 $src4 $src5 $src6
do
ls -l $f
f=${f%/}
dname=${f%/*}
dir2=${f##*/}
dir1=${dname##*/}
#and now you can use $dir1 and $dir2:
archive_file="$dir1_$dir2_$date.tar.gz"
...
sleep 1
done
 
1 members found this post helpful.
Old 09-02-2014, 01:28 AM   #12
_mz
Member
 
Registered: Jul 2013
Posts: 33

Original Poster
Rep: Reputation: Disabled
Here is the script

Quote:
#!/bin/bash

# What to backup
src1="/interfaces/data/trace/in/"
src2="/interfaces/data/trace/out/"
src3="/interfaces/data/test/in/"
src4="/interfaces/data/test/out/"
src5="/interfaces/data/error/in/"
src6="/interfaces/data/error/out/"

# Where to backup to.
dest="/interfaces/data/archive"

# Retention period
max_days=30

# Create archive filename.
date=$(date '+%Y''%m''%d')
#archive_file="$dir1_$dir2_$date.tar.gz"

#insert set -xv at the beginning to check what's happening
set -xv
#....
for f in $src1 $src2 $src3 $src4 $src5 $src6
do
ls -l $f
f=${f%/}
dname=${f%/*}
dir2=${f##*/}
dir1=${dname##*/}
#and now you can use $dir1 and $dir2:
archive_file=$dir1_$dir2_$date.tar.gz
#...
#find $f -mtime +$max_days | xargs tar cvzf $dest/$archive_file $f
find $f -mtime +$max_days | xargs tar cvzf $dest/$archive_file -T -
sleep 1
done
From the background (set -xv), it seems variables are detected as example below:
+ f=/interfaces/data/error/out
+ dname=/interfaces/data/error
+ dir2=out
+ dir1=error
+ archive_file=20140902.tar.gz
+ find /interfaces/data/error/out -mtime +30
+ xargs tar cvzf /interfaces/data/archive/20140902.tar.gz -T -

but the archive_file still 20140902.tar.gz. Once the script completed, I tar tvf 20140902.tar.gz to see the content, it backed up only interfaces/data/error/out. It should be more than /error. Dir /test/*, /trace/* have files that older than 30 days as well.
 
Old 09-02-2014, 01:40 AM   #13
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 9,439

Rep: Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770
to fix the archive name:
archive_file=${dir1}_${dir2}_$date.tar.gz
 
Old 09-02-2014, 08:44 PM   #14
_mz
Member
 
Registered: Jul 2013
Posts: 33

Original Poster
Rep: Reputation: Disabled
Thanks. At least it works for the main part.

There is a small adjustment that I need to do as when I untar the archive file, the scripts actually tar'ed the whole/full dir like:

interfaces/data/error/in

It should only tar error/in, error/out etc. This is due to for loop for the variable $src1, $src2 etc. I will try to adjust it accordingly.
 
Old 09-03-2014, 12:35 AM   #15
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 9,439

Rep: Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770Reputation: 2770
in this command: find $f -mtime +$max_days | xargs tar cvzf $dest/$archive_file -T -
you should remove xargs, that is not required. It may cause errors. (see post #9)
you may try the tar flag: --strip-components=NUMBER to remove leading directories (see man page)

If you really want to say thanks just press YES...
 
1 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] BASH read text list zip files into a single archive --possible? SilversleevesX Programming 9 03-10-2014 02:43 AM
[SOLVED] how to archive torrent file + the iso files , like the ova archive of virtual box jheengut Linux - Software 2 12-22-2013 05:04 PM
BASH: Unzip files/folders from archive one at a time using a loop cade Programming 3 05-16-2010 10:48 PM
Script to Add files to a Zip archive with out unzipping it !! prashix Programming 12 10-23-2007 09:19 AM
Automation Script !! Create automated archive of files based on name of file! yoshima Programming 5 09-28-2007 05:17 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 05:52 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration