LinuxQuestions.org
Page 1 of 2 1 2
Show 50 post(s) from this thread on one page

LinuxQuestions.org (/questions/)
-   Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   Bash script to archive files (https://www.linuxquestions.org/questions/linux-newbie-8/bash-script-to-archive-files-4175516728/)

_mz 08-29-2014 02:52 AM
Bash script to archive files
 
Hi script experts,

# What to backup
/interfaces/data/trace/in/*
/interfaces/data/trace/out/*
/interfaces/data/test/in/*
/interfaces/data/test/out/*
/interfaces/data/error/in/*
/interfaces/data/error/out/*

# Where to backup to.
/interfaces/data/archive

I need to tar those files/folders in those directories that older than 30days with file name as /interfaces/data/archive/$dir1_$dir2_$date.tar.gz

where:
dir1 = 3rd directory like /trace, /test, /error;
dir2 = in or out;
date = 20140829 format

example of file: /interfaces/data/archive/trace_in_20140829.tar.gz

Thank you guys.

pan64 08-29-2014 03:32 AM
and what is your problem? http://www.linuxquestions.org/linux/...Ask_a_Question

_mz 08-29-2014 03:56 AM
Quote:

I need to tar those files/folders in those directories that older than 30days with file name as /interfaces/data/archive/$dir1_$dir2_$date.tar.gz
This is my problem. How to get $dir, $dir so that I can create file like /interfaces/data/archive/$dir1_$dir2_$date.tar.gz

pan64 08-29-2014 04:25 AM
can you post your script? You need to construct the target directory from your variables using parameter substitution.

_mz 08-29-2014 05:01 AM
Well, I am very very new to scripting and maybe I need to learn more about for loop, if statement and etc. I just started to write the script on my own and there are a lot of tries and errors. Here is my script.

--------
#!/bin/bash

# What to backup
src1="/interfaces/data/trace/in/"
src2="/interfaces/data/trace/out/"
src3="/interfaces/data/test/in/"
src4="/interfaces/data/test/out/"
src5="/interfaces/data/error/in/"
src6="/interfaces/data/error/out/"

# Where to backup to.
dest="/interfaces/data/archive"

# Retention period
max_days=30

# Create archive filename.
date=$(date '+%Y''%m''%d')
archive_file="$dir1_$dir2_$date.tar.gz"

# Log file
LOG_FILE="/interfaces/data/script/archive.$date.log"

# Print start status message.
echo "Backing up $src5 to $dest"
date
echo

# Backup the files using tar.
#cd $src5
#find . -mtime +$max_days | xargs tar cvzf $dest/error_in_$date.tar.gz

# ----------

#for f in $src1 $src2 $src3 $src4 $src5 $src6
#do
#ls -l $f
# tar cvzf $dest/trace_in_$date.tar.gz $src1
#sleep 1
#done

# -----------

#cd /interfaces/data/
#for i in *
#do
#[ -d "$i" ] && zip -r "$dest/$i.zip" "$i"
#done

# --------

#cd /interfaces/data/
for i in /interfaces/data/*
do
[ -d == "trace" ]

#zip -r "$dest/$i.zip" "$i"
#if [ "${f}" == "trace" && "${f}" == "test" && "${f}" == "error"]
#if [ "$string" == "trace" ]
#then
#echo "I am in"
ls -lh $i
#zip -r "$dest/$string.zip" "$string"

done

# Print end status message.
echo
echo "Backup finished"
date
echo

# Long listing of files in $dest to check file sizes.
echo "Checking the file sizes in $dest"
ls -lh $dest
-------


The easy and simplest way that I could think of is go to each directory and tar it with hard coded .../error_in_$date.tar.gz as below:
#cd $src5
#find . -mtime +$max_days | xargs tar cvzf $dest/error_in_$date.tar.gz

Again, I am a beginner and my way of scripting is in beginner level as well. Maybe there is more flexible way that I am looking for now.

Please advise guru :)

pan64 08-29-2014 05:07 AM
as I told you you need to use parameter substitution. You can check it on the man page of bash.
Code:
#insert set -xv at the beginning to check what's happening
set -xv
....
for f in $src1 $src2 $src3 $src4 $src5 $src6
do
ls -l $f
dname=${f%/*}
dir2=${f##*/}
dir1=${dname##*/}
#and now you can use $dir1 and $dir2:
archive_file="$dir1_$dir2_$date.tar.gz"
...
sleep 1
done

_mz 08-29-2014 05:25 AM
new for me. Where should I insert full tar command? Could you please insert for me so that I can test it and hence see how your code manipulates the files.. This way I will learn your code.

_mz 08-29-2014 05:44 AM
I tried insert this code above "sleep 1" line:

find $f -mtime +$max_days | xargs tar cvzf $dest/$archive_file $f

Am I doing correct way?


However, the file created as 20140829.tar.gz. I untar the file, Here you go:

# ll interfaces/
total 4
drwxrwxr-x 3 root root 4096 Aug 29 18:38 data

# ll interfaces/data/
total 4
drwxrwxr-x 3 root root 4096 Aug 29 18:38 error

# ll interfaces/data/error/
total 4
drwxrwxr-x 2 2037 sapsys 4096 Jul 25 08:48 out

# ll interfaces/data/error/out/
total 224
-rw-rw---- 1 2031 sapsys 369 Jun 23 12:05 NDC900APF.NDC900APF_20140623_120526
-rw-rw---- 1 2031 sapsys 1107 Jun 25 09:20 NDC900APF.NDC900APF_20140625_092033
-rw-rw---- 1 2031 sapsys 1107 Jun 25 09:26 NDC900APF.NDC900APF_20140625_092658
-rw-rw---- 1 2031 sapsys 1107 Jun 25 10:14 NDC900APF.NDC900APF_20140625_101453
-rw-rw---- 1 2031 sapsys 1107 Jun 25 10:19 NDC900APF.NDC900APF_20140625_101954

pan64 08-29-2014 06:51 AM
you can try:
find $f -mtime +$max_days | tar cvzf $dest/$archive_file -T -

_mz 09-01-2014 09:27 PM
Hi pan64,

I tried execute the script and this is what happened:

+ for f in '$src1' '$src2' '$src3' '$src4' '$src5' '$src6'
+ ls -l /interfaces/data/error/out/
total 224
...
files are here
...
...
+ dir2=
+ dir1=out
+ archive_file=20140902.tar.gz
+ find /interfaces/data/error/out/ -mtime +30
+ tar cvzf /interfaces/data/archive/20140902.tar.gz -T -

The loop for all the source directories are working fine. However, it looks like $dir2 could not retrieve the value. Maybe this is the reason why is the file created as 20140902.tar.gz. Actually what I need is:

dir1=error
dir2=out # but your code dir1=out
archive_file=$dir1_$dir2_$date.tar.gz

Some more only files in /interfaces/data/error/out are tar'ed. The rest of file are not despite they are more than 30 days old.

pan64 09-02-2014 12:19 AM
actually you have not posted your script so I have no idea what is the reason. As I wrote you can put set -xv at the beginning of the script and you will see what's happening inside.
I can only guess the problem is that you put a final / at the end of the directories. You can remove it by:
Code:
for f in $src1 $src2 $src3 $src4 $src5 $src6
do
ls -l $f
f=${f%/}
dname=${f%/*}
dir2=${f##*/}
dir1=${dname##*/}
#and now you can use $dir1 and $dir2:
archive_file="$dir1_$dir2_$date.tar.gz"
...
sleep 1
done

_mz 09-02-2014 01:28 AM
Here is the script

Quote:
#!/bin/bash

# What to backup
src1="/interfaces/data/trace/in/"
src2="/interfaces/data/trace/out/"
src3="/interfaces/data/test/in/"
src4="/interfaces/data/test/out/"
src5="/interfaces/data/error/in/"
src6="/interfaces/data/error/out/"

# Where to backup to.
dest="/interfaces/data/archive"

# Retention period
max_days=30

# Create archive filename.
date=$(date '+%Y''%m''%d')
#archive_file="$dir1_$dir2_$date.tar.gz"

#insert set -xv at the beginning to check what's happening
set -xv
#....
for f in $src1 $src2 $src3 $src4 $src5 $src6
do
ls -l $f
f=${f%/}
dname=${f%/*}
dir2=${f##*/}
dir1=${dname##*/}
#and now you can use $dir1 and $dir2:
archive_file=$dir1_$dir2_$date.tar.gz
#...
#find $f -mtime +$max_days | xargs tar cvzf $dest/$archive_file $f
find $f -mtime +$max_days | xargs tar cvzf $dest/$archive_file -T -
sleep 1
done
From the background (set -xv), it seems variables are detected as example below:
+ f=/interfaces/data/error/out
+ dname=/interfaces/data/error
+ dir2=out
+ dir1=error
+ archive_file=20140902.tar.gz
+ find /interfaces/data/error/out -mtime +30
+ xargs tar cvzf /interfaces/data/archive/20140902.tar.gz -T -

but the archive_file still 20140902.tar.gz. Once the script completed, I tar tvf 20140902.tar.gz to see the content, it backed up only interfaces/data/error/out. It should be more than /error. Dir /test/*, /trace/* have files that older than 30 days as well.

pan64 09-02-2014 01:40 AM
to fix the archive name:
archive_file=${dir1}_${dir2}_$date.tar.gz

_mz 09-02-2014 08:44 PM
Thanks. At least it works for the main part.

There is a small adjustment that I need to do as when I untar the archive file, the scripts actually tar'ed the whole/full dir like:

interfaces/data/error/in

It should only tar error/in, error/out etc. This is due to for loop for the variable $src1, $src2 etc. I will try to adjust it accordingly.

pan64 09-03-2014 12:35 AM
in this command: find $f -mtime +$max_days | xargs tar cvzf $dest/$archive_file -T -
you should remove xargs, that is not required. It may cause errors. (see post #9)
you may try the tar flag: --strip-components=NUMBER to remove leading directories (see man page)

If you really want to say thanks just press YES...


All times are GMT -5. The time now is 12:03 PM.
Page 1 of 2 1 2
Show 50 post(s) from this thread on one page