LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 02-28-2011, 02:45 PM   #1
chandkur
LQ Newbie
 
Registered: Feb 2011
Posts: 16

Rep: Reputation: 0
archiving log files


Hello all,
I am sure this question might have been asked before, but I need a solution to this very badly.

I need a script that accepts two parameters inputDir and outputDir.
This script should copy all the log files in the inputDir to a folder like <BackupLogs-currentDaysDate>
The new folder with the log files should be tarred and gzipped <BackupLogs-currentDaysDate>.tgz
And this new <BackupLogs-currentDaysDate>.tgz file should be copied to the outputDir.
Also all the log files in the inputDir should be deleted.

Please help and thanks to all in advance.

Thanks
Chandra
 
Old 02-28-2011, 03:09 PM   #2
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,577
Blog Entries: 14

Rep: Reputation: 969Reputation: 969Reputation: 969Reputation: 969Reputation: 969Reputation: 969Reputation: 969Reputation: 969
Linux already has a log rotation infrastructure built in.

Type "man logrotate" for details.
 
Old 02-28-2011, 03:23 PM   #3
devUnix
Member
 
Registered: Oct 2010
Location: Bengaluru, India
Distribution: RHEL 5.1 on My PC, & SunOS / Sun Solaris, RHEL, SuSe, Debian, FreeBSD and other Linux flavors @ Work
Posts: 576

Rep: Reputation: 48
Chandra, I do not think that you only mean to or want to backup logs. By having such a script in place you can specify any source directory which may contain just anything- logs, songs, movies, photos, documents, etc.

Well, I can help you with it. I have done a similar task before. But I will have to write the script again as I do not have it available with me at the moment. So, wait fomr some time while I write andf test it before posting here for your use.

It is a small task, in fact. And you yourself can do it. Guess? Okay, you already know the steps- that is what the script needs to know to follow them one by one. Try it doing yourself in the meanwhile I do it for you. Okay, man?
 
Old 02-28-2011, 03:39 PM   #4
chandkur
LQ Newbie
 
Registered: Feb 2011
Posts: 16

Original Poster
Rep: Reputation: 0
Hey guys thanks for responding to my thread.

Quote:
MensaWater Linux already has a log rotation infrastructure built in.

Type "man logrotate" for details.
can you give me some examples of how to use it for my requirement ?

Quote:
devUnix
It is a small task, in fact. And you yourself can do it. Guess? Okay, you already know the steps- that is what the script needs to know to follow them one by one. Try it doing yourself in the meanwhile I do it for you. Okay, man?
Hey thanks, much appreciate your help. I am trying to do it on my end too, but I have to accept that my shell scripting knowledge is limited and am using forums like this and google to come up with a good solution.
Plus I have a release this weekend and getting pulled in different directions.

I am stuck at this step, wherein I need to be able to copy all the files ending with .log ( i,e log files) from the input directory.

Thanks
Chandra
 
Old 02-28-2011, 04:18 PM   #5
devUnix
Member
 
Registered: Oct 2010
Location: Bengaluru, India
Distribution: RHEL 5.1 on My PC, & SunOS / Sun Solaris, RHEL, SuSe, Debian, FreeBSD and other Linux flavors @ Work
Posts: 576

Rep: Reputation: 48
Chandra, I have written it now. Let me first check it. So, you have to wait for some 2 or 5 minutes more. Okay? I do not have much work to do right now in the office.
 
Old 02-28-2011, 04:24 PM   #6
chandkur
LQ Newbie
 
Registered: Feb 2011
Posts: 16

Original Poster
Rep: Reputation: 0
Quote:
devUnix Chandra, I have written it now. Let me first check it. So, you have to wait for some 2 or 5 minutes more. Okay? I do not have much work to do right now in the office.
Thank you very much for your time and help.
 
Old 02-28-2011, 04:27 PM   #7
devUnix
Member
 
Registered: Oct 2010
Location: Bengaluru, India
Distribution: RHEL 5.1 on My PC, & SunOS / Sun Solaris, RHEL, SuSe, Debian, FreeBSD and other Linux flavors @ Work
Posts: 576

Rep: Reputation: 48
Chandra, it is done. I have tested the script and it works fine. You also check it out. Okay?

First, let me give you the script here and then I will give you some explanation. I have not used any comments at the moment. But yuo can figure out everything easily.

-bash-2.05b# cat backItUp.sh
Code:
#!/bin/bash
if [ "$#" -lt 2 ] || [ "$#" -gt 2 ]; then
        echo "$0 source destination"
        exit 1
fi
if [ ! -e "$1" ]; then
        echo "Source: $1 does not exist."
        exit 1
fi
tar -vcf ${2}-$(date +%d-%m-%Y).tar $1
if [ "$?" != "0" ]; then
        echo "tar failed."
        exit 1
else
        echo "Archiving completed successfully."
fi
rm -rf $1
if [ "$?" != "0" ]; then
        echo "Deleting $1 failed."
        exit 1
else
        echo "$1 successfully deleted."
fi
gzip -v ${2}-$(date +%d-%m-%Y).tar
if [ "$?" != "0" ]; then
        echo "gzip failed."
        exit 1
else
        echo "Compressing completed successfully."
fi
gzip -vt ${2}-$(date +%d-%m-%Y).tar.gz
if [ "$?" != "0" ]; then
        echo "gzip testing failed."
        exit 1
else
        echo "gzip testing completed successfully."
fi
ls -l ${2}-$(date +%d-%m-%Y).tar.gz
if [ "$?" != "0" ]; then
        echo "Some unknown errors have occured."
        exit 1
else
        echo "#######DONE#######."
fi
exit 0

Chandra, I am in a hurry as my shift is over.

I give you the test I have used:

Code:
-bash-2.05b# ls /tmp
logs
-bash-2.05b# backItUp.sh
/root/bin/backItUp.sh source destination
-bash-2.05b# backItUp.sh /tmp/logs/
/root/bin/backItUp.sh source destination
-bash-2.05b# backItUp.sh /tmp/logs/ backup hi
/root/bin/backItUp.sh source destination
-bash-2.05b# backItUp.sh /tmp/log backup
Source: /tmp/log does not exist.
-bash-2.05b# backItUp.sh /tmp/logs/ backup-tmp-logs
tar: Removing leading `/' from member names
tmp/logs/
tmp/logs/08022011-Mike.log
tmp/logs/09022011-Mariyam.log
tmp/logs/09022011-Mike.log
tmp/logs/28022011-Ken.log
tmp/logs/09022011-Ken.log
tmp/logs/10022011-Mariyam.log
tmp/logs/11022011-Mike.log
Archiving completed successfully.
/tmp/logs/ successfully deleted.
backup-tmp-logs-28-02-2011.tar:  93.2% -- replaced with backup-tmp-logs-28-02-2011.tar.gz
Compressing completed successfully.
backup-tmp-logs-28-02-2011.tar.gz:       OK
gzip testing completed successfully.
-rw-r--r--  1 root root 747 Feb 28 21:58 backup-tmp-logs-28-02-2011.tar.gz
#######DONE#######.
-bash-2.05b# ls -ltr
total 60
drwx------  2 root root  4096 Feb  4 21:46 Mail
-rwxr-xr-x  1 root root    95 Feb  9 20:45 print-mails-sender
-rwxr-xr-x  1 root root   152 Feb  9 22:00 mail.sh
-rw-r--r--  1 root root   159 Feb 11 21:52 out
-rw-------  1 root root     0 Feb 11 21:53 mbox
-rw-r--r--  1 root root 17658 Feb 28 21:11 --help.tgz
-rw-r--r--  1 root root  8515 Feb 28 21:44 backup-28-02-2011.tar.gz
drwxr-xr-x  3 root root  4096 Feb 28 21:54 bin
-rwxr-xr-x  1 root root   946 Feb 28 21:57 backItUp.sh
-rw-r--r--  1 root root   747 Feb 28 21:58 backup-tmp-logs-28-02-2011.tar.gz
-bash-2.05b# ls -l /tmp
total 0
-bash-2.05b# file backup-tmp-logs-28-02-2011.tar.gz
backup-tmp-logs-28-02-2011.tar.gz: gzip compressed data, was "backup-tmp-logs-28-02-2011.tar", from Unix
-bash-2.05b# gzip -l backup-tmp-logs-28-02-2011.tar.gz
         compressed        uncompressed  ratio uncompressed_name
                747               10240  93.2% backup-tmp-logs-28-02-2011.tar
-bash-2.05b# gunzip backup-tmp-logs-28-02-2011.tar.gz
-bash-2.05b# ls -l
total 68
-rwxr-xr-x  1 root root   946 Feb 28 21:57 backItUp.sh
-rw-r--r--  1 root root  8515 Feb 28 21:44 backup-28-02-2011.tar.gz
-rw-r--r--  1 root root 10240 Feb 28 21:58 backup-tmp-logs-28-02-2011.tar
drwxr-xr-x  3 root root  4096 Feb 28 21:54 bin
-rw-r--r--  1 root root 17658 Feb 28 21:11 --help.tgz
drwx------  2 root root  4096 Feb  4 21:46 Mail
-rwxr-xr-x  1 root root   152 Feb  9 22:00 mail.sh
-rw-------  1 root root     0 Feb 11 21:53 mbox
-rw-r--r--  1 root root   159 Feb 11 21:52 out
-rwxr-xr-x  1 root root    95 Feb  9 20:45 print-mails-sender
-bash-2.05b# tar -vxf backup-tmp-logs-28-02-2011.tar
tmp/logs/
tmp/logs/08022011-Mike.log
tmp/logs/09022011-Mariyam.log
tmp/logs/09022011-Mike.log
tmp/logs/28022011-Ken.log
tmp/logs/09022011-Ken.log
tmp/logs/10022011-Mariyam.log
tmp/logs/11022011-Mike.log
-bash-2.05b# ls -ltr
total 72
drwx------  2 root root  4096 Feb  4 21:46 Mail
-rwxr-xr-x  1 root root    95 Feb  9 20:45 print-mails-sender
-rwxr-xr-x  1 root root   152 Feb  9 22:00 mail.sh
-rw-r--r--  1 root root   159 Feb 11 21:52 out
-rw-------  1 root root     0 Feb 11 21:53 mbox
-rw-r--r--  1 root root 17658 Feb 28 21:11 --help.tgz
-rw-r--r--  1 root root  8515 Feb 28 21:44 backup-28-02-2011.tar.gz
drwxr-xr-x  3 root root  4096 Feb 28 21:54 bin
-rwxr-xr-x  1 root root   946 Feb 28 21:57 backItUp.sh
-rw-r--r--  1 root root 10240 Feb 28 21:58 backup-tmp-logs-28-02-2011.tar
drwxr-xr-x  3 root root  4096 Feb 28 22:00 tmp
-bash-2.05b# ls -l tmp
total 4
drwxrwxrwx  2 root root 4096 Feb 28 20:50 logs
-bash-2.05b# ls -lR tmp
tmp:
total 4
drwxrwxrwx  2 root root 4096 Feb 28 20:50 logs

tmp/logs:
total 28
-rw-r--r--  1 apache apache  144 Feb  8 21:46 08022011-Mike.log
-rw-r--r--  1 apache apache  143 Feb  9 14:33 09022011-Ken.log
-rw-r--r--  1 apache apache  267 Feb 10 21:36 09022011-Mariyam.log
-rw-r--r--  1 apache apache  188 Feb  9 14:22 09022011-Mike.log
-rw-r--r--  1 apache apache  687 Feb 10 21:37 10022011-Mariyam.log
-rw-r--r--  1 apache apache 1525 Feb 11 15:36 11022011-Mike.log
-rw-r--r--  1 apache apache  176 Feb 28 20:50 28022011-Ken.log
-bash-2.05b#


Just go through the above test to see how I have used the script.

When you hjave copied and pasted the script do not forget to make it executable by:

Code:
chmod +x scriptName.sh
Let me know if you have any problem. I will see it tomorrow. Take care and good night!

Last edited by devUnix; 02-28-2011 at 04:29 PM.
 
Old 02-28-2011, 04:54 PM   #8
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,577
Blog Entries: 14

Rep: Reputation: 969Reputation: 969Reputation: 969Reputation: 969Reputation: 969Reputation: 969Reputation: 969Reputation: 969
You should be able to see examples in your /etc/logrotate.conf and if it includes /etc/logrotate.d (as RHEL/CentOS do) then in the files under that directory.
 
Old 03-01-2011, 09:29 AM   #9
chandkur
LQ Newbie
 
Registered: Feb 2011
Posts: 16

Original Poster
Rep: Reputation: 0
Thank you very much. I have gone through the script that you posted and i didn't find anywhere that you are trying to find the files ending in .log extension in the source directory.
I maybe wrong, can you please clarify.
 
Old 03-01-2011, 09:42 AM   #10
devUnix
Member
 
Registered: Oct 2010
Location: Bengaluru, India
Distribution: RHEL 5.1 on My PC, & SunOS / Sun Solaris, RHEL, SuSe, Debian, FreeBSD and other Linux flavors @ Work
Posts: 576

Rep: Reputation: 48
Quote:
Originally Posted by chandkur View Post
Thank you very much. I have gone through the script that you posted and i didn't find anywhere that you are trying to find the files ending in .log extension in the source directory.
I maybe wrong, can you please clarify.

I got reminded of it when I was going back by cab that you would ask me that question. Wait for some minutes.
 
Old 03-01-2011, 11:38 AM   #11
chandkur
LQ Newbie
 
Registered: Feb 2011
Posts: 16

Original Poster
Rep: Reputation: 0
Hello devUnix, did you get a chance to modify your script ?
basically we want to be doing this :
1. in the Source directory, we need to create a tar of all the .log files(there will be other files in this source directory like *.properties etc).
2. make a gzip of the above created tar file.
3. copy this gzip file to the destination directory.
4. delete all the *.log files in the source directory.

Thanks
Chandra
 
Old 03-01-2011, 12:01 PM   #12
devUnix
Member
 
Registered: Oct 2010
Location: Bengaluru, India
Distribution: RHEL 5.1 on My PC, & SunOS / Sun Solaris, RHEL, SuSe, Debian, FreeBSD and other Linux flavors @ Work
Posts: 576

Rep: Reputation: 48
Chandra:

Note that you can use any type of file globbing:


*.log
*.txt
*.tmp
*.blah-blah


You can speficy even a complete directory as shown in the test below.

Have a look at this test.

Code:
-bash-2.05b# backItUp.sh /tmp/logs/*.txt
Destination Backup File Name: text
/tmp/logs/a.txt
/tmp/logs/b.txt
Archiving completed successfully.

/tmp/logs/a.txt successfully deleted.

/tmp/logs/b.txt successfully deleted.

text-01-03-2011.tar:     98.9% -- replaced with text-01-03-2011.tar.gz
Compressing completed successfully.

text-01-03-2011.tar.gz:  OK
gzip testing completed successfully.

-rw-r--r--  1 root root 152 Mar  1 17:24 text-01-03-2011.tar.gz
#######DONE#######.
-bash-2.05b# ls -ltr /tmp/logs/
total 0
-rw-r--r--  1 root root 0 Mar  1 17:22 b.log
-rw-r--r--  1 root root 0 Mar  1 17:22 a.log
-rw-r--r--  1 root root 0 Mar  1 17:22 2.tmp
-rw-r--r--  1 root root 0 Mar  1 17:22 2.log
-rw-r--r--  1 root root 0 Mar  1 17:22 1.tmp
-rw-r--r--  1 root root 0 Mar  1 17:22 1.log
-bash-2.05b# backItUp.sh /tmp/logs
Destination Backup File Name: logs
/tmp/logs/
/tmp/logs/1.log
/tmp/logs/2.log
/tmp/logs/1.tmp
/tmp/logs/2.tmp
/tmp/logs/a.log
/tmp/logs/b.log
Archiving completed successfully.

/tmp/logs successfully deleted.

logs-01-03-2011.tar:     98.3% -- replaced with logs-01-03-2011.tar.gz
Compressing completed successfully.

logs-01-03-2011.tar.gz:  OK
gzip testing completed successfully.

-rw-r--r--  1 root root 214 Mar  1 17:25 logs-01-03-2011.tar.gz
#######DONE#######.
-bash-2.05b# ls -ltr /tmp/logs/
ls: /tmp/logs/: No such file or directory
-bash-2.05b#

Isn't that what you wanted?


The first version of my script can be used to backup any directory that may contain any types of files.

But this second version is more advanced and enhanced. I give you the script here and recommend that you test it before you trust it.

Note that the format of executing the script is:

backItUp.sh /myDir/*.log <ENTER>

OR

backItUp.sh /myDir/ <ENTER>

As soon as you hit the Enter / Return key, you will be prompted to specify a file name which will be used to name the compressed archive or backup file because we want to store everything in one single container- the archive.

Basch Script Name: backItUp.sh

Code:
###############################################################################
####### Script Name     :       backItUp.sh                             #######
####### Purpose         :       Taking Backup of Specified Files        #######
####### Developer       :       Dev                                     #######
####### Email           :       dk_mahadeva@yahoo.com                   #######
###############################################################################
#!/bin/bash
if [ `echo $* | awk '{print length}'` -eq 0 ]; then
        echo -e "Source is Required.\n"
        echo -e `basename $0` " source\n"
        exit 1
fi
for File in $*; do
        if [ ! -e $File ]; then
                echo -e "$File does not exist.\n"
                exit 1
        fi
done
echo -n "Destination Backup File Name: "
read
if [ `echo $REPLY | awk '{print length}'` -eq 0 ]; then
        echo -e "Destination Backup File Name is Required.\n"
        exit 1
fi
BackupFile=${REPLY}-$(date +%d-%m-%Y).tar
tar -vPcf ${BackupFile} $*
if [ "$?" != "0" ]; then
        echo -e "tar failed.\n"
        exit 1
else
        echo -e "Archiving completed successfully.\n"
fi
for File in $*; do
        rm -rf $File
        if [ "$?" != "0" ]; then
                echo -e "Deleting $File failed.\n"
        else
                echo -e "$File successfully deleted.\n"
        fi
done
gzip -v ${BackupFile}
if [ "$?" != "0" ]; then
        echo -e "gzip failed.\n"
        exit 1
else
        echo -e "Compressing completed successfully.\n"
fi
gzip -vt ${BackupFile}.gz
if [ "$?" != "0" ]; then
        echo -e "gzip testing failed.\n"
        exit 1
else
        echo -e "gzip testing completed successfully.\n"
fi
ls -l ${BackupFile}.gz
if [ "$?" != "0" ]; then
        echo -e "Some unknown errors have occured.\n"
        exit 1
else
        echo "#######DONE#######."
fi
exit 0

The above script, as I have created, is almost fully error-protected. It can handle possible errors such as incorrect file or directory name or path and other errors that may come up when tar and gzip utilities are used.

If you find it useful then consider sending me $ 100000000... do not faint - I am just kidding.

Last edited by devUnix; 03-01-2011 at 12:05 PM.
 
Old 03-01-2011, 12:08 PM   #13
devUnix
Member
 
Registered: Oct 2010
Location: Bengaluru, India
Distribution: RHEL 5.1 on My PC, & SunOS / Sun Solaris, RHEL, SuSe, Debian, FreeBSD and other Linux flavors @ Work
Posts: 576

Rep: Reputation: 48
Chandra, bhai mene tumhari last comment nahi padhi. Lekin mene jo script likhi hai tum use check karo. It will do the job efficiently. If you need any further modifications or changes or enhancement, let me know it. Agar comments ki jarurat ho script me samjhne ke liye to bata dena, mein samjadunga. It is not much difficult. You can easily get it. By the way, where do you work?

I just did some more tests backuping directories and files separately. The script works fantastically!

I am going to automate the restore activity which however requires only two steps / commands:

Code:
gunzip bash-scripts-01-03-2011.tar.gz
and

Code:
tar -vxf bash-scripts-01-03-2011.tar
You can view the contents of the archive without extracting them from it (I was once asked in an interview how to do that):

Code:
tar -vtf bash-scripts-01-03-2011.tar

Last edited by devUnix; 03-01-2011 at 12:19 PM.
 
Old 03-01-2011, 12:26 PM   #14
chandkur
LQ Newbie
 
Registered: Feb 2011
Posts: 16

Original Poster
Rep: Reputation: 0
hi I got this error while executing your script. I was attempting to try on files with .log1 extension.

[ddmsuser@WTXTT02 download-data]$ ./backItUp.sh /home/ddmsuser/DDMSPurge/*.log1
/home/ddmsuser/DDMSPurge/*.log1 does not exist.
 
Old 03-01-2011, 12:31 PM   #15
chandkur
LQ Newbie
 
Registered: Feb 2011
Posts: 16

Original Poster
Rep: Reputation: 0
Hi yaar, thanks for all the help.. maine aapki hindi mein comments nahin padatha.. sach mein accha laga aapke comments pad kar..
array yaar.. mein ne test kiya aapki nayi script se.. par wo error aya hai..
mein buffalo NY mein kaam karta hoon yaar.. aap kaha se ho ?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Log file and command output archiving software harry edwards Linux - Server 2 11-01-2008 07:28 PM
archiving files in c++ mkrems Programming 2 04-03-2008 10:18 PM
archiving rotated log files tjainsworth Linux - Security 4 07-12-2006 02:00 PM
Webserver Log Rotation, Stats and Archiving. graq Linux - Enterprise 1 02-24-2006 07:57 PM
Archiving Email Files ? dolphans1 Mandriva 6 06-15-2005 11:12 AM


All times are GMT -5. The time now is 09:11 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration