Linux - NewbieThis Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place!
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Hello all,
I am sure this question might have been asked before, but I need a solution to this very badly.
I need a script that accepts two parameters inputDir and outputDir.
This script should copy all the log files in the inputDir to a folder like <BackupLogs-currentDaysDate>
The new folder with the log files should be tarred and gzipped <BackupLogs-currentDaysDate>.tgz
And this new <BackupLogs-currentDaysDate>.tgz file should be copied to the outputDir.
Also all the log files in the inputDir should be deleted.
Chandra, I do not think that you only mean to or want to backup logs. By having such a script in place you can specify any source directory which may contain just anything- logs, songs, movies, photos, documents, etc.
Well, I can help you with it. I have done a similar task before. But I will have to write the script again as I do not have it available with me at the moment. So, wait fomr some time while I write andf test it before posting here for your use.
It is a small task, in fact. And you yourself can do it. Guess? Okay, you already know the steps- that is what the script needs to know to follow them one by one. Try it doing yourself in the meanwhile I do it for you. Okay, man?
MensaWater Linux already has a log rotation infrastructure built in.
Type "man logrotate" for details.
can you give me some examples of how to use it for my requirement ?
Quote:
devUnix
It is a small task, in fact. And you yourself can do it. Guess? Okay, you already know the steps- that is what the script needs to know to follow them one by one. Try it doing yourself in the meanwhile I do it for you. Okay, man?
Hey thanks, much appreciate your help. I am trying to do it on my end too, but I have to accept that my shell scripting knowledge is limited and am using forums like this and google to come up with a good solution.
Plus I have a release this weekend and getting pulled in different directions.
I am stuck at this step, wherein I need to be able to copy all the files ending with .log ( i,e log files) from the input directory.
Chandra, I have written it now. Let me first check it. So, you have to wait for some 2 or 5 minutes more. Okay? I do not have much work to do right now in the office.
devUnix Chandra, I have written it now. Let me first check it. So, you have to wait for some 2 or 5 minutes more. Okay? I do not have much work to do right now in the office.
Chandra, it is done. I have tested the script and it works fine. You also check it out. Okay?
First, let me give you the script here and then I will give you some explanation. I have not used any comments at the moment. But yuo can figure out everything easily.
-bash-2.05b# cat backItUp.sh
Code:
#!/bin/bash
if [ "$#" -lt 2 ] || [ "$#" -gt 2 ]; then
echo "$0 source destination"
exit 1
fi
if [ ! -e "$1" ]; then
echo "Source: $1 does not exist."
exit 1
fi
tar -vcf ${2}-$(date +%d-%m-%Y).tar $1
if [ "$?" != "0" ]; then
echo "tar failed."
exit 1
else
echo "Archiving completed successfully."
fi
rm -rf $1
if [ "$?" != "0" ]; then
echo "Deleting $1 failed."
exit 1
else
echo "$1 successfully deleted."
fi
gzip -v ${2}-$(date +%d-%m-%Y).tar
if [ "$?" != "0" ]; then
echo "gzip failed."
exit 1
else
echo "Compressing completed successfully."
fi
gzip -vt ${2}-$(date +%d-%m-%Y).tar.gz
if [ "$?" != "0" ]; then
echo "gzip testing failed."
exit 1
else
echo "gzip testing completed successfully."
fi
ls -l ${2}-$(date +%d-%m-%Y).tar.gz
if [ "$?" != "0" ]; then
echo "Some unknown errors have occured."
exit 1
else
echo "#######DONE#######."
fi
exit 0
Chandra, I am in a hurry as my shift is over.
I give you the test I have used:
Code:
-bash-2.05b# ls /tmp
logs
-bash-2.05b# backItUp.sh
/root/bin/backItUp.sh source destination
-bash-2.05b# backItUp.sh /tmp/logs/
/root/bin/backItUp.sh source destination
-bash-2.05b# backItUp.sh /tmp/logs/ backup hi
/root/bin/backItUp.sh source destination
-bash-2.05b# backItUp.sh /tmp/log backup
Source: /tmp/log does not exist.
-bash-2.05b# backItUp.sh /tmp/logs/ backup-tmp-logs
tar: Removing leading `/' from member names
tmp/logs/
tmp/logs/08022011-Mike.log
tmp/logs/09022011-Mariyam.log
tmp/logs/09022011-Mike.log
tmp/logs/28022011-Ken.log
tmp/logs/09022011-Ken.log
tmp/logs/10022011-Mariyam.log
tmp/logs/11022011-Mike.log
Archiving completed successfully.
/tmp/logs/ successfully deleted.
backup-tmp-logs-28-02-2011.tar: 93.2% -- replaced with backup-tmp-logs-28-02-2011.tar.gz
Compressing completed successfully.
backup-tmp-logs-28-02-2011.tar.gz: OK
gzip testing completed successfully.
-rw-r--r-- 1 root root 747 Feb 28 21:58 backup-tmp-logs-28-02-2011.tar.gz
#######DONE#######.
-bash-2.05b# ls -ltr
total 60
drwx------ 2 root root 4096 Feb 4 21:46 Mail
-rwxr-xr-x 1 root root 95 Feb 9 20:45 print-mails-sender
-rwxr-xr-x 1 root root 152 Feb 9 22:00 mail.sh
-rw-r--r-- 1 root root 159 Feb 11 21:52 out
-rw------- 1 root root 0 Feb 11 21:53 mbox
-rw-r--r-- 1 root root 17658 Feb 28 21:11 --help.tgz
-rw-r--r-- 1 root root 8515 Feb 28 21:44 backup-28-02-2011.tar.gz
drwxr-xr-x 3 root root 4096 Feb 28 21:54 bin
-rwxr-xr-x 1 root root 946 Feb 28 21:57 backItUp.sh
-rw-r--r-- 1 root root 747 Feb 28 21:58 backup-tmp-logs-28-02-2011.tar.gz
-bash-2.05b# ls -l /tmp
total 0
-bash-2.05b# file backup-tmp-logs-28-02-2011.tar.gz
backup-tmp-logs-28-02-2011.tar.gz: gzip compressed data, was "backup-tmp-logs-28-02-2011.tar", from Unix
-bash-2.05b# gzip -l backup-tmp-logs-28-02-2011.tar.gz
compressed uncompressed ratio uncompressed_name
747 10240 93.2% backup-tmp-logs-28-02-2011.tar
-bash-2.05b# gunzip backup-tmp-logs-28-02-2011.tar.gz
-bash-2.05b# ls -l
total 68
-rwxr-xr-x 1 root root 946 Feb 28 21:57 backItUp.sh
-rw-r--r-- 1 root root 8515 Feb 28 21:44 backup-28-02-2011.tar.gz
-rw-r--r-- 1 root root 10240 Feb 28 21:58 backup-tmp-logs-28-02-2011.tar
drwxr-xr-x 3 root root 4096 Feb 28 21:54 bin
-rw-r--r-- 1 root root 17658 Feb 28 21:11 --help.tgz
drwx------ 2 root root 4096 Feb 4 21:46 Mail
-rwxr-xr-x 1 root root 152 Feb 9 22:00 mail.sh
-rw------- 1 root root 0 Feb 11 21:53 mbox
-rw-r--r-- 1 root root 159 Feb 11 21:52 out
-rwxr-xr-x 1 root root 95 Feb 9 20:45 print-mails-sender
-bash-2.05b# tar -vxf backup-tmp-logs-28-02-2011.tar
tmp/logs/
tmp/logs/08022011-Mike.log
tmp/logs/09022011-Mariyam.log
tmp/logs/09022011-Mike.log
tmp/logs/28022011-Ken.log
tmp/logs/09022011-Ken.log
tmp/logs/10022011-Mariyam.log
tmp/logs/11022011-Mike.log
-bash-2.05b# ls -ltr
total 72
drwx------ 2 root root 4096 Feb 4 21:46 Mail
-rwxr-xr-x 1 root root 95 Feb 9 20:45 print-mails-sender
-rwxr-xr-x 1 root root 152 Feb 9 22:00 mail.sh
-rw-r--r-- 1 root root 159 Feb 11 21:52 out
-rw------- 1 root root 0 Feb 11 21:53 mbox
-rw-r--r-- 1 root root 17658 Feb 28 21:11 --help.tgz
-rw-r--r-- 1 root root 8515 Feb 28 21:44 backup-28-02-2011.tar.gz
drwxr-xr-x 3 root root 4096 Feb 28 21:54 bin
-rwxr-xr-x 1 root root 946 Feb 28 21:57 backItUp.sh
-rw-r--r-- 1 root root 10240 Feb 28 21:58 backup-tmp-logs-28-02-2011.tar
drwxr-xr-x 3 root root 4096 Feb 28 22:00 tmp
-bash-2.05b# ls -l tmp
total 4
drwxrwxrwx 2 root root 4096 Feb 28 20:50 logs
-bash-2.05b# ls -lR tmp
tmp:
total 4
drwxrwxrwx 2 root root 4096 Feb 28 20:50 logs
tmp/logs:
total 28
-rw-r--r-- 1 apache apache 144 Feb 8 21:46 08022011-Mike.log
-rw-r--r-- 1 apache apache 143 Feb 9 14:33 09022011-Ken.log
-rw-r--r-- 1 apache apache 267 Feb 10 21:36 09022011-Mariyam.log
-rw-r--r-- 1 apache apache 188 Feb 9 14:22 09022011-Mike.log
-rw-r--r-- 1 apache apache 687 Feb 10 21:37 10022011-Mariyam.log
-rw-r--r-- 1 apache apache 1525 Feb 11 15:36 11022011-Mike.log
-rw-r--r-- 1 apache apache 176 Feb 28 20:50 28022011-Ken.log
-bash-2.05b#
Just go through the above test to see how I have used the script.
When you hjave copied and pasted the script do not forget to make it executable by:
Code:
chmod +x scriptName.sh
Let me know if you have any problem. I will see it tomorrow. Take care and good night!
You should be able to see examples in your /etc/logrotate.conf and if it includes /etc/logrotate.d (as RHEL/CentOS do) then in the files under that directory.
Thank you very much. I have gone through the script that you posted and i didn't find anywhere that you are trying to find the files ending in .log extension in the source directory.
I maybe wrong, can you please clarify.
Thank you very much. I have gone through the script that you posted and i didn't find anywhere that you are trying to find the files ending in .log extension in the source directory.
I maybe wrong, can you please clarify.
I got reminded of it when I was going back by cab that you would ask me that question. Wait for some minutes.
Hello devUnix, did you get a chance to modify your script ?
basically we want to be doing this :
1. in the Source directory, we need to create a tar of all the .log files(there will be other files in this source directory like *.properties etc).
2. make a gzip of the above created tar file.
3. copy this gzip file to the destination directory.
4. delete all the *.log files in the source directory.
You can speficy even a complete directory as shown in the test below.
Have a look at this test.
Code:
-bash-2.05b# backItUp.sh /tmp/logs/*.txt
Destination Backup File Name: text
/tmp/logs/a.txt
/tmp/logs/b.txt
Archiving completed successfully.
/tmp/logs/a.txt successfully deleted.
/tmp/logs/b.txt successfully deleted.
text-01-03-2011.tar: 98.9% -- replaced with text-01-03-2011.tar.gz
Compressing completed successfully.
text-01-03-2011.tar.gz: OK
gzip testing completed successfully.
-rw-r--r-- 1 root root 152 Mar 1 17:24 text-01-03-2011.tar.gz
#######DONE#######.
-bash-2.05b# ls -ltr /tmp/logs/
total 0
-rw-r--r-- 1 root root 0 Mar 1 17:22 b.log
-rw-r--r-- 1 root root 0 Mar 1 17:22 a.log
-rw-r--r-- 1 root root 0 Mar 1 17:22 2.tmp
-rw-r--r-- 1 root root 0 Mar 1 17:22 2.log
-rw-r--r-- 1 root root 0 Mar 1 17:22 1.tmp
-rw-r--r-- 1 root root 0 Mar 1 17:22 1.log
-bash-2.05b# backItUp.sh /tmp/logs
Destination Backup File Name: logs
/tmp/logs/
/tmp/logs/1.log
/tmp/logs/2.log
/tmp/logs/1.tmp
/tmp/logs/2.tmp
/tmp/logs/a.log
/tmp/logs/b.log
Archiving completed successfully.
/tmp/logs successfully deleted.
logs-01-03-2011.tar: 98.3% -- replaced with logs-01-03-2011.tar.gz
Compressing completed successfully.
logs-01-03-2011.tar.gz: OK
gzip testing completed successfully.
-rw-r--r-- 1 root root 214 Mar 1 17:25 logs-01-03-2011.tar.gz
#######DONE#######.
-bash-2.05b# ls -ltr /tmp/logs/
ls: /tmp/logs/: No such file or directory
-bash-2.05b#
Isn't that what you wanted?
The first version of my script can be used to backup any directory that may contain any types of files.
But this second version is more advanced and enhanced. I give you the script here and recommend that you test it before you trust it.
Note that the format of executing the script is:
backItUp.sh /myDir/*.log <ENTER>
OR
backItUp.sh /myDir/ <ENTER>
As soon as you hit the Enter / Return key, you will be prompted to specify a file name which will be used to name the compressed archive or backup file because we want to store everything in one single container- the archive.
Basch Script Name: backItUp.sh
Code:
###############################################################################
####### Script Name : backItUp.sh #######
####### Purpose : Taking Backup of Specified Files #######
####### Developer : Dev #######
####### Email : dk_mahadeva@yahoo.com #######
###############################################################################
#!/bin/bash
if [ `echo $* | awk '{print length}'` -eq 0 ]; then
echo -e "Source is Required.\n"
echo -e `basename $0` " source\n"
exit 1
fi
for File in $*; do
if [ ! -e $File ]; then
echo -e "$File does not exist.\n"
exit 1
fi
done
echo -n "Destination Backup File Name: "
read
if [ `echo $REPLY | awk '{print length}'` -eq 0 ]; then
echo -e "Destination Backup File Name is Required.\n"
exit 1
fi
BackupFile=${REPLY}-$(date +%d-%m-%Y).tar
tar -vPcf ${BackupFile} $*
if [ "$?" != "0" ]; then
echo -e "tar failed.\n"
exit 1
else
echo -e "Archiving completed successfully.\n"
fi
for File in $*; do
rm -rf $File
if [ "$?" != "0" ]; then
echo -e "Deleting $File failed.\n"
else
echo -e "$File successfully deleted.\n"
fi
done
gzip -v ${BackupFile}
if [ "$?" != "0" ]; then
echo -e "gzip failed.\n"
exit 1
else
echo -e "Compressing completed successfully.\n"
fi
gzip -vt ${BackupFile}.gz
if [ "$?" != "0" ]; then
echo -e "gzip testing failed.\n"
exit 1
else
echo -e "gzip testing completed successfully.\n"
fi
ls -l ${BackupFile}.gz
if [ "$?" != "0" ]; then
echo -e "Some unknown errors have occured.\n"
exit 1
else
echo "#######DONE#######."
fi
exit 0
The above script, as I have created, is almost fully error-protected. It can handle possible errors such as incorrect file or directory name or path and other errors that may come up when tar and gzip utilities are used.
If you find it useful then consider sending me $ 100000000... do not faint - I am just kidding.
Chandra, bhai mene tumhari last comment nahi padhi. Lekin mene jo script likhi hai tum use check karo. It will do the job efficiently. If you need any further modifications or changes or enhancement, let me know it. Agar comments ki jarurat ho script me samjhne ke liye to bata dena, mein samjadunga. It is not much difficult. You can easily get it. By the way, where do you work?
I just did some more tests backuping directories and files separately. The script works fantastically!
I am going to automate the restore activity which however requires only two steps / commands:
Code:
gunzip bash-scripts-01-03-2011.tar.gz
and
Code:
tar -vxf bash-scripts-01-03-2011.tar
You can view the contents of the archive without extracting them from it (I was once asked in an interview how to do that):
Hi yaar, thanks for all the help.. maine aapki hindi mein comments nahin padatha.. sach mein accha laga aapke comments pad kar..
array yaar.. mein ne test kiya aapki nayi script se.. par wo error aya hai..
mein buffalo NY mein kaam karta hoon yaar.. aap kaha se ho ?
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.