Linux - NewbieThis Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place!
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I install you all sorts of Linux cluster and configure the world, but what I never really learned (about time I know) is shell / bash scripting
Once a month I go through my Linux server and zip up the weblogs with an awefully simply poor writen DIY script.
Code:
custnum=1
for customer in `ls /home | grep -v mysql | grep -v titan`
do
echo $custnum
cd /home/$customer/logs
tar cvfz jun2008.tar.gz log.200806*
for logfile in log.200806*
do
rm $logfile
echo $logfile
done
custnum=`expr $custnum + 1`
done
What I am looking for is finding a way to use a cronjob instead without manually changing these lines
Code:
tar cvfz jun2008.tar.gz log.200806*
for logfile in log.200806*
I need a way for the script to query the previous month (script should run 1st of the month) and "zips" up the logfiles accordingly. So if i.e. the script runs February 1st, is should automaticlly go through every folder and tar up all log.201001*
Makes sense ?
I came so far to show the last month as number
Code:
date +"%m" --date='1 month ago'
and written word
Code:
date --date='1 month ago' +%B
Now I have to somehow throw everything in one pot and make a script out of it
When testing, run your script with sh -x yourscript to see what's going on.
jlinkels
wow thanks a lot ...
That seems to work almost perfectly.
Just does a weird thing and saves the tar in the specific log folder, i.e. /home/customer but also in the home folder directly using
Code:
mask=log.$(date -d "1 month ago" +%Y%m)*
name=$(date -d "1 month ago" +%b%Y).tar.gz
custnum=1
for customer in `ls /home | grep -v mysql | grep -v titan`
do
echo $custnum
cd /home/$customer/logs
tar cvzf $name $mask
for logfile in $mask
do
rm $logfile
echo $logfile
done
custnum=`expr $custnum + 1`
done
custnum=1
for customer in `ls /home | grep -v mysql | grep -v titan`
do
echo $custnum
cd /home/$customer/logs
tar cvfz jun2008.tar.gz log.200806*
for logfile in log.200806*
do
rm $logfile
echo $logfile
done
custnum=`expr $custnum + 1`
done
for customer in `ls /home | grep -v mysql | grep -v titan`
Add a correct shebang in the first line:
Code:
#!/bin/bash
Then use the real bash capabilities:
Code:
shopt -s extglob
for CUSTOMER in /home/!(*mysql*|*titan*)
do...
bash has also some mathematical capabilities,
Code:
custnum=`expr $custnum + 1`
could be changed to
Code:
custnum=$(($custnum+1))
Postcript: you should really use quotation marks when dealing with file and path names that are declared using variables, or you are going to have a hard time in the future:
Code:
custnum=1
for customer in `ls /home | grep -v mysql | grep -v titan`
do
echo $custnum
cd "/home/$customer/logs"
tar cvfz jun2008.tar.gz log.200806*
for logfile in log.200806*
do
rm "$logfile"
echo "$logfile"
done
custnum=`expr $custnum + 1`
done
Thanks guys for your input .. it is all working now ..
Only a big bug which I need another solution for .. if you accidentally run the script twice you end up overwriting the .tar.gz file with an empty one ..
Not really back to the drawing board 100% -- just add some basic sanity checking to the script:
If the destination file exists already, then either
A) abort entirely
B) backup the existing file, and proceed writing the new one (even if it's empty)
C) write the new file with a different name, or a suffix of some sort.
D) Prompt the user for advice.
Since it appears that the script is interactive, I would go with option (D). If the script is NOT interactive, I'd go with option (B). There are probably more than these ABCD options you could do, but these should give you ideas.
First of all thanks guys that you bear with me on this one
Actually not all working now :/
Quote:
Originally Posted by jlinkels
In scripting a lot is possible, but not create 2 tar files in 2 locations with just one tar command.
Something I don't want - but it happened
As for i92guboj' suggestions - I tried to change the script accordingly but it didn't work at all, probably did something wrong .. Like I say - I am a noob
Let me show what I have done so far
Code:
#!/bin/bash
mask=log.$(date -d "1 month ago" +%Y%m)*
name=$(date -d "1 month ago" +%b%Y).tar.gz
custnum=1
for customer in `ls /home | grep -v mysql | grep -v titan`
do
echo $custnum
cd "/home/$customer/logs"
tar cvzf /home/$customer/logs/$name $mask
for logfile in $mask
do
rm $logfile
echo $logfile
done
custnum=`expr $custnum + 1`
done
So basically I need to compress all logfiles within each customer folder underneath /home, i.e.
/home/somewebsite/logs
/home/anotherwebsite/logs
where
/home/mysql
/home/titan
need to be excluded, where titan is a program which has its own logfiles and must not be touched.
This is the current situation prior executing the script on a test server :
Here you can see multiple logfiles from Nov/Dec 2009 and January 2010
So if I would now execute the same script again it would overwrite "Dec2009.tar.gz" with an empty file.
Quote:
Originally Posted by GrapefruiTgirl
Not really back to the drawing board 100% -- just add some basic sanity checking to the script:
If the destination file exists already, then either
A) abort entirely
B) backup the existing file, and proceed writing the new one (even if it's empty)
C) write the new file with a different name, or a suffix of some sort.
D) Prompt the user for advice.
Since it appears that the script is interactive, I would go with option (D). If the script is NOT interactive, I'd go with option (B). There are probably more than these ABCD options you could do, but these should give you ideas.
Sasha
(A) of some sort where (C) is an option. On the server where I need a script like this urgently, the original compressed file has to remain under /home/<customer>/logs and should not be copied to anywhere. I don't mind if it simply creates a new file with an additional character, for example December2009.1.tar.gz or something like this. When I said "some sort" (A) I mean not abort the whole script, simply skip this file and jump to the next logs folder etc.
Makes sense ?
Like I say - I am completely new to bash scripting and stuff which may seem easy for you is a whole different world to me
All I remember when it comes to scripting is
10 Print "Hello"
20 Goto 10
Last edited by utw-mephisto; 01-26-2010 at 05:25 PM.
(for option A) Before the tar command, add a line:
Code:
[ -e "$mask" ] && break
This way, if the destination file already exists, the script will skip the rest of the loop,and move to the next customer.
I'm not sure at this moment, why it's looking in those 'weird' locations, but I speculate it may have something to do with the way you are still using the `ls` command to grab all the folder names.
Honestly, two things are making it hard (for me anyway) to figure out what's going on:
1) You aren't formatting your code, like indenting the contents of loops, if/then statements, etc.., and
2) running the script with -x is making the output brutally confusing. It is good for TESTING, like in a small environment with only a couple folders, but continuing to use +x after establishing the script is basically working, is making output hard to read (for you as well as us )
(for option A) Before the tar command, add a line:
Code:
[ -e "$mask" ] && break
This way, if the destination file already exists, the script will skip the rest of the loop,and move to the next customer.
I'm not sure at this moment, why it's looking in those 'weird' locations, but I speculate it may have something to do with the way you are still using the `ls` command to grab all the folder names.
Honestly, two things are making it hard (for me anyway) to figure out what's going on:
1) You aren't formatting your code, like indenting the contents of loops, if/then statements, etc.., and
2) running the script with -x is making the output brutally confusing. It is good for TESTING, like in a small environment with only a couple folders, but continuing to use +x after establishing the script is basically working, is making output hard to read (for you as well as us )
Sasha
Fair enough, jlinkels asked for it which was the reason I run it with +x
I know my formatting is horrible .. it is really the first script I ever wrote .. My last script contained just multiple wget lines to download multiple files
I mean I am willing to learn no doubt but if someone has the time to show me how you would do a little script like that properly I am all "ear"
I am just a bit tired of going through 80 server each month to compress those weblogs which is the reason I was looking into automating the whole thing in the first place
Last edited by utw-mephisto; 01-26-2010 at 06:03 PM.
Distribution: Debian /Jessie/Stretch/Sid, Linux Mint DE
Posts: 5,195
Rep:
Quote:
Originally Posted by utw-mephisto
Fair enough, jlinkels asked for it which was the reason I run it with +x
Why not, it reads easier than the NY Times!
A number of problems:
- If you check for existence of the tar file, you should not check:
if [ -e "$mask" ]
but if [ ! -e $name ]
because $name is the name of the tar file.
In pseudo code:
Code:
if [ ! -e $name ]
then
... do the tar thing
fi
Furthermore, you are trying to descend into files which are not directories. So, in your for loop, take the current name and check if it is a directory
Code:
if [ -d $customer ]
then
...
fi
-d checks if it is directory but you guessed that.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.