Some help with some basic scripting needed please

utw-mephisto · 01-25-2010, 07:52 PM

I install you all sorts of Linux cluster and configure the world, but what I never really learned (about time I know) is shell / bash scripting

Once a month I go through my Linux server and zip up the weblogs with an awefully simply poor writen DIY script.

Code:

custnum=1
for customer in `ls /home | grep -v mysql | grep -v titan`
do
echo $custnum
cd /home/$customer/logs
tar cvfz jun2008.tar.gz log.200806*
for logfile in log.200806*
do
rm $logfile
echo $logfile
done
custnum=`expr $custnum + 1`
done

What I am looking for is finding a way to use a cronjob instead without manually changing these lines

Code:

tar cvfz jun2008.tar.gz log.200806*
for logfile in log.200806*

I need a way for the script to query the previous month (script should run 1st of the month) and "zips" up the logfiles accordingly. So if i.e. the script runs February 1st, is should automaticlly go through every folder and tar up all log.201001*

Makes sense ?

I came so far to show the last month as number

Code:

date +"%m" --date='1 month ago'

and written word

Code:

date  --date='1 month ago' +%B

Now I have to somehow throw everything in one pot and make a script out of it

Any help is highly appreciated ...

jlinkels · 01-25-2010, 08:20 PM

For the mask of log.200806* use this:

Code:

mask=log.$(date -d "1 month ago" +%Y%m)*

and then you simply do

Code:

for logfile in $mask
do
....
done

$(command) is the same as `command` but it is more readable.

To get you backup file name:

Code:

name=$(date -d "1 month ago" +%b%Y).tar.gz and use it
tar cvzf $name $mask

If you don't like the first capital, make it lowercase:

Code:

name=$(echo $(date -d "1 month ago" +%b%Y).tar.gz | tr [:upper:] [:lower:])

When testing, run your script with sh -x yourscript to see what's going on.

jlinkels

utw-mephisto · 01-26-2010, 12:18 PM

Quote:

Originally Posted by jlinkels

For the mask of log.200806* use this:

Code:

mask=log.$(date -d "1 month ago" +%Y%m)*

and then you simply do

Code:

for logfile in $mask
do
....
done

$(command) is the same as `command` but it is more readable.

To get you backup file name:

Code:

name=$(date -d "1 month ago" +%b%Y).tar.gz and use it
tar cvzf $name $mask

If you don't like the first capital, make it lowercase:

Code:

name=$(echo $(date -d "1 month ago" +%b%Y).tar.gz | tr [:upper:] [:lower:])

When testing, run your script with sh -x yourscript to see what's going on.

jlinkels

wow thanks a lot ...

That seems to work almost perfectly.
Just does a weird thing and saves the tar in the specific log folder, i.e. /home/customer but also in the home folder directly using

Code:

mask=log.$(date -d "1 month ago" +%Y%m)*
name=$(date -d "1 month ago" +%b%Y).tar.gz
custnum=1
for customer in `ls /home | grep -v mysql | grep -v titan`
do
echo $custnum
cd /home/$customer/logs
tar cvzf $name $mask
for logfile in $mask
do
rm $logfile
echo $logfile
done
custnum=`expr $custnum + 1`
done

jlinkels · 01-26-2010, 12:36 PM

Quote:

Originally Posted by utw-mephisto

Just does a weird thing and saves the tar in the specific log folder, i.e. /home/customer but also in the home folder directly using

In scripting a lot is possible, but not create 2 tar files in 2 locations with just one tar command.

Can you create a test directory with a small number of files and run this script with sh -x?

To avoid any mistakes it is best to create the tar file using an absolute path. Just prepend the tar file name with /home/$customer/logs/

jlinkels

GrapefruiTgirl · 01-26-2010, 12:43 PM

Quote:

Originally Posted by jlinkels

In scripting a lot is possible, but not create 2 tar files in 2 locations with just one tar command.

Oh, but whyyy?

Code:

tar cz <some-folder> | tee > tar1.tgz loc2/tar2.tgz

The above will tar-zip <some-folder> to two destinations, those being tar1 in the $CWD and tar2 in a subfolder called loc2.

Sasha

i92guboj · 01-26-2010, 12:56 PM

Code:

custnum=1
for customer in `ls /home | grep -v mysql | grep -v titan`
do
echo $custnum
cd /home/$customer/logs
tar cvfz jun2008.tar.gz log.200806*
for logfile in log.200806*
do
rm $logfile
echo $logfile
done
custnum=`expr $custnum + 1`
done

Listen to Sasha above, about the tar stuff.

I'll just add one of my favorites. Why you shouldn't parse the output from ls like that in this line:

Code:

for customer in `ls /home | grep -v mysql | grep -v titan`

Add a correct shebang in the first line:

Code:

#!/bin/bash

Then use the real bash capabilities:

Code:

shopt -s extglob
for CUSTOMER in /home/!(*mysql*|*titan*)
do...

bash has also some mathematical capabilities,

Code:

custnum=`expr $custnum + 1`

could be changed to

Code:

custnum=$(($custnum+1))

Postcript: you should really use quotation marks when dealing with file and path names that are declared using variables, or you are going to have a hard time in the future:

Code:

custnum=1
for customer in `ls /home | grep -v mysql | grep -v titan`
do
echo $custnum
cd "/home/$customer/logs"
tar cvfz jun2008.tar.gz log.200806*
for logfile in log.200806*
do
rm "$logfile"
echo "$logfile"
done
custnum=`expr $custnum + 1`
done

utw-mephisto · 01-26-2010, 02:10 PM

Thanks guys for your input .. it is all working now ..

Only a big bug which I need another solution for .. if you accidentally run the script twice you end up overwriting the .tar.gz file with an empty one ..

Back to the drawing board

GrapefruiTgirl · 01-26-2010, 03:04 PM

Not really back to the drawing board 100% -- just add some basic sanity checking to the script:

If the destination file exists already, then either

A) abort entirely
B) backup the existing file, and proceed writing the new one (even if it's empty)
C) write the new file with a different name, or a suffix of some sort.
D) Prompt the user for advice.

Since it appears that the script is interactive, I would go with option (D). If the script is NOT interactive, I'd go with option (B). There are probably more than these ABCD options you could do, but these should give you ideas.

Sasha

utw-mephisto · 01-26-2010, 05:24 PM

First of all thanks guys that you bear with me on this one

Actually not all working now :/

Quote:

Originally Posted by jlinkels

In scripting a lot is possible, but not create 2 tar files in 2 locations with just one tar command.

Something I don't want - but it happened

As for i92guboj' suggestions - I tried to change the script accordingly but it didn't work at all, probably did something wrong .. Like I say - I am a noob

Let me show what I have done so far

Code:

#!/bin/bash
mask=log.$(date -d "1 month ago" +%Y%m)*
name=$(date -d "1 month ago" +%b%Y).tar.gz
custnum=1
for customer in `ls /home | grep -v mysql | grep -v titan`
do
echo $custnum
cd "/home/$customer/logs"
tar cvzf /home/$customer/logs/$name $mask
for logfile in $mask
do
rm $logfile
echo $logfile
done
custnum=`expr $custnum + 1`
done

So basically I need to compress all logfiles within each customer folder underneath /home, i.e.

/home/somewebsite/logs
/home/anotherwebsite/logs

where

/home/mysql
/home/titan

need to be excluded, where titan is a program which has its own logfiles and must not be touched.

This is the current situation prior executing the script on a test server :

Here you can see multiple logfiles from Nov/Dec 2009 and January 2010

Code:

[root@centoslamp logs]# pwd
/home/new-website/logs
[root@centoslamp logs]# ls
error_log     log.20091117  log.20091122  log.20091125  log.20091202  log.20091207  log.20091214  log.20091218  log.20091231  log.20100107  log.20100115  log.20100126
log.20091109  log.20091120  log.20091123  log.20091130  log.20091203  log.20091209  log.20091215  log.20091221  log.20100104  log.20100111  log.20100124
log.20091110  log.20091121  log.20091124  log.20091201  log.20091206  log.20091211  log.20091217  log.20091222  log.20100106  log.20100112  log.20100125

Now I execute the script above :

Code:

[root@centoslamp home]# sh -x tidy.sh 
++ date -d '1 month ago' +%Y%m
+ mask='log.200912*'
++ date -d '1 month ago' +%b%Y
+ name=Dec2009.tar.gz
+ custnum=1
++ ls /home
++ grep -v mysql
++ grep -v titan
+ for customer in '`ls /home | grep -v mysql | grep -v titan`'
+ echo 1
1
+ cd /home/new-website/logs
+ tar cvzf /home/new-website/logs/Dec2009.tar.gz log.20091201 log.20091202 log.20091203 log.20091206 log.20091207 log.20091209 log.20091211 log.20091214 log.20091215 log.20091217 log.20091218 log.20091221 log.20091222 log.20091231
log.20091201
log.20091202
log.20091203
log.20091206
log.20091207
log.20091209
log.20091211
log.20091214
log.20091215
log.20091217
log.20091218
log.20091221
log.20091222
log.20091231
+ for logfile in '$mask'
+ rm log.20091201
+ echo log.20091201
log.20091201
+ for logfile in '$mask'
+ rm log.20091202
+ echo log.20091202
log.20091202
+ for logfile in '$mask'
+ rm log.20091203
+ echo log.20091203
log.20091203
+ for logfile in '$mask'
+ rm log.20091206
+ echo log.20091206
log.20091206
+ for logfile in '$mask'
+ rm log.20091207
+ echo log.20091207
log.20091207
+ for logfile in '$mask'
+ rm log.20091209
+ echo log.20091209
log.20091209
+ for logfile in '$mask'
+ rm log.20091211
+ echo log.20091211
log.20091211
+ for logfile in '$mask'
+ rm log.20091214
+ echo log.20091214
log.20091214
+ for logfile in '$mask'
+ rm log.20091215
+ echo log.20091215
log.20091215
+ for logfile in '$mask'
+ rm log.20091217
+ echo log.20091217
log.20091217
+ for logfile in '$mask'
+ rm log.20091218
+ echo log.20091218
log.20091218
+ for logfile in '$mask'
+ rm log.20091221
+ echo log.20091221
log.20091221
+ for logfile in '$mask'
+ rm log.20091222
+ echo log.20091222
log.20091222
+ for logfile in '$mask'
+ rm log.20091231
+ echo log.20091231
log.20091231
++ expr 1 + 1
+ custnum=2
+ for customer in '`ls /home | grep -v mysql | grep -v titan`'
+ echo 2
2
+ cd /home/tidy.sh/logs
tidy.sh: line 8: cd: /home/tidy.sh/logs: Not a directory
+ tar cvzf /home/tidy.sh/logs/Dec2009.tar.gz 'log.200912*'
tar: /home/tidy.sh/logs/Dec2009.tar.gz: Cannot open: Not a directory
tar: Error is not recoverable: exiting now
tar: log.200912*: Cannot stat: No such file or directory
tar: Child returned status 2
tar: Error exit delayed from previous errors
+ for logfile in '$mask'
+ rm 'log.200912*'
rm: cannot remove `log.200912*': No such file or directory
+ echo 'log.200912*'
log.200912*
++ expr 2 + 1
+ custnum=3
+ for customer in '`ls /home | grep -v mysql | grep -v titan`'
+ echo 3
3
+ cd /home/wibble.txt/logs
tidy.sh: line 8: cd: /home/wibble.txt/logs: Not a directory
+ tar cvzf /home/wibble.txt/logs/Dec2009.tar.gz 'log.200912*'
tar: /home/wibble.txt/logs/Dec2009.tar.gz: Cannot open: Not a directory
tar: Error is not recoverable: exiting now
tar: log.200912*: Cannot stat: No such file or directory
tar: Child returned status 2
tar: Error exit delayed from previous errors
+ for logfile in '$mask'
+ rm 'log.200912*'
rm: cannot remove `log.200912*': No such file or directory
+ echo 'log.200912*'
log.200912*
++ expr 3 + 1
+ custnum=4

To be perfectly honest with you the output is a bit chinese for me. It seems to check random paths, i.e. /home/tidy.sh/logs/

Which obviously doesn't exist.

It did seem to have created the archive though and deleted the files accordingly

Code:

[root@centoslamp home]# ls new-website/logs/
Dec2009.tar.gz  log.20091109  log.20091117  log.20091121  log.20091123  log.20091125  log.20100104  log.20100107  log.20100112  log.20100124  log.20100126
error_log       log.20091110  log.20091120  log.20091122  log.20091124  log.20091130  log.20100106  log.20100111  log.20100115  log.20100125

So if I would now execute the same script again it would overwrite "Dec2009.tar.gz" with an empty file.

Quote:

Originally Posted by GrapefruiTgirl

Not really back to the drawing board 100% -- just add some basic sanity checking to the script:

If the destination file exists already, then either

A) abort entirely
B) backup the existing file, and proceed writing the new one (even if it's empty)
C) write the new file with a different name, or a suffix of some sort.
D) Prompt the user for advice.

Since it appears that the script is interactive, I would go with option (D). If the script is NOT interactive, I'd go with option (B). There are probably more than these ABCD options you could do, but these should give you ideas.

Sasha

(A) of some sort where (C) is an option. On the server where I need a script like this urgently, the original compressed file has to remain under /home/<customer>/logs and should not be copied to anywhere. I don't mind if it simply creates a new file with an additional character, for example December2009.1.tar.gz or something like this. When I said "some sort" (A) I mean not abort the whole script, simply skip this file and jump to the next logs folder etc.

Makes sense ?

Like I say - I am completely new to bash scripting and stuff which may seem easy for you is a whole different world to me

All I remember when it comes to scripting is

10 Print "Hello"
20 Goto 10

GrapefruiTgirl · 01-26-2010, 05:57 PM

(for option A) Before the tar command, add a line:

Code:

[ -e "$mask" ] && break

This way, if the destination file already exists, the script will skip the rest of the loop,and move to the next customer.

I'm not sure at this moment, why it's looking in those 'weird' locations, but I speculate it may have something to do with the way you are still using the `ls` command to grab all the folder names.

Honestly, two things are making it hard (for me anyway) to figure out what's going on:
1) You aren't formatting your code, like indenting the contents of loops, if/then statements, etc.., and
2) running the script with -x is making the output brutally confusing. It is good for TESTING, like in a small environment with only a couple folders, but continuing to use +x after establishing the script is basically working, is making output hard to read (for you as well as us

)

Sasha

utw-mephisto · 01-26-2010, 06:01 PM

Quote:

Originally Posted by GrapefruiTgirl

(for option A) Before the tar command, add a line:

Code:

[ -e "$mask" ] && break

This way, if the destination file already exists, the script will skip the rest of the loop,and move to the next customer.

I'm not sure at this moment, why it's looking in those 'weird' locations, but I speculate it may have something to do with the way you are still using the `ls` command to grab all the folder names.

Honestly, two things are making it hard (for me anyway) to figure out what's going on:
1) You aren't formatting your code, like indenting the contents of loops, if/then statements, etc.., and
2) running the script with -x is making the output brutally confusing. It is good for TESTING, like in a small environment with only a couple folders, but continuing to use +x after establishing the script is basically working, is making output hard to read (for you as well as us

)

Sasha

Fair enough, jlinkels asked for it which was the reason I run it with +x

I know my formatting is horrible .. it is really the first script I ever wrote .. My last script contained just multiple wget lines to download multiple files

I mean I am willing to learn no doubt but if someone has the time to show me how you would do a little script like that properly I am all "ear"

I am just a bit tired of going through 80 server each month to compress those weblogs which is the reason I was looking into automating the whole thing in the first place

GrapefruiTgirl · 01-26-2010, 06:08 PM

Code:

#!/bin/bash

mask=log.$(date -d "1 month ago" +%Y%m)*
name=$(date -d "1 month ago" +%b%Y).tar.gz
custnum=1

for customer in `ls /home | grep -v mysql | grep -v titan`; do
  echo $custnum
  cd "/home/$customer/logs"
  tar cvzf /home/$customer/logs/$name $mask
 
  for logfile in $mask; do
    rm $logfile
    echo $logfile
  done

  custnum=`expr $custnum + 1`
done

Here's how I would format it (this is your script from post #9)

Cheers!
Sasha

jlinkels · 01-26-2010, 08:22 PM

Quote:

Originally Posted by utw-mephisto

Fair enough, jlinkels asked for it which was the reason I run it with +x

Why not, it reads easier than the NY Times!

A number of problems:

- If you check for existence of the tar file, you should not check:
if [ -e "$mask" ]
but
if [ ! -e $name ]
because $name is the name of the tar file.
In pseudo code:

Code:

if [ ! -e $name ]
then
... do the tar thing
fi

Furthermore, you are trying to descend into files which are not directories. So, in your for loop, take the current name and check if it is a directory

Code:

if [ -d $customer ]
then
...
fi

-d checks if it is directory but you guessed that.

jlinkels

GrapefruiTgirl · 01-26-2010, 08:27 PM

Quote:

- If you check for existence of the tar file, you should not check:
if [ -e "$mask" ]
but
if [ ! -e $name ]
because $name is the..

My mistake -- thanks for catching that jlinkels