LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 01-25-2010, 07:52 PM   #1
utw-mephisto
Member
 
Registered: Apr 2005
Posts: 93

Rep: Reputation: 16
Question Some help with some basic scripting needed please


I install you all sorts of Linux cluster and configure the world, but what I never really learned (about time I know) is shell / bash scripting

Once a month I go through my Linux server and zip up the weblogs with an awefully simply poor writen DIY script.

Code:
custnum=1
for customer in `ls /home | grep -v mysql | grep -v titan`
do
echo $custnum
cd /home/$customer/logs
tar cvfz jun2008.tar.gz log.200806*
for logfile in log.200806*
do
rm $logfile
echo $logfile
done
custnum=`expr $custnum + 1`
done
What I am looking for is finding a way to use a cronjob instead without manually changing these lines

Code:
tar cvfz jun2008.tar.gz log.200806*
for logfile in log.200806*
I need a way for the script to query the previous month (script should run 1st of the month) and "zips" up the logfiles accordingly. So if i.e. the script runs February 1st, is should automaticlly go through every folder and tar up all log.201001*

Makes sense ?

I came so far to show the last month as number
Code:
date +"%m" --date='1 month ago'
and written word
Code:
date  --date='1 month ago' +%B
Now I have to somehow throw everything in one pot and make a script out of it

Any help is highly appreciated ...
 
Old 01-25-2010, 08:20 PM   #2
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire, Leeuwarden
Distribution: Debian /Jessie/Stretch/Sid, Linux Mint DE
Posts: 4,706

Rep: Reputation: 741Reputation: 741Reputation: 741Reputation: 741Reputation: 741Reputation: 741Reputation: 741
For the mask of log.200806* use this:

Code:
mask=log.$(date -d "1 month ago" +%Y%m)*
and then you simply do
Code:
for logfile in $mask
do
....
done
$(command) is the same as `command` but it is more readable.

To get you backup file name:

Code:
name=$(date -d "1 month ago" +%b%Y).tar.gz and use it
tar cvzf $name $mask
If you don't like the first capital, make it lowercase:
Code:
name=$(echo $(date -d "1 month ago" +%b%Y).tar.gz | tr [:upper:] [:lower:])
When testing, run your script with sh -x yourscript to see what's going on.

jlinkels
 
Old 01-26-2010, 12:18 PM   #3
utw-mephisto
Member
 
Registered: Apr 2005
Posts: 93

Original Poster
Rep: Reputation: 16
Quote:
Originally Posted by jlinkels View Post
For the mask of log.200806* use this:

Code:
mask=log.$(date -d "1 month ago" +%Y%m)*
and then you simply do
Code:
for logfile in $mask
do
....
done
$(command) is the same as `command` but it is more readable.

To get you backup file name:

Code:
name=$(date -d "1 month ago" +%b%Y).tar.gz and use it
tar cvzf $name $mask
If you don't like the first capital, make it lowercase:
Code:
name=$(echo $(date -d "1 month ago" +%b%Y).tar.gz | tr [:upper:] [:lower:])
When testing, run your script with sh -x yourscript to see what's going on.

jlinkels
wow thanks a lot ...

That seems to work almost perfectly.
Just does a weird thing and saves the tar in the specific log folder, i.e. /home/customer but also in the home folder directly using

Code:
mask=log.$(date -d "1 month ago" +%Y%m)*
name=$(date -d "1 month ago" +%b%Y).tar.gz
custnum=1
for customer in `ls /home | grep -v mysql | grep -v titan`
do
echo $custnum
cd /home/$customer/logs
tar cvzf $name $mask
for logfile in $mask
do
rm $logfile
echo $logfile
done
custnum=`expr $custnum + 1`
done
 
Old 01-26-2010, 12:36 PM   #4
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire, Leeuwarden
Distribution: Debian /Jessie/Stretch/Sid, Linux Mint DE
Posts: 4,706

Rep: Reputation: 741Reputation: 741Reputation: 741Reputation: 741Reputation: 741Reputation: 741Reputation: 741
Quote:
Originally Posted by utw-mephisto View Post
Just does a weird thing and saves the tar in the specific log folder, i.e. /home/customer but also in the home folder directly using
In scripting a lot is possible, but not create 2 tar files in 2 locations with just one tar command.

Can you create a test directory with a small number of files and run this script with sh -x?

To avoid any mistakes it is best to create the tar file using an absolute path. Just prepend the tar file name with /home/$customer/logs/

jlinkels
 
Old 01-26-2010, 12:43 PM   #5
GrapefruiTgirl
LQ Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551
Quote:
Originally Posted by jlinkels View Post
In scripting a lot is possible, but not create 2 tar files in 2 locations with just one tar command.
Oh, but whyyy?

Code:
tar cz <some-folder> | tee > tar1.tgz loc2/tar2.tgz
The above will tar-zip <some-folder> to two destinations, those being tar1 in the $CWD and tar2 in a subfolder called loc2.

Sasha
 
Old 01-26-2010, 12:56 PM   #6
i92guboj
Gentoo support team
 
Registered: May 2008
Location: Lucena, Córdoba (Spain)
Distribution: Gentoo
Posts: 4,074

Rep: Reputation: 386Reputation: 386Reputation: 386Reputation: 386
Code:
custnum=1
for customer in `ls /home | grep -v mysql | grep -v titan`
do
echo $custnum
cd /home/$customer/logs
tar cvfz jun2008.tar.gz log.200806*
for logfile in log.200806*
do
rm $logfile
echo $logfile
done
custnum=`expr $custnum + 1`
done
Listen to Sasha above, about the tar stuff.

I'll just add one of my favorites. Why you shouldn't parse the output from ls like that in this line:

Code:
for customer in `ls /home | grep -v mysql | grep -v titan`
Add a correct shebang in the first line:

Code:
#!/bin/bash
Then use the real bash capabilities:

Code:
shopt -s extglob
for CUSTOMER in /home/!(*mysql*|*titan*)
do...
bash has also some mathematical capabilities,

Code:
custnum=`expr $custnum + 1`
could be changed to

Code:
custnum=$(($custnum+1))
Postcript: you should really use quotation marks when dealing with file and path names that are declared using variables, or you are going to have a hard time in the future:

Code:
custnum=1
for customer in `ls /home | grep -v mysql | grep -v titan`
do
echo $custnum
cd "/home/$customer/logs"
tar cvfz jun2008.tar.gz log.200806*
for logfile in log.200806*
do
rm "$logfile"
echo "$logfile"
done
custnum=`expr $custnum + 1`
done

Last edited by i92guboj; 01-26-2010 at 01:08 PM.
 
Old 01-26-2010, 02:10 PM   #7
utw-mephisto
Member
 
Registered: Apr 2005
Posts: 93

Original Poster
Rep: Reputation: 16
Thanks guys for your input .. it is all working now ..

Only a big bug which I need another solution for .. if you accidentally run the script twice you end up overwriting the .tar.gz file with an empty one ..

Back to the drawing board
 
Old 01-26-2010, 03:04 PM   #8
GrapefruiTgirl
LQ Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551
Not really back to the drawing board 100% -- just add some basic sanity checking to the script:

If the destination file exists already, then either

A) abort entirely
B) backup the existing file, and proceed writing the new one (even if it's empty)
C) write the new file with a different name, or a suffix of some sort.
D) Prompt the user for advice.

Since it appears that the script is interactive, I would go with option (D). If the script is NOT interactive, I'd go with option (B). There are probably more than these ABCD options you could do, but these should give you ideas.

Sasha
 
Old 01-26-2010, 05:24 PM   #9
utw-mephisto
Member
 
Registered: Apr 2005
Posts: 93

Original Poster
Rep: Reputation: 16
First of all thanks guys that you bear with me on this one
Actually not all working now :/

Quote:
Originally Posted by jlinkels View Post
In scripting a lot is possible, but not create 2 tar files in 2 locations with just one tar command.
Something I don't want - but it happened


As for i92guboj' suggestions - I tried to change the script accordingly but it didn't work at all, probably did something wrong .. Like I say - I am a noob

Let me show what I have done so far
Code:
#!/bin/bash
mask=log.$(date -d "1 month ago" +%Y%m)*
name=$(date -d "1 month ago" +%b%Y).tar.gz
custnum=1
for customer in `ls /home | grep -v mysql | grep -v titan`
do
echo $custnum
cd "/home/$customer/logs"
tar cvzf /home/$customer/logs/$name $mask
for logfile in $mask
do
rm $logfile
echo $logfile
done
custnum=`expr $custnum + 1`
done
So basically I need to compress all logfiles within each customer folder underneath /home, i.e.

/home/somewebsite/logs
/home/anotherwebsite/logs

where

/home/mysql
/home/titan

need to be excluded, where titan is a program which has its own logfiles and must not be touched.

This is the current situation prior executing the script on a test server :

Here you can see multiple logfiles from Nov/Dec 2009 and January 2010

Code:
[root@centoslamp logs]# pwd
/home/new-website/logs
[root@centoslamp logs]# ls
error_log     log.20091117  log.20091122  log.20091125  log.20091202  log.20091207  log.20091214  log.20091218  log.20091231  log.20100107  log.20100115  log.20100126
log.20091109  log.20091120  log.20091123  log.20091130  log.20091203  log.20091209  log.20091215  log.20091221  log.20100104  log.20100111  log.20100124
log.20091110  log.20091121  log.20091124  log.20091201  log.20091206  log.20091211  log.20091217  log.20091222  log.20100106  log.20100112  log.20100125
Now I execute the script above :
Code:
[root@centoslamp home]# sh -x tidy.sh 
++ date -d '1 month ago' +%Y%m
+ mask='log.200912*'
++ date -d '1 month ago' +%b%Y
+ name=Dec2009.tar.gz
+ custnum=1
++ ls /home
++ grep -v mysql
++ grep -v titan
+ for customer in '`ls /home | grep -v mysql | grep -v titan`'
+ echo 1
1
+ cd /home/new-website/logs
+ tar cvzf /home/new-website/logs/Dec2009.tar.gz log.20091201 log.20091202 log.20091203 log.20091206 log.20091207 log.20091209 log.20091211 log.20091214 log.20091215 log.20091217 log.20091218 log.20091221 log.20091222 log.20091231
log.20091201
log.20091202
log.20091203
log.20091206
log.20091207
log.20091209
log.20091211
log.20091214
log.20091215
log.20091217
log.20091218
log.20091221
log.20091222
log.20091231
+ for logfile in '$mask'
+ rm log.20091201
+ echo log.20091201
log.20091201
+ for logfile in '$mask'
+ rm log.20091202
+ echo log.20091202
log.20091202
+ for logfile in '$mask'
+ rm log.20091203
+ echo log.20091203
log.20091203
+ for logfile in '$mask'
+ rm log.20091206
+ echo log.20091206
log.20091206
+ for logfile in '$mask'
+ rm log.20091207
+ echo log.20091207
log.20091207
+ for logfile in '$mask'
+ rm log.20091209
+ echo log.20091209
log.20091209
+ for logfile in '$mask'
+ rm log.20091211
+ echo log.20091211
log.20091211
+ for logfile in '$mask'
+ rm log.20091214
+ echo log.20091214
log.20091214
+ for logfile in '$mask'
+ rm log.20091215
+ echo log.20091215
log.20091215
+ for logfile in '$mask'
+ rm log.20091217
+ echo log.20091217
log.20091217
+ for logfile in '$mask'
+ rm log.20091218
+ echo log.20091218
log.20091218
+ for logfile in '$mask'
+ rm log.20091221
+ echo log.20091221
log.20091221
+ for logfile in '$mask'
+ rm log.20091222
+ echo log.20091222
log.20091222
+ for logfile in '$mask'
+ rm log.20091231
+ echo log.20091231
log.20091231
++ expr 1 + 1
+ custnum=2
+ for customer in '`ls /home | grep -v mysql | grep -v titan`'
+ echo 2
2
+ cd /home/tidy.sh/logs
tidy.sh: line 8: cd: /home/tidy.sh/logs: Not a directory
+ tar cvzf /home/tidy.sh/logs/Dec2009.tar.gz 'log.200912*'
tar: /home/tidy.sh/logs/Dec2009.tar.gz: Cannot open: Not a directory
tar: Error is not recoverable: exiting now
tar: log.200912*: Cannot stat: No such file or directory
tar: Child returned status 2
tar: Error exit delayed from previous errors
+ for logfile in '$mask'
+ rm 'log.200912*'
rm: cannot remove `log.200912*': No such file or directory
+ echo 'log.200912*'
log.200912*
++ expr 2 + 1
+ custnum=3
+ for customer in '`ls /home | grep -v mysql | grep -v titan`'
+ echo 3
3
+ cd /home/wibble.txt/logs
tidy.sh: line 8: cd: /home/wibble.txt/logs: Not a directory
+ tar cvzf /home/wibble.txt/logs/Dec2009.tar.gz 'log.200912*'
tar: /home/wibble.txt/logs/Dec2009.tar.gz: Cannot open: Not a directory
tar: Error is not recoverable: exiting now
tar: log.200912*: Cannot stat: No such file or directory
tar: Child returned status 2
tar: Error exit delayed from previous errors
+ for logfile in '$mask'
+ rm 'log.200912*'
rm: cannot remove `log.200912*': No such file or directory
+ echo 'log.200912*'
log.200912*
++ expr 3 + 1
+ custnum=4
To be perfectly honest with you the output is a bit chinese for me. It seems to check random paths, i.e. /home/tidy.sh/logs/

Which obviously doesn't exist.

It did seem to have created the archive though and deleted the files accordingly
Code:
[root@centoslamp home]# ls new-website/logs/
Dec2009.tar.gz  log.20091109  log.20091117  log.20091121  log.20091123  log.20091125  log.20100104  log.20100107  log.20100112  log.20100124  log.20100126
error_log       log.20091110  log.20091120  log.20091122  log.20091124  log.20091130  log.20100106  log.20100111  log.20100115  log.20100125
So if I would now execute the same script again it would overwrite "Dec2009.tar.gz" with an empty file.

Quote:
Originally Posted by GrapefruiTgirl View Post
Not really back to the drawing board 100% -- just add some basic sanity checking to the script:

If the destination file exists already, then either

A) abort entirely
B) backup the existing file, and proceed writing the new one (even if it's empty)
C) write the new file with a different name, or a suffix of some sort.
D) Prompt the user for advice.

Since it appears that the script is interactive, I would go with option (D). If the script is NOT interactive, I'd go with option (B). There are probably more than these ABCD options you could do, but these should give you ideas.

Sasha
(A) of some sort where (C) is an option. On the server where I need a script like this urgently, the original compressed file has to remain under /home/<customer>/logs and should not be copied to anywhere. I don't mind if it simply creates a new file with an additional character, for example December2009.1.tar.gz or something like this. When I said "some sort" (A) I mean not abort the whole script, simply skip this file and jump to the next logs folder etc.

Makes sense ?

Like I say - I am completely new to bash scripting and stuff which may seem easy for you is a whole different world to me

All I remember when it comes to scripting is

10 Print "Hello"
20 Goto 10


Last edited by utw-mephisto; 01-26-2010 at 05:25 PM.
 
Old 01-26-2010, 05:57 PM   #10
GrapefruiTgirl
LQ Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551
(for option A) Before the tar command, add a line:

Code:
[ -e "$mask" ] && break
This way, if the destination file already exists, the script will skip the rest of the loop,and move to the next customer.

I'm not sure at this moment, why it's looking in those 'weird' locations, but I speculate it may have something to do with the way you are still using the `ls` command to grab all the folder names.

Honestly, two things are making it hard (for me anyway) to figure out what's going on:
1) You aren't formatting your code, like indenting the contents of loops, if/then statements, etc.., and
2) running the script with -x is making the output brutally confusing. It is good for TESTING, like in a small environment with only a couple folders, but continuing to use +x after establishing the script is basically working, is making output hard to read (for you as well as us )

Sasha
 
Old 01-26-2010, 06:01 PM   #11
utw-mephisto
Member
 
Registered: Apr 2005
Posts: 93

Original Poster
Rep: Reputation: 16
Quote:
Originally Posted by GrapefruiTgirl View Post
(for option A) Before the tar command, add a line:

Code:
[ -e "$mask" ] && break
This way, if the destination file already exists, the script will skip the rest of the loop,and move to the next customer.

I'm not sure at this moment, why it's looking in those 'weird' locations, but I speculate it may have something to do with the way you are still using the `ls` command to grab all the folder names.

Honestly, two things are making it hard (for me anyway) to figure out what's going on:
1) You aren't formatting your code, like indenting the contents of loops, if/then statements, etc.., and
2) running the script with -x is making the output brutally confusing. It is good for TESTING, like in a small environment with only a couple folders, but continuing to use +x after establishing the script is basically working, is making output hard to read (for you as well as us )

Sasha
Fair enough, jlinkels asked for it which was the reason I run it with +x

I know my formatting is horrible .. it is really the first script I ever wrote .. My last script contained just multiple wget lines to download multiple files

I mean I am willing to learn no doubt but if someone has the time to show me how you would do a little script like that properly I am all "ear"

I am just a bit tired of going through 80 server each month to compress those weblogs which is the reason I was looking into automating the whole thing in the first place

Last edited by utw-mephisto; 01-26-2010 at 06:03 PM.
 
Old 01-26-2010, 06:08 PM   #12
GrapefruiTgirl
LQ Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551
Code:
#!/bin/bash

mask=log.$(date -d "1 month ago" +%Y%m)*
name=$(date -d "1 month ago" +%b%Y).tar.gz
custnum=1

for customer in `ls /home | grep -v mysql | grep -v titan`; do
  echo $custnum
  cd "/home/$customer/logs"
  tar cvzf /home/$customer/logs/$name $mask
 
  for logfile in $mask; do
    rm $logfile
    echo $logfile
  done

  custnum=`expr $custnum + 1`
done
Here's how I would format it (this is your script from post #9)

Cheers!
Sasha
 
Old 01-26-2010, 08:22 PM   #13
jlinkels
Senior Member
 
Registered: Oct 2003
Location: Bonaire, Leeuwarden
Distribution: Debian /Jessie/Stretch/Sid, Linux Mint DE
Posts: 4,706

Rep: Reputation: 741Reputation: 741Reputation: 741Reputation: 741Reputation: 741Reputation: 741Reputation: 741
Quote:
Originally Posted by utw-mephisto View Post
Fair enough, jlinkels asked for it which was the reason I run it with +x
Why not, it reads easier than the NY Times!

A number of problems:

- If you check for existence of the tar file, you should not check:
if [ -e "$mask" ]
but
if [ ! -e $name ]
because $name is the name of the tar file.
In pseudo code:
Code:
if [ ! -e $name ]
then
... do the tar thing
fi
Furthermore, you are trying to descend into files which are not directories. So, in your for loop, take the current name and check if it is a directory
Code:
if [ -d $customer ]
then
...
fi
-d checks if it is directory but you guessed that.

jlinkels

Last edited by jlinkels; 01-26-2010 at 08:23 PM.
 
Old 01-26-2010, 08:27 PM   #14
GrapefruiTgirl
LQ Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551
Quote:
- If you check for existence of the tar file, you should not check:
if [ -e "$mask" ]
but
if [ ! -e $name ]
because $name is the..
My mistake -- thanks for catching that jlinkels
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Basic Scripting karlochacon Linux - Newbie 6 11-12-2009 07:22 PM
Basic shell scripting help zipper777 Linux - Newbie 6 03-13-2006 01:58 PM
Basic bach scripting BruceC Linux - Newbie 4 02-25-2005 10:31 AM
very basic scripting question spatter9999 Programming 3 09-18-2003 02:36 AM
Basic BASH scripting question Dark_Helmet Programming 3 06-26-2003 06:45 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 10:09 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration