LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 08-08-2004, 05:12 AM   #1
aLiVe
LQ Newbie
 
Registered: Jul 2004
Posts: 5

Rep: Reputation: 0
Unhappy Need help from pros in shell scripting?


I am new to linux and so unfortunately i was ask to create a script that can zip up the web logs that are generated by Apache and Tomcat web server. After zipping up the logs, it clears the old logs and send the zipped logs to a ftp server. Lastly, the script has to be executed once a month. I really have no clue on how to do it. Can anyone help me?
 
Old 08-08-2004, 06:07 AM   #2
frob23
Senior Member
 
Registered: Jan 2004
Location: Roughly 29.467N / 81.206W
Distribution: OpenBSD, Ubuntu, FreeBSD
Posts: 1,449

Rep: Reputation: 48
Well, it isn't very complicated... let's take it a step at a time.

Be sure to replace the proper values with the ones you need. Hope this helps.

Code:
#!/bin/bash
# Note change the line above to point to the location of bash on your system
REMOTESERVER='ftp.server.net'
USERNAME='username'
PASSWORD='password'


# First let's move to the area where the logs are
cd /var/log

# Now, let's zip the files we want.
bzip2 -k nameoflogfile1.log
bzip2 -k nameoflogfile2.log

# Zero out the logs for next time.  We put this here so we don't lose
# any data stored in the time it takes to transfer the logs by ftp.
cat /dev/null > nameoffile1.log
cat /dev/null > nameoffile2.log

# Now we need to ftp the files to the server
ftp -n $REMOTESERVER << EOF
quote USER $USERNAME
quote PASS $PASSWORD
cd /directorytoplace/backups/in
bin
put nameoffile1.log.bzip2
put nameoffile2.log.bzip2
quit
EOF

# Remove the bzip'ed logs
rm nameoffile1.log.bzip2
rm nameoffile2.log.bzip2

# And that is it.

Last edited by frob23; 08-08-2004 at 10:30 AM.
 
Old 08-08-2004, 06:13 AM   #3
frob23
Senior Member
 
Registered: Jan 2004
Location: Roughly 29.467N / 81.206W
Distribution: OpenBSD, Ubuntu, FreeBSD
Posts: 1,449

Rep: Reputation: 48
Oh, and to execute it once a month.

type:
crontab -e

And add the following entry to your crontab
Code:
# Min Hr Day Mn Week command
0  0  1  *  * /path/to/shell/script.sh
This will execute it at midnight on the first day of the month every month. Make sure the script is executable... it will run as the user who has it in their crontab so put it in as root.

Last edited by frob23; 08-08-2004 at 08:08 AM.
 
Old 08-08-2004, 06:29 AM   #4
pablowablo
Member
 
Registered: Apr 2004
Posts: 131

Rep: Reputation: 15
hello frob23,

just some questions...

What do these lines do?

Code:
cat /dev/null > nameoffile1.log
cat /dev/null > nameoffile2.log
And how come cron -e doesn't work on me? It says -e is an invalid option...

Thanks
 
Old 08-08-2004, 07:43 AM   #5
Hjalte
LQ Newbie
 
Registered: May 2004
Location: Denmark
Distribution: Debian
Posts: 28

Rep: Reputation: 15
I'm not frob23, but I can answar those questions


about those two lines
"cat" prints the content of a file.
the ">", replaces the content of the file after, with the output of what is before the ">".

så what is happening is, the content of /dev/null is put into nameoffil1.log.
Therefore the file is emptied...


I can't make "cron -e" work either.
but on my box, "crontab -e" works the way the other command is supposed to. Try that
 
Old 08-08-2004, 07:51 AM   #6
frob23
Senior Member
 
Registered: Jan 2004
Location: Roughly 29.467N / 81.206W
Distribution: OpenBSD, Ubuntu, FreeBSD
Posts: 1,449

Rep: Reputation: 48
lol, sorry about that.

The problem of writing advice when sitting at a windows computer without access to a *nix system.

Yes, crontab -e is what I meant to say. I will go back and edit that in for you. The explaination of the other lines above is correct. Remember to run the crontab command as root (and if you want to run the backup at another time and need help doing it just ask).
 
Old 08-08-2004, 08:02 AM   #7
pablowablo
Member
 
Registered: Apr 2004
Posts: 131

Rep: Reputation: 15
ahhh I see I see... I have seen output redirected to /dev/null before but never seen input used that way... Hmmm but what will happen to the files? Is it erased or the contents are just removed?


Oh and crontab -e works! only I don't know what to do with it... hehe Ima do some researching, but basically what I want to do is for it to perform a script I made every 24 hours....

Last edited by pablowablo; 08-08-2004 at 08:04 AM.
 
Old 08-08-2004, 08:07 AM   #8
frob23
Senior Member
 
Registered: Jan 2004
Location: Roughly 29.467N / 81.206W
Distribution: OpenBSD, Ubuntu, FreeBSD
Posts: 1,449

Rep: Reputation: 48
The contents will be removed. We don't want to erase the files because they would be recreated with the wrong permissions, or owner, or maybe the programs would fail. It is safer to zero out the file instead.

You should run these commands singularly (not in the script) at least once so you can get a feel for how it will work.

Note:I am adding an option to the bzip2 command to make sure we don't lose the original files... pay attention to my original post... it is going to be updated in a second.
 
Old 08-08-2004, 08:17 AM   #9
frob23
Senior Member
 
Registered: Jan 2004
Location: Roughly 29.467N / 81.206W
Distribution: OpenBSD, Ubuntu, FreeBSD
Posts: 1,449

Rep: Reputation: 48
Quote:
Originally posted by pablowablo
ahhh I see I see... I have seen output redirected to /dev/null before but never seen input used that way... Hmmm but what will happen to the files? Is it erased or the contents are just removed?


Oh and crontab -e works! only I don't know what to do with it... hehe Ima do some researching, but basically what I want to do is for it to perform a script I made every 24 hours....
Okay the file is ordered like this:
minute hour day_of_month month day_of_week command

The way it works is that each time a field matches the command will be run. If we want a command to run at 08:15 each day we would use

15 8 * * * command

And if we want it to run once a day at 3am would would use:

0 3 * * * command

Note: Don't use:

* 3 * * * command

Because that will run EVERY minute from 3 to 3:59 remember that EVERY match will run the command. Read the man page for more information "man 5 crontab"
 
Old 08-08-2004, 08:33 AM   #10
pablowablo
Member
 
Registered: Apr 2004
Posts: 131

Rep: Reputation: 15
I tried it out myself ( cat /dev/null > something) ... neat trick! I'm gonna remember this.

Thanks for the tutorial on cron, I'm going to try some things.... hope it goes well...


P.S. to the original poster.. sorry if I hijacked your thread
 
Old 08-08-2004, 10:20 AM   #11
aLiVe
LQ Newbie
 
Registered: Jul 2004
Posts: 5

Original Poster
Rep: Reputation: 0
Thanks for the help guys. And also thanks pablowablo for his questions which also benefited me. I have learnt quite a lot. Thanks so much!
 
Old 08-08-2004, 10:26 AM   #12
frob23
Senior Member
 
Registered: Jan 2004
Location: Roughly 29.467N / 81.206W
Distribution: OpenBSD, Ubuntu, FreeBSD
Posts: 1,449

Rep: Reputation: 48
Glad we helped.

[Just a note: remember to make sure this file (the script) is readable only by root -- or else anyone can check your username and password in the script for the remote server.]
 
Old 08-08-2004, 11:52 AM   #13
Hjalte
LQ Newbie
 
Registered: May 2004
Location: Denmark
Distribution: Debian
Posts: 28

Rep: Reputation: 15
Just something I thought about.

When you just empty the file, it'll have the same name. Therefore the bzipped file will have the same name, as the one on the ftp server, and it'll be overwritten.

Or have I missed something
 
Old 08-08-2004, 12:33 PM   #14
frob23
Senior Member
 
Registered: Jan 2004
Location: Roughly 29.467N / 81.206W
Distribution: OpenBSD, Ubuntu, FreeBSD
Posts: 1,449

Rep: Reputation: 48
Yes, you are correct... if the files are not moved or renamed than it will cause the files to have the same name on the server.

Here is one solution.

Code:
#!/bin/bash
# Note change the line above to point to the location of bash on your system
# Remember to change all the values and filenames below to reflect your system.
REMOTESERVER='ftp.server.net'
USERNAME='username'
PASSWORD='password'


# First let's move to the area where the logs are
cd /var/log

# Now, let's zip the files we want.
bzip2 -k nameoflogfile1.log
bzip2 -k nameoflogfile2.log

#Insert this after the bzip2 step to make sure the files are different by date.
FILE1=`date +%Y%m%d`.nameoflogfile1.log.bzip2
FILE2=`date +%Y%m%d`.nameoflogfile2.log.bzip2
mv nameoflogfile1.log.bzip2 $FILE1
mv nameoflogfile2.log.bzip2 $FILE2

# Zero out the logs for next time.  We put this here so we don't lose
# any data stored in the time it takes to transfer the logs by ftp.
cat /dev/null > nameoffile1.log
cat /dev/null > nameoffile2.log

# Now we need to ftp the files to the server
ftp -n $REMOTESERVER << EOF
quote USER $USERNAME
quote PASS $PASSWORD
cd /directorytoplace/backups/in
bin
put $FILE1
put $FILE2
quit
EOF

# Remove the bzip'ed logs
rm $FILE1
rm $FILE2

# And that is it.
This will create files in the format YYYYMMDD.nameoflogfile.log.bzip2
example: 20040808.foo.log

We format the date backwards so that it makes the files sort nicer. This might be a little overkill for what the poster wants though. And those are ` back ticks... which means it will replace the output of the command on the command line. Does that make sense?
 
Old 08-08-2004, 12:50 PM   #15
frob23
Senior Member
 
Registered: Jan 2004
Location: Roughly 29.467N / 81.206W
Distribution: OpenBSD, Ubuntu, FreeBSD
Posts: 1,449

Rep: Reputation: 48
This is a better complete script that uses variables for most things. So you can change them once at the top of the file for easier access. This is probably the version I would recommend you (or anyone) would use. The rest really were done one bit at a time and didn't represent a complete final script.

Edit: I also fixed the fact that I put .bzip2 when the real extension should have been .bz2 -- my bad... sorry.

Code:
#!/bin/bash
# Note change the line above to point to the location of bash on your system
# Remember to change all the values and filenames below to reflect your system.
# We have moved all these to the top so you only have to change them in one
# place.
REMOTESERVER='172.23.23.25'     # This is the server to store the logs on
USERNAME='backup'               # The username on the server
PASSWORD='xyzzy123'             # The password for the username
LOGFILE1=httpd-access.log       # This is one log file to backup
LOGFILE2=ipfw.today             # This is the other
LOCALLOGDIR=/var/log            # This is where the logs are locally
REMOTELOGDIR=/var/backups/foo   # This is where we are going to store them

# You should not need to change anything below unless you want to
# change how this script behaves (for example adding a third file).
# First let's move to the area where the logs are
cd $LOCALLOGDIR

# Now, let's zip the files we want. Leaving the old copies.
bzip2 -k $LOGFILE1
bzip2 -k $LOGFILE2

#Insert this after the bzip2 step to make sure the files are different by date.
FILE1=`date +%Y%m%d`.$LOGFILE1.bz2
FILE2=`date +%Y%m%d`.$LOGFILE2.bz2
mv $LOGFILE1.bz2 $FILE1
mv $LOGFILE2.bz2 $FILE2

# Zero out the logs for next time.  We put this here so we don't lose
# any data stored in the time it takes to transfer the logs by ftp.
cat /dev/null > $LOGFILE1
cat /dev/null > $LOGFILE2

# Now we need to ftp the files to the server
ftp -n $REMOTESERVER << EOF
quote USER $USERNAME
quote PASS $PASSWORD
cd $REMOTELOGDIR
bin
put $FILE1
put $FILE2
quit
EOF

# Remove the bzip'ed logs
rm $FILE1
rm $FILE2

# And that is it.

Last edited by frob23; 08-08-2004 at 12:58 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Shell Scripting: Getting a pid and killing it via a shell script topcat Programming 15 10-28-2007 03:14 AM
shell interface vs shell scripting? I'm confused jcchenz Linux - Software 1 10-26-2005 04:32 PM
Shell scripting pros herve2001 Linux - Newbie 2 02-24-2005 02:13 PM
Need help from pros in shell script aLiVe Programming 1 08-08-2004 05:50 AM
Shell Scripting, I think... AMDPwred Linux - General 1 04-06-2002 01:01 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 01:52 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration