LinuxQuestions.org - Need help from pros in shell scripting?

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - Need help from pros in shell scripting? (https://www.linuxquestions.org/questions/linux-newbie-8/need-help-from-pros-in-shell-scripting-214929/)

Need help from pros in shell scripting?

I am new to linux and so unfortunately i was ask to create a script that can zip up the web logs that are generated by Apache and Tomcat web server. After zipping up the logs, it clears the old logs and send the zipped logs to a ftp server. Lastly, the script has to be executed once a month. I really have no clue on how to do it. Can anyone help me?

Well, it isn't very complicated... let's take it a step at a time.

Be sure to replace the proper values with the ones you need. Hope this helps.

Code:

#!/bin/bash

# Note change the line above to point to the location of bash on your system

REMOTESERVER='ftp.server.net'

USERNAME='username'

PASSWORD='password'





# First let's move to the area where the logs are

cd /var/log



# Now, let's zip the files we want.

bzip2 -k nameoflogfile1.log

bzip2 -k nameoflogfile2.log



# Zero out the logs for next time.  We put this here so we don't lose

# any data stored in the time it takes to transfer the logs by ftp.

cat /dev/null > nameoffile1.log

cat /dev/null > nameoffile2.log



# Now we need to ftp the files to the server

ftp -n $REMOTESERVER << EOF

quote USER $USERNAME

quote PASS $PASSWORD

cd /directorytoplace/backups/in

bin

put nameoffile1.log.bzip2

put nameoffile2.log.bzip2

quit

EOF



# Remove the bzip'ed logs

rm nameoffile1.log.bzip2

rm nameoffile2.log.bzip2



# And that is it.

Oh, and to execute it once a month.

type:
crontab -e

And add the following entry to your crontab

Code:

# Min Hr Day Mn Week command

0  0  1  *  * /path/to/shell/script.sh

This will execute it at midnight on the first day of the month every month. Make sure the script is executable... it will run as the user who has it in their crontab so put it in as root.

hello frob23,

just some questions...

What do these lines do?

Code:

cat /dev/null > nameoffile1.log

cat /dev/null > nameoffile2.log

And how come cron -e doesn't work on me? It says -e is an invalid option...

Thanks

I'm not frob23, but I can answar those questions :D

about those two lines
"cat" prints the content of a file.
the ">", replaces the content of the file after, with the output of what is before the ">".

så what is happening is, the content of /dev/null is put into nameoffil1.log.
Therefore the file is emptied...

I can't make "cron -e" work either.
but on my box, "crontab -e" works the way the other command is supposed to. Try that :)

lol, sorry about that.

The problem of writing advice when sitting at a windows computer without access to a *nix system.

Yes, crontab -e is what I meant to say. I will go back and edit that in for you. The explaination of the other lines above is correct. Remember to run the crontab command as root (and if you want to run the backup at another time and need help doing it just ask).

ahhh I see I see... I have seen output redirected to /dev/null before but never seen input used that way... Hmmm but what will happen to the files? Is it erased or the contents are just removed?

Oh and crontab -e works! only I don't know what to do with it... hehe Ima do some researching, but basically what I want to do is for it to perform a script I made every 24 hours....

The contents will be removed. We don't want to erase the files because they would be recreated with the wrong permissions, or owner, or maybe the programs would fail. It is safer to zero out the file instead.

You should run these commands singularly (not in the script) at least once so you can get a feel for how it will work.

Note:I am adding an option to the bzip2 command to make sure we don't lose the original files... pay attention to my original post... it is going to be updated in a second.

Quote:

Originally posted by pablowablo
ahhh I see I see... I have seen output redirected to /dev/null before but never seen input used that way... Hmmm but what will happen to the files? Is it erased or the contents are just removed?

Oh and crontab -e works! only I don't know what to do with it... hehe Ima do some researching, but basically what I want to do is for it to perform a script I made every 24 hours....

Okay the file is ordered like this:
minute hour day_of_month month day_of_week command

The way it works is that each time a field matches the command will be run. If we want a command to run at 08:15 each day we would use

15 8 * * * command

And if we want it to run once a day at 3am would would use:

0 3 * * * command

Note: Don't use:

* 3 * * * command

Because that will run EVERY minute from 3 to 3:59 remember that EVERY match will run the command. Read the man page for more information "man 5 crontab"

I tried it out myself ( cat /dev/null > something) ... neat trick! I'm gonna remember this.

Thanks for the tutorial on cron, I'm going to try some things.... hope it goes well...

P.S. to the original poster.. sorry if I hijacked your thread :(

Thanks for the help guys. And also thanks pablowablo for his questions which also benefited me. I have learnt quite a lot. Thanks so much!

Glad we helped.

[Just a note: remember to make sure this file (the script) is readable only by root -- or else anyone can check your username and password in the script for the remote server.]

Just something I thought about.

When you just empty the file, it'll have the same name. Therefore the bzipped file will have the same name, as the one on the ftp server, and it'll be overwritten.

Or have I missed something :confused:

Yes, you are correct... if the files are not moved or renamed than it will cause the files to have the same name on the server.

Here is one solution.

Code:

#!/bin/bash

# Note change the line above to point to the location of bash on your system

# Remember to change all the values and filenames below to reflect your system.

REMOTESERVER='ftp.server.net'

USERNAME='username'

PASSWORD='password'





# First let's move to the area where the logs are

cd /var/log



# Now, let's zip the files we want.

bzip2 -k nameoflogfile1.log

bzip2 -k nameoflogfile2.log



#Insert this after the bzip2 step to make sure the files are different by date.

FILE1=`date +%Y%m%d`.nameoflogfile1.log.bzip2

FILE2=`date +%Y%m%d`.nameoflogfile2.log.bzip2

mv nameoflogfile1.log.bzip2 $FILE1

mv nameoflogfile2.log.bzip2 $FILE2



# Zero out the logs for next time.  We put this here so we don't lose

# any data stored in the time it takes to transfer the logs by ftp.

cat /dev/null > nameoffile1.log

cat /dev/null > nameoffile2.log



# Now we need to ftp the files to the server

ftp -n $REMOTESERVER << EOF

quote USER $USERNAME

quote PASS $PASSWORD

cd /directorytoplace/backups/in

bin

put $FILE1

put $FILE2

quit

EOF



# Remove the bzip'ed logs

rm $FILE1

rm $FILE2



# And that is it.

This will create files in the format YYYYMMDD.nameoflogfile.log.bzip2
example: 20040808.foo.log

We format the date backwards so that it makes the files sort nicer. This might be a little overkill for what the poster wants though. And those are ` back ticks... which means it will replace the output of the command on the command line. Does that make sense?

This is a better complete script that uses variables for most things. So you can change them once at the top of the file for easier access. This is probably the version I would recommend you (or anyone) would use. The rest really were done one bit at a time and didn't represent a complete final script.

Edit: I also fixed the fact that I put .bzip2 when the real extension should have been .bz2 -- my bad... sorry.

Code:

#!/bin/bash

# Note change the line above to point to the location of bash on your system

# Remember to change all the values and filenames below to reflect your system.

# We have moved all these to the top so you only have to change them in one

# place.

REMOTESERVER='172.23.23.25'    # This is the server to store the logs on

USERNAME='backup'              # The username on the server

PASSWORD='xyzzy123'            # The password for the username

LOGFILE1=httpd-access.log      # This is one log file to backup

LOGFILE2=ipfw.today            # This is the other

LOCALLOGDIR=/var/log            # This is where the logs are locally

REMOTELOGDIR=/var/backups/foo  # This is where we are going to store them



# You should not need to change anything below unless you want to

# change how this script behaves (for example adding a third file).

# First let's move to the area where the logs are

cd $LOCALLOGDIR



# Now, let's zip the files we want. Leaving the old copies.

bzip2 -k $LOGFILE1

bzip2 -k $LOGFILE2



#Insert this after the bzip2 step to make sure the files are different by date.

FILE1=`date +%Y%m%d`.$LOGFILE1.bz2

FILE2=`date +%Y%m%d`.$LOGFILE2.bz2

mv $LOGFILE1.bz2 $FILE1

mv $LOGFILE2.bz2 $FILE2



# Zero out the logs for next time.  We put this here so we don't lose

# any data stored in the time it takes to transfer the logs by ftp.

cat /dev/null > $LOGFILE1

cat /dev/null > $LOGFILE2



# Now we need to ftp the files to the server

ftp -n $REMOTESERVER << EOF

quote USER $USERNAME

quote PASS $PASSWORD

cd $REMOTELOGDIR

bin

put $FILE1

put $FILE2

quit

EOF



# Remove the bzip'ed logs

rm $FILE1

rm $FILE2



# And that is it.