LinuxQuestions.org - Writing a shell script for logs movement to remote location.

- Programming (https://www.linuxquestions.org/questions/programming-9/)

- - Writing a shell script for logs movement to remote location. (https://www.linuxquestions.org/questions/programming-9/writing-a-shell-script-for-logs-movement-to-remote-location-4175450618/)

Writing a shell script for logs movement to remote location.

Hi,

I have to write a shell script for automatic logs movement and compression weekly and monthly on remote server.
So, I'm facing some problems as I am not expert in shell scripting.

I have to move my sarg logs weekly from source server to destination server.
Sarg logs generate daily, weekly and monthly with different names as per dates, for example--

daily logs name format---
2013Feb02-2013Feb02
2013Feb01-2013Feb01

weekly logs name format---
2013Jan27-2013Feb02

monthly logs name format---
2013Jan01-2013Jan31

Now, how will I write a shell script which will move such named logs from source server to destination server? Daily and weekly logs at the end of each week and monthly logs at the end of month.
What will be the syntax or pattern that will be used to catch the dates?

I was trying to write a script but I got struck at one step-

Quote:

rsync -avze ssh (?????) root@<ip_address_of_remote_server>:/logs/Linux_Server_Logs/sarg/Feb-2013/

where, ????? is the format of the dates which I need to get, that I used as an example in my first post.

To get date formats, you can use the date cmd with the relevant options http://linux.die.net/man/1/date eg

Code:

date +%y%b%d

13Feb21

Quote:

rsync -avze ssh 2013Feb05-2013Feb05 2013Feb06-2013Feb06 2013Feb07-2013Feb07 2013Feb08-2013Feb08 2013Feb09-2013Feb09 2013Feb10-2013Feb10 2013Feb11-2013Feb11 <IP_address_of_remote_server>:/logs/Linux_Server_Logs/sarg/Feb-2013/daily/

This I do weekly and same case is for the weekly logs also. But what would be the code for the script where it chooses the date automatically for one week, like this--

2013Feb01-2013Feb01 - 2013Feb07-2013Feb07

Then again for the next week-

2013Feb08-2013Feb08 - 2013Feb14-2013Feb14

Is there any update from anyone?

You could place all log files in 1 directory and once your script is done with movement at remote server it should move all these files from this directory to another directory so next time script will pick up whatever it founds at this directory. By doing this you dont need to worry about applying date formats of these files.

Quote:

Originally Posted by Satyaveer Arya (Post 4896508)

Hi chrism,

I got it. I already know that date format. But my concern is what would be the code if I have to write from 1st to 7th date.

For example--
I do move logs manually using this command mentioned below:

This I do weekly and same case is for the weekly logs also. But what would be the code for the script where it chooses the date automatically for one week, like this--

2013Feb01-2013Feb01 - 2013Feb07-2013Feb07

Then again for the next week-

2013Feb08-2013Feb08 - 2013Feb14-2013Feb14

More Date Magic:

Code:

Today=`date +"%Y_%m_%d"`

OneDayAgo=`(date +"%Y_%m_%d" --date="1 days ago")`

TwoDaysAgo=`(date +"%Y_%m_%d" --date="2 days ago")`

ThreeDaysAgo=`(date +"%Y_%m_%d" --date="3 days ago")`

FourDaysAgo=`(date +"%Y_%m_%d" --date="4 days ago")`

FiveDaysAgo=`(date +"%Y_%m_%d" --date="5 days ago")`

SixDaysAgo=`(date +"%Y_%m_%d" --date="6 days ago")`

SevenDaysAgo=`(date +"%Y_%m_%d" --date="7 days ago")`

EightDaysAgo=`(date +"%Y_%m_%d" --date="8 days ago")`

NineDaysAgo=`(date +"%Y_%m_%d" --date="9 days ago")`

TenDaysAgo=`(date +"%Y_%m_%d" --date="10 days ago")`



FirstDay

date -d "-1 month -$(($(date +%d)-1)) days"

Lastday

date -d "-$(date +%d) days -1 month"

Stolen from http://www.linuxquestions.org/questi...9/#post4789927

HTH.

Hi Habitual,

Code:

[root@proxy ~]# cat sarg_script

ssh <dest._server_ip> 'cd /backup && if [ -d $(date +"%b-%Y") ]; 

then 

echo "directory exists"

else 

mkdir -p $(date +"%b-%Y")/{daily,weekly,monthly} 

fi'



Today=`date +"%Y%b%d"`

OneDayAgo=`(date +"%Y%b%d" --date="1 days ago")`

TwoDaysAgo=`(date +"%Y%b%d" --date="2 days ago")`

ThreeDaysAgo=`(date +"%Y%b%d" --date="3 days ago")`

FourDaysAgo=`(date +"%Y%b%d" --date="4 days ago")`

FiveDaysAgo=`(date +"%Y%b%d" --date="5 days ago")`

SixDaysAgo=`(date +"%Y%b%d" --date="6 days ago")`

SevenDaysAgo=`(date +"%Y%b%d" --date="7 days ago")`



rsync -avze ssh /var/www/sarg/daily/$OneDayAgo-$OneDayAgo $TwoDaysAgo-$TwoDaysAgo root@<dest._server_ip>:/backup/$(date +"%b-%Y")/daily



ssh <dest._server_ip> 'cd /backup/$(date +"%b-%Y")/daily && tar -zvcf $OneDayAgo-$OneDayAgo.tar.gz $OneDayAgo-$OneDayAgo && tar -zvcf $TwoDaysAgo-$TwoDaysAgo.tar.gz $TwoDaysAgo-$TwoDaysAgo && rm -rf $OneDayAgo-$OneDayAgo $TwoDaysAgo-$TwoDaysAgo'

I created this above mentioned script for testing of two days logs movement and compression. If I run this script it gives me this below mentioned error:

Quote:

rsync error: some files could not be transferred (code 23) at main.c(892) [sender=2.6.8]
tar: -: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors

This script copies only one date's directory and do not compresses on destination server.

The output on destination server in the destination directory is as follows :

Quote:

2013Mar08-2013Mar08 -.tar.gz

It copies the 2013Mar08-2013Mar08 directory and creates -.tar.gz file.

Quote:

du -sh ./-.tar.gz
4.0K ./-.tar.gz

du -sh 2013Mar08-2013Mar08
577M 2013Mar08-2013Mar08

If I create the same script for only one date's directory then script executes successfully.

So, my concern is why the above mentioned script is not running properly for two or more than two date's directories?
I will be very thankful to you.

Code:

ssh <dest._server_ip> 'cd /backup/$(date +"%b-%Y")/

should maybe be

Code:

ssh <dest._server_ip> 'cd /backup/$(date +"%Y%b%d")/

sticks out the most.

subscribed with interest...

This bash?

Edit: I would
tar+gz locally
then
move stuff around via rsync.

but that's just "me".

Hi Habitual,

Quote:

ssh <dest._server_ip> 'cd /backup && if [ -d $(date +"%b-%Y") ];
then
echo "directory exists"
else
mkdir -p $(date +"%b-%Y")/{daily,weekly,monthly}
fi'

Today=`date +"%Y%b%d"`
OneDayAgo=`(date +"%Y%b%d" --date="1 days ago")`
TwoDaysAgo=`(date +"%Y%b%d" --date="2 days ago")`
ThreeDaysAgo=`(date +"%Y%b%d" --date="3 days ago")`
FourDaysAgo=`(date +"%Y%b%d" --date="4 days ago")`
FiveDaysAgo=`(date +"%Y%b%d" --date="5 days ago")`
SixDaysAgo=`(date +"%Y%b%d" --date="6 days ago")`
SevenDaysAgo=`(date +"%Y%b%d" --date="7 days ago")`

rsync -avze ssh /var/www/sarg/daily/$OneDayAgo-$OneDayAgo root@<dest._server_ip>:/backup/$(date +"%b-%Y")/daily
rsync -avze ssh /var/www/sarg/daily/$TwoDaysAgo-$TwoDaysAgo root@<dest._server_ip>:/backup/$(date +"%b-%Y")/daily
rsync -avze ssh /var/www/sarg/daily/$ThreeDaysAgo-$ThreeDaysAgo root@<dest._server_ip>:/backup/$(date +"%b-%Y")/daily
rsync -avze ssh /var/www/sarg/daily/$FourDaysAgo-$FourDaysAgo root@<dest._server_ip>:/backup/$(date +"%b-%Y")/daily
rsync -avze ssh /var/www/sarg/daily/$FiveDaysAgo-$FiveDaysAgo root@<dest._server_ip>:/backup/$(date +"%b-%Y")/daily
rsync -avze ssh /var/www/sarg/daily/$SixDaysAgo-$SixDaysAgo root@<dest._server_ip>:/backup/$(date +"%b-%Y")/daily
rsync -avze ssh /var/www/sarg/daily/$SevenDaysAgo-$SevenDaysAgo root@<dest._server_ip>:/backup/$(date +"%b-%Y")/daily

ssh <dest._server_ip> 'cd /backup/$(date +"%b-%Y")/daily
tar -zvcf $(date +"%Y%b%d" --date="1 days ago")-$(date +"%Y%b%d" --date="1 days ago").tar.gz $(date +"%Y%b%d" --date="1 days ago")-$(date +"%Y%b%d" --date="1 days ago")
tar -zvcf $(date +"%Y%b%d" --date="2 days ago")-$(date +"%Y%b%d" --date="2 days ago").tar.gz $(date +"%Y%b%d" --date="2 days ago")-$(date +"%Y%b%d" --date="2 days ago")
tar -zvcf $(date +"%Y%b%d" --date="3 days ago")-$(date +"%Y%b%d" --date="3 days ago").tar.gz $(date +"%Y%b%d" --date="3 days ago")-$(date +"%Y%b%d" --date="3 days ago")
tar -zvcf $(date +"%Y%b%d" --date="4 days ago")-$(date +"%Y%b%d" --date="4 days ago").tar.gz $(date +"%Y%b%d" --date="4 days ago")-$(date +"%Y%b%d" --date="4 days ago")
tar -zvcf $(date +"%Y%b%d" --date="5 days ago")-$(date +"%Y%b%d" --date="5 days ago").tar.gz $(date +"%Y%b%d" --date="5 days ago")-$(date +"%Y%b%d" --date="5 days ago")
tar -zvcf $(date +"%Y%b%d" --date="6 days ago")-$(date +"%Y%b%d" --date="6 days ago").tar.gz $(date +"%Y%b%d" --date="6 days ago")-$(date +"%Y%b%d" --date="6 days ago")
tar -zvcf $(date +"%Y%b%d" --date="7 days ago")-$(date +"%Y%b%d" --date="7 days ago").tar.gz $(date +"%Y%b%d" --date="7 days ago")-$(date +"%Y%b%d" --date="7 days ago")'

ssh <dest._server_ip> 'cd /backup/$(date +"%b-%Y")/daily && rm -rf $(date +"%Y%b%d" --date="1 days ago")-$(date +"%Y%b%d" --date="1 days ago") $(date +"%Y%b%d" --date="2 days ago")-$(date +"%Y%b%d" --date="2 days ago") $(date +"%Y%b%d" --date="3 days ago")-$(date +"%Y%b%d" --date="3 days ago") $(date +"%Y%b%d" --date="4 days ago")-$(date +"%Y%b%d" --date="4 days ago") $(date +"%Y%b%d" --date="5 days ago")-$(date +"%Y%b%d" --date="5 days ago") $(date +"%Y%b%d" --date="6 days ago")-$(date +"%Y%b%d" --date="6 days ago") $(date +"%Y%b%d" --date="7 days ago")-$(date +"%Y%b%d" --date="7 days ago")'

This above mentioned script is working fine. I made myself correct. You can check from my previous post.
I was wondering if I can make this script short. Can anyone help me, please?

If you say it is working, who am I to argue? :)

Shorter? Probably!

Have a Great Day!

Hi Habitual,

Quote:

Edit: I would
tar+gz locally
then
move stuff around via rsync.

I already know that idea but problem on my side is that our proxy server do run out of space at the end of the week. So, that's why I went for this idea in the above script. Make me correct if I'm wrong or please give me some better solution so that I can work on that too.

Thanks for your suggestion in advance. :)