LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 02-18-2013, 03:03 AM   #1
Satyaveer Arya
Senior Member
 
Registered: May 2010
Location: Dehradun, Uttarakhand, India
Distribution: RHEL, CentOS, Debian, Oracle Solaris 10
Posts: 1,412

Rep: Reputation: 303Reputation: 303Reputation: 303Reputation: 303
Writing a shell script for logs movement to remote location.


Hi,

I have to write a shell script for automatic logs movement and compression weekly and monthly on remote server.
So, I'm facing some problems as I am not expert in shell scripting.

I have to move my sarg logs weekly from source server to destination server.
Sarg logs generate daily, weekly and monthly with different names as per dates, for example--

daily logs name format---
2013Feb02-2013Feb02
2013Feb01-2013Feb01

weekly logs name format---
2013Jan27-2013Feb02

monthly logs name format---
2013Jan01-2013Jan31

Now, how will I write a shell script which will move such named logs from source server to destination server? Daily and weekly logs at the end of each week and monthly logs at the end of month.
What will be the syntax or pattern that will be used to catch the dates?

Last edited by Satyaveer Arya; 02-21-2013 at 02:23 AM.
 
Old 02-18-2013, 03:09 AM   #2
Satyaveer Arya
Senior Member
 
Registered: May 2010
Location: Dehradun, Uttarakhand, India
Distribution: RHEL, CentOS, Debian, Oracle Solaris 10
Posts: 1,412

Original Poster
Rep: Reputation: 303Reputation: 303Reputation: 303Reputation: 303
I was trying to write a script but I got struck at one step-

Quote:
rsync -avze ssh (?????) root@<ip_address_of_remote_server>:/logs/Linux_Server_Logs/sarg/Feb-2013/
where, ????? is the format of the dates which I need to get, that I used as an example in my first post.
 
Old 02-21-2013, 12:20 AM   #3
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,264

Rep: Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028
To get date formats, you can use the date cmd with the relevant options http://linux.die.net/man/1/date eg
Code:
date +%y%b%d
13Feb21
 
Old 02-21-2013, 02:37 AM   #4
Satyaveer Arya
Senior Member
 
Registered: May 2010
Location: Dehradun, Uttarakhand, India
Distribution: RHEL, CentOS, Debian, Oracle Solaris 10
Posts: 1,412

Original Poster
Rep: Reputation: 303Reputation: 303Reputation: 303Reputation: 303
Hi chrism,

I got it. I already know that date format. But my concern is what would be the code if I have to write from 1st to 7th date.

For example--
I do move logs manually using this command mentioned below:

Quote:
rsync -avze ssh 2013Feb05-2013Feb05 2013Feb06-2013Feb06 2013Feb07-2013Feb07 2013Feb08-2013Feb08 2013Feb09-2013Feb09 2013Feb10-2013Feb10 2013Feb11-2013Feb11 <IP_address_of_remote_server>:/logs/Linux_Server_Logs/sarg/Feb-2013/daily/
This I do weekly and same case is for the weekly logs also. But what would be the code for the script where it chooses the date automatically for one week, like this--

2013Feb01-2013Feb01 - 2013Feb07-2013Feb07

Then again for the next week-

2013Feb08-2013Feb08 - 2013Feb14-2013Feb14
 
Old 03-07-2013, 07:55 AM   #5
Satyaveer Arya
Senior Member
 
Registered: May 2010
Location: Dehradun, Uttarakhand, India
Distribution: RHEL, CentOS, Debian, Oracle Solaris 10
Posts: 1,412

Original Poster
Rep: Reputation: 303Reputation: 303Reputation: 303Reputation: 303
Is there any update from anyone?
 
Old 03-07-2013, 08:41 PM   #6
p2006.prashant
LQ Newbie
 
Registered: Dec 2007
Location: India
Distribution: Redhat Linux 5,6.2, Solaris
Posts: 16
Blog Entries: 1

Rep: Reputation: 5
You could place all log files in 1 directory and once your script is done with movement at remote server it should move all these files from this directory to another directory so next time script will pick up whatever it founds at this directory. By doing this you dont need to worry about applying date formats of these files.
 
Old 03-08-2013, 06:04 AM   #7
Habitual
Senior Member
 
Registered: Jan 2011
Distribution: Undecided
Posts: 3,310
Blog Entries: 5

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
Quote:
Originally Posted by Satyaveer Arya View Post
Hi chrism,

I got it. I already know that date format. But my concern is what would be the code if I have to write from 1st to 7th date.

For example--
I do move logs manually using this command mentioned below:



This I do weekly and same case is for the weekly logs also. But what would be the code for the script where it chooses the date automatically for one week, like this--

2013Feb01-2013Feb01 - 2013Feb07-2013Feb07

Then again for the next week-

2013Feb08-2013Feb08 - 2013Feb14-2013Feb14
More Date Magic:
Code:
Today=`date +"%Y_%m_%d"`
OneDayAgo=`(date +"%Y_%m_%d" --date="1 days ago")`
TwoDaysAgo=`(date +"%Y_%m_%d" --date="2 days ago")`
ThreeDaysAgo=`(date +"%Y_%m_%d" --date="3 days ago")`
FourDaysAgo=`(date +"%Y_%m_%d" --date="4 days ago")`
FiveDaysAgo=`(date +"%Y_%m_%d" --date="5 days ago")`
SixDaysAgo=`(date +"%Y_%m_%d" --date="6 days ago")`
SevenDaysAgo=`(date +"%Y_%m_%d" --date="7 days ago")`
EightDaysAgo=`(date +"%Y_%m_%d" --date="8 days ago")`
NineDaysAgo=`(date +"%Y_%m_%d" --date="9 days ago")`
TenDaysAgo=`(date +"%Y_%m_%d" --date="10 days ago")`

FirstDay
date -d "-1 month -$(($(date +%d)-1)) days"
Lastday
date -d "-$(date +%d) days -1 month"
Stolen from http://www.linuxquestions.org/questi...9/#post4789927

HTH.
 
Old 03-09-2013, 06:18 AM   #8
Satyaveer Arya
Senior Member
 
Registered: May 2010
Location: Dehradun, Uttarakhand, India
Distribution: RHEL, CentOS, Debian, Oracle Solaris 10
Posts: 1,412

Original Poster
Rep: Reputation: 303Reputation: 303Reputation: 303Reputation: 303
Hi Habitual,

Code:
[root@proxy ~]# cat sarg_script
ssh <dest._server_ip> 'cd /backup && if [ -d $(date +"%b-%Y") ]; 
then 
echo "directory exists"
else 
mkdir -p $(date +"%b-%Y")/{daily,weekly,monthly} 
fi'

Today=`date +"%Y%b%d"`
OneDayAgo=`(date +"%Y%b%d" --date="1 days ago")`
TwoDaysAgo=`(date +"%Y%b%d" --date="2 days ago")`
ThreeDaysAgo=`(date +"%Y%b%d" --date="3 days ago")`
FourDaysAgo=`(date +"%Y%b%d" --date="4 days ago")`
FiveDaysAgo=`(date +"%Y%b%d" --date="5 days ago")`
SixDaysAgo=`(date +"%Y%b%d" --date="6 days ago")`
SevenDaysAgo=`(date +"%Y%b%d" --date="7 days ago")`

rsync -avze ssh /var/www/sarg/daily/$OneDayAgo-$OneDayAgo $TwoDaysAgo-$TwoDaysAgo root@<dest._server_ip>:/backup/$(date +"%b-%Y")/daily

ssh <dest._server_ip> 'cd /backup/$(date +"%b-%Y")/daily && tar -zvcf $OneDayAgo-$OneDayAgo.tar.gz $OneDayAgo-$OneDayAgo && tar -zvcf $TwoDaysAgo-$TwoDaysAgo.tar.gz $TwoDaysAgo-$TwoDaysAgo && rm -rf $OneDayAgo-$OneDayAgo $TwoDaysAgo-$TwoDaysAgo'
I created this above mentioned script for testing of two days logs movement and compression. If I run this script it gives me this below mentioned error:

Quote:
rsync error: some files could not be transferred (code 23) at main.c(892) [sender=2.6.8]
tar: -: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors
This script copies only one date's directory and do not compresses on destination server.

The output on destination server in the destination directory is as follows :

Quote:
2013Mar08-2013Mar08 -.tar.gz
It copies the 2013Mar08-2013Mar08 directory and creates -.tar.gz file.

Quote:
du -sh ./-.tar.gz
4.0K ./-.tar.gz

du -sh 2013Mar08-2013Mar08
577M 2013Mar08-2013Mar08
If I create the same script for only one date's directory then script executes successfully.

So, my concern is why the above mentioned script is not running properly for two or more than two date's directories?
I will be very thankful to you.
 
Old 03-09-2013, 07:57 AM   #9
Habitual
Senior Member
 
Registered: Jan 2011
Distribution: Undecided
Posts: 3,310
Blog Entries: 5

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
Code:
ssh <dest._server_ip> 'cd /backup/$(date +"%b-%Y")/
should maybe be
Code:
ssh <dest._server_ip> 'cd /backup/$(date +"%Y%b%d")/
sticks out the most.

subscribed with interest...

This bash?

Edit: I would
tar+gz locally
then
move stuff around via rsync.

but that's just "me".

Last edited by Habitual; 03-09-2013 at 08:00 AM.
 
Old 03-09-2013, 09:05 AM   #10
Satyaveer Arya
Senior Member
 
Registered: May 2010
Location: Dehradun, Uttarakhand, India
Distribution: RHEL, CentOS, Debian, Oracle Solaris 10
Posts: 1,412

Original Poster
Rep: Reputation: 303Reputation: 303Reputation: 303Reputation: 303
Hi Habitual,

Quote:
ssh <dest._server_ip> 'cd /backup && if [ -d $(date +"%b-%Y") ];
then
echo "directory exists"
else
mkdir -p $(date +"%b-%Y")/{daily,weekly,monthly}
fi'

Today=`date +"%Y%b%d"`
OneDayAgo=`(date +"%Y%b%d" --date="1 days ago")`
TwoDaysAgo=`(date +"%Y%b%d" --date="2 days ago")`
ThreeDaysAgo=`(date +"%Y%b%d" --date="3 days ago")`
FourDaysAgo=`(date +"%Y%b%d" --date="4 days ago")`
FiveDaysAgo=`(date +"%Y%b%d" --date="5 days ago")`
SixDaysAgo=`(date +"%Y%b%d" --date="6 days ago")`
SevenDaysAgo=`(date +"%Y%b%d" --date="7 days ago")`

rsync -avze ssh /var/www/sarg/daily/$OneDayAgo-$OneDayAgo root@<dest._server_ip>:/backup/$(date +"%b-%Y")/daily
rsync -avze ssh /var/www/sarg/daily/$TwoDaysAgo-$TwoDaysAgo root@<dest._server_ip>:/backup/$(date +"%b-%Y")/daily
rsync -avze ssh /var/www/sarg/daily/$ThreeDaysAgo-$ThreeDaysAgo root@<dest._server_ip>:/backup/$(date +"%b-%Y")/daily
rsync -avze ssh /var/www/sarg/daily/$FourDaysAgo-$FourDaysAgo root@<dest._server_ip>:/backup/$(date +"%b-%Y")/daily
rsync -avze ssh /var/www/sarg/daily/$FiveDaysAgo-$FiveDaysAgo root@<dest._server_ip>:/backup/$(date +"%b-%Y")/daily
rsync -avze ssh /var/www/sarg/daily/$SixDaysAgo-$SixDaysAgo root@<dest._server_ip>:/backup/$(date +"%b-%Y")/daily
rsync -avze ssh /var/www/sarg/daily/$SevenDaysAgo-$SevenDaysAgo root@<dest._server_ip>:/backup/$(date +"%b-%Y")/daily

ssh <dest._server_ip> 'cd /backup/$(date +"%b-%Y")/daily
tar -zvcf $(date +"%Y%b%d" --date="1 days ago")-$(date +"%Y%b%d" --date="1 days ago").tar.gz $(date +"%Y%b%d" --date="1 days ago")-$(date +"%Y%b%d" --date="1 days ago")
tar -zvcf $(date +"%Y%b%d" --date="2 days ago")-$(date +"%Y%b%d" --date="2 days ago").tar.gz $(date +"%Y%b%d" --date="2 days ago")-$(date +"%Y%b%d" --date="2 days ago")
tar -zvcf $(date +"%Y%b%d" --date="3 days ago")-$(date +"%Y%b%d" --date="3 days ago").tar.gz $(date +"%Y%b%d" --date="3 days ago")-$(date +"%Y%b%d" --date="3 days ago")
tar -zvcf $(date +"%Y%b%d" --date="4 days ago")-$(date +"%Y%b%d" --date="4 days ago").tar.gz $(date +"%Y%b%d" --date="4 days ago")-$(date +"%Y%b%d" --date="4 days ago")
tar -zvcf $(date +"%Y%b%d" --date="5 days ago")-$(date +"%Y%b%d" --date="5 days ago").tar.gz $(date +"%Y%b%d" --date="5 days ago")-$(date +"%Y%b%d" --date="5 days ago")
tar -zvcf $(date +"%Y%b%d" --date="6 days ago")-$(date +"%Y%b%d" --date="6 days ago").tar.gz $(date +"%Y%b%d" --date="6 days ago")-$(date +"%Y%b%d" --date="6 days ago")
tar -zvcf $(date +"%Y%b%d" --date="7 days ago")-$(date +"%Y%b%d" --date="7 days ago").tar.gz $(date +"%Y%b%d" --date="7 days ago")-$(date +"%Y%b%d" --date="7 days ago")'

ssh <dest._server_ip> 'cd /backup/$(date +"%b-%Y")/daily && rm -rf $(date +"%Y%b%d" --date="1 days ago")-$(date +"%Y%b%d" --date="1 days ago") $(date +"%Y%b%d" --date="2 days ago")-$(date +"%Y%b%d" --date="2 days ago") $(date +"%Y%b%d" --date="3 days ago")-$(date +"%Y%b%d" --date="3 days ago") $(date +"%Y%b%d" --date="4 days ago")-$(date +"%Y%b%d" --date="4 days ago") $(date +"%Y%b%d" --date="5 days ago")-$(date +"%Y%b%d" --date="5 days ago") $(date +"%Y%b%d" --date="6 days ago")-$(date +"%Y%b%d" --date="6 days ago") $(date +"%Y%b%d" --date="7 days ago")-$(date +"%Y%b%d" --date="7 days ago")'
This above mentioned script is working fine. I made myself correct. You can check from my previous post.
I was wondering if I can make this script short. Can anyone help me, please?
 
1 members found this post helpful.
Old 03-09-2013, 09:12 AM   #11
Habitual
Senior Member
 
Registered: Jan 2011
Distribution: Undecided
Posts: 3,310
Blog Entries: 5

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
If you say it is working, who am I to argue?

Shorter? Probably!

Have a Great Day!
 
Old 03-09-2013, 09:14 AM   #12
Satyaveer Arya
Senior Member
 
Registered: May 2010
Location: Dehradun, Uttarakhand, India
Distribution: RHEL, CentOS, Debian, Oracle Solaris 10
Posts: 1,412

Original Poster
Rep: Reputation: 303Reputation: 303Reputation: 303Reputation: 303
Hi Habitual,

Quote:
Edit: I would
tar+gz locally
then
move stuff around via rsync.
I already know that idea but problem on my side is that our proxy server do run out of space at the end of the week. So, that's why I went for this idea in the above script. Make me correct if I'm wrong or please give me some better solution so that I can work on that too.

Thanks for your suggestion in advance.
 
Old 04-08-2013, 07:05 PM   #13
Habitual
Senior Member
 
Registered: Jan 2011
Distribution: Undecided
Posts: 3,310
Blog Entries: 5

Rep: Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783Reputation: 783
http://www.cyberciti.biz/faq/howto-u...r-ssh-session/ ?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Need help with writing a c shell script? Ajit Gunge Linux - Newbie 5 06-21-2012 12:58 AM
How to call Shell Script on a remote server with remote servers env variables need Linux - Server 1 10-14-2009 08:37 PM
how can a shell script determine its own location when being sourced? DeuceNegative Linux - Software 5 02-21-2008 01:57 PM
Remote Script execution from the central location anil_garg5 Linux - General 5 05-11-2006 03:29 PM
Need Help in Writing a Shell Script. KptnKrill Linux - General 2 05-19-2003 05:44 AM


All times are GMT -5. The time now is 09:14 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration