LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-05-2016, 03:02 PM   #1
blktux
LQ Newbie
 
Registered: Apr 2016
Posts: 11

Rep: Reputation: Disabled
Newbie Cronjob Question


Hello all,

I am trying to archive log files from my RabbitMQ server. The current plan is to use logrotate to rotate logs daily or if they grow to over 100M and then compress them after they are five days old keeping them in the same directory.

I was able to setup logrotate without issue however as the plan progressed to sending those compressed log archives to another server, I was unable to find out how to use crontab to transfer those files. Currently the cronjob is set up for moving files every Friday at 8PM from the rabbitmq directory that are older than 5 days. My problem is, I don't know how to copy the log files that are older than 5 days to the new server. I have tried to search on Google for ways to use the find command and scp in a cronjob but I have been unsuccessful.

Here is the command I have so far:
0 20 * * 5 find /var/log/rabbitmq/ -type f -mtime +5

I am not sure how to finish the command to scp those files to the other server.

Any advice is appreciated.

Thanks,
 
Old 04-05-2016, 03:21 PM   #2
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,552

Rep: Reputation: 2112Reputation: 2112Reputation: 2112Reputation: 2112Reputation: 2112Reputation: 2112Reputation: 2112Reputation: 2112Reputation: 2112Reputation: 2112Reputation: 2112
Your best bet is to write a script with all of the necessary commands in it, and then run that from cron.
 
Old 04-05-2016, 03:34 PM   #3
blktux
LQ Newbie
 
Registered: Apr 2016
Posts: 11

Original Poster
Rep: Reputation: Disabled
Is scp in the script the way to go? I'm trying to figure out how to do that best and have considered rsync.

I'm not sure if it matters but I plan on doing something similar for other services such as tomcat and other applications. Am I wrong in thinking that will require a script per application?
 
Old 04-05-2016, 03:43 PM   #4
suicidaleggroll
LQ Guru
 
Registered: Nov 2010
Location: Colorado
Distribution: OpenSUSE, CentOS
Posts: 5,552

Rep: Reputation: 2112Reputation: 2112Reputation: 2112Reputation: 2112Reputation: 2112Reputation: 2112Reputation: 2112Reputation: 2112Reputation: 2112Reputation: 2112Reputation: 2112
Either scp or rsync would work fine, they both work over ssh so just set up your ssh keys and you're golden.

Any 1-2 line jobs could be put straight in the crontab, much more than that and you're better off making a script.
 
Old 04-05-2016, 11:58 PM   #5
JJJCR
Senior Member
 
Registered: Apr 2010
Posts: 1,344

Rep: Reputation: 226Reputation: 226Reputation: 226
Not tested, before rolling out to production test with a dummy data.

0 20 * * 5 find /var/log/rabbitmq/ -type f -mtime +5 | mv $(xargs) /archive_log_file/my_other_directory

Try it and let us know. Cheers!!!
 
Old 04-06-2016, 02:21 AM   #6
MadeInGermany
Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 503

Rep: Reputation: 235Reputation: 235Reputation: 235
Isn't the $(xargs) run before the pipe, and mv doesn't read from the pipe?
Also, mv has -t option so can be used with find -exec
Code:
0 20 * * 5 find /var/log/rabbitmq/ -type f -mtime +5 -exec mv -t /archive_log_file/my_other_directory {} +
But, wasn't the question about scp?
 
Old 04-06-2016, 05:11 AM   #7
JJJCR
Senior Member
 
Registered: Apr 2010
Posts: 1,344

Rep: Reputation: 226Reputation: 226Reputation: 226
There are a lot of ways to do a task, actually I did test this command (0 20 * * 5 find /var/log/rabbitmq/ -type f -mtime +5 | mv $(xargs) /archive_log_file/my_other_directory) on a production server and it works fine.

But of course, I don't want to be responsible for any mistakes made if the command behave differently on another environment.

Test and test before applying the command to make sure it works as expected.
 
Old 04-06-2016, 06:41 AM   #8
MadeInGermany
Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 503

Rep: Reputation: 235Reputation: 235Reputation: 235
Pardon, I thought that the shell substitutes the $(xargs) before it creates the pipe.
Now I have tested it - and I was wrong: it works, at least in principle. (What happens if there are too many arguments?)

Still, for mv the -t option is preferrable (e.g. there are no problems with too many arguments or with special characters in file names).
 
1 members found this post helpful.
Old 04-06-2016, 08:15 AM   #9
blktux
LQ Newbie
 
Registered: Apr 2016
Posts: 11

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by JJJCR View Post
Not tested, before rolling out to production test with a dummy data.

0 20 * * 5 find /var/log/rabbitmq/ -type f -mtime +5 | mv $(xargs) /archive_log_file/my_other_directory

Try it and let us know. Cheers!!!
I'm confident that would work however I don't think I can use the mv command to get the files to a different machine.

I have log rotate setup to rotate the file daily and compress (.gz) files older than 5 days. Below is the cron job I plan on testing to move the compressed files to a different server for analysis at a later time:
0 20 * * 5 rsync -av /var/log/rabbitmq/*.gz <remote_server>:/logs/rabbitmq

I also plan on having a cron on the same machine to delete the compressed log files once they have been rsync'd to the remote server:
0 20 * * 6 find /var/log/rabbitmq/ type -f -mtime +7 -name "*.gz" -delete

Thoughts?
 
Old 04-06-2016, 12:51 PM   #10
MadeInGermany
Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 503

Rep: Reputation: 235Reputation: 235Reputation: 235
One problem you can solve by means of one cron job:
Code:
0 20 * * 5 rsync -av /var/log/rabbitmq/*.gz <remote_server>:/logs/rabbitmq && find /var/log/rabbitmq/ type -f -mtime +7 -name "*.gz" -delete
I.e. run the delete job only after the transfer ended successfully.
 
Old 04-06-2016, 01:02 PM   #11
MadeInGermany
Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 503

Rep: Reputation: 235Reputation: 235Reputation: 235
Perhaps your rsync has the wonderfull --remove-source-files option, then you can solve all potential problems:
Code:
0 20 * * 5 rsync --remove-source-files -av /var/log/rabbitmq/*.gz <remote_server>:/logs/rabbitmq
 
Old 04-06-2016, 01:48 PM   #12
hortageno
Member
 
Registered: Aug 2015
Distribution: Ubuntu 16.04 LTS
Posts: 234

Rep: Reputation: 65
... and if you really want to archive these log files, you should rename them by adding a time stamp or similar to the filename. Otherwise you will overwrite your logfiles with the new version every day.
 
Old 04-06-2016, 10:25 PM   #13
JJJCR
Senior Member
 
Registered: Apr 2010
Posts: 1,344

Rep: Reputation: 226Reputation: 226Reputation: 226
you can try this:

Quote:
tar cvzf - /path_to/local/folder | ssh <user_name>@<remote_server> "dd of=/path_on/remote/server/file_name_data.tar.gz"
But it you set it on cron you need to use ssh keys or sshpass to supply the password without user intervention.
 
Old 04-07-2016, 08:41 AM   #14
blktux
LQ Newbie
 
Registered: Apr 2016
Posts: 11

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by hortageno View Post
... and if you really want to archive these log files, you should rename them by adding a time stamp or similar to the filename. Otherwise you will overwrite your logfiles with the new version every day.
I realized this with my test. Unfortunately I am unable to get the size limit to work with logrotate so I am working on that before I am able to copy the log archives to another machine.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Linux Newbie stuck with cronjob oracle765 Linux - Newbie 1 01-08-2014 09:11 PM
question regarding a cronjob? Ajit Gunge Linux - Newbie 5 05-08-2012 12:54 AM
question regarding a cronjob? Ajit Gunge Linux - Software 3 05-03-2012 04:17 AM
Cronjob - Newbie - Please Help! Nanook Linux - Software 6 07-13-2005 11:44 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 05:33 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration