LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 11-16-2015, 10:57 AM   #1
ndenial
LQ Newbie
 
Registered: Nov 2015
Posts: 7

Rep: Reputation: Disabled
Cron Job help to move directories older than 90 days


I have been frequenting this site more and more since I have become the owner/maintainer of a Debian based server. Previous experience has been with Ubuntu and Edubuntu LTSP. This will be my first post here and hope it is in the right area. So far I have been able to understand the basics but have yet to fully dive into any scripting, this will be my first foray into cron tasks as well.

System: Debian 8 based


I would like to Move directories containing logs older than 90 days to backup drive, maintaining folder structure. I seem to only be able to move files and a single child directory.


Folder structure (logs/year/month/day/hour):
/var/example/logs/2014/12/01/01/*files*
/var/example/logs/2014/12/01/01/sensor1/*files*
/var/example/logs/2014/12/01/01/sensor2/*files*

Issue: using
Code:
find /var/example/logs/* -type d -mtime +90 -exec -mv '{}' /mnt/archivedrive/archive/example/logs/ \;
appears to dump all files into a /example/logs/ directory and do not maintain structure.


I did not run the above code as is, instead I used ls -l where the -mv is. It returns
Code:
/var/example/logs/2014/12/01/01:
total 948
drwxrwxr-x 2 *exampleuser* *examplegroup*   4096 Dec 01 2014 *sensor1dir*
drwxrwxr-x 2 *exampleuser* *examplegroup*   4096 Dec 01 2014 *sensor1dir*
-rw-rw-r-- 1 *exampleuser* *examplegroup*   9268 Dec 01 2014 *file.1*
-rw-rw-r-- 1 *exampleuser* *examplegroup* 944609 Dec 01 2014 *file.2*
/mnt/archivedrive/archive/example/logs/:
total 0
That repeats for each date/hour and it appears it will dump the results into the /mnt/archivedrive/archive/example/logs/ folder without maintaining their folder structure.


I am intending to put this into my chron.daily, then follow up with a another which removes files older than 365 days from the archive drive.

Any help would be appreciated as I have spent a few days trying to come up with a solution.

Thank you!

Last edited by ndenial; 11-16-2015 at 12:18 PM.
 
Old 11-16-2015, 11:10 AM   #2
HMW
Member
 
Registered: Aug 2013
Location: Sweden
Distribution: Debian, Arch, Red Hat, CentOS
Posts: 773
Blog Entries: 3

Rep: Reputation: 369Reputation: 369Reputation: 369Reputation: 369
Quote:
Originally Posted by ndenial View Post
I did not run the above code as is, instead I used ls -l where the -mv is
Hi!

Ehm... so, why didn't you run your find command? Sorry, but I cannot make head or tails from your question!

Can you show the actual output of your find command?

Best regards,
HMW
 
Old 11-16-2015, 12:12 PM   #3
ndenial
LQ Newbie
 
Registered: Nov 2015
Posts: 7

Original Poster
Rep: Reputation: Disabled
Sorry for the confusion...

I ran this
Code:
find /var/example/logs/* -type d -mtime +90 -exec ls -l '{}' /mnt/archivedrive/archive/example/logs/ \;
to simulate this
Code:
find /var/example/logs/* -type d -mtime +90 -exec -mv '{}' /mnt/archivedrive/archive/example/logs/ \;
and got this:

Code:
Logserver1:~# find /var/ossim/logs/* -type d -mtime +347 -exec ls -l '{}' /mnt/wd2tb/test/ \;
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/09:
total 936
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9145 Dec  3  2014 data.stats
-rw-rw-r-- 1 logserver ossim 936420 Dec  4  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/04:
total 936
drwxrwxr-x 2 logserver ossim   4096 Dec  3  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9019 Dec  2  2014 data.stats
-rw-rw-r-- 1 logserver ossim 936420 Dec  3  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/02:
total 936
drwxrwxr-x 2 logserver ossim   4096 Dec  3  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9189 Dec  2  2014 data.stats
-rw-rw-r-- 1 logserver ossim 936420 Dec  3  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/12:
total 932
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9080 Dec  3  2014 data.stats
-rw-rw-r-- 1 logserver ossim 932324 Dec  4  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/13:
total 940
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9260 Dec  3  2014 data.stats
-rw-rw-r-- 1 logserver ossim 940516 Dec  4  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/06:
total 936
drwxrwxr-x 2 logserver ossim   4096 Dec  3  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9121 Dec  2  2014 data.stats
-rw-rw-r-- 1 logserver ossim 936420 Dec  3  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/11:
total 932
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9059 Dec  3  2014 data.stats
-rw-rw-r-- 1 logserver ossim 932324 Dec  4  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/00:
total 932
drwxrwxr-x 2 logserver ossim   4096 Dec  3  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9046 Dec  2  2014 data.stats
-rw-rw-r-- 1 logserver ossim 932324 Dec  3  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/08:
total 940
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9255 Dec  3  2014 data.stats
-rw-rw-r-- 1 logserver ossim 940516 Dec  4  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/14:
total 948
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor1
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor2
-rw-rw-r-- 1 logserver ossim   9397 Dec  3  2014 data.stats
-rw-rw-r-- 1 logserver ossim 944609 Dec  4  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/10:
total 936
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9121 Dec  3  2014 data.stats
-rw-rw-r-- 1 logserver ossim 936420 Dec  4  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/07:
total 964
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor1
-rw-rw-r-- 1 logserver ossim  10043 Dec  3  2014 data.stats
-rw-rw-r-- 1 logserver ossim 965092 Dec  4  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/01:
total 932
drwxrwxr-x 2 logserver ossim   4096 Dec  3  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9097 Dec  2  2014 data.stats
-rw-rw-r-- 1 logserver ossim 932324 Dec  3  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/03:
total 932
drwxrwxr-x 2 logserver ossim   4096 Dec  3  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9120 Dec  2  2014 data.stats
-rw-rw-r-- 1 logserver ossim 932324 Dec  3  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/15:
total 948
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor1
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor2
-rw-rw-r-- 1 logserver ossim   9268 Dec  3  2014 data.stats
-rw-rw-r-- 1 logserver ossim 944609 Dec  4  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/05:
total 932
drwxrwxr-x 2 logserver ossim   4096 Dec  3  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9053 Dec  2  2014 data.stats
-rw-rw-r-- 1 logserver ossim 932324 Dec  3  2014 mindex.inx

Which makes me think that it will only copy the files and directories listed to /mnt/wd2tb/test/ not /mnt/wd2tb/test/2014/12/03/05, /mnt/wd2tb/test/2014/12/03/06 etc etc.

I don't want to do the actual move yet until I am sure that I wont screw myself and make a mess of the directory. I do not have a test environment for this.

Last edited by ndenial; 11-16-2015 at 01:18 PM.
 
Old 11-16-2015, 01:40 PM   #4
hoes
Member
 
Registered: Sep 2005
Distribution: debian, linux from scratch
Posts: 177

Rep: Reputation: 38
Actually you run the command

Code:
ls -l dir /mnt/wd2tb/test/
every time. The files in the specific dir pop up, so you want to do an action with this directory.
Instead of ls -l do something useful.
Run
Code:
    find /var/example/logs/* -type d -mtime +90 -exec echo mv '{}' /mnt/archivedrive/archive/example/logs/ \;
to check if the move commands are as you would expect.
 
Old 11-16-2015, 01:57 PM   #5
ndenial
LQ Newbie
 
Registered: Nov 2015
Posts: 7

Original Poster
Rep: Reputation: Disabled
Hoes,

Thank you, I searched for a command that would better "simulate" this for me but I could not find it for the life of me.

The result of that is:

Quote:
Logserver:~# find /var/ossim/logs/* -type d -mtime +347 -exec echo mv '{}' /mnt/wd2tb/test/ \;
mv /var/ossim/logs/2014/12/03/09 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/04 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/02 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/12 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/13 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/06 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/11 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/16 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/00 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/08 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/14 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/10 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/07 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/01 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/17 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/03 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/15 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/18 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/05 /mnt/wd2tb/test/
Logserver:~#
If I understand this output, it does not appear to carry over the file structure right?

Perhaps I am going at this the wrong way and since the file paths are standardized I could do a script which takes the current date - 90 days = the folder name to move?

Last edited by ndenial; 11-16-2015 at 01:59 PM.
 
Old 11-16-2015, 02:35 PM   #6
ndenial
LQ Newbie
 
Registered: Nov 2015
Posts: 7

Original Poster
Rep: Reputation: Disabled
I think I fixed my issue. Does this look right?

Code:
logserver:~# find /var/ossim/logs/* -type d -mtime +347 -exec echo mv '{}' /mnt/wd2tb/test/'{}' \;
mv /var/ossim/logs/2014/12/03/09 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/09
mv /var/ossim/logs/2014/12/03/04 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/04
mv /var/ossim/logs/2014/12/03/02 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/02
mv /var/ossim/logs/2014/12/03/12 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/12
mv /var/ossim/logs/2014/12/03/13 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/13
mv /var/ossim/logs/2014/12/03/06 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/06
mv /var/ossim/logs/2014/12/03/11 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/11
mv /var/ossim/logs/2014/12/03/16 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/16
mv /var/ossim/logs/2014/12/03/00 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/00
mv /var/ossim/logs/2014/12/03/08 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/08
mv /var/ossim/logs/2014/12/03/14 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/14
mv /var/ossim/logs/2014/12/03/10 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/10
mv /var/ossim/logs/2014/12/03/07 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/07
mv /var/ossim/logs/2014/12/03/01 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/01
mv /var/ossim/logs/2014/12/03/17 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/17
mv /var/ossim/logs/2014/12/03/19 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/19
mv /var/ossim/logs/2014/12/03/03 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/03
mv /var/ossim/logs/2014/12/03/15 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/15
mv /var/ossim/logs/2014/12/03/18 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/18
mv /var/ossim/logs/2014/12/03/05 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/05
logserver:~#
Had to add '{}' to the target
 
Old 11-16-2015, 02:50 PM   #7
hoes
Member
 
Registered: Sep 2005
Distribution: debian, linux from scratch
Posts: 177

Rep: Reputation: 38
I guess the current command will not work as the subdirectory /mnt/wd2tb/test//var/ossim/logs/2014/12/03/ might be nonexistent.
In that case you need a script that checks for the directory and makes it if needed.
But if you want to do such things than you might just make a shell script.
In that case, it might also be easier to make a move of the files every month or every three months.
In that case you can just copy the directories for those months.
 
Old 11-16-2015, 03:16 PM   #8
ndenial
LQ Newbie
 
Registered: Nov 2015
Posts: 7

Original Poster
Rep: Reputation: Disabled
Yes,

I also had an error in the previous command as it had to // in the destination.

testing with one folder at a time, tried using cp and mv but they both give errors due to directory not existing. Tried using cp -parents and still didn't work (ommitted directory).

Im just trying to automate this so I can free up space on the main drive, so daily cron job would be best as some logs can be quite large. Some days are around 2G

I dont want to do this on a monthly basis as that could be up to 300G. (We are tuning our syslog / ossim / and flow logs, but in the mean time they are rather large) Smaller chunks will be preferred.
 
Old 11-16-2015, 03:28 PM   #9
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Yawnstown, Ohio
Distribution: Mojave
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
How about use find and use the results to rsync the data to the target?
then clean up the source...

Just an idea...
 
Old 11-16-2015, 03:37 PM   #10
ndenial
LQ Newbie
 
Registered: Nov 2015
Posts: 7

Original Poster
Rep: Reputation: Disabled
Not entirely sure but I looked at RSYNC and it does not look like it will remove the files from the source once complete. Im not looking to synchronize or back up this data, just move it once it ages, then remove it completely once it goes stale (90 days and then 365days)


If you have an example I would be more than happy to try.
 
Old 11-16-2015, 06:04 PM   #11
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.10, Centos 7.5
Posts: 17,670

Rep: Reputation: 2487Reputation: 2487Reputation: 2487Reputation: 2487Reputation: 2487Reputation: 2487Reputation: 2487Reputation: 2487Reputation: 2487Reputation: 2487Reputation: 2487
It looks like post #6 should work. The '//' should get collapsed into '/'; I did a quick n dirty test of something similar and it worked
Code:
ls
test.pl  tmp2  t.pl  t.t

 mkdir tmp3
 mv tmp2 tmp3//tmp2

 ls
test.pl  tmp3  t.pl  t.t

 ls tmp3
tmp2
 
Old 11-18-2015, 03:48 PM   #12
ndenial
LQ Newbie
 
Registered: Nov 2015
Posts: 7

Original Poster
Rep: Reputation: Disabled
Chris,

I agree, but the problem that I am having with #6 is that it will not create folders that do not exist, therefor I would need to create all the folders ahead of time unless I create a function to include the mv and mkdir -p maybe.
 
Old 11-18-2015, 06:36 PM   #13
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.10, Centos 7.5
Posts: 17,670

Rep: Reputation: 2487Reputation: 2487Reputation: 2487Reputation: 2487Reputation: 2487Reputation: 2487Reputation: 2487Reputation: 2487Reputation: 2487Reputation: 2487Reputation: 2487
Ok; now I see your problem. Yeah, you're going to have to script a short fn to create the target dir structure with 'mkdir -p'.
Judging by your prev posts I don't think you'll have any problem figuring that out
 
Old 11-18-2015, 08:49 PM   #14
Habitual
LQ Veteran
 
Registered: Jan 2011
Location: Yawnstown, Ohio
Distribution: Mojave
Posts: 9,374
Blog Entries: 37

Rep: Reputation: Disabled
Quote:
Originally Posted by ndenial View Post
Not entirely sure but I looked at RSYNC and it does not look like it will remove the files from the source once complete. Im not looking to synchronize or back up this data, just move it once it ages, then remove it completely once it goes stale (90 days and then 365days)


If you have an example I would be more than happy to try.
Something like:
Code:
find /var/example/logs/* -type d -mtime +90 -exec rsync -avzn '{}' /mnt/archivedrive/archive/example/logs/ \;
-n is a dry-run, so easy to debug/check...and prolly could use a tweak or two to enhance it...
should copy all candidate files and their directory to the target...

Just an idea...

Last edited by Habitual; 11-18-2015 at 08:51 PM.
 
Old 12-11-2015, 03:36 PM   #15
cesarbergara
Member
 
Registered: Feb 2012
Location: Buenos Aires, Argentina
Distribution: Debian, Suse, Mandrake,
Posts: 73

Rep: Reputation: Disabled
Hi. Do you try with:

mv -pr {}
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Find the directories which is older than x days delete and zip those directories ramesh pagadala Linux - Newbie 2 08-29-2013 08:17 AM
How to compress and move log older than 30 days from one folder to another LittleMaster Linux - Newbie 6 09-20-2012 12:30 AM
[SOLVED] Attempting to delete files that are older than 90 days, but not directories caiphn Linux - Newbie 12 07-07-2012 07:11 PM
Need to move folders older than 5 days to a bad up directory calicowboy54 Linux - Newbie 4 01-15-2012 09:27 PM
How to move files older than 30 days gfem AIX 8 11-08-2006 04:58 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 11:09 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration