Cron Job help to move directories older than 90 days

ndenial · 11-16-2015, 10:57 AM

I have been frequenting this site more and more since I have become the owner/maintainer of a Debian based server. Previous experience has been with Ubuntu and Edubuntu LTSP. This will be my first post here and hope it is in the right area. So far I have been able to understand the basics but have yet to fully dive into any scripting, this will be my first foray into cron tasks as well.

System: Debian 8 based

I would like to Move directories containing logs older than 90 days to backup drive, maintaining folder structure. I seem to only be able to move files and a single child directory.

Folder structure (logs/year/month/day/hour):
/var/example/logs/2014/12/01/01/*files*
/var/example/logs/2014/12/01/01/sensor1/*files*
/var/example/logs/2014/12/01/01/sensor2/*files*

Issue: using

Code:

find /var/example/logs/* -type d -mtime +90 -exec -mv '{}' /mnt/archivedrive/archive/example/logs/ \;

appears to dump all files into a /example/logs/ directory and do not maintain structure.

I did not run the above code as is, instead I used ls -l where the -mv is. It returns

Code:

/var/example/logs/2014/12/01/01:
total 948
drwxrwxr-x 2 *exampleuser* *examplegroup*   4096 Dec 01 2014 *sensor1dir*
drwxrwxr-x 2 *exampleuser* *examplegroup*   4096 Dec 01 2014 *sensor1dir*
-rw-rw-r-- 1 *exampleuser* *examplegroup*   9268 Dec 01 2014 *file.1*
-rw-rw-r-- 1 *exampleuser* *examplegroup* 944609 Dec 01 2014 *file.2*
/mnt/archivedrive/archive/example/logs/:
total 0

That repeats for each date/hour and it appears it will dump the results into the /mnt/archivedrive/archive/example/logs/ folder without maintaining their folder structure.

I am intending to put this into my chron.daily, then follow up with a another which removes files older than 365 days from the archive drive.

Any help would be appreciated as I have spent a few days trying to come up with a solution.

Thank you!

HMW · 11-16-2015, 11:10 AM

Quote:

Originally Posted by ndenial

I did not run the above code as is, instead I used ls -l where the -mv is

Hi!

Ehm... so, why didn't you run your find command? Sorry, but I cannot make head or tails from your question!

Can you show the actual output of your find command?

Best regards,
HMW

ndenial · 11-16-2015, 12:12 PM

Sorry for the confusion...

I ran this

Code:

find /var/example/logs/* -type d -mtime +90 -exec ls -l '{}' /mnt/archivedrive/archive/example/logs/ \;

to simulate this

Code:

find /var/example/logs/* -type d -mtime +90 -exec -mv '{}' /mnt/archivedrive/archive/example/logs/ \;

and got this:

Code:

Logserver1:~# find /var/ossim/logs/* -type d -mtime +347 -exec ls -l '{}' /mnt/wd2tb/test/ \;
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/09:
total 936
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9145 Dec  3  2014 data.stats
-rw-rw-r-- 1 logserver ossim 936420 Dec  4  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/04:
total 936
drwxrwxr-x 2 logserver ossim   4096 Dec  3  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9019 Dec  2  2014 data.stats
-rw-rw-r-- 1 logserver ossim 936420 Dec  3  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/02:
total 936
drwxrwxr-x 2 logserver ossim   4096 Dec  3  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9189 Dec  2  2014 data.stats
-rw-rw-r-- 1 logserver ossim 936420 Dec  3  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/12:
total 932
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9080 Dec  3  2014 data.stats
-rw-rw-r-- 1 logserver ossim 932324 Dec  4  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/13:
total 940
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9260 Dec  3  2014 data.stats
-rw-rw-r-- 1 logserver ossim 940516 Dec  4  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/06:
total 936
drwxrwxr-x 2 logserver ossim   4096 Dec  3  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9121 Dec  2  2014 data.stats
-rw-rw-r-- 1 logserver ossim 936420 Dec  3  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/11:
total 932
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9059 Dec  3  2014 data.stats
-rw-rw-r-- 1 logserver ossim 932324 Dec  4  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/00:
total 932
drwxrwxr-x 2 logserver ossim   4096 Dec  3  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9046 Dec  2  2014 data.stats
-rw-rw-r-- 1 logserver ossim 932324 Dec  3  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/08:
total 940
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9255 Dec  3  2014 data.stats
-rw-rw-r-- 1 logserver ossim 940516 Dec  4  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/14:
total 948
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor1
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor2
-rw-rw-r-- 1 logserver ossim   9397 Dec  3  2014 data.stats
-rw-rw-r-- 1 logserver ossim 944609 Dec  4  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/10:
total 936
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9121 Dec  3  2014 data.stats
-rw-rw-r-- 1 logserver ossim 936420 Dec  4  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/07:
total 964
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor1
-rw-rw-r-- 1 logserver ossim  10043 Dec  3  2014 data.stats
-rw-rw-r-- 1 logserver ossim 965092 Dec  4  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/01:
total 932
drwxrwxr-x 2 logserver ossim   4096 Dec  3  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9097 Dec  2  2014 data.stats
-rw-rw-r-- 1 logserver ossim 932324 Dec  3  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/03:
total 932
drwxrwxr-x 2 logserver ossim   4096 Dec  3  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9120 Dec  2  2014 data.stats
-rw-rw-r-- 1 logserver ossim 932324 Dec  3  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/15:
total 948
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor1
drwxrwxr-x 2 logserver ossim   4096 Dec  4  2014 sensor2
-rw-rw-r-- 1 logserver ossim   9268 Dec  3  2014 data.stats
-rw-rw-r-- 1 logserver ossim 944609 Dec  4  2014 mindex.inx
/mnt/wd2tb/test/:
total 0

/var/ossim/logs/2014/12/03/05:
total 932
drwxrwxr-x 2 logserver ossim   4096 Dec  3  2014 sensor1
-rw-rw-r-- 1 logserver ossim   9053 Dec  2  2014 data.stats
-rw-rw-r-- 1 logserver ossim 932324 Dec  3  2014 mindex.inx

Which makes me think that it will only copy the files and directories listed to /mnt/wd2tb/test/ not /mnt/wd2tb/test/2014/12/03/05, /mnt/wd2tb/test/2014/12/03/06 etc etc.

I don't want to do the actual move yet until I am sure that I wont screw myself and make a mess of the directory. I do not have a test environment for this.

hoes · 11-16-2015, 01:40 PM

Actually you run the command

Code:

ls -l dir /mnt/wd2tb/test/

every time. The files in the specific dir pop up, so you want to do an action with this directory.
Instead of ls -l do something useful.
Run

Code:

    find /var/example/logs/* -type d -mtime +90 -exec echo mv '{}' /mnt/archivedrive/archive/example/logs/ \;

to check if the move commands are as you would expect.

ndenial · 11-16-2015, 01:57 PM

Hoes,

Thank you, I searched for a command that would better "simulate" this for me but I could not find it for the life of me.

The result of that is:

Quote:

Logserver:~# find /var/ossim/logs/* -type d -mtime +347 -exec echo mv '{}' /mnt/wd2tb/test/ \;
mv /var/ossim/logs/2014/12/03/09 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/04 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/02 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/12 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/13 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/06 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/11 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/16 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/00 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/08 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/14 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/10 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/07 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/01 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/17 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/03 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/15 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/18 /mnt/wd2tb/test/
mv /var/ossim/logs/2014/12/03/05 /mnt/wd2tb/test/
Logserver:~#

If I understand this output, it does not appear to carry over the file structure right?

Perhaps I am going at this the wrong way and since the file paths are standardized I could do a script which takes the current date - 90 days = the folder name to move?

ndenial · 11-16-2015, 02:35 PM

I think I fixed my issue. Does this look right?

Code:

logserver:~# find /var/ossim/logs/* -type d -mtime +347 -exec echo mv '{}' /mnt/wd2tb/test/'{}' \;
mv /var/ossim/logs/2014/12/03/09 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/09
mv /var/ossim/logs/2014/12/03/04 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/04
mv /var/ossim/logs/2014/12/03/02 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/02
mv /var/ossim/logs/2014/12/03/12 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/12
mv /var/ossim/logs/2014/12/03/13 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/13
mv /var/ossim/logs/2014/12/03/06 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/06
mv /var/ossim/logs/2014/12/03/11 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/11
mv /var/ossim/logs/2014/12/03/16 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/16
mv /var/ossim/logs/2014/12/03/00 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/00
mv /var/ossim/logs/2014/12/03/08 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/08
mv /var/ossim/logs/2014/12/03/14 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/14
mv /var/ossim/logs/2014/12/03/10 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/10
mv /var/ossim/logs/2014/12/03/07 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/07
mv /var/ossim/logs/2014/12/03/01 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/01
mv /var/ossim/logs/2014/12/03/17 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/17
mv /var/ossim/logs/2014/12/03/19 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/19
mv /var/ossim/logs/2014/12/03/03 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/03
mv /var/ossim/logs/2014/12/03/15 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/15
mv /var/ossim/logs/2014/12/03/18 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/18
mv /var/ossim/logs/2014/12/03/05 /mnt/wd2tb/test//var/ossim/logs/2014/12/03/05
logserver:~#

Had to add '{}' to the target

hoes · 11-16-2015, 02:50 PM

I guess the current command will not work as the subdirectory /mnt/wd2tb/test//var/ossim/logs/2014/12/03/ might be nonexistent.
In that case you need a script that checks for the directory and makes it if needed.
But if you want to do such things than you might just make a shell script.
In that case, it might also be easier to make a move of the files every month or every three months.
In that case you can just copy the directories for those months.

ndenial · 11-16-2015, 03:16 PM

Yes,

I also had an error in the previous command as it had to // in the destination.

testing with one folder at a time, tried using cp and mv but they both give errors due to directory not existing. Tried using cp -parents and still didn't work (ommitted directory).

Im just trying to automate this so I can free up space on the main drive, so daily cron job would be best as some logs can be quite large. Some days are around 2G

I dont want to do this on a monthly basis as that could be up to 300G. (We are tuning our syslog / ossim / and flow logs, but in the mean time they are rather large) Smaller chunks will be preferred.

Habitual · 11-16-2015, 03:28 PM

How about use find and use the results to rsync the data to the target?
then clean up the source...

Just an idea...

ndenial · 11-16-2015, 03:37 PM

Not entirely sure but I looked at RSYNC and it does not look like it will remove the files from the source once complete. Im not looking to synchronize or back up this data, just move it once it ages, then remove it completely once it goes stale (90 days and then 365days)

If you have an example I would be more than happy to try.

chrism01 · 11-16-2015, 06:04 PM

It looks like post #6 should work. The '//' should get collapsed into '/'; I did a quick n dirty test of something similar and it worked

Code:

ls
test.pl  tmp2  t.pl  t.t

 mkdir tmp3
 mv tmp2 tmp3//tmp2

 ls
test.pl  tmp3  t.pl  t.t

 ls tmp3
tmp2

ndenial · 11-18-2015, 03:48 PM

Chris,

I agree, but the problem that I am having with #6 is that it will not create folders that do not exist, therefor I would need to create all the folders ahead of time unless I create a function to include the mv and mkdir -p maybe.

chrism01 · 11-18-2015, 06:36 PM

Ok; now I see your problem. Yeah, you're going to have to script a short fn to create the target dir structure with 'mkdir -p'.
Judging by your prev posts I don't think you'll have any problem figuring that out

Habitual · 11-18-2015, 08:49 PM

Quote:

Originally Posted by ndenial

Not entirely sure but I looked at RSYNC and it does not look like it will remove the files from the source once complete. Im not looking to synchronize or back up this data, just move it once it ages, then remove it completely once it goes stale (90 days and then 365days)

If you have an example I would be more than happy to try.

Something like:

Code:

find /var/example/logs/* -type d -mtime +90 -exec rsync -avzn '{}' /mnt/archivedrive/archive/example/logs/ \;

-n is a dry-run, so easy to debug/check...and prolly could use a tweak or two to enhance it...
should copy all candidate files and their directory to the target...

Just an idea...

cesarbergara · 12-11-2015, 03:36 PM

Hi. Do you try with:

mv -pr {}