Cron Job help to move directories older than 90 days
I have been frequenting this site more and more since I have become the owner/maintainer of a Debian based server. Previous experience has been with Ubuntu and Edubuntu LTSP. This will be my first post here and hope it is in the right area. So far I have been able to understand the basics but have yet to fully dive into any scripting, this will be my first foray into cron tasks as well.
System: Debian 8 based I would like to Move directories containing logs older than 90 days to backup drive, maintaining folder structure. I seem to only be able to move files and a single child directory. Folder structure (logs/year/month/day/hour): /var/example/logs/2014/12/01/01/*files* /var/example/logs/2014/12/01/01/sensor1/*files* /var/example/logs/2014/12/01/01/sensor2/*files* Issue: using Code:
find /var/example/logs/* -type d -mtime +90 -exec -mv '{}' /mnt/archivedrive/archive/example/logs/ \; I did not run the above code as is, instead I used ls -l where the -mv is. It returns Code:
/var/example/logs/2014/12/01/01: I am intending to put this into my chron.daily, then follow up with a another which removes files older than 365 days from the archive drive. Any help would be appreciated as I have spent a few days trying to come up with a solution. Thank you! |
Quote:
Ehm... so, why didn't you run your find command? Sorry, but I cannot make head or tails from your question! Can you show the actual output of your find command? Best regards, HMW |
Sorry for the confusion...
I ran this Code:
find /var/example/logs/* -type d -mtime +90 -exec ls -l '{}' /mnt/archivedrive/archive/example/logs/ \; Code:
find /var/example/logs/* -type d -mtime +90 -exec -mv '{}' /mnt/archivedrive/archive/example/logs/ \; Code:
Logserver1:~# find /var/ossim/logs/* -type d -mtime +347 -exec ls -l '{}' /mnt/wd2tb/test/ \; Which makes me think that it will only copy the files and directories listed to /mnt/wd2tb/test/ not /mnt/wd2tb/test/2014/12/03/05, /mnt/wd2tb/test/2014/12/03/06 etc etc. I don't want to do the actual move yet until I am sure that I wont screw myself and make a mess of the directory. I do not have a test environment for this. |
Actually you run the command
Code:
ls -l dir /mnt/wd2tb/test/ Instead of ls -l do something useful. Run Code:
find /var/example/logs/* -type d -mtime +90 -exec echo mv '{}' /mnt/archivedrive/archive/example/logs/ \; |
Hoes,
Thank you, I searched for a command that would better "simulate" this for me but I could not find it for the life of me. The result of that is: Quote:
Perhaps I am going at this the wrong way and since the file paths are standardized I could do a script which takes the current date - 90 days = the folder name to move? |
I think I fixed my issue. Does this look right?
Code:
logserver:~# find /var/ossim/logs/* -type d -mtime +347 -exec echo mv '{}' /mnt/wd2tb/test/'{}' \; |
I guess the current command will not work as the subdirectory /mnt/wd2tb/test//var/ossim/logs/2014/12/03/ might be nonexistent.
In that case you need a script that checks for the directory and makes it if needed. But if you want to do such things than you might just make a shell script. In that case, it might also be easier to make a move of the files every month or every three months. In that case you can just copy the directories for those months. |
Yes,
I also had an error in the previous command as it had to // in the destination. testing with one folder at a time, tried using cp and mv but they both give errors due to directory not existing. Tried using cp -parents and still didn't work (ommitted directory). Im just trying to automate this so I can free up space on the main drive, so daily cron job would be best as some logs can be quite large. Some days are around 2G I dont want to do this on a monthly basis as that could be up to 300G. (We are tuning our syslog / ossim / and flow logs, but in the mean time they are rather large) Smaller chunks will be preferred. |
How about use find and use the results to rsync the data to the target?
then clean up the source... Just an idea... |
Not entirely sure but I looked at RSYNC and it does not look like it will remove the files from the source once complete. Im not looking to synchronize or back up this data, just move it once it ages, then remove it completely once it goes stale (90 days and then 365days)
If you have an example I would be more than happy to try. |
It looks like post #6 should work. The '//' should get collapsed into '/'; I did a quick n dirty test of something similar and it worked
Code:
ls |
Chris,
I agree, but the problem that I am having with #6 is that it will not create folders that do not exist, therefor I would need to create all the folders ahead of time unless I create a function to include the mv and mkdir -p maybe. |
Ok; now I see your problem. Yeah, you're going to have to script a short fn to create the target dir structure with 'mkdir -p'.
Judging by your prev posts I don't think you'll have any problem figuring that out :) |
Quote:
Code:
find /var/example/logs/* -type d -mtime +90 -exec rsync -avzn '{}' /mnt/archivedrive/archive/example/logs/ \; should copy all candidate files and their directory to the target... Just an idea... |
Hi. Do you try with:
mv -pr {} |
All times are GMT -5. The time now is 03:41 PM. |