rm by date
 

Guys,

Im stumped on 1 last line of a script im writing, i need to have the script go into each directory called test* and rm anything older than 18 days from the day its executed. Cron will run it daily.

Heres what i have except what will determine the old than 18 days for removal. Sorry im a newb. Thanks!


#!/bin/bash

for i in test01 test02 test03 test04 test05 test06 test07 test08 test09 test10 test11 test12 ; do
rm -rf anything older than 30 days
done

Use find to identify files with an mtime older than 30 days and then feed the identified files to rm with xargs:

Thanks Matthew! So it should look like:

#!/bin/bash

for i in test01 test02 test03 test04 test05 test06 test07 test08 test09 test10 test11 test12 ; do
find . -name 'file*' -mtime +10 -type f -print0 | xargs -0 rm -rf
done

File equals:

file_backup_backup14_20080106.tgz
file_backup_backup14_20080107.tgz
etc...

You don't really need xargs or the for loop (and the -type parameter, while probably good practice, isn't strictly necessary either if you are sure these files are all regular files)...

I think the xargs approach is faster because it supplies all the arguments to rm that applies to all the files at once. However using -exec will supply the arguments one at a time and thus a bit slow. Besides I don't think you need -print0 either.

find . -name 'test*' -mtime +30 -type f |xargs -0 rm -f

With the xargs method, you don't need to use the "for i in ... do ... done", just the single line I provided should do it. The only thing you might have to watch for is that find will recurse into sub-directories, and that the pattern passed to -name is specific enough to only get the files you want.

As a test to see if it is going to operate on the right files, do this first:
If that lists only the files you want to delete, replace the "ls -l" with "rm -f".

Thanks again Matthew, I wanted to try a variation of this, if it is possible. I would like to take files for the past two months and have a script that will keep the past two weeks and then every Sunday only. Is that possible?

I thought using xargs is faster due to the fact that using -exec will fork a child of find for every argument, whereas xargs simply passes the argument to the resulting command?

Not with the -mtime option to find. If you want such complex behavior, you will probably want to write a perl script or something like that.

*edit* actually you might be able to do it with the printf option, and the -mtime option, ORed together with -o.

Actually this would be pretty simple to do in the shell if you're using GNU versions of 'ls'.

will display the weekday of the modification time on the files.

There are other formats for the style (man ls for specifics, and man date for the formats)

Using that info and the find commands listed above you can construct a simple shell script to do what you want.

Matthew/Raconteur,

Thanks for the response, but i am confused what the line in the script would be to remove all files but for the last 2 weeks and every Sunday for a two month period. Could you post an example? Thanks a million for the time and the help!

This is quick and dirty, and only cursorily tested, the usual caveats apply. If I were writing this for keepsies I'd probably do a few things differently.

Change the initial file pattern to match whatever your files are named. I called it filter.sh but of course you can name it whatever you wish...

Remove the sharps if you want a very verbose version.

hth

Do you want to descend into sub-directories, or just remove files from one directory?

Good point, my stab at this assumes one directory.
However, I think if you added a -type f parameter to the initial find it would pass the multi-level test.

OK, if you want to descend into sub-directories as well as working in the current directory, here's the command for you:
Please backup before doing this - I take no responsibility if it deletes the wrong files because I might have mis-interpreted your needs or made a mistake.

It is probably a good idea to run it without the "| xargs rm -f" on the end first and verifiy that the files it lists are indeed the ones you want to delete, and only the ones you want to delete. i.e.
OK, now for an explanation.

The -printf option to find can be used to print more information that the default (which is just to print the file path & name. In this case I used the format string "%Cw %p\n". Please look at the find manual page and find %C. If you read this section of the page, you should be able to see that "%Cw" means the day of week number (0-6) the file was last modified. (0 is Sunday). I could also have used %Ca which would print the day of week like "Sun", "Mon" and so on, but this is locale dependent, so it would not work on a machine with German language settings for example.

The next bit, "%p\n" just means the file path & name with a new line character after it.

The "-mtime +14" means to print only files older than 14 days since the last modification.

The "-type f" means to only print files.

The "." as the first option to find means "start in the current directory".

In you run the find command without the following pipeline, you will see something like this:

The number is the weekday of modification of the file (0=Sun, 3=Wed, 6=Sat), then the file name in question.

We do not want to delete files which were modified on a Sunday, so we exclude the lines beginning with 0 with the next part of the pipeline:
If the example output of the find command above is what you get, after the grep part, it will look like
The next part of the pipeline:
...prints only characters from column 3 onwards - it cuts off the day of week and the space. We now have just a list of files which we want to remove:
This is the input to xargs which calls the specified command (rm -f) for groups of strings read from standard input - i.e. remove the listed files.

I hope that is clear. If you have further questions about it, feel free to ask.

P.S. if you do not want to descend into sub-directories, change the find part of the command, adding

Heh, I was just now goofing around to see if I could get all of this on one line (just to see if it could be done)... but I gave it up because the phone rang.

Your example doesn't quite meet the requirements, if I read this correctly:

"[...]remove all files but for the last 2 weeks and every Sunday for a two month period[...]"

But it is still well done, closer than I got in the time I fiddled with it.

I might still be able to do it all on one line, but its Friday and I have a date ;-) Have fun!

I missed out the two month period, didn't I?

You just need to add
...to the find command, assuming you think of a month as 30 days. You might need some brackets too to make it work right.

Something like this (I've used \ to break the lines to make it more readable - maybe...)
Have a nice date. Don't talk about Linux too much.

not tested. To test, remove xargs

Thanks everyone for the time answering my questions. So the way i would try to run this is in a directory called /home/anyuser will live the script along with the directories the script will need to go into and then perform the function we're talking about. A ls -l would look like:

dir1
dir2
dir3
dir4
filecleanup.sh

In each of the directories called dir1, 2, etc..., etc are .tgz files called:
misc.20080101.gz
misc.20080102.gz
misc.20080103.gz
up to
misc.20080315.gz

So these .gz files will increment everyday and the script will run by cron once a day and keep only everything from the past two weeks from when the script will run plus every sunday from the past 60 days. (since the question posed earlier was if im using a 30 day period as a month)

You want to make sure the script does not delete itself. Also, when running from cron, you get a very minimal environment with a limited PATH, so you need to make sure any programs you call are in that limited PATH, or make sure you modify the PATH in your script.

I would recommend it looks something like this (note the \ at the end of lines must be the last character on the line - no spaces or anything should follow it) :

Note that by explicitly telling find to start in /home/anyuser/dir* instead of "." it will not see the script itself and thus avoid problems with removing itself.

Matthew,

thanks for the response. What you suggested does keep everything from the past 14 days but then removes everything else but the first 4 days. so theres a big gap be between. Lets say for example we have files 01-60that are from june22-aug21, it removes everything but files 01,02,03,04 (sat-tues) and then 49-60.

It should remove everything but files 02,09,16,23,30,37,44,(all Sundays) and 49-60 (the last two weeks).

Thanks again for the help!

Maybe the modification date on the files is not set right - have the files been modified since they were created? Note that *nix filesystems do not store a creation timestamp, only access time and last modification time.

So your command should work in that scenario? What i did was create test directories with test files in them and then modified the dates with touch -t date testfile to create a test scenario before running them on legit files just in case.

I thought it would, but I didn't test it. Like I said - there may be mistakes... :-) You get the rough idea though?

No one mentioned this, but while you are experimenting, change 'rm' to 'echo'. Unless you are a coder who never makes any mistakes, it is likely that you make just that tiny little mistake which wipes out your hard drive.

Always substitute rm with echo while developing!

jlinkels

Hi Matthew,

I dont get the rough idea unfortunately. im a novice to linux. But if you give up, give me the break down of what your last command does and ill work on it from here and try to get it to work. Very much appreciate all your help!

Bump! a d bump!

All the breakdown I can provide I did already is post #15.

Thanks Matthew for trying. Maybe someone else can hit this nail on the head. Id really like to get this to work.

A couple of changes were required to matthewg42's code. The idea was correct, just a minor and/or mixup, and should have used modification time instead of status change time:

Ok, let's generate the test files, and set the modification dates:


Now, let's find the correct range of files to delete:
Now, let's exclude Sundays, and delete the remainding files:

Perfect! Thanks a million Mr.C, I created the test env you listed in your post and ran it and it ran perfectly! I also then ran it in the test env i created, just to double check and it was perfect again. If you can, will you give a break down of your final find command that did the actual delete and how it determined what days were what and how removed exactly what was requested?

I could use it to understand Linux better and to write my own script next time. If not, thank you for your time.

Also, thanks to Matthew and rab for trying to help me out with this! Hopefully Mr.C post helped you too!