How to find and delete all files with specific name

khandu · 06-16-2014, 05:54 PM

Hey Guys

We have files with the following format

Quote:

name1.20130101-0000323.gz
name2.20130501-12300012.gz
...
name3.20140503-12322.gz

Basically a filename.date-number.gz

We want to create a script to delete all files with date more than a year old.. Was trying this..

Code:

find /file/location/ -type f ! \( -iname "*$(date -d -1year +'%Y%m*')" \) -exec echo {} \;

But is not coming out right.. Something to do with trying to capture * on filename before the date..

Can someone please help out and tell me what will be the correct syntax?

Cheers

khandu · 06-16-2014, 06:19 PM

So basically want to delete all files with datestamp older than 1 year from today..

the -mtime won't work in this case.. have to capture via the filename..

sag47 · 06-16-2014, 06:40 PM

What do you mean -mtime won't work? I don't understand what you mean capture via the filename. Typically the find command would look like this...

Code:

#test
find /location -type f -mtime +365
#remove the files
find /location -type f -mtime +365 -exec rm {} \;

Did something happen to the files where you can't use the timestamp? If you need to do it based off of the date in the filename then you would need to use a more advanced loop like the following... Note, this assumes *all* files end with [0-9]{8}.gz.

Code:

find . -type f -name '*[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9].gz' | while read x;do
filedate="$(echo $x | sed 's/.*\.\([0-9]\{8\}\).gz/\1/')"
if [ "$(date -d "$filedate" +%s)" -lt "$(date -d -1year +%s)" ];then
  echo "$x is older than 1 year from now.  Created on $(date -d "$filedate")"
fi
done

It's best to explicitly state the format you expect the files to be otherwise this method would likely fail. You should check the date in the file name against the output of the date -d "$filedate" command. On my system, date command automatically interprets the date format to be YYYYMMDD.

khandu · 06-16-2014, 06:50 PM

Quote:

Originally Posted by sag47

What do you mean -mtime won't work? I don't understand what you mean capture via the filename. Typically the find command would look like this...

Code:

#test
find /location -type f -mtime +365
#remove the files
find /location -type f -mtime +365 -exec rm {} \;

Did something happen to the files where you can't use the timestamp? If you need to do it based off of the date in the filename then you would need to use a more advanced loop like the following... Note, this assumes *all* files end with [0-9]{8}.gz.

Code:

find . -type f -name '*[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9].gz' | while read x;do
filedate="$(echo $x | sed 's/.*\.\([0-9]\{8\}\).gz/\1/')"
if [ "$(date -d "$filedate" +%s)" -lt "$(date -d -1year +%s)" ];then
  echo "$x is older than 1 year from now.  Created on $(date -d "$filedate")"
fi
done

It's best to explicitly state the format you expect the files to be otherwise this method would likely fail. You should check the date in the file name against the output of the date -d "$filedate" command. On my system, date command automatically interprets the date format to be YYYYMMDD.

Yeah the modify timestamp is screwed up

actually format is this (modified first post).. sorry had given the wrong format earlier

Quote:

filename.date-*.gz

the "date" is in format "YYYYMMDD" and that is what we want to act on to delete all files older than one year

khandu · 06-16-2014, 07:18 PM

I am trying this

Quote:

filedate="$(echo $x | sed 's/.*\.$[0-9]{8}$-.*.gz/\1/')"
echo $filedate

But the result is spitting out the whole filename instead of just the group captured..

So

name1.20130101-0000323.gz

instead of

20130101

khandu · 06-16-2014, 10:09 PM

I am trying to do this but the array is not working.. any ideas

Quote:

#!/bin/bash
find /files/ -type f -name '*[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]-*.gz' | while read x;
do
filedate="$(echo $x | grep -Eo '[0-9]{8}')"
if [ "$filedate" -lt "$(date -d -1year +%Y%m%d)" ];then
array1=$x

fi
done
echo ${#array1[@]}

The result is coming 0

khandu · 06-16-2014, 10:30 PM

I realise that it is due to the | symbol opening a subshell..

But basically want to store all results in a array which then I can
1) email via mailx command in one email like a report (if i currently will put it in loop it will send 100 emails individually for each find)and
2) delete using -exec rm command

sag47 · 06-16-2014, 11:02 PM

Perhaps, you should step back from the problem and figure out what it is you actually want. Currently your last post is entirely different from your original request. Please organize your thoughts and properly express what you're trying to do from start to finish.

The original answer I gave you seemed sufficient in that it could have been lightly modified by you to accomplish removing files older than a year when the date timestamp is part of the file name. One could update the expression or change how they're obtaining the date from the file name with a small modification. If that's not what you want then see my former statement in this post.

If you need to email the output before removing then don't mess with an array. Simply pipe stdout from the while loop in my original posted program to stdin of the mail program of your choosing.

khandu · 06-16-2014, 11:07 PM

Sorry.. Will try to explain again..

Have a directory with the following file structure

Quote:

name1.20130101-0000323.gz
name2.20130501-12300012.gz
...
name3.20140503-12322.gz

FORMAT: filename.DATE-TIME.gz

Also modified timestamp is not proper for these files so cannot rely on that

I want to

1) Find all the files for the DATE more than a year old from today

2) Delete all those found files

3) Send a report email of all the files matched above and deleted.

sag47 · 06-16-2014, 11:27 PM

Quote:

Originally Posted by khandu

FORMAT: filename.DATE-TIME.gz

Your format for time does not make sense and is not a standard time format. Since you only care about things older than a year then the day should be precise enough (i.e. we don't care about the time).

Quote:

Originally Posted by khandu

Also modified timestamp is not proper for these files so cannot rely on that

Understood.

Quote:

Originally Posted by khandu

I want to

1) Find all the files for the DATE more than a year old from today

2) Delete all those found files

3) Send a report email of all the files matched above and deleted.

Okay then you can lightly modify my original program. You should break it down into two steps. The first is a for loop which will mail you what *would* be deleted. Then execute the same program slightly modified to actually delete it. Here is my last post pretty much literal. Considering you don't seem to understand the original program I wrote I highly recommend against you running any code on your system until you fully understand what it does. Use at your own risk.

Here's what I talked about breaking it into two steps.

Code:

find . -type f -name '*.gz' | while read x;do
filedate="$(echo $x | sed 's/.*\.\([0-9]\{8\}\)-[0-9]\+.gz/\1/')"
if [ "$(date -d "$filedate" +%s)" -lt "$(date -d -1year +%s)" ];then
  echo "$x is older than 1 year from now.  Created on $(date -d "$filedate")"
fi
done | mail -s "files to be deleted" user@email.example

Code:

find . -type f -name '*.gz' | while read x;do
filedate="$(echo $x | sed 's/.*\.\([0-9]\{8\}\)-[0-9]\+.gz/\1/')"
if [ "$(date -d "$filedate" +%s)" -lt "$(date -d -1year +%s)" ];then
  \rm "$x"
fi
done

You should learn bash, read the date man page to understand what I'm doing, and backup any files before you attempt this yourself.

khandu · 06-16-2014, 11:52 PM

Thanks

I did understand your first code.. i just didn't realise that i can pipe the mail after the done..

I have a similar script running for another use where I am doing two step (one for mail and one for actual delete) but the difference was that it was in a single line command and thus mailx command was working at the end of it.. this had loop and therefore I didn't realize the mailx had to be outside the done with a |

I am testing it in dev..

thanks