LinuxQuestions.org
Help answer threads with 0 replies.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 10-21-2008, 05:56 AM   #1
Penn
LQ Newbie
 
Registered: Apr 2008
Posts: 25

Rep: Reputation: 15
How do i remove files from date to date.


Hi....
Im using debian etch. I have 5000 mails. I would like to remove all mails from 2008-01 to 2008-08 (yyyy-mm)

Last edited by Penn; 10-21-2008 at 06:06 AM.
 
Old 10-21-2008, 06:22 AM   #2
sr_sanjeev
LQ Newbie
 
Registered: Mar 2008
Posts: 6

Rep: Reputation: 0
Hi
Either you can write a small script or use like

ls -alrt | grep 2008 | rm -rf *

but before doing that , please ensure that first 2 commands are serving your purpose or not?
 
Old 10-21-2008, 07:46 AM   #3
Penn
LQ Newbie
 
Registered: Apr 2008
Posts: 25

Original Poster
Rep: Reputation: 15
OK......Im going to move the mails to another directory first before i remove it. When i use the following:
"ls -alrt | grep 2008 | mv -rf * /directory"

then i recieve

"Argument list too long"
 
Old 10-21-2008, 08:22 AM   #4
ledow
Member
 
Registered: Apr 2005
Location: UK
Distribution: Slackware 13.0
Posts: 241

Rep: Reputation: 34
I always have that problem with ls - you have to plumb it through something like xargs first.

The above is extremely primitive and (although it will probably do the job in this case because it's only used for mail files) it will also get any files owned by user 2008 etc. You'll probably also hit problems on filenames with spaces and other strange characters. I'd say it's a poor script to then try and plumb through rm!

find . -iname "*2008*" | sed s/.*/\"\&\"/ |xargs -n1 echo

(Obviously, replace the last echo with a sane remove command once you have tested that it lists only the files you want). This would be a bit better and would properly handle spaces in filenames and isn't reliant on ls's output being sane (which it isn't in a lot of cases of long filenames, etc.). The find gets the filenames, the sed encloses all filenames in quotes, the xargs pipes them in one at a time to the final command (ls/rm/mv or whatever). Because it uses find, you can easily narrow by actual file creation date, etc., instead of filename.

Much easier would be to load up konqueror, sort by date/filename and do it manually (if you have that graphical route available to you).

Last edited by ledow; 10-21-2008 at 08:30 AM.
 
Old 10-21-2008, 10:22 AM   #5
BrianSnook
LQ Newbie
 
Registered: Jan 2008
Location: Atlanta, GA
Distribution: FreeBSD, RH9, RHEL5, CentOS5
Posts: 3

Rep: Reputation: 1
use "date --date" and shell math to pass file ages to "find"

You'll want to use find with the "-mtime" flag to limit files based on modification time. Use it like this:

Code:
    find STARTDIR -mtime -LESSTHANTHISAGE -mtime +OLDERTHANTHISAGE
You'll then want to use the "-exec COMMAND {} \;" syntax to execute a command on the file(s), where "{}" designates the files find finds and the "\;" tells find that the exec command is done. You'll probably want to use "mv" to move the file to another directory, as you suggested, rather than deleting right away, but if you test enough and have enough confidence, you can do it either way. Remember, you can never test too many times and be too careful!

Here's a quick example of how to use "find" to give you just the files you want in a directory (or tree of directories). If you're impatient, you can jump down to the code and start trying it out, but hopefully this explanation is useful. I've tested it in /bin/sh (the real one, not the one some installs link to bash), /bin/bash, and /bin/zsh. If you use a csh-derived shell, you'll need to re-interpret the variables and math, or just invoke a Bourne-style shell to execute the example. A few things to note:

I used '-name "*" -prune' to try to limit my find to my home directory, and to prune (not search) past any subdirectories, strictly to avoid crawling a bunch of junk in a bunch of directories that are overdue some "housekeeping" and show my messiness. If all your files are in a bottom-level directory (you have no subdirectories), you don't need this, and since this still gave me contents of one directory deep, if you do have subdirectories, depending on whether you want to move those files too may mean that this will or will not meet your needs by itself (in fact, if you want to remove files from subdirectories, you would also want to remove the prune clause). That is to say, if all your mail files are in one directory or you want to remove files from subdirectories too, you can take out the '-name "*" -prune'.

Also, you'll need a version of "date" that supports the "--date" argument to pass an arbitrary date in the format of "--date MM/DD/[YY]YY". Most modern versions of Linux and Solaris should support this. If they don't, you could always find a calendar and manually count the days from today to the dates you want... (or install gnu coreutils/gnudate).

Finally, the one big caveat is that in order to get maximum precision, you may need to play around with your dates/ages a bit. Because I had files one day older than my start date and one day younger than my end date, I arbitrarily chose those dates as examples, but I had to subtract two days from my end date. I expected I would have to subtract one day, but it appears that the find is probably calculating age based on the total time (date/time), and I have not yet reached the time of day when the file "becomes a day older". You might need to do the same thing in reverse (adding a day) depending on when your first file was modified and when you're running this.

To be clearer: today is 21 Oct, and I want to get files younger than 26 Jun and older than 02 Oct. I use the "+%j" argument to format the date as the day of the year (DOY) (ranging from 1-366), and then subtract my begin and end days of the year from today's day of the year. Today is DOY 295 and 2 October is DOY 276. The difference is 295-276=19. I don't want anything older than 19 days, however, since that would miss the file _on_ the last date, so I need to say anything older than 19-1 days, or 18 days. Well, I'm checking this in the morning, and I apparently modified my targeted file in the afternoon, so it is not yet 18 days old, so I needed to subtract _two_ days to get anything older than 17 days old. You'll want to play around with the find that does _not_ have the "-exec mv {} NEWDIR \;" (OR especially test before using "-exec rm {} \;") before you actually move or remove anything, to be sure that you get what you want. Keeping the "-ls" in there ensures that you also get a listing of what would be moved/removed, and piping the whole thing through "tee LOGFILENAME" ensures that you have a file you can review for what you want in case your data scrolls off your screen and out of your scrollback buffer.

Code:
sh-2.05b$ DOY_BEGIN=`date --date 06/26/2008 +%j`
sh-2.05b$ DOY_END=`date --date 10/2/2008 +%j`
sh-2.05b$ DOY_TODAY=`date +%j`
sh-2.05b$ let MOD_BEGIN=${DOY_TODAY}-${DOY_BEGIN}
sh-2.05b$ let MOD_END=${DOY_TODAY}-${DOY_END}-2
sh-2.05b$ echo $MOD_BEGIN
117
sh-2.05b$ echo $MOD_END
17
sh-2.05b$ find . -name "*" -prune -type f -mtime +$MOD_END -mtime -$MOD_BEGIN -ls
[...]
sh-2.05b$ mkdir tmp_mail_to_move-remove
sh-2.05b$ find . -name "*" -prune -type f -mtime +$MOD_END -mtime -$MOD_BEGIN -ls \
  -exec cp -p {} tmp_mail_to_move-remove/ \; | tee log.find.mail_to_move-remove.txt
1860698    4 -rw-rw-r--   1 bmsnook  bmsnook      1507 Aug  7 12:54 ./.mozilla/appreg
474875    4 -rw-r--r--   1 bmsnook  bmsnook      2816 Jul 28 12:53 ./.ssh/authorized_keys
1012168    0 -rw-r--r--   1 bmsnook  bmsnook         0 Sep 12 15:10 ./test.scp.ssh1
1012707    0 -rw-r--r--   1 bmsnook  bmsnook         0 Sep 12 15:10 ./test.scp.ssh2
1012708    4 -rw-r--r--   1 bmsnook  bmsnook      2834 Sep 12 15:14 ./log_script.devhost.scp.ssh1.txt
1013370    4 -rw-r--r--   1 bmsnook  bmsnook      1135 Jul 21 18:37 ./compare.uue
1012711    4 -rw-r--r--   1 bmsnook  bmsnook      3590 Sep 19 13:39 ./fubar.csv
1012709   12 -rw-r--r--   1 bmsnook  bmsnook      8904 Sep 12 15:14 ./log_script.wicket.scp.ssh2.txt
1013378   12 -rw-r--r--   1 bmsnook  bmsnook      8506 Jun 26 18:11 ./servers.SExxxxxxxx.txt.noc.nss.stm
1013412   20 -rw-r--r--   1 bmsnook  bmsnook     17765 Jun 26 17:52 ./servers.SExxxxxxxx.txt
1013411    4 -rw-r--r--   1 bmsnook  bmsnook       766 Jun 26 18:11 ./servers.SExxxxxxxx.txt.east
1012715    4 -rw-r--r--   1 bmsnook  bmsnook       197 Sep 19 14:22 ./list.db
1013413    4 -rw-r--r--   1 bmsnook  bmsnook       783 Jun 26 18:11 ./servers.SExxxxxxxx.txt.internal
1012710    4 -rw-r--r--   1 bmsnook  bmsnook       150 Sep 19 14:20 ./list.seth.sorted
1012713    4 -rw-r--r--   1 bmsnook  bmsnook       150 Sep 19 14:20 ./list.seth
1013414    4 -rw-r--r--   1 bmsnook  bmsnook      3851 Jun 26 18:11 ./servers.SExxxxxxxx.txt.west
1013415    4 -rw-r--r--   1 bmsnook  bmsnook      1899 Jun 26 18:11 ./servers.SExxxxxxxx.txt.misc
1013418    4 -rw-r--r--   1 bmsnook  bmsnook       208 Jun 27 17:18 ./servers.SExxxxxxxx.all.small
1013417    4 -rw-r--r--   1 bmsnook  bmsnook      1098 Jun 27 17:13 ./servers.SExxxxxxxx.all
1012714    4 -rw-r--r--   1 bmsnook  bmsnook       197 Sep 19 14:22 ./list.db.sorted
1013419    4 -rw-r--r--   1 bmsnook  bmsnook      1205 Jun 30 15:04 ./locations.cubes.bldg1.txt
1013420    4 -rw-r--r--   1 bmsnook  bmsnook       112 Jul  2 10:53 ./ips.host.company
1013333    4 -rw-r--r--   1 bmsnook  bmsnook       294 Oct  2 12:42 ./delete-me_oct02_file.txt
sh-2.05b$ ls -al tmp_mail_to_move-remove
[...]

Last edited by BrianSnook; 10-21-2008 at 01:25 PM. Reason: Added code tags
 
1 members found this post helpful.
Old 10-21-2008, 12:00 PM   #6
BrianSnook
LQ Newbie
 
Registered: Jan 2008
Location: Atlanta, GA
Distribution: FreeBSD, RH9, RHEL5, CentOS5
Posts: 3

Rep: Reputation: 1
Exclamation DANGER: " | rm -rf *" does NOT do what you think

Quote:
Originally Posted by sr_sanjeev View Post
Hi
Either you can write a small script or use like

ls -alrt | grep 2008 | rm -rf *

but before doing that , please ensure that first 2 commands are serving your purpose or not?
Quote:
Originally Posted by Penn View Post
OK......Im going to move the mails to another directory first before i remove it. When i use the following:
"ls -alrt | grep 2008 | mv -rf * /directory"

then i recieve

"Argument list too long"
Do yourself a favour and do not use this syntax. However well-intentioned we may presume this suggestion was, there are a few problems:

sr_sanjeev, while "ls -alrt" will certainly provide a long (-l) listing of all (-a) sorted by reversed (-r) time (-t), this in no way actually filters based on any dates at all. Using the "-rt" certainly helps you the human get a listing that's easier to see file modification dates, but it doesn't help the computer and provides no help for the subsequent "grep" or "mv"/"rm" commands.

Penn, you were wise to change the originally-suggested "rm" to a "mv", but the "-r" flag doesn't make sense for a "mv" (any directory moved will automatically "take" its files with it). Grepping 2008 out of the output will give you all the lines that have 2008 in them, which means that you could conceivably get files before or after you intend. This also ignores the fact that most copies of "ls" will only show the year for long listings over a certain age, so most listings will have date strings like "Mar 17", "Aug 14", or "Oct 2" that don't include the 2008 this command would pick up on.

In addition, even if you did grep a long listing, grep is providing the _whole_ line, including the file permissions, owner, etc., in addition to the filename.

ledow, you're on the right track as far as passing filenames to "rm" via a pipe: you would need to use "xargs" to pass multiple options from the ls/grep output streams to the rm command, but you'd also need to remove the "*" (details below).

Unfortunately, if you passed the grep output to "xargs rm" you'd be telling "rm" to not only delete the filenames that grep returns, but also any filenames that happened to match anything else on those returned lines, including the username that owns them, the directory permissions, the parts of the datestamps, and anything else on the line separated by whitespace, really.

Finally, and most importantly, the wildcard "*" is expanded by the shell _before_ any of the commands are executed, which means that "*" is still the wildcard for all the files in your current directory, regardless of what other commands you pass to "rm", either directly or via a pipe. Adding "rm -rf *" to _anything_ means you're trying to delete _all_ the files and directories in your current directory (wherever that is), regardless of any flags, arguments, or piped input. If you want an illustration of what you're trying to delete, add an "echo" in front of the original "rm" (instead of the "mv" you put), and you'll see what you were telling it to delete.

If you want to see what I mean, just do this (please KEEP the echos where I put them to ensure you're not actually deleting anything accidentally):

Code:
ls -l | grep 2008 | echo rm *
or this:
Code:
ls -l | grep 2008 | xargs echo rm *
Here's an illustration in a "safe" directory for testing purposes:

Code:
% mkdir tmp_never_pipe_to_rm-rf-star
% cd tmp_never_pipe_to_rm-rf-star
% touch all in zdirectory
% touch -r ../.bashrc files
% touch -r ../.gnome here
% touch -r ../.xscreensaver this
% ls -l
total 0
-rw-r--r--    1 bmsnook  bmsnook         0 Oct 21 11:59 all
-rw-r--r--    1 bmsnook  bmsnook         0 May 23 12:32 files
-rw-r--r--    1 bmsnook  bmsnook         0 Jun 29  2007 here
-rw-r--r--    1 bmsnook  bmsnook         0 Oct 21 11:59 in
-rw-r--r--    1 bmsnook  bmsnook         0 Mar 21  2007 this
-rw-r--r--    1 bmsnook  bmsnook         0 Oct 21 11:59 zdirectory
%
Now look what happens when we pipe a grep of this listing to "rm *":

Code:
% ls -l | grep 2007 | echo rm *
rm all files here in this zdirectory
Now, note that not only did it not take any filtering based on our grep, but it just plain wants to delete everything. You'd need "xargs", but you'd also want to remove the "*". Compare with and without the wildcard:

Code:
## With the wildcard
% ls -l | grep 2007 | xargs echo rm *
rm all files here in this zdirectory -rw-r--r-- 1 bmsnook bmsnook 0 Jun 29 2007 here -rw-r--r-- 1 bmsnook bmsnook 0 Mar 21 2007 this

## Without the wildcard
% ls -l | grep 2007 | xargs echo rm
rm -rw-r--r-- 1 bmsnook bmsnook 0 Jun 29 2007 here -rw-r--r-- 1 bmsnook bmsnook 0 Mar 21 2007 this

Obviously, grep alone isn't what we want. The simplest way to get the filename if you're doing a long listing-based filter is to use awk and print "$NF" (the field in the position of the number of fields in the line, which is the last field):

Code:
% ls -l | awk '/2007/{print $NF}' | xargs echo rm
rm here this
That's closer, but you're still deleting everything based on a year, without any granularity for month or day. Here is a straightforward way to get the granularity you want with the "date" and "find" commands built into your system. Please refer to my response in Post#5 ('use "date --date" and shell math to pass file ages to "find"') for a full-blown explanation of all the "whys" about the date stuff. Here are the commands to conduct a full-on test on your own system without deleting your own stuff.

Code:
mkdir -p tmp_find_date_delete/tmp_stuff_to_delete
cd  tmp_find_date_delete/tmp_stuff_to_delete
touch -t 200801010001 zfilebegin01
for month in {01..10}; do for day in 01 15 28; do \
  for time in 0030 1130 1330 1830 2330; do \
  touch -t 2008${month}${day}${time} zfiledelete${month}${day}${time}; \
  done; done; done
touch -t 200808310001 zfileend0001
touch -t 200808312359 zfileend2359
touch -t 200712310001 zfilekeep.200712310001
touch -t 200712311130 zfilekeep.200712311130
touch -t 200712311330 zfilekeep.200712311330
touch -t 200712312359 zfilekeep.20071231
touch -t 200809010001 zfilekeep.20080901
touch -t 200809011130 zfilekeep.20081130
touch -t 200809011330 zfilekeep.20081330
touch -t 200809012359 zfilekeep.20082359

DOY_BEGIN=`date --date 01/01/2008 '+%j'`
DOY_END=`date --date 08/31/2008 '+%j'`
DOY_TODAY=`date '+%j'`
let AGE_OLDEST=${DOY_TODAY}-${DOY_BEGIN}
let AGE_YOUNGEST=${DOY_TODAY}-${DOY_END}
echo ${AGE_OLDEST}
echo ${AGE_YOUNGEST}
mkdir ../tmp_move-remove_testdir
find . -mtime -${AGE_OLDEST} -mtime +${AGE_YOUNGEST} -ls
find . -mtime -${AGE_OLDEST} -mtime +${AGE_YOUNGEST} -ls -exec mv {} ../tmp_move-remove_testdir/ \;
Depending on what you find, you may need to play around with options like this:
Code:
let AGE_OLDEST=${DOY_TODAY}-${DOY_BEGIN}+1
let AGE_YOUNGEST=${DOY_TODAY}-${DOY_END}-1
and you may need to move files back before deleting the rest to avoid collateral damage, based on file modification times and when you run the script, which is another excellent reason to use mv instead of rm initially (i.e., I end up either missing some files from 2008-01-01 or inadvertently getting some files from 2007-12-31, depending on which I use, so I need to move some back or away).

Last edited by BrianSnook; 10-21-2008 at 01:40 PM. Reason: Added code tags
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
shell script to remove old files based on date WindozBytes Linux - General 12 06-04-2012 01:21 AM
shell script to find modified date and last accessed date of any file. parasdua Linux - Newbie 6 04-22-2008 09:59 AM
Setting system date and time affecting the clock and date on BIOS satimis Ubuntu 7 09-21-2007 08:02 AM
what is the correct syntax order for tar with --after-date DATE, --newer DAT farhan Linux - General 1 03-16-2007 08:43 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 12:39 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration