I need a script to delete oldest files in directory to keep from using up all space

Clutch2 · 01-25-2024, 02:14 PM

My security system sends alert video to a couple pi's in case the pc that does the monitoring is stolen. The problem is sometimes I space off and forget to delete files before I run out of storage.

Anyone have a script that I can run as a cron job to keep the directory contents under a specified size by deleting the oldest files. The directory right now is half full and has 6000 files in it to give you an idea.

Thanks,

rtmistler · 01-25-2024, 02:32 PM

A job for logrotate

scasey · 01-25-2024, 02:54 PM

Or multilog…’tho log rotate is likely extant. multilog is not likely installed, unless OP is a DJB fan.

Clutch2 · 01-26-2024, 09:27 AM

I found a script that works for me. It was on another site. Am I allowed to post a link to it, I don't want to break any rules here.

////// · 01-26-2024, 09:32 AM

Quote:

Originally Posted by Clutch2

I found a script that works for me. It was on another site. Am I allowed to post a link to it, I don't want to break any rules here.

yes, you can post that script.

use code tags.

Clutch2 · 01-26-2024, 09:59 AM

From askunbuntu. I added a check to make sure all three parameters were supplied. It seems to work for me.

Code:

#!/bin/bash
#Usage = sh limit-directory-size.sh /media/computer/mypartition 88 120 ("/media/computer/mypartition" = the directory to be limited / "88"=the percentage of the total partition this directory is allowed to use / "120"=the number of files to be deleted every time the script loops (while $Directory_Percentage > $Max_Directory_Percentage)

if [ $# -lt 3 ]; then
  echo 1>&2 "$0: not enough arguments"
  exit 2
elif [ $# -gt 3 ]; then
  echo 1>&2 "$0: too many arguments"
  exit 2
fi
# The three arguments are available as "$1", "$2", "$3"

#Directory to limit
Watched_Directory=$1
echo "Directory to limit="$Watched_Directory

#Percentage of partition this directory is allowed to use
Max_Directory_Percentage=$2
echo "Percentage of partition this directory is allowed to use="$Max_Directory_Percentage

#Current size of this directory
Directory_Size=$( du -sk "$Watched_Directory" | cut -f1 )
echo "Current size of this directory="$Directory_Size

#Total space of the partition = Used+Available
Disk_Size=$(( $(df $Watched_Directory | tail -n 1 | awk '{print $3}')+$(df $Watched_Directory | tail -n 1 | awk '{print $4}') ))       
echo "Total space of the partition="$Disk_Size

#Curent percentage used by the directory
Directory_Percentage=$(echo "scale=2;100*$Directory_Size/$Disk_Size+0.5" | bc | awk '{printf("%d\n",$1 + 0.5)}')
echo "Curent percentage used by the directory="$Directory_Percentage

#number of files to be deleted every time the script loops (can be set to "1" if you want to be very accurate but the script is slower)
Number_Files_Deleted_Each_Loop=$3
echo "number of files to be deleted every time the script loops="$Number_Files_Deleted_Each_Loop

#While the current percentage is higher than allowed percentage, we delete the oldest files
while [ $Directory_Percentage -gt $Max_Directory_Percentage ] ; do
    #we delete the files
    find $Watched_Directory -type f -printf "%T@ %p\n" | sort -nr | tail -$Number_Files_Deleted_Each_Loop | cut -d' ' -f 2- | xargs rm
    #we delete the empty directories
    find $Watched_Directory -type d -empty -delete
    #we re-calculate $Directory_Percentage
    Directory_Size=$( du -sk "$Watched_Directory" | cut -f1 )
    Directory_Percentage=$(echo "scale=2;100*$Directory_Size/$Disk_Size+0.5" | bc | awk '{printf("%d\n",$1 + 0.5)}')
done

MadeInGermany · 01-26-2024, 10:17 AM

In the current directory leave the newest 5000 files and delete all the older ones:

Code:

ls -t . | sed '1,5000d' | xargs -rd '\n' rm

Quick and not too dirty. But does not consider the disk space.

Note that

xargs -rd '\n' rm

safely handles filenames with space characters, in contrast to just

xargs rm

Please correct that in your script!

Clutch2 · 01-26-2024, 11:33 AM

Thank you for the improvement. I edited the script.

michaelk · 01-26-2024, 11:57 AM

It depends on how much time you might want to spend to simplify the script.

Without knowing anything about your security system, files are usually named using a timestamp of YYYYMMDD... so they naturally sort without using their real timestamp. I also assume the alert is a certain number of seconds so I would guess their size is fairly the same. If true you should be able to calculate the number of files to delete and delete them using just one rm command.

logrotate may not be enough in this case because the number of files per day is random.

On the other hand, I would guess you really only need to keep a certain number of days of files so that would still keep it more efficient.

chrism01 · 02-06-2024, 09:25 PM

I'd say logrotate should be able to handle it, see eg https://opensource.com/article/21/10/linux-logrotate.
Just select a reasonable min num of files to keep. (ie the newest N files) using the 'rotate' option.
The rest gets deleted automatically unless otherwise specified eg call a script to scp them ....

Obviously try it on a test machine (create dummy file sets to test) until you're sure you have all the correct settings.

sundialsvcs · 02-06-2024, 09:42 PM

“logrotate” will already be in your crib tab sequence for other reasons and it is quite a general-purpose tool. It (or one of its brethren) should be ideal for this. Simply edit the configuration file.

jmccue · 02-07-2024, 09:09 AM

I am trying to understand your request, so I will rephrase:

You want to delete the oldest 'n' files no matter the how old? For example, if all your files that directory are less than say 1 day old, you want to delete the 'n' oldest files ?

If so, I would say MadeInGermany's should be fine.

But since this directory is a kind of security directory, I would suggest you delete all files older that a specified number of days.

For example, this will delete files with a mod date older than 2 days. This will allow you to resend files if the pi person said they did not get today's file(s).

Code:

find YOURDIR -type f -mtime +2 -exec rm '{}' \;

Clutch2 · 02-09-2024, 08:46 AM

I'm keeping alert video clips as long as I can given storage space on the machine I'm sending a copy to. The script I posted uses a percentage of storage and deletes in age order as I use up that percentage.

At this time I consider my problem solved.

Thanks ever one for the various approaches.

MadeInGermany · 02-09-2024, 10:57 AM

If something else fills up your disk, would your script delete all your alert video clips?
Maybe a second criterium makes sense, for ex only consider +2 days old files for deletion.