delete 1 month-old files except when if only a number of files are left

vincix · 05-15-2018, 12:03 PM

I'm doing a daily backup for a mysql database and once a month I delete all files that are older than a month. So there are about 30-31 files per month, and double that, the moment I run the clean-up command.

Code:

if [ $(ls /path/to/dir/mysql* | wc -l) -lt 31 ]; then 
    echo "There are too few files. Not proceeding." && exit 2
else
    find /path/to/dir/ -name "mysql*" -mtime +30 -exec rm rf {} \;
fi

Is there a better, cleaner and safer way to do this than using wc to count the backup files?

Thanks

rtmistler · 05-15-2018, 12:21 PM

I would say that solution for the very custom decision you've reached is fine. However, I feel that older files no matter how many of them are either worthless or they are not depending upon the age. Therefore I might lend towards solely the line in your else clause and leave it at that. I'm sure you have your reasons for wishing to retain some small number of files.

michaelk · 05-15-2018, 12:37 PM

Are you creating daily backups? There are many ways to delete old files. How do you run your script? How are your files named?

One method would be to label your backup files as date_mysql_backup where the date could be the current month i.e. 05_xyx_mysql_backup. Create a cron job to delete your scripts on the 1st day of the month.

Code:

MM HH 1 * * /path/to/my/script

#!/bin/bash
old_files=$(date +'%m' -d 'last month')
rm old_files*

Do you care about keeping just a months worth of backups or an entire month as defined by the days of the month (1-31). If just the past 31 days of backups regardless of the exact day then run a cron script everyday.

Code:

MM HH * * * find /path/to/dir/ -name "mysql*" -mtime +31 -exec rm rf {} \;

Most do not understand that with ASCII sorting labeling files using the date-name (YYYYMMDD-name) is easier to script and automatically sorts versus name-MMDDYYYY.

rtmistler · 05-15-2018, 12:43 PM

I supposed a real other way to contend with this is to use logrotate and specify the age and number of files limits for rotation, along with the number of backups intended to be kept. Plus you can zip the backups so that the results will be smaller, assuming they are text files in the first place. Logrotate is not just for system logs, it also does not have to be for files ending in .log, it can be for any directory, any file extension, or even no extension, it is that flexible.

michaelk · 05-15-2018, 12:45 PM

logrotate would be the obvious answer....

MadeInGermany · 05-15-2018, 02:20 PM

Quote:

Originally Posted by vincix

I'm doing a daily backup for a mysql database and once a month I delete all files that are older than a month. So there are about 30-31 files per month, and double that, the moment I run the clean-up command.

Code:

if [ $(ls /path/to/dir/mysql* | wc -l) -lt 31 ]; then 
    echo "There are too few files. Not proceeding." && exit 2
else
    find /path/to/dir/ -name "mysql*" -mtime +30 -exec rm rf {} \;
fi

Is there a better, cleaner and safer way to do this than using wc to count the backup files?

Thanks

You are deleting files and directories that are older than 30 days.
ls is okay, but a shell builtin is elegant+efficient

Code:

argcnt(){ echo $#; }
if [ $(argcnt /path/to/dir/mysql*) -lt 31 ]
then
    echo "There are too few files. Not proceeding." && exit 2
else
    find /path/to/dir/ -maxdepth 1 -type f -name "mysql*" -mtime +30 -delete
fi

As was stated already, in most practical cases (one file per day is produced) the find statement is sufficient.

vincix · 05-16-2018, 08:56 AM

This is how I create my mysqldump backup files:

Code:

export mysqlpwd=mypassword
mysqldump -u root -p$mysqlpwd --single-transaction --routines --triggers --databases my_db > /mnt/backup_orange_nfs/mysql/my_db_$(date +%F.%H:%M:%S).sql

@michaelk So I guess my naming method is name-YYYYMMDD, which is neither. But your point is that, besides the format, the date should stand in front of the name.
So given this, the find command actually looks for my_db* (not mysql, as state initially - I simply showed how I did it in my first post, wasn't entirely accurate in this respect)

I'll have a look at logrotate. It seems logic to make use of a system which is already in place in linux. It's just that I don't have lots of experience with logrotate and sometimes I feel it's a little bit daunting, not sure why. It doesn't seem that hard to understand

@MadeInGermany
I don't understand exactly how your function works. How does the number of position parameters ($#) gives the number of files in this context exactly? Can you use $# to count the number of files without using a function, just for the sake of demonstration?

MadeInGermany · 05-16-2018, 12:58 PM

The shell looks into the file system and expands /path/to/dir/mysql* to words.
The function is run with them as arguments. They are mapped to positional parameters within the function (these are indeed local to the function, not overwriting the positional parameters in the main script).

There might be a way to count words without using a function - but I don't know a good one.

keefaz · 05-16-2018, 03:36 PM

You can set positional parameters without affecting the script in the command substitution as it is executed in a subshell

Code:

if [ $(set -- /path/to/dir/mysql*; echo $#) -lt 31 ]; then...

vincix · 05-17-2018, 05:02 AM

Please remind me, what does -- stand for in this context?

Turbocapitalist · 05-17-2018, 05:07 AM

Quote:

Originally Posted by vincix

Please remind me, what does -- stand for in this context?

From the bash manual page, the function "set" under the section "shell builtin commands"

-- If no arguments follow this option, then the positional
parameters are unset. Otherwise, the positional parame‐
ters are set to the args, even if some of them begin
with a -.

See the manual about what + and - do for each parameter

MadeInGermany · 05-17-2018, 05:44 AM

The -- option means it is the last option. In the case of
set -- $var
and $var being "-brave world" the set command should not see options -b -r -a -v -e

Here we know that the arguments start with / so we can omit the --

keefaz · 05-17-2018, 08:29 AM

I noticed the trick to use positional parameters to count elements in a directory using globing gives wrong result when a directory is empty

Because when globbing doesn't occur then bash uses the globing string as first parameter, so $# will contain 1...

Code:

> echo $(set /path/to/empty/dir/*; echo $#)
1

Maybe we can add a test like

Code:

echo $(set /path/to/empty/dir/*; [ "${1: -1}" = '*' ] && echo 0 || echo $#)

Or just use nullglob

Code:

shopt -s nullglob
echo $(set -- /path/to/empty/dir/*; echo $#)

here the -- that follows set is required because without it, set will output all env variables

Code:

help set
...
If no ARGs are given, all shell variables are printed.

I know wrong count result with empty directory is irrelevant in this application, but just wanted to point this

vincix · 05-17-2018, 09:21 AM

Well, even if it might not affect the script itself, it doesn't seem like something you can easily ignore. Perhaps the 'ls' solution might be cleaner.

keefaz · 05-17-2018, 09:33 AM

I found pipes with ls don't always end nicely (filenames with spaces...)
Globbing is cool, keeping in mind its default behaviour with non results