LinuxQuestions.org - [SOLVED] Centos 7 BASH find command not working from cron

- Linux - Server (https://www.linuxquestions.org/questions/linux-server-73/)

- - Centos 7 BASH find command not working from cron (https://www.linuxquestions.org/questions/linux-server-73/centos-7-bash-find-command-not-working-from-cron-4175597293/)

Centos 7 BASH find command not working from cron

Hi all

I'm using this command to find .sql.7z files on a network share and delete files older than three days:

Code:

find /mnt/voiceArchive/*.sql.7z -type f -mtime +4 -exec rm -rf {} \;

This command runs fine if ran from the BASH shell via an xterm GUI session, on the same NFS-mounted share.

However, if I run the same above command from a crontab, it does not work. Find returns:

Code:

find: ‘/mnt/voiceArchive/*.sql.7z’: No such file or directory

It is as if the network share just isn't there for find to access, when find is ran from a crontab.

Since I am using an absolute path, why can't find see network shares if run from cron?

I've established that the share is mounted at the time the crontab runs by doing a df -mh and piping the results to a file.

How can I get BASH find, when run from a crontab, to "see into" mounted network shares?

If I move my test data off a network share onto a "real" directory in the local filing system, find works just fine in trimming files older than 3 days, when called from a crontab.

Any help or suggestions appreciated!

Thanks

Having the "*" in your path relies on the wildcard expansion. What happens if you create a small script:

Code:

#!/bin/bash

/bin/find /mnt/voiceArchive/*.sql.7z -type f -mtime +4 -exec /bin/rm -rf {} \;

and then call this script from cron?

Is there any reason you're using -exec rm -rf {} \; rather than -delete

To be honest, the more elegant:

Code:

find /mnt/voiceArchive/ -name "*.sql.7z" -type f -mtime +4 -delete

might fill your needs.

Quote:

Originally Posted by rylan76 (Post 5653818)

Code:

find /mnt/voiceArchive/*.sql.7z -type f -mtime +4 -exec rm -rf {} \;

are those files or folders?
are you sure you want to use

Code:

rm -rf

???
(not actually sure what happens when using '-r' on files, not folders)

Quote:

Originally Posted by TenTenths (Post 5653842)

Having the "*" in your path relies on the wildcard expansion. What happens if you create a small script:

Code:

#!/bin/bash

/bin/find /mnt/voiceArchive/*.sql.7z -type f -mtime +4 -exec /bin/rm -rf {} \;

and then call this script from cron?

Is there any reason you're using -exec rm -rf {} \; rather than -delete

To be honest, the more elegant:

Code:

find /mnt/voiceArchive/ -name "*.sql.7z" -type f -mtime +4 -delete

might fill your needs.

Hi thx for replying.

If I try the first suggestion, I get the behaviour that nothing exists (exactly the failure mode cron exhibits with my original script):

Code:

/bin/find: ‘/mnt/voiceArchive/*.sql.7z’: No such file or directory

I'm using rm -rf {} as that is what the example I found looks like.

I have tried -delete (the second statement you suggest) - only difference is it now fails silently instead of emitting an error as it does when "exec rm -rf {} \;" is used... i. e. it fails apparently for the same reason (trying to "find" on a NFS network share) in exactly the same way.

Files weeks old still stick in the network folder.

I find it very strange that it appears true to state that find fails completely on any date-related processing when the filesystem it accesses to check file dates on is mounted via NFS... the NFS mounted filesystem is ext4 (just like the local physically mounted filesystem.)

The moment any of the two above statements is run via cron on a locally mounted ext4 filesystem (physically connected to the motherboard, not mounted via NFS / network) it works perfectly.

Thanks for the assistance - any other pointers?

I think I'll have to give up on find or BASH in total and write a C application or a JAVA app to do this - it seems find is incapable of finding files older than 3 days if you specify a NFS share to be searched.

As you explained in your first post .sql.7z are filenames, not directories - the syntax you tried is incorrect. Instead try this:

first try to use find only, without delete:

Code:

find <dir> -type -f -name <pattern>

where <dir> is probably /mnt/voiceArchive if you want to look for files inside this directory. use '*.sql.7z' as pattern if you want to find files named something.sql.7z.
You cannot use: find <file> ..... (also seem man find about that)

If this worked for you you can specify time, add -mtime 4 or whatever you need and check if that prints really what you need.
Finally if you want to remove those files add -delete. Do not use rm -rf because it will silently delete anything found (and since you did not specify any filter on filenames it will delete every file without questions).

Hi Pan64

Thx for replying!

I tried what you recommended, but I still have exactly the same problem.

E. g. find works correctly, finding files older than 4 days.

It then deletes them if I use -delete

But this ONLY works if I execute the script containing the find command from a terminal session, manually, by hand, on an NFS mounted share that contains the files.

The moment I test or try to run this in a crontab, on an NFS mounted share that contains the files, both the "just list the files" test script and the "actual delete" test script using the find command fails - but ONLY if I try to run it on a NFS mounted share.

If I run the script on a local directory, it works fine in both cron and the terminal. The find command itself successfully finds all files it is supposed to delete.

If I run the script on an NFS mounted directory, it works only in a terminal session, and never in cron. The find command itself fails to find any files to delete.

Other BASH commands (like a simple ls with a pattern) work in both cron and terminal.

What gives? Environment variables - what environment variable(s) does find require to be able to list files on NFS shares?

Quote:

Originally Posted by rylan76 (Post 5659887)

Other BASH commands (like a simple ls with a pattern) work in both cron and terminal.

For what it's worth, bash is an interface and a scripting language, so are ksh and zsh. None have much to do with the programs you are using them to invoke, such as find, except maybe that programs available in the directories listed in the $PATH environment variable don't need their full paths typed out. The contents of the $PATH variable are one of the differences between cron and the interactive shell.

Please post the exact find command you are using in the interactive shell (terminal) including the exact options. My guess is that you are putting find something in cron instead of using the full path like with /bin/find something

http://www.linuxquestions.org/questi...gs-4175464257/
yes, please post the real commands you tried, what's happened, what did you wanted to achieve exactly and how do you know that did not work?

Code:

/bin/find /mnt/voiceArchive/ -iname '*.sql.7z' -type f -mtime +4 -delete

Hi guys

Thanks for replying.

Ok, here's it all in detail.

All of this is on a Centos 7 box (System A) working on an NFS share /mnt/voiceArchive, shared from a Centos 6.5 box (System B)

System A is where the data is generated (MySQL server), to be backed up to System B (the archive machine)

System A

Code:

df -mh



Filesystem                                  Size  Used Avail Use% Mounted on

/dev/sda2                                  1.8T  37G  1.7T  3% /

devtmpfs                                    3.8G    0  3.8G  0% /dev

tmpfs                                      3.8G    0  3.8G  0% /dev/shm

tmpfs                                      3.8G  387M  3.5G  10% /run

tmpfs                                      3.8G    0  3.8G  0% /sys/fs/cgroup

192.168.10.1:/mnt/Development/VoiceRecording  1.8T  1.5T  306G  83% /mnt/voiceArchive

tmpfs                                      777M    0  777M  0% /run/user/1001

tmpfs                                      777M    0  777M  0% /run/user/0

System B

Code:

cat /etc/exports



/mnt/Development/VoiceRecording        192.168.10.2(rw,sync,no_root_squash)

Crontab on System A

Code:

SHELL=/bin/bash

45 17 * * 1,2,3,4,5 /bin/bash -f /root/backup_mysql_to_voice_archive_disk_xtrabackup.sh

/root/backup_mysql_to_voice_archive_disk_xtrabackup.sh on System A:

Code:

#!/bin/bash

clear

NOW=$(date +"%Y-%m-%d %H:%M:%S")

FILEDATE=$(date +"%Y-%m-%d")

BACKUPDUMPPATH="/mnt/voiceArchive/server_mysql_bak"

BACKUPDUMPFILE="$BACKUPDUMPPATH/server_db_bak_$FILEDATE.sql"

XTRABACKUPPATH="$BACKUPDUMPPATH/xtrabackup/"

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/bin:$BACKUPDUMPPATH:$XTRABACKUPPATH



LOGDETAILPATH=/root/mysql_backup_detail.txt

LOGXTRABACKUPPATH_P1=/root/xtrabackup_result_detail_phase_1.txt

LOGXTRABACKUPPATH_P2=/root/xtrabackup_result_detail_phase_2.txt



echo "$NOW - Starting... see mysql_backup_detail.txt for details."



echo "$NOW - Begin MySQL backup sequence." > $LOGDETAILPATH



echo "$NOW - Remove expired mysqldump dump files..." >> $LOGDETAILPATH



#find '$BACKUPDUMPPATH/*.sql.7z' -type f -mtime +4 -exec rm -rf {} \; >> $LOGDETAILPATH 2>&1

find $BACKUPDUMPPATH -name "*.sql.7z" -type f -mtime +4 -delete >> $LOGDETAILPATH 2>&1



RMRESULT=$?



NOW=$(date +"%Y-%m-%d %H:%M:%S")



if [ $RMRESULT -ne 0 ]; then

        echo "$NOW - ERROR returned from clearing expired dump files: $RMRESULT" >> $LOGDETAILPATH



        echo "$NOW - ERROR returned from clearing expired dump files: $RMRESULT"

else

        echo "$NOW - Removing expired dump files successful." >> $LOGDETAILPATH

fi



echo "$NOW - Dumping MySQL data..." >> $LOGDETAILPATH



/usr/bin/mysqldump --single-transaction=TRUE --quick=TRUE -uroot -psecretpassword sourcedb --events --routines --triggers > $BACKUPDUMPFILE



DUMPRESULT=$?



NOW=$(date +"%Y-%m-%d %H:%M:%S")



if [ $DUMPRESULT -ne 0 ]; then

        echo "$NOW - ERROR returned from mysqldump: $DUMPRESULT" >> $LOGDETAILPATH



        echo "$NOW - ERROR returned from mysqldump: $DUMPRESULT"



        exit

else

        echo "$NOW - Mysqldump successful." >> $LOGDETAILPATH

fi



echo "$NOW - Compressing $BACKUPDUMPFILE..." >> $LOGDETAILPATH



/usr/bin/7za a -t7z -m0=lzma -mx=9 -mfb=64 -md=64m -ms=on $BACKUPDUMPFILE.7z $BACKUPDUMPFILE >> $LOGDETAILPATH 2>&1



SEVENZRESULT=$?



NOW=$(date +"%Y-%m-%d %H:%M:%S")



if [ $SEVENZRESULT -ne 0 ]; then

        echo "$NOW - ERROR returned from 7zip of mysqldump file: $SEVENZRESULT" >> $LOGDETAILPATH



        echo "$NOW - ERROR returned from 7zip of mysqldump file: $SEVENZRESULT"



        exit

else

        echo "$NOW - 7zip of $BACKUPDUMPFILE successful." >> $LOGDETAILPATH



        echo "$NOW - Removing source dumpfile after compression..." >> $LOGDETAILPATH



        rm -rf $BACKUPDUMPFILE



        RMRESULT=$?

        if [ $RMRESULT -ne 0 ]; then

                echo "$NOW - ERROR returned when removing source dumpfile $BACKUPDUMPFILE after compression: $RMRESULT" >> $LOGDETAILPATH



                echo "$NOW - ERROR returned when removing source dumpfile $BACKUPDUMPFILE after compression: $RMRESULT"



                exit

        else

                echo "$NOW - Removal of source dumpfile $BACKUPDUMPFILE after compression successful." >> $LOGDETAILPATH

        fi

fi



echo "$NOW - Removing expired xtrabackup 7z files..." >> $LOGDETAILPATH



#find '$XTRABACKUPPATH*.7z' -type f -mtime +4 -exec rm -rf {} \; >> $LOGDETAILPATH 2>&1

find $XTRABACKUPPPATH -name "*.7z" -type f -mtime +4 -delete >> $LOGDETAILPATH 2>&1



RMRESULT=$?



NOW=$(date +"%Y-%m-%d %H:%M:%S")



if [ $RMRESULT -ne 0 ]; then

        echo "$NOW - ERROR returned when removing expired xtrabackup 7z files: $RMRESULT" >> $LOGDETAILPATH



        echo "$NOW - ERROR returned when removing expired xtrabackup 7z files: $RMRESULT"

else

        echo "$NOW - Removal of expired xtrabackup 7z files successful." >> $LOGDETAILPATH

fi



echo "$NOW - Running innobackupex phase 1..." >> $LOGDETAILPATH



innobackupex --user=root --password=secretpass $XTRABACKUPPATH 2> $LOGXTRABACKUPPATH_P1



NOW=$(date +"%Y-%m-%d %H:%M:%S")



echo "$NOW - Innobackupex phase 1 complete, checking if successful..." >> $LOGDETAILPATH



grep -R "completed OK" $LOGXTRABACKUPPATH_P1



GREPRESULT1=$?



NOW=$(date +"%Y-%m-%d %H:%M:%S")



if [ $GREPRESULT1 -ne 0 ]; then

        echo "$NOW - ERROR returned when running innobackupex phase 1: $GREPRESULT1" >> $LOGDETAILPATH



        echo "$NOW - Cannot proceed with backup!" >> $LOGDETAILPATH



        echo "$NOW - ERROR returned when running innobackupex phase 1: $GREPRESULT1"



        echo "$NOW - Cannot proceed with backup!"



        exit

else

        echo "$NOW - Innobackupex phase 1 successful." >> $LOGDETAILPATH

fi



NEWESTDIR=$(ls -t $XTRABACKUPPATH | head -1)



echo "$NOW - Running innobackupex phase 2 on $XTRABACKUPPATH$NEWESTDIR ..." >> $LOGDETAILPATH



innobackupex --apply-log $XTRABACKUPPATH$NEWESTDIR 2> $LOGXTRABACKUPPATH_P2



NOW=$(date +"%Y-%m-%d %H:%M:%S")



echo "$NOW - Innobackupex phase 2 complete, checking if successful..." >> $LOGDETAILPATH



grep -R "completed OK" $LOGXTRABACKUPPATH_P2



GREPRESULT1=$?



NOW=$(date +"%Y-%m-%d %H:%M:%S")



if [ $GREPRESULT1 -ne 0 ]; then

        echo "$NOW - ERROR returned when running innobackupex phase 2: $GREPRESULT1" >> $LOGDETAILPATH



        echo "$NOW - BACKUP HAS FAILED!!!" >> $LOGDETAILPATH



        echo "$NOW - ERROR returned when running innobackupex phase 2: $GREPRESULT1"



        echo "$NOW - BACKUP HAS FAILED!!!"



        exit

else

        echo "$NOW - Innobackupex phase 2 successful." >> $LOGDETAILPATH



        NOW=$(date +"%Y-%m-%d %H:%M:%S")



        echo "$NOW - Compressing $XTRABACKUPPATH$NEWESTDIR ..." >> $LOGDETAILPATH



        tar cvf $XTRABACKUPPATH$NEWESTDIR.tar $XTRABACKUPPATH$NEWESTDIR



        /usr/bin/7za a -t7z -m0=lzma -mx=9 -mfb=64 -md=64m -ms=on $XTRABACKUPPATH$NEWESTDIR.tar.7z $XTRABACKUPPATH$NEWESTDIR.tar >> $LOGDETAILPATH 2>&1

        SEVENZRESULT=$?



        NOW=$(date +"%Y-%m-%d %H:%M:%S")



        if [ $SEVENZRESULT -ne 0 ]; then

                echo "$NOW - ERROR returned from 7zip of xtrabackup directory: $XTRABACKUPPATH$NEWESTDIR" >> $LOGDETAILPATH



                echo "$NOW - ERROR returned from 7zip of xtrabackup directory: $XTRABACKUPPATH$NEWESTDIR"



                else

                echo "$NOW - 7zip of $XTRABACKUPPATH$NEWESTDIR successful." >> $LOGDETAILPATH



                echo "$NOW - Removing .tar file of directory after compression." >> $LOGDETAILPATH



                rm -rf $XTRABACKUPPATH$NEWESTDIR.tar



                echo "$NOW - Removing source directory after compression..." >> $LOGDETAILPATH



                rm -rf $XTRABACKUPPATH$NEWESTDIR



                RMRESULT=$?

                if [ $RMRESULT -ne 0 ]; then

                        echo "$NOW - ERROR returned when removing source xtrabackup directory after compression: $RMRESULT" >> $LOGDETAILPATH



                        exit

                else

                        echo "$NOW - Removal of source xtrabackup directory after compression successful." >> $LOGDETAILPATH

                fi

        fi



        echo "$NOW - Backup complete." >> $LOGDETAILPATH



        echo "$NOW - Backup complete."

fi

The salient lines are

Code:

find $BACKUPDUMPPATH -name "*.sql.7z" -type f -mtime +4 -delete >> $LOGDETAILPATH 2>&1



find $XTRABACKUPPPATH -name "*.7z" -type f -mtime +4 -delete >> $LOGDETAILPATH 2>&1

The above two lines work (deleting *.sql.7z and *.7z files, older than four days), if the script is called from a terminal session via

Code:

# bash -f /root/backup_mysql_to_voice_archive_disk_xtrabackup.sh

However, if put in a crontab, e.g.:

Code:

SHELL=/bin/bash

45 17 * * 1,2,3,4,5 /bin/bash -f /root/backup_mysql_to_voice_archive_disk_xtrabackup.sh

The above two lines do not work and cannot find any *.sql.7z and *.7z files, older than four days, to delete, on the /mnt/voiceArchive NFS share.

If put in a crontab, as shown above, and the target files referenced by the script /root/backup_mysql_to_voice_archive_disk_xtrabackup.sh in the lines

Code:

find $BACKUPDUMPPATH -name "*.sql.7z" -type f -mtime +4 -delete >> $LOGDETAILPATH 2>&1



find $XTRABACKUPPPATH -name "*.7z" -type f -mtime +4 -delete >> $LOGDETAILPATH 2>&1

are NOT on an NFS share, the find operation to delete files older than four days DOES work.

So the factor that breaks the above two bash find statements -appears- to be that find in Centos 7 on my system A, cannot find files by date on the NFS share shared from my system B.

This is ludicrous and cannot be, so I'm doing something wrong.

But what?

The error determination procedure is that if the crontab is left to run over a few weeks, files accumulate on the NFS share that are days and weeks older than four days in the past. Files older than four days are never deleted - e. g. the find commands fail to find files older than four days on the NFS share, and fails to delete them.

Again, thanks for your time, if you need more specific info please advise I'm happy to provide it.

Regards

You might insert these as the second and third lines, to aid in the debugging of the script.

Code:

set -e

set -x

Hi Turbocapitalist

Ok, I'll insert those for tonight's run and see what pops up.

Thank you

:)

date is ok on your hosts?

I think so:

System A (Centos 7)

Code:

# date

Wed Jan 25 15:40:56 SAST 2017

System B (Centos 6)

Code:

# date

Wed Jan 25 15:40:48 SAST 2017

The files on System B:

Code:

# ls -l --full-time

total 1430556

-rwxr-xr-x 1 root root 330491013 2017-01-20 18:12:16.815201573 +0200 verdi_ho_server_db_bak_2017-01-20.sql.7z

-rwxr-xr-x 1 root root 358683926 2017-01-23 18:11:59.003875748 +0200 verdi_ho_server_db_bak_2017-01-23.sql.7z

-rwxr-xr-x 1 root root 381712761 2017-01-24 18:15:51.026692831 +0200 verdi_ho_server_db_bak_2017-01-24.sql.7z



#pwd

/mnt/voiceArchive/verdi_ho_server_mysql_bak

E. g. today being Jan 25 it appears logically that at least

Code:

-rwxr-xr-x 1 root root 330491013 2017-01-20 18:12:16.815201573 +0200 verdi_ho_server_db_bak_2017-01-20.sql.7z

having a date of 2017-01-20, must be caught by

Code:

find /mnt/voiceArchive/verdi_ho_server_mysql_bak -name "*.sql.7z" -type f -mtime +4 -delete

in a crontab, but it is only caught if ran in a terminal session, by hand.

Thx again.