Find command returns files I don't want

keefaz · 02-21-2018, 11:21 AM

Quote:

Originally Posted by harboa

As I said in my original post..

For example, looking for a filename "*ER1_100*", it finds "EDI1160_Z_SHP_OBDLV_SAVE_REPLICA_22648345"

The error we got in the script was "file not found" as the "EDI1160" mentioned had already been moved to another directory bu another script and, in any case, wasn't a file that I wanted to perform any actions on.

Yes I read original post

But I still have not the complete picture, sorry I just want to know which exact command triggers the error, not a speculative reason

Ie, when you say find has returned EDI1160_Z_SHP_OBDLV_SAVE_REPLICA_22648345 while it has been told to search for *ER1_100*
I don't know where EDI1160_Z_SHP_OBDLV_SAVE_REPLICA_22648345 comes from, is there a ' echo $file ' executed first in the loop?
(if so, I know find returned wrong filename)

rknichols · 02-21-2018, 02:14 PM

Quote:

Originally Posted by pan64

As it was already explained by rknichols find first reads the directory and next it processes this list. It is a common problem if you modifies the directory during that process. find may fail just because it cannot follow that change. For example find still thinks the first entry in a directory matches, but in the meantime you removed that file from that dir, but a new one was created - so find will print the first file, which is now incorrect.

It won't do that. It stores the names, not some index number, in a list, then processes the names in the list. If some of those names don't exist, then it will get a "No such file or directory" error when it tries to stat() that name (which it needs to do for the other tests). There is no way to refer to a directory entry by anything other than the name.

I just realized, it might be find itself that is complaining that it can't stat() that moved file. In that case, it's just a harmless diagnostic, and your script will never see that name. Switching to "-exec" won't help with that. The name was there when find first read the directory, now it's gone.

You might try redirecting the stderr from just the find command onto a file. That way you can be sure of the source of the message.

pan64 · 02-22-2018, 05:50 AM

Quote:

Originally Posted by rknichols

It won't do that.

Yes, probably, I don't know the internals. But the other statements are still valid, modifying the directory content during the execution of find makes the result unreliable. That's why the whole process is still unsafe.
And there is still no answer to the question: why did find print a non-matching name? (see post #14)

MadeInGermany · 02-22-2018, 07:30 AM

Quote:

Originally Posted by harboa

...

Code:

for file in `find ${SOURCEDIR} -maxdepth 1 -name "${FILEMASK}" -type f -mmin +${FILEMIN}`; do

...

As was said already, the list in a for loop is not robust against special characters.
You can handle wildcards * ? [ ] with

Code:

set -f # do not glob file names

before the loop, but this still cannot handle space characters.
More robust is a while loop:

Code:

find ${SOURCEDIR} -maxdepth 1 -name "${FILEMASK}" -type f -mmin +${FILEMIN} |
while IFS= read -r file; do something with "$file"; done

This introduces a potential new problem because a while loop runs in a sub shell.
Also it does not solve a delay problem.
Using find -exec delays less, but of course the number of commands is limited.
If it is only an mv command:

Code:

find ${SOURCEDIR} -maxdepth 1 -name "${FILEMASK}" -type f -mmin +${FILEMIN} -exec mv -v {} targetdir \; >>logfile 2>&1

In the worst case you can chain a few commands

Code:

find ${SOURCEDIR} -maxdepth 1 -name "${FILEMASK}" -type f -mmin +${FILEMIN} -exec echo "moving" {} \; -exec mv {} targetdir \; -exec echo "moved" {} \; >>logfile 2>&1

harboa · 02-22-2018, 07:35 AM

Quote:

Originally Posted by MadeInGermany

As was said already, the list in a for loop is not robust against special characters.
You can handle wildcards * ? [ ] with

Code:

set -f # do not glob file names

before the loop, but this still cannot handle space characters.
More robust is a while loop:

Code:

find ${SOURCEDIR} -maxdepth 1 -name "${FILEMASK}" -type f -mmin +${FILEMIN} |
while IFS= read -r file; do something with "$file"; done

This introduces a potential new problem because a while loop runs in a sub shell.
Also it does not solve a delay problem.
Using find -exec delays less, but of course the number of commands is limited.
If it is only an mv command:

Code:

find ${SOURCEDIR} -maxdepth 1 -name "${FILEMASK}" -type f -mmin +${FILEMIN} -exec mv -v {} targetdir \; >>logfile 2>&1

In the worst case you can chain a few commands

Code:

find ${SOURCEDIR} -maxdepth 1 -name "${FILEMASK}" -type f -mmin +${FILEMIN} -exec echo "moving" {} \; -exec mv {} targetdir \; -exec echo "moved" {} \; >>logfile 2>&1

Thanks for that though I think I'm going to change approach on this now since it appears that find gets a list of all the files before applying the filters. I'll use ls FILEMASK (correctly coded of course) and some extra coding as this seems to properly achieve what I am after.

harboa · 02-22-2018, 07:45 AM

Quote:

Originally Posted by keefaz

Yes I read original post

But I still have not the complete picture, sorry I just want to know which exact command triggers the error, not a speculative reason

Ie, when you say find has returned EDI1160_Z_SHP_OBDLV_SAVE_REPLICA_22648345 while it has been told to search for *ER1_100*
I don't know where EDI1160_Z_SHP_OBDLV_SAVE_REPLICA_22648345 comes from, is there a ' echo $file ' executed first in the loop?
(if so, I know find returned wrong filename)

OK - probably not my best use of English! Basically, the script fails with a "file not found" statement. I happen to know that there is another script that runs for some files output from SAP that already moves those files to another area. So..
a) the find in my view shouldn't have "found" that file since it was not in the mask
b) when I look in the search directory the file is not there but is in the correct directory so the other script has correctly moved it

From a point of view of the code, bar some setting up and validation of some variable, all it had was this :-

Code:

            for file in `find ${SOURCEDIR} -maxdepth 1 -name "${FILEMASK}" -type f -mmin +${FILEMIN}`; do
                echo "                       Moving file ${file} from ${SOURCEDIR} to ${TARGETDIR}" >> ${LOGFILE}
                mv ${file} ${TARGETDIR}
            done

So I was fairly sure the find was returning the wrong filenames since there was nothing else it was doing..

harboa · 02-26-2018, 09:44 AM

Hi guys,

Thanks for all the help and suggestions on this..! I had a rethink and went for the following coding :-

Code:

CHKDATE=$(date -d "-${FILEMIN} min" +"%s")
for file in `ls ${SOURCEDIR}/${FILEMASK}`; do
	ModDate=$(date -r ${file} +%s)
	if [ "${ModDate}" -lt "${CHKDATE}" ] ; then
		mv ${file} ${TARGETDIR}
	fi
done

To re-cap - coding is to move a file (indicated by the FILEMASK) from SOURCEDIR to TARGETDIR if older than FILEMIN

Feel free to critique and suggest improvements..!

Cheers

keefaz · 02-26-2018, 11:09 AM

You could loop filenames with shell globing avoiding use of ls

Code:

shopt -s nullglob
for file in ${SOURCEDIR}/${FILEMASK}; do ...

pan64 · 02-26-2018, 11:11 AM

I don't think it will solve the problem what was described earlier. If the directory was modified during the execution of this script it will still fail.
And, if find found something strange I think you need to find the reason otherwise you may just jump into the same issue again. But you know if that works for you.

Additionally I would suggest you to use set -xv at the beginning of your script and you will see what's happening (will not solve anyting, just print a lot of information about the execution, program flow).
Additionally I would suggest you to use shellcheck (www.shellcheck.net) to check your script.