LinuxQuestions.org - How to search for missing files and pass their names on to another shell script

- Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)

- - How to search for missing files and pass their names on to another shell script (https://www.linuxquestions.org/questions/linux-newbie-8/how-to-search-for-missing-files-and-pass-their-names-on-to-another-shell-script-745621/)

How to search for missing files and pass their names on to another shell script

Hi all,

I'm pretty n00bish, If you can help me with this question, I will be eternally grateful. I have a folder full of data, but occasionally some data goes missing. I need to write a script that scans through my data directory, finds the missing files, and sends the file name to a script that downloads it. The file names follow a particular pattern. Here's an example:

~/data/032/BRIB.Q1.03209000000.d
~/data/032/BRIB.Q2.03209000000.d
~/data/032/BRIB.Q4.03209000000.d

In this case, my script should notice that Q3 is missing and run a shell script (that I've already written) to get it. It needs to be able to search through all of the folders in in the data directory. I'm thinking I can create a list of file "prefixes" like "BRIB.Q1", "BRIB.Q2"... etc that it needs to match up with.

TL,DR:
What kind of commands do I need to look at to write this script? How do I recursively search through directories, checking to make sure a list of files following a particular pattern exist?

Danny

Quote:

I have a folder full of data, but occasionally some data goes missing.

A more logical approach might be to find out just why "some data goes missing".

Then you could fix the problem that is causing the problem (rather than "patching it up") and the problem would go away. Writing scripts to mask a problem you have not completely understood is not going to help you in the long term.

You need to investigate exactly why, or how "some data goes missing", and fix that.

Then your problem will be solved.

If you need further advice, please post more details.

It depends how general you need the script to be. For the example you've given, (and generalizable to any situation where the pattern is simply an incrimenting number, you could:

Code:

#!/bin/bash



pattern="~/data/032/BRIB.Q\$n.03209000000.d"

get_missing=<insert path and script name here>



for n in $(seq 1 4); do

  file=$(eval echo "$pattern")

  [ ! -e "$file" ] && $get_missing "$file"

done

Since you are new to this, here is a bit of a "blow by blow" explanation of the script:

I told the script to use bash because I've run into trouble on Ubuntu when it uses dash for sh. When I defined $pattern, I escaped the dollar sign on $n with a backslash so substitution was not done at that step. seq, in this case will generate the sequence "1 2 3 4" which the for loop will iterate through. Each time through the loop, $file will get assigned a new value based on $n. (If anybody knows of a cleaner way to do that step, please let us know.) Finally it tests for the existance of $file and if it doesn't exist, it calls your script with the missing file name. If the files you are looking for are regular files, you could use -f instead of -e. (Look at the bash man page for information on the builtin command test.)

More general scripts, or scripts that need to iterate through letters, will of course, be more complicated.

Quote:

You need to investigate exactly why, or how "some data goes missing", and fix that.

I wrote that because I didn't want to make the post too long with unnecessary information. I have a cron job running that downloads the data, which is being constantly generated, from a remote server. To get the data, I run someone else's program to generate it, which messes up sometimes. It's just weird and I don't have much control over it.