LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 08-06-2009, 02:21 PM   #1
djslothario
LQ Newbie
 
Registered: Mar 2009
Posts: 4

Rep: Reputation: 0
How to search for missing files and pass their names on to another shell script


Hi all,

I'm pretty n00bish, If you can help me with this question, I will be eternally grateful. I have a folder full of data, but occasionally some data goes missing. I need to write a script that scans through my data directory, finds the missing files, and sends the file name to a script that downloads it. The file names follow a particular pattern. Here's an example:

~/data/032/BRIB.Q1.03209000000.d
~/data/032/BRIB.Q2.03209000000.d
~/data/032/BRIB.Q4.03209000000.d

In this case, my script should notice that Q3 is missing and run a shell script (that I've already written) to get it. It needs to be able to search through all of the folders in in the data directory. I'm thinking I can create a list of file "prefixes" like "BRIB.Q1", "BRIB.Q2"... etc that it needs to match up with.

TL,DR:
What kind of commands do I need to look at to write this script? How do I recursively search through directories, checking to make sure a list of files following a particular pattern exist?

Danny
 
Old 08-06-2009, 04:45 PM   #2
tredegar
LQ 5k Club
 
Registered: May 2003
Location: London, UK
Distribution: Debian "Jessie"
Posts: 6,033

Rep: Reputation: 371Reputation: 371Reputation: 371Reputation: 371
Quote:
I have a folder full of data, but occasionally some data goes missing.
A more logical approach might be to find out just why "some data goes missing".

Then you could fix the problem that is causing the problem (rather than "patching it up") and the problem would go away. Writing scripts to mask a problem you have not completely understood is not going to help you in the long term.

You need to investigate exactly why, or how "some data goes missing", and fix that.

Then your problem will be solved.

If you need further advice, please post more details.
 
Old 08-06-2009, 04:51 PM   #3
blackhole54
Senior Member
 
Registered: Mar 2006
Posts: 1,896

Rep: Reputation: 61
It depends how general you need the script to be. For the example you've given, (and generalizable to any situation where the pattern is simply an incrimenting number, you could:

Code:
#!/bin/bash

pattern="~/data/032/BRIB.Q\$n.03209000000.d"
get_missing=<insert path and script name here>

for n in $(seq 1 4); do
   file=$(eval echo "$pattern")
   [ ! -e "$file" ] && $get_missing "$file"
done
Since you are new to this, here is a bit of a "blow by blow" explanation of the script:

I told the script to use bash because I've run into trouble on Ubuntu when it uses dash for sh. When I defined $pattern, I escaped the dollar sign on $n with a backslash so substitution was not done at that step. seq, in this case will generate the sequence "1 2 3 4" which the for loop will iterate through. Each time through the loop, $file will get assigned a new value based on $n. (If anybody knows of a cleaner way to do that step, please let us know.) Finally it tests for the existance of $file and if it doesn't exist, it calls your script with the missing file name. If the files you are looking for are regular files, you could use -f instead of -e. (Look at the bash man page for information on the builtin command test.)

More general scripts, or scripts that need to iterate through letters, will of course, be more complicated.

Last edited by blackhole54; 08-06-2009 at 04:54 PM. Reason: cleanup
 
Old 08-07-2009, 01:59 AM   #4
djslothario
LQ Newbie
 
Registered: Mar 2009
Posts: 4

Original Poster
Rep: Reputation: 0
Quote:
You need to investigate exactly why, or how "some data goes missing", and fix that.
I wrote that because I didn't want to make the post too long with unnecessary information. I have a cron job running that downloads the data, which is being constantly generated, from a remote server. To get the data, I run someone else's program to generate it, which messes up sometimes. It's just weird and I don't have much control over it.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Shell script help: How to pass a directory path to the script? dv502 Programming 13 09-10-2008 04:07 AM
pass php variables to shell script viveksnv Programming 1 03-05-2008 12:46 AM
shell script to move files from one system to another sytem with file names in a txt coral_km Linux - Newbie 3 02-13-2008 11:23 PM
Need a script to search and replace text in file using shell script unixlearner Programming 14 06-21-2007 11:37 PM
PHP pass argument to shell script monzter Programming 2 08-14-2004 07:16 AM


All times are GMT -5. The time now is 11:46 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration