[SOLVED] Script to check files in nested directories in Linux

zaayu87 · 12-17-2012, 07:22 AM

Hi,

I have a script to process files coming in at different time intervals under multiple nested directories. Each directory set is considered as an app. Different cron jobs have been set for these apps to process the files.

the dir structure for say app1 is something like this
ls -l /home/app1/
sub-dir1
sub-dir2
sub-dir3

and each sub-dir has another set of dirs like this(The below dirs are same for all the sub-dirs):
ls -l /home/app1/sub-dir1
real
complex

again, real and complex are split into the below set:
ls -l /home/app1/sub-dir1/real
raw
pending
wip
done
error

The file processing is based on staging concept. file(s) come(s) under raw/pending then are/is moved one by one to "wip" to process and on success is moved to "done" or to "error" otherwise.

My main target is to set up a script to be called inside my main script, which checks for files in each of the raw/pending directories in all the sub-dirs of an app i.e. my script should check in this order:

/home/app1/sub-dir1/real/raw/*.* or /home/app1/sub-dir1/real/pending/*.*
/home/app1/sub-dir2/real/raw/*.* or /home/app1/sub-dir1/real/pending/*.*
/home/app1/sub-dir3/real/raw/*.* or /home/app1/sub-dir1/real/pending/*.*

after it has checked in all the sub-dir's it should send a mail consiting of those sub-dir's which haven't received files for today.

I wrote a script to include general if-else condition to check for files. When there is no file found it should send same via mail but post the "find condition" the script simply exits.

#!/bin/bash
# script: checkfile.sh
#
# usage: checkfile.sh <dir>
#################################

dir=$1/pending
dir=$1/raw

if [ ! -d "$dir" ]; then
echo "$dir is not a directory"
exit 1
fi

if [ ! -d "$dir1" ]; then
echo "$dir1 is not a directory"
exit 1
fi

for file in `find "$dir" -name '*.*' -type f`
do
if [ ! -f $file ];
then
echo "No file in $dir" | mailx -s "file does not exist" xyz@nozone.com
fi
done

for file in `find "$dir1" -name '*.*' -type f`
do
if [ ! -f $file ];
then
echo "No file in $dir1" | mailx -s "file does not exist" xyz@nozone.com
fi
done

output:
$ sh -x checkfile.sh /home/app1/sub-dir1/real
+ dir=/home/app1/sub-dir1/real/pending
+ dir1=/home/app1/sub-dir1/real/raw
+ '[' '!' -d /home/app1/sub-dir1/real/pending ']'
+ '[' '!' -d /home/app1/sub-dir1/real/raw ']'
++ find /home/app1/sub-dir1/real/pending -name '*.*' -type f
++ find /home/app1/sub-dir1/real/raw -name '*.*' -type f

Please advice where i am doing it wrong

Thx in advance

shivaa · 12-17-2012, 08:07 AM

Seems ok, but some mistakes, like you've defined variable "dir" two times, whereas you're using "dir" and "dir1" in script.
Also once go through manpage of test operations to use correct test condition.

Code:

dir=$1/pending
dir1=$1/raw
 
if ([ ! -d "$dir" ];[ ! -d "$dir1" ];); then
echo "No directory found"
exit 1
fi

for file in `find $dir -name "*.*" -type f`
do
if [ ! -e $file ];
then
echo "No $file in $dir" | mailx -s "File $file does not exist in $dir" xyz@nozone.com
fi
done
 
for file in `find $dir1 -name "*.*" -type f`
do
if [ ! -e $file ];
then
echo "No $file in $dir1" | mailx -s "File $file does not exist in $dir1" xyz@nozone.com
fi
done

zaayu87 · 12-17-2012, 08:22 AM

Thank you.

My mistake in typing the dir names for "dir" and "dir1". In code they are as you mentioned.
But unluckily the changes provided still doesn't gimme the desired results. In-fact I had also tried the "-e" option too. But still..

shivaa · 12-17-2012, 08:36 AM

Actual problem is with find... command. When you save/use it as variable, its listing file in a horizontal manner, not each in one line. Once try this:

Code:

dir=$1/pending
dir1=$1/raw
 
if [ -d "$dir" ]; then
find $dir -name "*.*" -type f -print > /tmp/list1
if [ -d $dir1 ]; then
find $dir1 -name "*.*" -type f -print > /tmp/list2
else
echo "Directory $dir1 not found"
fi
else
echo "Directory $dir not found"
exit 1
fi

while read -r file
do
if [ ! -e $file ];
then
echo "No $file in $dir" | mailx -s "File $file does not exist in $dir" xyz@nozone.com
fi
done < /tmp/list1
 
while read -r file
do
if [ ! -e $file ];
then
echo "No $file in $dir" | mailx -s "File $file does not exist in $dir" xyz@nozone.com
fi
done < /tmp/list2

# Cleanup
\rm /tmp/list1 /tmp/list2

Note: I have modified the code little bit, but it's working ok.

zaayu87 · 12-18-2012, 12:33 AM

Hi Shivaa. Thanks for the changes.

But It still isn't working..

With your code, instead of the "mailx" part i added an "echo" statement. Like this:
...............

while read -r file
do
if [ ! -e $file ];
then
echo "No file in $dir"
fi
done <
...............

But in the output, it doesnt print the output on the terminal:
$ sh checkfile.sh /home/app1/sub-dir1/real
$

Output with "-x" option:
$ sh -x checkfile.sh /home/app1/sub-dir1/real
+ dir=/home/app1/sub-dir1/real/pending
+ dir1=/home/app1/sub-dir1/real/raw
+ '[' -d /home/app1/sub-dir1/real/pending ']'
+ find /home/app1/sub-dir1/real/pending -name '*.*' -type f -print
+ '[' -d /home/app1/sub-dir1/real/raw ']'
+ find /home/app1/sub-dir1/real/raw -name '*.*' -type f -print
+ read -r file
+ read -r file
+ rm /tmp/list1 /tmp/list2

shivaa · 12-18-2012, 01:26 AM

According to set conditions, it will not print any echo msg until any condition gets false.
By the by you can add:

Code:

if [ -d "$dir" ]; then
find $dir -name "*.*" -type f -print > /tmp/list1
echo "Listing done for $dir"  # You can add this echo msg
ls -la /tmp/list1  # To check list of files in $dir 
if [ -d $dir1 ]; then
find $dir1 -name "*.*" -type f -print > /tmp/list2
echo "Listing done for $dir1"  # You can add this echo msg
ls -la /tmp/list2  # To check list of files in $dir1
else
echo "Directory $dir1 not found"
fi
else
echo "Directory $dir not found"
exit 1
fi

David the H. · 12-19-2012, 11:41 AM

Could you please use ***[code][/code]*** tags around your code and data, to preserve the original formatting and to improve readability. Do not use quote tags, bolding, colors, "start/end" lines, or other creative techniques.

shivaa has spotted the problem (or at least a problem), but not the reason it's wrong. The real problem is that you Don't Read Lines With For!

The proper way to read the output of a command like find is to use a while+read loop, and ideally to use null characters to separate the entries.

I also think your loops have a logic error behind them. Since find has already determined that the files exist, there's no need to run the subsequent tests. The -name option in find is also probably pointless, or even wrong. Unless you actually need to match files that have "." in them?

But actually I doubt we even need to use find at all. I think perhaps you simply want something like this:

Code:

#loop through all the desired directories
for dname in /home/app*/sub-dir*/real/@(raw|pending); do

	#load all files in the directory into an array.
	files=( "$dname/"* )

	#if the files array is empty, then output a message.
	if (( ! ${#files[@]} )); then
		echo "No file in ${dname}xyz@nozone.com"
	fi

done  > >( mailx -s "file does not exist" )

Notice how I made a modification using an output process substitution to combine the entire output of the loop into a single mail command ( or you could also just use a pipe ). But actually I'm not familiar with mailx or your requirements, so it may not work for you.

A few more general points:

See globbing and extended globbing for details on how to match multiple directories at a time.

$(..) is highly recommended over `..`.

When using bash or ksh, it's recommended to use [[..]] for string/file tests, and ((..)) for numerical tests. Avoid using the old [..] test unless you specifically need POSIX-style portability.

http://wiki.bash-hackers.org/commands/classictest
http://wiki.bash-hackers.org/syntax/...nal_expression
http://mywiki.wooledge.org/ArithmeticExpression