LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 12-17-2012, 08:22 AM   #1
zaayu87
LQ Newbie
 
Registered: Sep 2012
Location: India
Posts: 6

Rep: Reputation: Disabled
Script to check files in nested directories in Linux


Hi,

I have a script to process files coming in at different time intervals under multiple nested directories. Each directory set is considered as an app. Different cron jobs have been set for these apps to process the files.

the dir structure for say app1 is something like this
ls -l /home/app1/
sub-dir1
sub-dir2
sub-dir3

and each sub-dir has another set of dirs like this(The below dirs are same for all the sub-dirs):
ls -l /home/app1/sub-dir1
real
complex

again, real and complex are split into the below set:
ls -l /home/app1/sub-dir1/real
raw
pending
wip
done
error

The file processing is based on staging concept. file(s) come(s) under raw/pending then are/is moved one by one to "wip" to process and on success is moved to "done" or to "error" otherwise.

My main target is to set up a script to be called inside my main script, which checks for files in each of the raw/pending directories in all the sub-dirs of an app i.e. my script should check in this order:

/home/app1/sub-dir1/real/raw/*.* or /home/app1/sub-dir1/real/pending/*.*
/home/app1/sub-dir2/real/raw/*.* or /home/app1/sub-dir1/real/pending/*.*
/home/app1/sub-dir3/real/raw/*.* or /home/app1/sub-dir1/real/pending/*.*

after it has checked in all the sub-dir's it should send a mail consiting of those sub-dir's which haven't received files for today.


I wrote a script to include general if-else condition to check for files. When there is no file found it should send same via mail but post the "find condition" the script simply exits.

#!/bin/bash
# script: checkfile.sh
#
# usage: checkfile.sh <dir>
#################################

dir=$1/pending
dir=$1/raw

if [ ! -d "$dir" ]; then
echo "$dir is not a directory"
exit 1
fi

if [ ! -d "$dir1" ]; then
echo "$dir1 is not a directory"
exit 1
fi

for file in `find "$dir" -name '*.*' -type f`
do
if [ ! -f $file ];
then
echo "No file in $dir" | mailx -s "file does not exist" xyz@nozone.com
fi
done

for file in `find "$dir1" -name '*.*' -type f`
do
if [ ! -f $file ];
then
echo "No file in $dir1" | mailx -s "file does not exist" xyz@nozone.com
fi
done

output:
$ sh -x checkfile.sh /home/app1/sub-dir1/real
+ dir=/home/app1/sub-dir1/real/pending
+ dir1=/home/app1/sub-dir1/real/raw
+ '[' '!' -d /home/app1/sub-dir1/real/pending ']'
+ '[' '!' -d /home/app1/sub-dir1/real/raw ']'
++ find /home/app1/sub-dir1/real/pending -name '*.*' -type f
++ find /home/app1/sub-dir1/real/raw -name '*.*' -type f

Please advice where i am doing it wrong

Thx in advance
 
Old 12-17-2012, 09:07 AM   #2
shivaa
Senior Member
 
Registered: Jul 2012
Location: Grenoble, Fr.
Distribution: Sun Solaris, RHEL, Ubuntu, Debian 6.0
Posts: 1,800
Blog Entries: 4

Rep: Reputation: 286Reputation: 286Reputation: 286
Seems ok, but some mistakes, like you've defined variable "dir" two times, whereas you're using "dir" and "dir1" in script.
Also once go through manpage of test operations to use correct test condition.
Code:
dir=$1/pending
dir1=$1/raw
 
if ([ ! -d "$dir" ];[ ! -d "$dir1" ];); then
echo "No directory found"
exit 1
fi

for file in `find $dir -name "*.*" -type f`
do
if [ ! -e $file ];
then
echo "No $file in $dir" | mailx -s "File $file does not exist in $dir" xyz@nozone.com
fi
done
 
for file in `find $dir1 -name "*.*" -type f`
do
if [ ! -e $file ];
then
echo "No $file in $dir1" | mailx -s "File $file does not exist in $dir1" xyz@nozone.com
fi
done
 
Old 12-17-2012, 09:22 AM   #3
zaayu87
LQ Newbie
 
Registered: Sep 2012
Location: India
Posts: 6

Original Poster
Rep: Reputation: Disabled
Thank you.

My mistake in typing the dir names for "dir" and "dir1". In code they are as you mentioned.
But unluckily the changes provided still doesn't gimme the desired results. In-fact I had also tried the "-e" option too. But still..
 
Old 12-17-2012, 09:36 AM   #4
shivaa
Senior Member
 
Registered: Jul 2012
Location: Grenoble, Fr.
Distribution: Sun Solaris, RHEL, Ubuntu, Debian 6.0
Posts: 1,800
Blog Entries: 4

Rep: Reputation: 286Reputation: 286Reputation: 286
Actual problem is with find... command. When you save/use it as variable, its listing file in a horizontal manner, not each in one line. Once try this:
Code:
dir=$1/pending
dir1=$1/raw
 
if [ -d "$dir" ]; then
find $dir -name "*.*" -type f -print > /tmp/list1
if [ -d $dir1 ]; then
find $dir1 -name "*.*" -type f -print > /tmp/list2
else
echo "Directory $dir1 not found"
fi
else
echo "Directory $dir not found"
exit 1
fi

while read -r file
do
if [ ! -e $file ];
then
echo "No $file in $dir" | mailx -s "File $file does not exist in $dir" xyz@nozone.com
fi
done < /tmp/list1
 
while read -r file
do
if [ ! -e $file ];
then
echo "No $file in $dir" | mailx -s "File $file does not exist in $dir" xyz@nozone.com
fi
done < /tmp/list2

# Cleanup
\rm /tmp/list1 /tmp/list2
Note: I have modified the code little bit, but it's working ok.

Last edited by shivaa; 12-17-2012 at 09:46 AM.
 
Old 12-18-2012, 01:33 AM   #5
zaayu87
LQ Newbie
 
Registered: Sep 2012
Location: India
Posts: 6

Original Poster
Rep: Reputation: Disabled
Unhappy

Hi Shivaa. Thanks for the changes.

But It still isn't working..

With your code, instead of the "mailx" part i added an "echo" statement. Like this:
...............

while read -r file
do
if [ ! -e $file ];
then
echo "No file in $dir"
fi
done <
...............

But in the output, it doesnt print the output on the terminal:
$ sh checkfile.sh /home/app1/sub-dir1/real
$

Output with "-x" option:
$ sh -x checkfile.sh /home/app1/sub-dir1/real
+ dir=/home/app1/sub-dir1/real/pending
+ dir1=/home/app1/sub-dir1/real/raw
+ '[' -d /home/app1/sub-dir1/real/pending ']'
+ find /home/app1/sub-dir1/real/pending -name '*.*' -type f -print
+ '[' -d /home/app1/sub-dir1/real/raw ']'
+ find /home/app1/sub-dir1/real/raw -name '*.*' -type f -print
+ read -r file
+ read -r file
+ rm /tmp/list1 /tmp/list2
 
Old 12-18-2012, 02:26 AM   #6
shivaa
Senior Member
 
Registered: Jul 2012
Location: Grenoble, Fr.
Distribution: Sun Solaris, RHEL, Ubuntu, Debian 6.0
Posts: 1,800
Blog Entries: 4

Rep: Reputation: 286Reputation: 286Reputation: 286
According to set conditions, it will not print any echo msg until any condition gets false.
By the by you can add:
Code:
if [ -d "$dir" ]; then
find $dir -name "*.*" -type f -print > /tmp/list1
echo "Listing done for $dir"  # You can add this echo msg
ls -la /tmp/list1  # To check list of files in $dir 
if [ -d $dir1 ]; then
find $dir1 -name "*.*" -type f -print > /tmp/list2
echo "Listing done for $dir1"  # You can add this echo msg
ls -la /tmp/list2  # To check list of files in $dir1
else
echo "Directory $dir1 not found"
fi
else
echo "Directory $dir not found"
exit 1
fi
 
Old 12-19-2012, 12:41 PM   #7
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian + kde 4 / 5
Posts: 6,837

Rep: Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981
Could you please use ***[code][/code]*** tags around your code and data, to preserve the original formatting and to improve readability. Do not use quote tags, bolding, colors, "start/end" lines, or other creative techniques.


shivaa has spotted the problem (or at least a problem), but not the reason it's wrong. The real problem is that you Don't Read Lines With For!

The proper way to read the output of a command like find is to use a while+read loop, and ideally to use null characters to separate the entries.

I also think your loops have a logic error behind them. Since find has already determined that the files exist, there's no need to run the subsequent tests. The -name option in find is also probably pointless, or even wrong. Unless you actually need to match files that have "." in them?


But actually I doubt we even need to use find at all. I think perhaps you simply want something like this:

Code:
#loop through all the desired directories
for dname in /home/app*/sub-dir*/real/@(raw|pending); do

	#load all files in the directory into an array.
	files=( "$dname/"* )

	#if the files array is empty, then output a message.
	if (( ! ${#files[@]} )); then
		echo "No file in ${dname}xyz@nozone.com"
	fi

done  > >( mailx -s "file does not exist" )

Notice how I made a modification using an output process substitution to combine the entire output of the loop into a single mail command ( or you could also just use a pipe ). But actually I'm not familiar with mailx or your requirements, so it may not work for you.


A few more general points:

See globbing and extended globbing for details on how to match multiple directories at a time.

$(..) is highly recommended over `..`.

When using bash or ksh, it's recommended to use [[..]] for string/file tests, and ((..)) for numerical tests. Avoid using the old [..] test unless you specifically need POSIX-style portability.

http://wiki.bash-hackers.org/commands/classictest
http://wiki.bash-hackers.org/syntax/...nal_expression
http://mywiki.wooledge.org/ArithmeticExpression
 
1 members found this post helpful.
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Removing multiple files in nested directories... cwwilson721 Slackware 7 04-02-2008 09:44 AM
Entering folders recursively to run script on files nested inside, ubuntu 7.10 aidansmoker Linux - Newbie 4 11-12-2007 12:39 AM
nested boot directories pugsley Linux - General 5 11-03-2006 07:04 PM
script to check $PATH directories exist Frustin Linux - Software 3 09-21-2004 01:20 PM
check for matching files when comparing directories WarriorWarren Linux - General 3 05-07-2003 02:40 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 06:54 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration