Need to loop script on subfolders in a directory

azurite · 06-25-2017, 02:51 PM

Hello everyone,

I'm am a newbie to coding so I am reaching out in hopes that I can get some help from this forum.

I am trying to run the script below from a single directory, however the directory has many subfolders where the file I need is located. I have bolded the file name below also.

Another thing is that for me to run this script successfully, at the moment I have to type in the actual path to the .ecclog file in terminal like so

Code:

scriptname.sh /TestFolder/.extension

Is there any way I can make the script run through all the subfolders and find the file it needs without having to type in the actual path in terminal? I have about 160 of these folders and manually running the script on each of those folders one by one would take forever. I'm looking for an easier way and I hope someone here can help me out!

BW-userx · 06-25-2017, 04:09 PM

not tested I just wrote it in here.

Code:

#!/bin/bash

working_dir="put search path here - parent/root dir name"

while read FILENAME
do

add code in here for file processing-
it will go through each file separately - recursively 

done< <(find "$working_dir" -type f -name "*.ecclog")

this way you can have the script anywhere in your system and when you run it it will still search wherever the path is set to look.

the value within FILENAME is a complete absolute path and filename. you might have to break it down to work with it using

Code:

f=$FILENAME
path=${f%/*}
xfile=${f##*/}
title=${xfile%.*}
ext=${xfile##*.}

Laserbeak · 06-25-2017, 04:28 PM

This should do it, or at least help you:

Code:

find . -name '*.ecclog' -exec scriptname.sh {} \;

That will run scriptname.sh with a parameter of every file ending in .ecclog from the current directory downwards.

azurite · 06-25-2017, 08:17 PM

Quote:

Originally Posted by BW-userx

not tested I just wrote it in here.

Code:

#!/bin/bash

working_dir="put search path here - parent/root dir name"

while read FILENAME
do

add code in here for file processing-
it will go through each file separately - recursively 

done< <(find "$working_dir" -type f -name "*.ecclog")

this way you can have the script anywhere in your system and when you run it it will still search wherever the path is set to look.

the value within FILENAME is a complete absolute path and filename. you might have to break it down to work with it using

Code:

f=$FILENAME
path=${f%/*}
xfile=${f##*/}
title=${xfile%.*}
ext=${xfile##*.}

Quote:

Originally Posted by Laserbeak

This should do it, or at least help you:

Code:

find . -name '*.ecclog' -exec scriptname.sh {} \;

That will run scriptname.sh with a parameter of every file ending in .ecclog from the current directory downwards.

If I understand correctly, your suggestions will make the script go into every subfolder and grab the .ecclog file unique to that subfolder and it will continue to do so until it has gone through and ran on all the 160 subfolder that I have? I ask because I forgot to mention that every subfolder that I have, has inside it a uniquely numbered file that ends in .ecclog.

grail · 06-25-2017, 08:31 PM

Yes it will find all files ending in 'ecclog' anywhere below the leading directory

Laserbeak · 06-26-2017, 01:11 AM

Quote:

Originally Posted by azurite

If I understand correctly, your suggestions will make the script go into every subfolder and grab the .ecclog file unique to that subfolder and it will continue to do so until it has gone through and ran on all the 160 subfolder that I have? I ask because I forgot to mention that every subfolder that I have, has inside it a uniquely numbered file that ends in .ecclog.

Yes, it will. If you want to test what it does, you can put echo in to see what will be run, like this:

Code:

find . -name '*.ecclog' -exec echo scriptname.sh {} \;

BW-userx · 06-26-2017, 07:47 AM

Quote:

Originally Posted by Laserbeak

Yes, it will. If you want to test what it does, you can put echo in to see what will be run, like this:

Code:

find . -name '*.ecclog' -exec echo scriptname.sh {} \;

the only big difference between what I wrote and this one is placement of the "command/script' when ran - my code was so you can run the script from anywhere, whereas this line of code you have to be within the parent directory, that can be ealiy changed to.

Code:

find /path/to/look/into -name '*.ecclog' -exec echo scriptname.sh {} \;

that too should work. if the while loop then within your coe a few lines could be removed to accommodate the loop, that basename part because the other chunk of code

Code:

f=$FILENAME
path=${f%/*}
xfile=${f##*/}
title=${xfile%.*}
ext=${xfile##*.}

gives you everything you need to work with, the path to the file "path", the file name "title" - the extension of same said file "ext"

so you can just sit in your home dir and run that one liner and or the code in while loop I gave you in a script.

Either way that should getter done

sundialsvcs · 06-26-2017, 08:53 AM

Don't try to do it in bash. Every "real" scripting language that I can think of has a file-finder. Simply choose the language you like, and insert a #!shebang line as the first line of your script identifying that language. Then, write the "shell script" in that language and you're done.

BW-userx · 06-26-2017, 09:27 AM

all of this:

Code:

#!/bin/sh

if [ $# -lt 1 ] ; then 
  echo "Usage: `basename $0` <eddy current ecclog file>"
  exit 1;
fi

logfile=$1;
basenm=`basename $logfile .ecclog`;



nums=`grep -n 'Final' $logfile | sed 's/:.*//'`;

basename gets you the prefex of the filename, yes?

Code:

userx%voider ⚡ ~ ⚡> testfile="home/dir/filename.egg"
userx%voider ⚡ ~ ⚡> basenm=$(basename $testfile .egg)
userx%voider ⚡ ~ ⚡> echo "$basenm"
filename
userx%voider ⚡ ~ ⚡>

becomes this:

Code:

#!/bin/bash

working_dir=

while find FILENAME
do

f=$FILENAME
path=${f%/*}
xfile=${f##*/}
title=${xfile%.*}
ext=${xfile##*.}

basenm=$title
#searching for the keyword 'Final' inside file using complete path and file name.
nums=$(grep -n 'Final' $FILENAME | sed 's/:.*//')

the rest of your code...

done< <(find "working_dir" -type f -name "*.ecclog" )

or this

Code:

#!/bin/bash

working_dir=

while find FILENAME
do

basenm=$(basename $FILENAME .ecclog)

#searching for the keyword 'Final' inside file using complete path and file name.

nums=$(grep -n 'Final' $FILENAME | sed 's/:.*//')

the rest of your code...

done< <(find "working_dir" -type f -name "*.ecclog" )

azurite · 06-26-2017, 01:24 PM

Quote:

Originally Posted by BW-userx

all of this:

Code:

#!/bin/sh

[strike]if [ $# -lt 1 ] ; then 
  echo "Usage: `basename $0` <eddy current ecclog>
....
....

becomes this:

Code:

#!/bin/bash

working_dir=

while find FILENAME
do

f=$FILENAME
path=${f%/*}
xfile=${f##*/}
title=${xfile%.*}
ext=${xfile##*.}
...
...

Quote:

Originally Posted by Laserbeak

Yes, it will. If you want to test what it does, you can put echo in to see what will be run, like this:

Code:

find . -name '*.ecclog' -exec echo scriptname.sh {} \;

Quote:

Originally Posted by grail

Yes it will find all files ending in 'ecclog' anywhere below the leading directory

Okay, I ran the following in terminal but received an error (see screenshot). I'm not sure what I'm doing wrong. Please excuse me if it's something blatantly obvious -- I'm very new to scripting so you guys might have to really walk me through it baby steps.

Code:

find . -name '*.ecclog' -exec ec_plot.sh {} \;

Here's a screenshot of my directory and the error I received in terminal.

Laserbeak · 06-26-2017, 01:40 PM

Quote:

Originally Posted by azurite

Okay, I ran the following in terminal but received an error (see screenshot). I'm not sure what I'm doing wrong. Please excuse me if it's something blatantly obvious -- I'm very new to scripting so you guys might have to really walk me through it baby steps.

Code:

find . -name '*.ecclog' -exec ec_plot.sh {} \;

Here's a screenshot of my directory and the error I received in terminal.

Several possible issues:

You may need to put the full pathname of the script in the find command, so instead of just ec_plot.sh, it'd be /path/to/ec_plot.sh

Or maybe your shebang isn't right in ec_plot.sh itself. Make sure you have

Code:

#!/bin/bash

or whatever is appropriate as the first line in your script. You can find out where bash is installed on your system by typing in:

Code:

which bash

Plus make sure the script is set to be executable:

Code:

chmod 755 ec_plot.sh

edit:

Ooh, I just saw your script, and the shebang problem is almost certainly it! You have #!/bin/sh which probably doesn't exist on your system, replace it with #!/bin/bash

BW-userx · 06-26-2017, 01:47 PM

make sure that the ec_plot.sh is in the very same dir you are in when you are running that

Code:

find . -name '*.ecclog' -exec ec_plot.sh {} \;

it looks like you are right clicking desktop -> open terminal. that gets you the current dir is desktop.
if you do that then just

Code:

cd /path to where your db files are at[enter]

and that ec_plot.sh has to be in that dir too. else you need to add absolute path within the

Code:

find /path/where/to/look -name '*.ecclog' -exec /path/where/this/is/at/ec_plot.sh {} \;

azurite · 06-26-2017, 01:50 PM

Quote:

Originally Posted by Laserbeak

Several possible issues:

You may need to put the full pathname of the script in the find command, so instead of just ec_plot.sh, it'd be /path/to/ec_plot.sh

Or maybe your shebang isn't right in ec_plot.sh itself. Make sure you have

edit:

Ooh, I just saw your script, and the shebang problem is almost certainly it! You have #!/bin/sh which probably doesn't exist on your system, replace it with #!/bin/bash

the script is executable and I fixed the shebang as you suggested. Now, I get the following error.

Code:

username@computer:~/Desktop/DWITESTSUB$ find . -name '*.ecclog' -exec /home/natasha/Desktop/
DWITESTSUB/ec_plot.sh
find: missing argument to `-exec'

Laserbeak · 06-26-2017, 01:53 PM

If I'm right, and /bin/sh isn't there, you can also copy /bin/bash to /bin/sh instead of changing your script. Bash actually changes its behavior slightly when ran as sh to be as compatible as possible with the original sh. If the script was written to run in sh, you could run into problems when running it in bash when not in sh emulation mode. (Although the differences are not very many)

BW-userx · 06-26-2017, 01:57 PM

example
script named
runFindFromAfar

Code:

#!/bin/bash

echo $1

Code:

userx%slackwhere ⚡ ~ ⚡> find /home/userx/scripts/testing/testdata -type f -exec /home/userx/scripts/testing/runFindFromAfar {} \;
/home/userx/scripts/testing/testdata/File-May-4
/home/userx/scripts/testing/testdata/File-May-7
/home/userx/scripts/testing/testdata/File-May-6
/home/userx/scripts/testing/testdata/File-May-3
/home/userx/scripts/testing/testdata/File-May-8
/home/userx/scripts/testing/testdata/File-May-0
/home/userx/scripts/testing/testdata/File-May-5
/home/userx/scripts/testing/testdata/File-May-2
/home/userx/scripts/testing/testdata/File-May-1