Loop command on folders in a directory

azurite · 05-31-2016, 12:35 AM

Hello all,

I'm fairly new to scripting so please bear with me. I will try to explain as best as I can but if there's anything that is not clear, please let me know. Also, if this is the wrong forum to post, please move the thread.

I have a directory (see below) with numerous folders and within each of those folders are various files. I would like to run a script that will go into each folder and then select the specified files and then repeat this process for all the folders that are in a given directory. Here is an example of how the path is set up:

Code:

~/W/PPA/Projects/DWI/subject#/random#'s/###_DTI_siemens_TClessdistort/multiplefiles.extension

(ex: ~/W/PPA/Projects/DWI/100604-200_2/3t_2010-04-05_22-40/003_DTI_siemens_TClessdistort/
20100405_224029DTIsiemensTClessdistorts_eddy_corrected_brain.nii.gz )

The folders contain multiple files with various extensions. I would like the script to go into random#1 folder, look for the specified files, execute the command, and then move onto random#2 folder and do the same.

Inside each folder are the following files:

Code:

random#'s_eddy_corrected_brain.nii.gz
random#'s_eddy_corrected_brain_mask.nii.gz
random#'sDTIsiemensTClessdistorts003a001.bvec
random#'sDTIsiemensTClessdistorts003a001.bval

I have the following script that only works on one specific type of file inside each folder but I need to write a new one that works on specified files (above) located inside each folder in the directory.

Code:

#! /bin/csh

foreach CASE ( `cat DWI_cases_image_path` )
cd `echo $CASE | cut -d/ -f1-3`
dtifit -k eddy_corrected_brain file -o output filename_dtifit -m mask eddy_corrected_brain_mask file 
-r bvecs .bvec extension file -b bvals .bval extension file
cd ../../../
end

In the above case, DWI_cases_image_path is a text file with the path to the one specific file like so

Code:

100604-200_2/3t_2010-04-05_22-40/003_DTI_siemens_TClessdistort/20100405_224029DTIsiemensTClessdistorts003a001.nii.gz
100604-200_3/3t_2015-10-16_11-38/003_DTI_siemens_TClessdistort/20151016_113816DTIsiemensTClessdistorts003a001A.nii.gz
etc

How can I make it so that I can replace the above to make a list of all the folders and then run a script that goes inside each folder and looks for the specified files?

Basically, I need the script to go into the directory, into the subdirectory, into the folders, and look for specified files in those folders. And repeat for every folder there is.

I have exhausted all options of self-teaching/experimenting to figure this out on my own and therefore, I'm hoping I can find some solution here.

Thank you in advance.

Turbocapitalist · 05-31-2016, 01:13 AM

A few things about the script. Unless you have a specific reason to use csh, it would be better to use something more portable. #!/bin/sh gets you the POSIX-compatible shell. Also, the backticks can be replaced by $( ) instead. Those are a little easier to read and can be nested. Then there is a tool "basename" which might be useful with what you have written instead of "cut".

About finding files or directories, there are the utilities "find" and "xargs". Try looking at the manual pages, they may be overwhelming but are a good reference. Some examples:

Code:

find /some/path/ -type f -print;

find /some/path/ -type d -print0 | xargs -0 /usr/local/bin/somescript;
find /some/path/ -type f -print0 | xargs -0 /usr/local/bin/somescript;
find /some/path/ -type f -print0 | xargs -I {} -0 /usr/local/bin/somescript -f "{}" -v;

find /some/path/ -type f -exec /usr/local/bin/somescript "{}" \; ;

Would using "find" make it easier?

JJJCR · 05-31-2016, 04:08 AM

Files that you are trying to find has a specific file extension or a file name with a pattern?

If yes, then I think a simple recursive grep command can do the job or maybe mixed with grep and awk.

Here's some links to get you started:

http://quickbytesstuff.blogspot.sg/2...-tutorial.html

http://www.cyberciti.biz/faq/grep-co...ries-examples/

azurite · 05-31-2016, 01:56 PM

Quote:

Originally Posted by Turbocapitalist

A few things about the script. Unless you have a specific reason to use csh, it would be better to use something more portable. #!/bin/sh gets you the POSIX-compatible shell. Also, the backticks can be replaced by $( ) instead. Those are a little easier to read and can be nested. Then there is a tool "basename" which might be useful with what you have written instead of "cut".

About finding files or directories, there are the utilities "find" and "xargs". Try looking at the manual pages, they may be overwhelming but are a good reference. Some examples:

Code:

find /some/path/ -type f -print;

find /some/path/ -type d -print0 | xargs -0 /usr/local/bin/somescript;
find /some/path/ -type f -print0 | xargs -0 /usr/local/bin/somescript;
find /some/path/ -type f -print0 | xargs -I {} -0 /usr/local/bin/somescript -f "{}" -v;

find /some/path/ -type f -exec /usr/local/bin/somescript "{}" \; ;

Would using "find" make it easier?

Quote:

Originally Posted by JJJCR

Files that you are trying to find has a specific file extension or a file name with a pattern?

If yes, then I think a simple recursive grep command can do the job or maybe mixed with grep and awk.

Here's some links to get you started:

http://quickbytesstuff.blogspot.sg/2...-tutorial.html

http://www.cyberciti.biz/faq/grep-co...ries-examples/

So I have a directory with about 100 folders inside and each of those folders has the following 4 files inside it. I want the script to go into the first folder, look for the files, run the analysis on those files, and then go through the 2nd folder and the 3rd and so on.

Files in each folder.

Code:

random#'s_eddy_corrected_brain.nii.gz
random#'s_eddy_corrected_brain_mask.nii.gz
random#'sDTIsiemensTClessdistorts003a001.bvec
random#'sDTIsiemensTClessdistorts003a001.bval

I am trying to input those files into this code to automate the analysis instead of having to manually input files using the gui for the program I am using.

Code:

foreach CASE ( `cat text file with list of folders` )
cd `echo $CASE`

dtifit < this tells the program to run the analysis
--data I need to tell it to input the file with the pattern "eddy_corrected_brain.nii.gz" located inside the folder
--output filename_dtifit 
--mask I need to tell it to input the file with the pattern "eddy_corrected_brain_mask.nii.gz" located inside the folder
--bvecs I need to tell it to input the file with the ".bvec" extension located inside the folder 
--bvals I need to tell it to input the file with the ".bval" extension located inside the folder 
cd ../../../
end

Can I just put the script inside the directory with the folder and the text file (with list of folders) and run it from there?

Turbocapitalist · 06-01-2016, 10:25 AM

Quote:

Originally Posted by azurite

Can I just put the script inside the directory with the folder and the text file (with list of folders) and run it from there?

The script does not have to be in any particular directory as long as you use paths with the file names. "find" can provide your script with the directory names.

Code:

find /some/path/ -name '*.eddy_corrected_brain_mask.nii.gz' -printf '%h\0' | xargs -0 /usr/local/bin/yourscript

That example would find all files with the pattern and send the directory name as an argument to a script.
Then the script can look in the given directory for the four files and pass them (with full path names) to the analysis program.

azurite · 06-01-2016, 09:22 PM

Quote:

Originally Posted by Turbocapitalist

The script does not have to be in any particular directory as long as you use paths with the file names. "find" can provide your script with the directory names.

Code:

find /some/path/ -name '*.eddy_corrected_brain_mask.nii.gz' -printf '%h\0' | xargs -0 /usr/local/bin/yourscript

That example would find all files with the pattern and send the directory name as an argument to a script.
Then the script can look in the given directory for the four files and pass them (with full path names) to the analysis program.

Could you expand a bit more on this? If the files are in my Documents/DWI folder do I just changed the command like so:

Code:

find /home/tom/documents/dwi -name '*.eddy_corrected_brain_mask.nii.gz' -printf '%h\0' | xargs -0 /usr/tom/bin/dti_script

I see that the above code has only one file 'eddy_corrected_brain_mask.nii.gz' and not the other 3, would script still look for the other 3 and run all of them together? I ask because the analysis program needs to get all 4 files and run them together.

JJJCR · 06-01-2016, 10:48 PM

Quote:

I ask because the analysis program needs to get all 4 files and run them together.

The input for the program are 4 files? If one of the file is missing the program will not run?

Or the program process each file separately in sequence?

azurite · 06-01-2016, 11:11 PM

Quote:

Originally Posted by JJJCR

The input for the program are 4 files? If one of the file is missing the program will not run?

Or the program process each file separately in sequence?

The program takes all 4 files and calculates an overall image out of the data from the 4 files.

Turbocapitalist · 06-02-2016, 12:35 AM

Quote:

Originally Posted by azurite

Could you expand a bit more on this? If the files are in my Documents/DWI folder do I just changed the command like so:

Code:

find /home/tom/documents/dwi -name '*.eddy_corrected_brain_mask.nii.gz' -printf '%h\0' | xargs -0 /usr/tom/bin/dti_script

I see that the above code has only one file 'eddy_corrected_brain_mask.nii.gz' and not the other 3, would script still look for the other 3 and run all of them together? I ask because the analysis program needs to get all 4 files and run them together.

"find" looks for one file, yes, but it passes only the directory name on to the script. It would be up to the script to figure out the four names and then pass them on to the analysis program along with the directory name. Finding the directories is easy. The script the would have to set which files in the directory to look for. From what you wrote above, there are some consistent patterns. So here is one possible way you could set the files:

Code:

#!/bin/sh -ex

# quit if a single directory is not given
if [ $# -ne 1 ]; then
        exit;
fi

directory=$1
echo Looking in $directory;

brain=$directory/*_eddy_corrected_brain.nii.gz
bval=$directory/*.bval

echo brain = $brain;
echo bval = $bval;

It's primitive and does not check if there are multiple files that could match the pattern.

The -x is for debugging and can be turned off later. The -e is a good idea to keep. They and nearly all else are in the manual page for "sh", except for the if [ ] part which is the manual page for "test". Does that get you started?

JJJCR · 06-02-2016, 03:05 AM

The four files that is required to run the program reside on the same directory and also found on other directories?

What I mean is, if either one of these files below is found; then the other files are also present on that same directory?

random#'s_eddy_corrected_brain.nii.gz
random#'s_eddy_corrected_brain_mask.nii.gz
random#'sDTIsiemensTClessdistorts003a001.bvec
random#'sDTIsiemensTClessdistorts003a001.bval

The reason why I'm asking, is if that is the case; then you just need to look for a single file then get the directory path. And same process is followed until all the folders are searched.

Then Store the directory path to an array or a text file, if a match is found.

Then run the command to retrieve the array's data or read the text file and do the analysis.

I'm sorry I can write a code for you for this one, but that would be the algorithm I will do if I'm at your side.

azurite · 06-11-2016, 06:11 PM

Quote:

Originally Posted by Turbocapitalist

"find" looks for one file, yes, but it passes only the directory name on to the script. It would be up to the script to figure out the four names and then pass them on to the analysis program along with the directory name. Finding the directories is easy. The script the would have to set which files in the directory to look for. From what you wrote above, there are some consistent patterns. So here is one possible way you could set the files:

Code:

#!/bin/sh -ex

# quit if a single directory is not given
if [ $# -ne 1 ]; then
        exit;
fi

directory=$1
echo Looking in $directory;

brain=$directory/*_eddy_corrected_brain.nii.gz
bval=$directory/*.bval

echo brain = $brain;
echo bval = $bval;

It's primitive and does not check if there are multiple files that could match the pattern.

The -x is for debugging and can be turned off later. The -e is a good idea to keep. They and nearly all else are in the manual page for "sh", except for the if [ ] part which is the manual page for "test". Does that get you started?

Quote:

Originally Posted by JJJCR

The four files that is required to run the program reside on the same directory and also found on other directories?

What I mean is, if either one of these files below is found; then the other files are also present on that same directory?

random#'s_eddy_corrected_brain.nii.gz
random#'s_eddy_corrected_brain_mask.nii.gz
random#'sDTIsiemensTClessdistorts003a001.bvec
random#'sDTIsiemensTClessdistorts003a001.bval

The reason why I'm asking, is if that is the case; then you just need to look for a single file then get the directory path. And same process is followed until all the folders are searched.

Then Store the directory path to an array or a text file, if a match is found.

Then run the command to retrieve the array's data or read the text file and do the analysis.

I'm sorry I can write a code for you for this one, but that would be the algorithm I will do if I'm at your side.

I think we're on the correct path here but I just want to make sure I run correct commands.
I have a folder with lots of subfolders (in the screenshot below, they are the 3 numerical named folders. Note: I only included 3 for the purposes of this example) and inside each of those numerical named folders are the 4 files that I need the analysis program to find and run together.

So I just need a script to go into all the subfolders in the main folder and grab the 4 files (.bvec/.bval/eddy_corrected/eddy_corrected_brain) that so the program can run the analysis.

My question is, what do I do to make the script go into each subfolder in the main folder and look for the files and repeat the process? Do I create a text file with the list of subfolders and use it with the 'cat' command? This is where I'm getting confused.

Screenshots:
Main Folder: http://i.imgur.com/EBWJ8I6.png
Subfolders inside main folder: http://i.imgur.com/Som5BBZ.png
Files inside each subfolder: http://i.imgur.com/yIdWvEW.png
Text file of name of subfolders: http://i.imgur.com/ZZMYpn5.png

Turbocapitalist · 06-11-2016, 11:57 PM

Quote:

Originally Posted by azurite

I think we're on the correct path here but I just want to make sure I run correct commands.
I have a folder with lots of subfolders (in the screenshot below, they are the 3 numerical named folders. Note: I only included 3 for the purposes of this example) and inside each of those numerical named folders are the 4 files that I need the analysis program to find and run together.

So I just need a script to go into all the subfolders in the main folder and grab the 4 files (.bvec/.bval/eddy_corrected/eddy_corrected_brain) that so the program can run the analysis.

My question is, what do I do to make the script go into each subfolder in the main folder and look for the files and repeat the process?

"find" will do that, in a first script or alone. If the four files are always found together, then you only need to have "find" locate one of them and then use that one to pass the name on to your second script. Then your second script does the work of figuring out the file names that should be in that directory.

Quote:

Originally Posted by azurite

Do I create a text file with the list of subfolders and use it with the 'cat' command? This is where I'm getting confused.

No, you don't have to. In #9 above, I have shown one possible way of finding the file names with a script given the name of the directory they are in. Again, that is on the basis of the patterns you have described to us for the file names. If that works, then it is only necessary to add a line calling the analysis program to that script. Then if that works, too, you can call that script with a first script containing "find"

allend · 06-12-2016, 12:15 AM

As I read this thread, the OP needs to visit each of the subdirectories containing the four files needed for analysis, and run the analysis program from that directory with appropriate inputs.

In a bash script run from the top directory

Code:

#!/bin/bash

# Save current directory
curdir=$PWD

# Set shopt to also look in subdirectories
shopt -s globstar

for f in **/*_eddy_corrected_brain.nii.gz; do
  # Change to subdirectory
  cd "${f%/*}"
# Do program stuff here
# dtifit < this tells the program to run the analysis
#--data I need to tell it to input the file with the pattern "eddy_corrected_brain.nii.gz" located inside the folder
#--output filename_dtifit 
#--mask I need to tell it to input the file with the pattern "eddy_corrected_brain_mask.nii.gz" located inside the folder
#--bvecs I need to tell it to input the file with the ".bvec" extension located inside the folder 
#--bvals I need to tell it to input the file with the ".bval" extension located inside the folder
  cd "$curdir"
done

# Undo change to shopt
shopt -u globstar

azurite · 06-12-2016, 04:46 PM

Quote:

Originally Posted by allend

As I read this thread, the OP needs to visit each of the subdirectories containing the four files needed for analysis, and run the analysis program from that directory with appropriate inputs.

In a bash script run from the top directory

Code:

#!/bin/bash

# Save current directory
curdir=$PWD

# Set shopt to also look in subdirectories
shopt -s globstar

for f in **/*_eddy_corrected_brain.nii.gz; do
  # Change to subdirectory
  cd "${f%/*}"
# Do program stuff here

# dtifit < this tells the program to run the analysis

#--data 'find -name *eddy_corrected_brain.nii.gz*`
#--output `filename_dtifit`
#--mask `find -name *eddy_corrected_brain_mask.nii.gz*`
#--bvecs `ls .bvec`
#--bvals `ls .bval`

  cd "$curdir"
done

# Undo change to shopt
shopt -u globstar

This is my first time with scripts so please bear with me. If I use this code, do I need to write the path to the directory anywhere in this code? Or will it work if I just copy and paste it into a text file and save it in the parent directory and run it from there? Could you please also check the bolded parts to see if I've written it correctly so that the end result is the script goes into each subfolder and looks for the 4 files.

allend · 06-12-2016, 09:07 PM

Quote:

Or will it work if I just copy and paste it into a text file and save it in the parent directory and run it from there?

If you do this, the code will work as written. Just remember to make the file executable with 'chmod +x <scriptanme>'.

Quote:

If I use this code, do I need to write the path to the directory anywhere in this code?

You could make the script more portable, in that it could be launched from any directory, by changing to

Code:

# Save current directory
curdir=$PWD
# Change to top directory
topdir="/path/to/top/directory"
cd "$topdir"
...
  cd "$curdir"
  cd "$topdir"

For setting the variables and calling the data analysis program, I suggest code like

Code:

data="$(stat -c %n *eddy_corrected_brain.nii.gz)"
...
dtifit -k "$data" -m "$mask" -r "$bvecs" -b "$bvals" -o "$output"

It is recommended to use 'stat' rather than 'ls' for getting filenames in scripts.