Loop command on folders in a directory

azurite · 06-13-2016, 01:07 AM

Quote:

Originally Posted by allend

If you do this, the code will work as written. Just remember to make the file executable with 'chmod +x <scriptanme>'.

You could make the script more portable, in that it could be launched from any directory, by changing to

Code:

# Save current directory
curdir=$PWD
# Change to top directory
topdir="/path/to/top/directory"
cd "$topdir"
...
  cd "$curdir"
  cd "$topdir"

For setting the variables and calling the data analysis program, I suggest code like

Code:

data="$(stat -c %n *eddy_corrected_brain.nii.gz)"
...
dtifit -k "$data" -m "$mask" -r "$bvecs" -b "$bvals" -o "$output"

It is recommended to use 'stat' rather than 'ls' for getting filenames in scripts.

I must be oblivious to something because I could not get it to work. Thus, I have recorded a bit of my process. Perhaps someone here can figure out what I'm doing wrong. The bit towards the end (around 3:50 min mark) is me doing the process using the program GUI instead of the script. But ultimately, I want to use a script to make the process faster/easier.

http://sendvid.com/y0a92ko4

allend · 06-13-2016, 02:12 AM

The script in the video does not seem to match what has been suggested. (I have attached a still from the video).

I was hoping that you would finish the script from the snippets provided.

Code:

#!/bin/bash

# Top of directory structure containing files to be processed.
topdir="/home/natasha/Documents/DWI"

cd "$topdir"

# Set shopt to also look in subdirectories
shopt -s globstar

for f in **/*_eddy_corrected_brain.nii.gz; do
  # Change to subdirectory
  cd "${f%/*}"

  # Set variables for program
  # TODO - Add error checking for missing or duplicate files
  data="$(stat -c %n *_eddy_corrected_brain.nii.gz)"
  mask="$(stat -c %n *_eddy_corrected_brain_mask.nii.gz)"
  bvecs="$(stat -c %n *.bvec)"
  bvals="$(stat -c %n *.bval)"
  # Build the output file name
  output="${f##/}"
  output="${output/_eddy_corrected_brain.nii.gz/}_dtifit"
  
  # Run the program
  dtifit -k "$data" -m "$mask" -r "$bvecs" -b "$bvals" -o "$output"

  # Change back to top directory
  cd "$topdir"
done

# Undo change to shopt
shopt -u globstar

JJJCR · 06-13-2016, 02:30 AM

Didn't know about stat.

I tried it on my box quite helpful no need to cut the string just to get the filename.

Quote:

[Quas@puas01x ~]$ stat -c %n *.txt
NewFile.txt
xinput.txt
xxq.txt

Looks like magic it gets all the filename of all the files ending with ".txt"

Thanks Allend!!

allend · 06-13-2016, 02:43 AM

Quote:

Didn't know about stat.

Taken from http://mywiki.wooledge.org/ParsingLs

JJJCR · 06-13-2016, 04:42 AM

Quote:

Originally Posted by allend

Taken from http://mywiki.wooledge.org/ParsingLs

Thanks for the link.

azurite · 06-13-2016, 03:21 PM

Quote:

Originally Posted by allend

The script in the video does not seem to match what has been suggested. (I have attached a still from the video).

I was hoping that you would finish the script from the snippets provided.

Code:

#!/bin/bash

# Top of directory structure containing files to be processed.
topdir="/home/natasha/Documents/DWI"

cd "$topdir"

# Set shopt to also look in subdirectories
shopt -s globstar

for f in **/*_eddy_corrected_brain.nii.gz; do
  # Change to subdirectory
  cd "${f%/*}"

  # Set variables for program
  # TODO - Add error checking for missing or duplicate files
  data="$(stat -c %n *_eddy_corrected_brain.nii.gz)"
  mask="$(stat -c %n *_eddy_corrected_brain_mask.nii.gz)"
  bvecs="$(stat -c %n *.bvec)"
  bvals="$(stat -c %n *.bval)"
  # Build the output file name
  output="${f##/}"
  output="${output/_eddy_corrected_brain.nii.gz/}_dtifit"
  
  # Run the program
  dtifit -k "$data" -m "$mask" -r "$bvecs" -b "$bvals" -o "$output"

  # Change back to top directory
  cd "$topdir"
done

# Undo change to shopt
shopt -u globstar

Hi allend,

I ran the modified script you so kindly posted and it seemed to have started working but I get the following error at the end. I'm not sure if it's something in the script or the actual files themselves. If you could be so kind as to again take a look and help with figure this out, I would really be grateful.

Code:

natasha@ubuntu:~/Documents$ cd /home/natasha/Documents/DWI
natasha@ubuntu:~/Documents/DWI$ chmod +X scriptb.sh
natasha@ubuntu:~/Documents/DWI$ ./scriptb.sh
0 128 0 128 0 72
0 slices processed
1 slices processed
2 slices processed
3 slices processed
4 slices processed
5 slices processed
6 slices processed
7 slices processed
8 slices processed
9 slices processed
10 slices processed
11 slices processed
12 slices processed
13 slices processed
14 slices processed
15 slices processed
16 slices processed
17 slices processed
18 slices processed
19 slices processed
20 slices processed
21 slices processed
22 slices processed
23 slices processed
24 slices processed
25 slices processed
26 slices processed
27 slices processed
28 slices processed
29 slices processed
30 slices processed
31 slices processed
32 slices processed
33 slices processed
34 slices processed
35 slices processed
36 slices processed
37 slices processed
38 slices processed
39 slices processed
40 slices processed
41 slices processed
42 slices processed
43 slices processed
44 slices processed
45 slices processed
46 slices processed
47 slices processed
48 slices processed
49 slices processed
50 slices processed
51 slices processed
52 slices processed
53 slices processed
54 slices processed
55 slices processed
56 slices processed
57 slices processed
58 slices processed
59 slices processed
60 slices processed
61 slices processed
62 slices processed
63 slices processed
64 slices processed
65 slices processed
66 slices processed
67 slices processed
68 slices processed
69 slices processed
70 slices processed
71 slices processed
Error: failed to open file 111180-100/3t_2016-01-07_21-42/003_DTI_siemens_TClessdistort/111180-100_dtifit_FA.nii.gz
Image Exception : #22 :: ERROR: Could not open image 111180-100/3t_2016-01-07_21-42/003_DTI_siemens_TClessdistort/111180-100_dtifit_FA
terminate called after throwing an instance of 'RBD_COMMON::BaseException'
./scriptb.sh: line 11:  5756 Aborted                 (core dumped) dtifit -k "$data" -o "$output" -m "$mask" -r "$bvecs" -b "$bvals"

JJJCR · 06-13-2016, 08:17 PM

When you were doing it manually? Do you need to extract ".gz" before you processed? Or what?

azurite · 06-13-2016, 09:16 PM

Quote:

Originally Posted by JJJCR

When you were doing it manually? Do you need to extract ".gz" before you processed? Or what?

No, when I did it manually, I didn't extract the .gz.
I think it might be something with how the output file is named. Is there anyway to write the code so that it names the output as something along the lines of 'subfolder name'_dti?

This is the code Iin terminnal when I successfully run it manually

Code:

/usr/share/fsl/5.0/bin/dtifit
 --data=/home/natasha/Documents/DWI/111180-100/3t_2016-01-07_21-42/003_DTI_siemens_TClessdistort/111180-100_eddy_corrected.nii.gz 
--out=/home/natasha/Documents/DWI/111180-100/3t_2016-01-07_21-42/003_DTI_siemens_TClessdistort/dti 
--mask=/home/natasha/Documents/DWI/111180-100/3t_2016-01-07_21-42/003_DTI_siemens_TClessdistort/111180-100_eddy_corrected_brain_mask.nii.gz 
--bvecs=/home/natasha/Documents/DWI/111180-100/3t_2016-01-07_21-42/003_DTI_siemens_TClessdistort/20160107_214213DTIsiemensTClessdistorts003a001.bvec 
--bvals=/home/natasha/Documents/DWI/111180-100/3t_2016-01-07_21-42/003_DTI_siemens_TClessdistort/20160107_214213DTIsiemensTClessdistorts003a001.bval
0 128 0 128 0 72
0 slices processed
1 slices processed
2 slices processed
3 slices processed
4 slices processed
5 slices processed
6 slices processed
7 slices processed
8 slices processed
9 slices processed
10 slices processed
11 slices processed
12 slices processed
13 slices processed
14 slices processed
15 slices processed
16 slices processed
17 slices processed
18 slices processed
19 slices processed
20 slices processed
21 slices processed
22 slices processed
23 slices processed
24 slices processed
25 slices processed
26 slices processed
27 slices processed
28 slices processed
29 slices processed
30 slices processed
31 slices processed
32 slices processed
33 slices processed
34 slices processed
35 slices processed
36 slices processed
37 slices processed
38 slices processed
39 slices processed
40 slices processed
41 slices processed
42 slices processed
43 slices processed
44 slices processed
45 slices processed
46 slices processed
47 slices processed
48 slices processed
49 slices processed
50 slices processed
51 slices processed
52 slices processed
53 slices processed
54 slices processed
55 slices processed
56 slices processed
57 slices processed
58 slices processed
59 slices processed
60 slices processed
61 slices processed
62 slices processed
63 slices processed
64 slices processed
65 slices processed
66 slices processed
67 slices processed
68 slices processed
69 slices processed
70 slices processed
71 slices processed
Done!

From what I can tell, the error has something to do with how the output file is saved/named.

Code:

Error: failed to open file 111440-100/3t_2016-01-27_20-58/003_DTI_siemens_TClessdistort/111440-100_eddy_corrected.nii.gz/_dtifit_FA.nii.gz
Image Exception : #22 :: ERROR: Could not open image 111440-100/3t_2016-01-27_20-58/003_DTI_siemens_TClessdistort/111440-100_eddy_corrected.nii.gz/_dtifit_FA
terminate called after throwing an instance of 'RBD_COMMON::BaseException'
./scriptb.sh: line 11:  6377 Aborted                 (core dumped) dtifit -k "$data" -o "$output" -m "$mask" -r "$bvecs" -b "$bvals"

instead it has to omit the bolded part and to save/name it like so

Code:

home/natasha/Documents/DWI/111440-100/3t_2016-01-27_20-58/003_DTI_siemens_TClessdistort/dti

How would I change the script accordingly?

JJJCR · 06-13-2016, 10:41 PM

try changing this line: output="${output/_eddy_corrected_brain.nii.gz/}_dtifit"

change to: output="${output/dti}_dtifit"

azurite · 06-14-2016, 12:57 AM

Quote:

Originally Posted by JJJCR

try changing this line: output="${output/_eddy_corrected_brain.nii.gz/}_dtifit"

change to: output="${output/dti}_dtifit"

Okay, so it's most probable that it's the bit of code that deals with the output name.
I modified it a bit to test my hypothesis and it sort of worked. Now I just have to figure out the correct syntax to put in so that I get the file name as the program requires it.
As you will see in the screenshot, I've experimented so that I got the output to be saved/named as "0_FA.nii.gz' etc. Now the goal is to change the code accordingly so that the output gets saved/named as "dti_fa.nii.gz" instead of the 0 in front. It feels like we're so close to getting this to work, I just want to be able to finally get it right.

JJJCR · 06-14-2016, 01:08 AM

add this line

Quote:

# Build the output file name
output="${f##/}"
str="dti"

change the output line to:

Quote:

output="${output/$str}_dtifit"

Good luck!!

azurite · 06-14-2016, 01:09 PM

Edit: Halfway figured it out.
Changed the output code to

Code:

output="${f/*}dti

azurite · 06-14-2016, 07:35 PM

The script is successful and I very much appreciate everyone's assistance and suggestions. If it's not too much to ask, I would like to understand the script line by line so that I can do this on my own in the future. I would like to get a good grasp of the basic foundation of writing scripts that, as this one, go into subfolders, look for files, and carry out commands but my knowledge is still not at a level where I can do this on my own. I was hoping that better understanding what each line does in the script allend provided would help me write simple scripts in the future. If anyone can write the explanations, please?

Code:

#!/bin/bash

# Top of directory structure containing files to be processed.
topdir="/home/natasha/Documents/DWI"

cd "$topdir"

# Set shopt to also look in subdirectories
shopt -s globstar

for f in **/*_eddy_corrected_brain.nii.gz; do
  # Change to subdirectory
  cd "${f%/*}"

  # Set variables for program
  # TODO - Add error checking for missing or duplicate files
  data="$(stat -c %n *_eddy_corrected_brain.nii.gz)"
  mask="$(stat -c %n *_eddy_corrected_brain_mask.nii.gz)"
  bvecs="$(stat -c %n *.bvec)"
  bvals="$(stat -c %n *.bval)"
  # Build the output file name
  output="${f##/}"
  output="${output/_eddy_corrected_brain.nii.gz/}_dtifit"
  
  # Run the program
  dtifit -k "$data" -m "$mask" -r "$bvecs" -b "$bvals" -o "$output"

  # Change back to top directory
  cd "$topdir"
done

# Undo change to shopt
shopt -u globstar

JJJCR · 06-14-2016, 09:22 PM

check out link below, it will explain some of the looping process.

http://wiki.bash-hackers.org/syntax/pe

http://www.linuxjournal.com/content/...lobbing-option

Good luck! Glad to know you finally run the script successfully.

Turbocapitalist · 06-14-2016, 09:32 PM

Well, most of it is buried in the "bash" manual page as shopt, for, and cd are built-in commands. I'd recommend following everything up with a browser through it to learn it as a reference source. No one can remember it all, or even most of it, but you can remember where you last saw something and be able to look it up as needed. "just in time" rather than "just in case". If you use features often enough, they will sink in.

"shopt -s globstar" is where the important change is. That allows ** to mean a recursive descent into directories and subdirectories. It is about the equivalent of "find" offered in #5 above.

The ${f%/*} trims the variable $f using substring removal matching from the end.

The "stat -c %n" prints just the file name and using the $( ... ) for command substitution the resulting output from the program "stat" is treated as if it were a string, allowing it to be assigned to a variable.

The ${f##/} is more substring removal but matching from the beginning. (Note that the slash is being matched it is not an operator.)

The second output=" line is just just a search and removal of '_eddy_corrected_brain.nii.gz/' and appending '_dtifit'

So, yes, "bash" is a scripting language.

Edit: JJJCR was more concise and posted the relevant links while I was slowly drafting the above.