LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 06-13-2016, 02:07 AM   #16
azurite
LQ Newbie
 
Registered: May 2016
Posts: 24

Original Poster
Rep: Reputation: Disabled

Quote:
Originally Posted by allend View Post
If you do this, the code will work as written. Just remember to make the file executable with 'chmod +x <scriptanme>'.

You could make the script more portable, in that it could be launched from any directory, by changing to
Code:
# Save current directory
curdir=$PWD
# Change to top directory
topdir="/path/to/top/directory"
cd "$topdir"
...
  cd "$curdir"
  cd "$topdir"
For setting the variables and calling the data analysis program, I suggest code like
Code:
data="$(stat -c %n *eddy_corrected_brain.nii.gz)"
...
dtifit -k "$data" -m "$mask" -r "$bvecs" -b "$bvals" -o "$output"
It is recommended to use 'stat' rather than 'ls' for getting filenames in scripts.
I must be oblivious to something because I could not get it to work. Thus, I have recorded a bit of my process. Perhaps someone here can figure out what I'm doing wrong. The bit towards the end (around 3:50 min mark) is me doing the process using the program GUI instead of the script. But ultimately, I want to use a script to make the process faster/easier.

http://sendvid.com/y0a92ko4
 
Old 06-13-2016, 03:12 AM   #17
allend
Senior Member
 
Registered: Oct 2003
Location: Melbourne
Distribution: Slackware-current
Posts: 4,661

Rep: Reputation: 1540Reputation: 1540Reputation: 1540Reputation: 1540Reputation: 1540Reputation: 1540Reputation: 1540Reputation: 1540Reputation: 1540Reputation: 1540Reputation: 1540
The script in the video does not seem to match what has been suggested. (I have attached a still from the video).

I was hoping that you would finish the script from the snippets provided.

Code:
#!/bin/bash

# Top of directory structure containing files to be processed.
topdir="/home/natasha/Documents/DWI"

cd "$topdir"

# Set shopt to also look in subdirectories
shopt -s globstar

for f in **/*_eddy_corrected_brain.nii.gz; do
  # Change to subdirectory
  cd "${f%/*}"

  # Set variables for program
  # TODO - Add error checking for missing or duplicate files
  data="$(stat -c %n *_eddy_corrected_brain.nii.gz)"
  mask="$(stat -c %n *_eddy_corrected_brain_mask.nii.gz)"
  bvecs="$(stat -c %n *.bvec)"
  bvals="$(stat -c %n *.bval)"
  # Build the output file name
  output="${f##/}"
  output="${output/_eddy_corrected_brain.nii.gz/}_dtifit"
  
  # Run the program
  dtifit -k "$data" -m "$mask" -r "$bvecs" -b "$bvals" -o "$output"

  # Change back to top directory
  cd "$topdir"
done

# Undo change to shopt
shopt -u globstar
Attached Thumbnails
Click image for larger version

Name:	wsnapshot.png
Views:	6
Size:	111.7 KB
ID:	22093  
 
Old 06-13-2016, 03:30 AM   #18
JJJCR
Senior Member
 
Registered: Apr 2010
Posts: 1,345

Rep: Reputation: 227Reputation: 227Reputation: 227
Didn't know about stat.

I tried it on my box quite helpful no need to cut the string just to get the filename.

Quote:
[Quas@puas01x ~]$ stat -c %n *.txt
NewFile.txt
xinput.txt
xxq.txt
Looks like magic it gets all the filename of all the files ending with ".txt"

Thanks Allend!!

Last edited by JJJCR; 06-13-2016 at 03:31 AM. Reason: edit
 
Old 06-13-2016, 03:43 AM   #19
allend
Senior Member
 
Registered: Oct 2003
Location: Melbourne
Distribution: Slackware-current
Posts: 4,661

Rep: Reputation: 1540Reputation: 1540Reputation: 1540Reputation: 1540Reputation: 1540Reputation: 1540Reputation: 1540Reputation: 1540Reputation: 1540Reputation: 1540Reputation: 1540
Quote:
Didn't know about stat.
Taken from http://mywiki.wooledge.org/ParsingLs
 
1 members found this post helpful.
Old 06-13-2016, 05:42 AM   #20
JJJCR
Senior Member
 
Registered: Apr 2010
Posts: 1,345

Rep: Reputation: 227Reputation: 227Reputation: 227
Lightbulb

Quote:
Originally Posted by allend View Post
Thanks for the link.
 
Old 06-13-2016, 04:21 PM   #21
azurite
LQ Newbie
 
Registered: May 2016
Posts: 24

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by allend View Post
The script in the video does not seem to match what has been suggested. (I have attached a still from the video).

I was hoping that you would finish the script from the snippets provided.

Code:
#!/bin/bash

# Top of directory structure containing files to be processed.
topdir="/home/natasha/Documents/DWI"

cd "$topdir"

# Set shopt to also look in subdirectories
shopt -s globstar

for f in **/*_eddy_corrected_brain.nii.gz; do
  # Change to subdirectory
  cd "${f%/*}"

  # Set variables for program
  # TODO - Add error checking for missing or duplicate files
  data="$(stat -c %n *_eddy_corrected_brain.nii.gz)"
  mask="$(stat -c %n *_eddy_corrected_brain_mask.nii.gz)"
  bvecs="$(stat -c %n *.bvec)"
  bvals="$(stat -c %n *.bval)"
  # Build the output file name
  output="${f##/}"
  output="${output/_eddy_corrected_brain.nii.gz/}_dtifit"
  
  # Run the program
  dtifit -k "$data" -m "$mask" -r "$bvecs" -b "$bvals" -o "$output"

  # Change back to top directory
  cd "$topdir"
done

# Undo change to shopt
shopt -u globstar
Hi allend,

I ran the modified script you so kindly posted and it seemed to have started working but I get the following error at the end. I'm not sure if it's something in the script or the actual files themselves. If you could be so kind as to again take a look and help with figure this out, I would really be grateful.

Code:
natasha@ubuntu:~/Documents$ cd /home/natasha/Documents/DWI
natasha@ubuntu:~/Documents/DWI$ chmod +X scriptb.sh
natasha@ubuntu:~/Documents/DWI$ ./scriptb.sh
0 128 0 128 0 72
0 slices processed
1 slices processed
2 slices processed
3 slices processed
4 slices processed
5 slices processed
6 slices processed
7 slices processed
8 slices processed
9 slices processed
10 slices processed
11 slices processed
12 slices processed
13 slices processed
14 slices processed
15 slices processed
16 slices processed
17 slices processed
18 slices processed
19 slices processed
20 slices processed
21 slices processed
22 slices processed
23 slices processed
24 slices processed
25 slices processed
26 slices processed
27 slices processed
28 slices processed
29 slices processed
30 slices processed
31 slices processed
32 slices processed
33 slices processed
34 slices processed
35 slices processed
36 slices processed
37 slices processed
38 slices processed
39 slices processed
40 slices processed
41 slices processed
42 slices processed
43 slices processed
44 slices processed
45 slices processed
46 slices processed
47 slices processed
48 slices processed
49 slices processed
50 slices processed
51 slices processed
52 slices processed
53 slices processed
54 slices processed
55 slices processed
56 slices processed
57 slices processed
58 slices processed
59 slices processed
60 slices processed
61 slices processed
62 slices processed
63 slices processed
64 slices processed
65 slices processed
66 slices processed
67 slices processed
68 slices processed
69 slices processed
70 slices processed
71 slices processed
Error: failed to open file 111180-100/3t_2016-01-07_21-42/003_DTI_siemens_TClessdistort/111180-100_dtifit_FA.nii.gz
Image Exception : #22 :: ERROR: Could not open image 111180-100/3t_2016-01-07_21-42/003_DTI_siemens_TClessdistort/111180-100_dtifit_FA
terminate called after throwing an instance of 'RBD_COMMON::BaseException'
./scriptb.sh: line 11:  5756 Aborted                 (core dumped) dtifit -k "$data" -o "$output" -m "$mask" -r "$bvecs" -b "$bvals"
 
Old 06-13-2016, 09:17 PM   #22
JJJCR
Senior Member
 
Registered: Apr 2010
Posts: 1,345

Rep: Reputation: 227Reputation: 227Reputation: 227
When you were doing it manually? Do you need to extract ".gz" before you processed? Or what?
 
Old 06-13-2016, 10:16 PM   #23
azurite
LQ Newbie
 
Registered: May 2016
Posts: 24

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by JJJCR View Post
When you were doing it manually? Do you need to extract ".gz" before you processed? Or what?
No, when I did it manually, I didn't extract the .gz.
I think it might be something with how the output file is named. Is there anyway to write the code so that it names the output as something along the lines of 'subfolder name'_dti?

This is the code Iin terminnal when I successfully run it manually
Code:
/usr/share/fsl/5.0/bin/dtifit
 --data=/home/natasha/Documents/DWI/111180-100/3t_2016-01-07_21-42/003_DTI_siemens_TClessdistort/111180-100_eddy_corrected.nii.gz 
--out=/home/natasha/Documents/DWI/111180-100/3t_2016-01-07_21-42/003_DTI_siemens_TClessdistort/dti 
--mask=/home/natasha/Documents/DWI/111180-100/3t_2016-01-07_21-42/003_DTI_siemens_TClessdistort/111180-100_eddy_corrected_brain_mask.nii.gz 
--bvecs=/home/natasha/Documents/DWI/111180-100/3t_2016-01-07_21-42/003_DTI_siemens_TClessdistort/20160107_214213DTIsiemensTClessdistorts003a001.bvec 
--bvals=/home/natasha/Documents/DWI/111180-100/3t_2016-01-07_21-42/003_DTI_siemens_TClessdistort/20160107_214213DTIsiemensTClessdistorts003a001.bval
0 128 0 128 0 72
0 slices processed
1 slices processed
2 slices processed
3 slices processed
4 slices processed
5 slices processed
6 slices processed
7 slices processed
8 slices processed
9 slices processed
10 slices processed
11 slices processed
12 slices processed
13 slices processed
14 slices processed
15 slices processed
16 slices processed
17 slices processed
18 slices processed
19 slices processed
20 slices processed
21 slices processed
22 slices processed
23 slices processed
24 slices processed
25 slices processed
26 slices processed
27 slices processed
28 slices processed
29 slices processed
30 slices processed
31 slices processed
32 slices processed
33 slices processed
34 slices processed
35 slices processed
36 slices processed
37 slices processed
38 slices processed
39 slices processed
40 slices processed
41 slices processed
42 slices processed
43 slices processed
44 slices processed
45 slices processed
46 slices processed
47 slices processed
48 slices processed
49 slices processed
50 slices processed
51 slices processed
52 slices processed
53 slices processed
54 slices processed
55 slices processed
56 slices processed
57 slices processed
58 slices processed
59 slices processed
60 slices processed
61 slices processed
62 slices processed
63 slices processed
64 slices processed
65 slices processed
66 slices processed
67 slices processed
68 slices processed
69 slices processed
70 slices processed
71 slices processed
Done!
From what I can tell, the error has something to do with how the output file is saved/named.

Code:
Error: failed to open file 111440-100/3t_2016-01-27_20-58/003_DTI_siemens_TClessdistort/111440-100_eddy_corrected.nii.gz/_dtifit_FA.nii.gz
Image Exception : #22 :: ERROR: Could not open image 111440-100/3t_2016-01-27_20-58/003_DTI_siemens_TClessdistort/111440-100_eddy_corrected.nii.gz/_dtifit_FA
terminate called after throwing an instance of 'RBD_COMMON::BaseException'
./scriptb.sh: line 11:  6377 Aborted                 (core dumped) dtifit -k "$data" -o "$output" -m "$mask" -r "$bvecs" -b "$bvals"
instead it has to omit the bolded part and to save/name it like so

Code:
home/natasha/Documents/DWI/111440-100/3t_2016-01-27_20-58/003_DTI_siemens_TClessdistort/dti
How would I change the script accordingly?

Last edited by azurite; 06-14-2016 at 11:52 PM.
 
Old 06-13-2016, 11:41 PM   #24
JJJCR
Senior Member
 
Registered: Apr 2010
Posts: 1,345

Rep: Reputation: 227Reputation: 227Reputation: 227
try changing this line: output="${output/_eddy_corrected_brain.nii.gz/}_dtifit"

change to: output="${output/dti}_dtifit"
 
Old 06-14-2016, 01:57 AM   #25
azurite
LQ Newbie
 
Registered: May 2016
Posts: 24

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by JJJCR View Post
try changing this line: output="${output/_eddy_corrected_brain.nii.gz/}_dtifit"

change to: output="${output/dti}_dtifit"
Okay, so it's most probable that it's the bit of code that deals with the output name.
I modified it a bit to test my hypothesis and it sort of worked. Now I just have to figure out the correct syntax to put in so that I get the file name as the program requires it.
As you will see in the screenshot, I've experimented so that I got the output to be saved/named as "0_FA.nii.gz' etc. Now the goal is to change the code accordingly so that the output gets saved/named as "dti_fa.nii.gz" instead of the 0 in front. It feels like we're so close to getting this to work, I just want to be able to finally get it right.

Last edited by azurite; 06-14-2016 at 11:52 PM.
 
Old 06-14-2016, 02:08 AM   #26
JJJCR
Senior Member
 
Registered: Apr 2010
Posts: 1,345

Rep: Reputation: 227Reputation: 227Reputation: 227
add this line

Quote:
# Build the output file name
output="${f##/}"
str="dti"

change the output line to:


Quote:
output="${output/$str}_dtifit"
Good luck!!
 
Old 06-14-2016, 02:09 PM   #27
azurite
LQ Newbie
 
Registered: May 2016
Posts: 24

Original Poster
Rep: Reputation: Disabled
Edit: Halfway figured it out.
Changed the output code to
Code:
output="${f/*}dti

Last edited by azurite; 06-14-2016 at 08:29 PM.
 
Old 06-14-2016, 08:35 PM   #28
azurite
LQ Newbie
 
Registered: May 2016
Posts: 24

Original Poster
Rep: Reputation: Disabled
The script is successful and I very much appreciate everyone's assistance and suggestions. If it's not too much to ask, I would like to understand the script line by line so that I can do this on my own in the future. I would like to get a good grasp of the basic foundation of writing scripts that, as this one, go into subfolders, look for files, and carry out commands but my knowledge is still not at a level where I can do this on my own. I was hoping that better understanding what each line does in the script allend provided would help me write simple scripts in the future. If anyone can write the explanations, please?

Code:
#!/bin/bash

# Top of directory structure containing files to be processed.
topdir="/home/natasha/Documents/DWI"

cd "$topdir"

# Set shopt to also look in subdirectories
shopt -s globstar

for f in **/*_eddy_corrected_brain.nii.gz; do
  # Change to subdirectory
  cd "${f%/*}"

  # Set variables for program
  # TODO - Add error checking for missing or duplicate files
  data="$(stat -c %n *_eddy_corrected_brain.nii.gz)"
  mask="$(stat -c %n *_eddy_corrected_brain_mask.nii.gz)"
  bvecs="$(stat -c %n *.bvec)"
  bvals="$(stat -c %n *.bval)"
  # Build the output file name
  output="${f##/}"
  output="${output/_eddy_corrected_brain.nii.gz/}_dtifit"
  
  # Run the program
  dtifit -k "$data" -m "$mask" -r "$bvecs" -b "$bvals" -o "$output"

  # Change back to top directory
  cd "$topdir"
done

# Undo change to shopt
shopt -u globstar
 
Old 06-14-2016, 10:22 PM   #29
JJJCR
Senior Member
 
Registered: Apr 2010
Posts: 1,345

Rep: Reputation: 227Reputation: 227Reputation: 227
check out link below, it will explain some of the looping process.

http://wiki.bash-hackers.org/syntax/pe

http://www.linuxjournal.com/content/...lobbing-option

Good luck! Glad to know you finally run the script successfully.

Last edited by JJJCR; 06-14-2016 at 10:23 PM. Reason: edit
 
1 members found this post helpful.
Old 06-14-2016, 10:32 PM   #30
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Ubuntu, Devuan, OpenBSD
Posts: 2,627
Blog Entries: 3

Rep: Reputation: 1157Reputation: 1157Reputation: 1157Reputation: 1157Reputation: 1157Reputation: 1157Reputation: 1157Reputation: 1157Reputation: 1157
Well, most of it is buried in the "bash" manual page as shopt, for, and cd are built-in commands. I'd recommend following everything up with a browser through it to learn it as a reference source. No one can remember it all, or even most of it, but you can remember where you last saw something and be able to look it up as needed. "just in time" rather than "just in case". If you use features often enough, they will sink in.

"shopt -s globstar" is where the important change is. That allows ** to mean a recursive descent into directories and subdirectories. It is about the equivalent of "find" offered in #5 above.

The ${f%/*} trims the variable $f using substring removal matching from the end.

The "stat -c %n" prints just the file name and using the $( ... ) for command substitution the resulting output from the program "stat" is treated as if it were a string, allowing it to be assigned to a variable.

The ${f##/} is more substring removal but matching from the beginning. (Note that the slash is being matched it is not an operator.)

The second output=" line is just just a search and removal of '_eddy_corrected_brain.nii.gz/' and appending '_dtifit'

So, yes, "bash" is a scripting language.

Edit: JJJCR was more concise and posted the relevant links while I was slowly drafting the above.

Last edited by Turbocapitalist; 06-14-2016 at 10:45 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
need command to move folders older than 4 days to a bad up directory calicowboy54 Linux - General 4 01-15-2012 10:24 PM
Include loop to get files from different folders that have same name Perseus Programming 9 10-10-2011 04:37 AM
[SOLVED] Bash question (loop though folders and run a command) takayama Programming 4 02-14-2011 06:20 AM
find -exec command to recursively delete files and folders in folders with X name Joan Murt Linux - Newbie 2 07-08-2009 05:35 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 10:35 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration