-   Programming (
-   -   Bash scripting using EOF to exec external programs (

PB0711 10-29-2009 02:12 PM

Bash scripting using EOF to exec external programs
Dear all,

As sad as it is I've always strayed away from bash scripting, I used Perl or even c-shell. However, I'm trying to set up a program on my university's cluster system so now I have to. I want to run a program called 'R' and run some commands that will depend upon the arguments in the script. In c-shelll I know I can do a redirect using >>EOF like syntax, can I do this is bash and if so what am I doing wrong?


#PBS -l walltime=6:00:00
#PBS -l mem=8000mb
#PBS -l ncpus=1

##module load R/2.9.1
##module load netcdf/3.2.6
##module load intel-suite/10.1

mzFiles=$(find . -iname '*.mz*' -print)
netCDFfiles=$(find . -iname '*.*cdf' -print) ##find all the files we need

fileString=$(echo $mzFiles $netCDFfiles) ##paste them together
filesArr=(`echo $foo | tr ' ' ' '`) ##make it into an array
date=$(date +%d%b) # get the date for unique file making

for f in ${filesArr[@]}
        if "$method" == "centWave"
                qsub $f $method $peakwidth
                qsub $f $method
        ##This hasn't been written yet
        ##This line will submit each file to the queing system

##Then we run R to paste everything together on one node.
##Check that the DIR are the same
/apps/R/2.9.1/bin/R --vanilla > $HOME/myReport$date.out <<EOF
rdfiles <- list.files(path="$TMPDIR/", recursive = TRUE, pattern="*.Rd")

There's more code to this but it's not the point. At the moment when I run the script it says "line 42" all of this code ": bad substitution"
I'm guessing that means that I can't do this. So hows the best way to do this? I've googled and read quite a few bash scripting tutorials but I havn't really seen an answer to this yet.

any help welcomed. thx :)

David the H. 10-29-2009 04:57 PM

By EOF, are you referring to here documents?

But "bad substitution" sounds more like a sed error or something to me.

BTW, what's this supposed to do? It doesn't look right to me. I don't think you want $foo, at the very least, and the tr command doesn't change anything.

filesArr=(`echo $foo | tr ' ' ' '`)

catkin 10-29-2009 05:10 PM

You need double quotes around fileString=$(echo $mzFiles $netCDFfiles) like

fileString="$(echo $mzFiles $netCDFfiles)" ##paste them together
If that doesn't work please post with line numbers.

ta0kira 10-29-2009 08:52 PM


Originally Posted by David the H. (Post 3737135)
BTW, what's this supposed to do? It doesn't look right to me. I don't think you want $foo, at the very least, and the tr command doesn't change anything.

filesArr=(`echo $foo | tr ' ' ' '`)

I thought that too, but if you count characters the replacement could well be a tab.
Kevin Barry

PB0711 11-04-2009 07:34 AM

Hello all,

Thank you for the replys. I've looked again at this and I think as you rightly point out that my $foo isn't doing anything because I forgot to translate between my testing and my script opps :o
However, I'm having a problem now with good old line spaces. I want $fileArr to be an array of $fileString, hence using the 'tr'. Is there another way to do make a list of files into an array

filesArr=(`echo $fileString | tr '' ' '`)
box:SDS user $ echo ${filesArr[3]}
box:SDS user$ echo $fileString
./SDS1/025 Sample 1.mzXML ./SDS1/046 Sample 1.mzXML ./SDS1/081 Sample 1.mzXML ./SDS2/030 Sample 2.mzXML ./SDS2/066 Sample 2.mzXML ./SDS2/079 Sample 2.mzXML ./SDS3/035 Sample 3.mzXML ./SDS3/044 Sample 3.mzXML ./SDS3/097 Sample 3.mzXML ./SDS4/018 Sample 4.mzXML ./SDS4/040 Sample 4.mzXML ./SDS4/071 Sample 4.mzXML ./SDS5/020 Sample 5.mzXML ./SDS5/057 Sample 5.mzXML ./SDS5/072 Sample 5.mzXML

I also tried something like this

box:SDS user$ mzFiles=$(find . -iname '*.mz*' -print $f!)
-bash: !: event not found
box:SDS user$ mzFiles=$(`find . -iname '*.mz*' -print $f!`)
-bash: !`: event not found
$ mzFiles=$(" find . -iname '*.mz*' -print $f! ")
-bash:  find . -iname '*.mz*' -print ! : command not found

catkin 11-04-2009 12:07 PM

If $fileString contains a space-separated list of files with names containing embedded spaces then there is no way to identify the individual files. If every file begins with ./ then it can be done but is not easy in shell script. Maybe better to take a step back and tell us how $fileString is generated; it may be easier to get the file names into an array at that point. You may find Greg's Wiki helpful.

David the H. 11-04-2009 02:16 PM

Probably the most convenient way when working with spaces in strings, especially file names, is to change the internal field separator so that it ignores them. Of course, it depends on how you've generated the strings in the first place. If you have a variable containing a list of names, and you change the IFS to newline, for example, then you just need to do is make sure that the individual entries are separated by newlines.


oldIFS=$IFS  # Save the original values in case you need to reset them later.
            # Default: space,tab, newline.

'            # Change the separator to newline only.

<your code>

IFS=$oldIFS  # To reset the value, if necessary.

You could also change the IFS to something else, such as a comma or colon, if you're using such characters as delimiters in your input.

All times are GMT -5. The time now is 10:27 PM.