[SOLVED] Help on using parallel shell command

kostas85 · 04-28-2014, 06:42 PM

Hello all,

I'm trying to parallelize the following shell script:

Code:

for ((i=0,i<=100,i+=4));do
 cat ${filelist1[@]:$i:4} ${filelist2[@]:$i:4} > ${filelist3[@]:$i:4} 
done

but I have troubles when I use the parallel command

Code:

for ((i=0,i<=100,i+=4));do
 f1in=("${f1in[@]}" "`echo ${filelist1[@]:$i:4}`")
 f2in=("${f2in[@]}" "`echo ${filelist2[@]:$i:4}`")
 fout=("${fout[@]}" "`echo ${filelist3[@]:$i:4}`")
done

parallel --xapply cat {1} {2} ">" {3} ::: "f1in[@]" ::: "f2in[@]" ::: "fout[@]"

The above interprets the files given in, say f1in[0], as a single file. So cat searches for a single file that doesn't exist and prints an error message. Probably the solution is really simple but I'm not an experienced shell programmer. Could you please propose some workarounds. Many thanks in advance!

Kostas

ntubski · 04-28-2014, 08:58 PM

`` is the inverse of echo, but echo joins all its arguments together with space. Just cancel them out, this keeps the arguments separated:

Code:

 f1in=("${f1in[@]}" "${filelist1[@]:$i:4}")

kostas85 · 04-29-2014, 02:40 AM

Thanks for your answer!!! I'm sorry I made a mistake in the code I posted... the 4+4 files should be concatenated to a single file so:

Code:

for ((i=0,i<=100,i+=4));do
 cat ${filelist1[@]:$i:4} ${filelist2[@]:$i:4} > ${filelist3[@]:$i} 
done

and similarly:

Code:

for ((i=0,i<=100,i+=4));do
 f1in=("${f1in[@]}" "`echo ${filelist1[@]:$i:4}`")
 f2in=("${f2in[@]}" "`echo ${filelist2[@]:$i:4}`")
 fout=("${fout[@]}" "`echo ${filelist3[@]:$i}`")
done

Each process should get 4 files from list filelist1, 4 from list filelist2 and join them to one file from filelist3

I used the echo to generate the "list" of items ${filelist1[@]:$i:4} inside the arrays f1in,f2in, since without the echo they are treated as seperate elements. The problem is that when I do as a test:

Code:

cat ${f1in[0]} ${f2in[0]} > ${fout[0]}

It works normally.. However, when processed from parallel it says that it cannot find the files since as you said the arguments are joined with space.
I'm sorry for the mistake... Any ideas ?

ntubski · 04-29-2014, 10:06 AM

Quote:

Originally Posted by kostas85

I used the echo to generate the "list" of items ${filelist1[@]:$i:4} inside the arrays f1in,f2in, since without the echo they are treated as seperate elements. The problem is that when I do as a test:

Code:

cat ${f1in[0]} ${f2in[0]} > ${fout[0]}

It works normally..

Ah, you really want a 2 dimensional array of file names. When you use ${f1in[0]} unquoted, the shell splits the value by whitespace, so that's a way of faking the inner array. Shell script doesn't have real 2D arrays, so you should consider writing this is in some other language. You can achieve the faking by explicitly telling parallel to call a shell:

Code:

parallel --xapply sh -c 'cat $0 $1 > "$2"' {1} {2} {3} ::: "f1in[@]" ::: "f2in[@]" ::: "fout[@]"

By the way, to join the arguments with space you can use "${filelist1[*]:$i:4}" instead of "`echo ${filelist1[@]:$i:4}`"

kostas85 · 04-29-2014, 03:08 PM

Thank you!!! It worked like a charm!!!

However I found in my previous post some errors... The part I posted is a part of a much larger script that I faulty reduced...

Here is the final working version with our suggestions:

--------
SERIAL
--------

Code:

#
# Generate three lists of files: filelist1 filelist2 filelist3
# Input to cat are: filelist1 and filelist2
# the output file should be named the ith element of filelist3
#
# filelist1 -> the files taken by 4 should be at the beginning of the resulting file
# filelist2 -> the files taken by 4 should follow
#
for ((i=0,i<=99,i+=4));do
 cat ${filelist1[@]:$i:4} ${filelist2[@]:$i:4} > ${filelist3[@]:$i} 
done

-----------
PARALLEL
-----------

Code:

#
# Generate three lists of files: filelist1 filelist2 filelist3
# Input to cat are: filelist1 and filelist2
# the output file should be named the ith element of  filelist3
#
# filelist1 -> the files taken by 4 should be at the beginning of the resulting file
# filelist2 -> the files taken by 4 should follow
#
function local_cat()
{
  cat $1 $2 > $3
}

export -f local_cat

for ((i=0,i<=99,i+=4));do
 f1in=("${f1in[@]}" "${filelist1[*]:$i:4}")
 f2in=("${f2in[@]}" "${filelist2[*]:$i:4}")
 fout=("${fout[@]}" "${filelist3[@]:$i}")
done

parallel --xapply local_cat {1} {2} {3} ::: "${f1in[@]}" ::: "${f2in[@]}" ::: "${fout[@]}"

Thanks again! You really helped me a lot!!!