[SOLVED] make code in bash go parallel

jheengut · 11-08-2012, 11:21 AM

Hi

I have a script that that needs the for loop to work on several items in a directory alas using

Code:

 command *

does not work.

I want to make the code more parallel

ie

from

Code:

for i in `seq $start  $finish` ; do
     command _something_ending_in_$i
done

to something like this

Code:

for j in ` seq $start $increment $finish ` ;  do 
  for i in `seq $I_start $j ` ; do
     command _something_ending_in_$i


	Quote:
	
	
		
			
				some code needs to be added here
			
		
	
	
  done

	Quote:
	
	
		
			
				and  here also
			
		
	
	
done

Quote:

$j should always be $increment greater than $I_start

I experimented with (( )) ARITHMETIC EVALUATION
to perform the job but it gets hard to understand and maintain.

Is my algorithm bad ??

Habitual · 11-08-2012, 01:14 PM

Code:

#!/bin/bash
for i in `seq 1 10`
do touch test.$i.txt
done
ls -al *.txt
exit 0
#EOF

and "command _something_ending_in_$i" should be "command _something_ending to $i"

See http://mywiki.wooledge.org/BashPitfalls

Here's some light reading that may interest you:
Various Linux Guides I have collected:
http://www.tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html
http://www.tldp.org/LDP/Bash-Beginne...tml/index.html
http://tldp.org/HOWTO/Bash-Prompt-HOWTO/
http://www.gnu.org/software/bash/man...ode/index.html
http://www.grymoire.com/Unix/Sh.html
http://www.grymoire.com/Unix/Sed.html
http://tldp.org/LDP/abs/abs-guide.pdf
http://www.tldp.org/LDP/abs/html/
http://mywiki.wooledge.org/BashFAQ
http://mywiki.wooledge.org/BashPitfalls
http://rute.2038bug.com/index.html.gz
http://mylinuxbook.com/linux-shell-environment/
http://www.subsignal.org/doc/AliensB...0Of%20Contents
http://www.rigacci.org/docs/biblio/o...TLCL-09.12.pdf

Where is David the H: when I need him.

schneidz · 11-08-2012, 01:26 PM

to make something run multithreaded you can try something like:

Code:

#!/bin/bash

i=0
while [ $i -le 10 ]
do
 i=`expr $i + 1`
 echo thread 1: i = $i
done &

j=100
while [ $j -le 110 ]
do
 j=`expr $j + 1`
 echo thread 2: j = $j
done &

lyle_s · 11-08-2012, 06:12 PM

Please have a look at http://www.gnu.org/software/parallel/

Lyle.

Reuti · 11-09-2012, 05:11 PM

Quote:

Originally Posted by jheengut

I have a script that that needs the for loop to work on several items in a directory alas using

On one and the same machine – how many cores are available? Or on several machines?

Depending what and how often you want to do it, you can also look into a small queuing system like GNUbatch.

jheengut · 11-25-2012, 11:45 AM

Quote:

Originally Posted by lyle_s

Please have a look at http://www.gnu.org/software/parallel/

Lyle.

It answers some questions?

propofol · 11-26-2012, 12:38 AM

I am not sure I follow exactly what you want to do. Is xargs and option?

Code:

seq 1 10 | xargs -P 4 -n 1 -I{} echo "Testing {}"

Regards,
Stefan