How can I execute a shell script with parallel command or other way to speed up processing

dc.901 · 09-05-2019, 06:59 AM

Please forgive me; it's early and perhaps my tea hasn't kicked in yet.

Quote:

Originally Posted by wikigeek

Code:

#!/bin/bash


while read line; do
         read a b c <<< $line
         echo "Checking for patterns...$a, $b, and $c"
         awk -v b1="$a" -v b2="$b" -v b3="$c" '$0~b1 && $0~b2 && $0~b3' win
         match=$(awk -v b1="$a" -v b2="$b" -v b3="$c" '$0~b1 && $0~b2 && $0~b3' win | wc -l)
         echo "Found $match games(s) with patterns $a, $b, and $c"
         done < mypatterns

Is that running awk twice? Second time to get the count?
Maybe, redirect output to a file first time then do wc -l <file>?

BW-userx · 09-05-2019, 07:11 AM

Code:

while read line; do
         read a b c <<< $line
         echo "Checking for patterns...$a, $b, and $c"
         awk -v b1="$a" -v b2="$b" -v b3="$c" '$0~b1 && $0~b2 && $0~b3' win
         match=$(awk -v b1="$a" -v b2="$b" -v b3="$c" '$0~b1 && $0~b2 && $0~b3' win | wc -l)
         echo "Found $match games(s) with patterns $a, $b, and $c"
         done < mypatterns

could be

Code:

while read a b c
echo "checking patterns now...
I do not know awk that well to know what that line is doing exactly. 
but you're repeating the reading in of data. you can split it on the first read.
done < mypatterns

example mypatterns

Code:

$ cat >> mypatterns
dobe bob sally
silly joey bakeoff

$ while read a b c ; do   echo "$a"; echo "$b"; echo "$c"; done < mypatterns
dobe
bob
sally
silly
joey
bakeoff

removing one read of the same data will cut some time off.

Firerat · 09-05-2019, 07:23 AM

I'll have to work out what your awk is doing ( the game rules )
and then come up with something else

awk is probably a waste here, since you are only comparing strings

and yeah, running it twice just to get the count is also a waste
so you can half the time right there

Firerat · 09-05-2019, 07:57 AM

what is in win?

just 3 random numbers?
or say like 30 random numbers 3 per line

all you are doing is seeing if 3 random numbers match in any order as fast a possible

So you should probably do it the other way round
i.e read the wins and check if they are in the big list
but you put that fist one in the background and go on to the next
and so on
after they have all finished you can then total the results

you may want to keep count of the ones you are putting in background and limit them idk so only four are running at anyone time

This is homework yes?

wikigeek · 09-05-2019, 08:39 AM

BW-userx · 09-05-2019, 08:55 AM

create a bunch of randoms in sets of three per line

Code:

 for i in {1..100} ; do  shuf -i 1-1000 -n 3 | tr '\n' " " >> randoms ; c=$((c>=3?0:c++)) ;  echo >> randoms ;  done
##just make a bigger list to compar againt
 for i in {1..200} ; do  shuf -i 1-10000 -n 3 | tr '\n' " " >> ckrandoms ; c=$((c>=3?0:c++)) ;  echo >> ckrandoms ;  done

you can run them line of commands in the terminal then check the output files to see what the samples looks like.

Code:

#!/usr/bin/env bash
#set -xv 

Acount=0
Bcount=0
Ccount=0

while read a b c 
do
	while read A
	do
		[[ "$A" =~ "$a" ]] && ((Acount++))
	done < ckrandoms #win
	
	while read B 
	do
		[[ $B =~ $b ]] && ((Bcount++))
	done < ckrandoms #win
	
	while read C
	do
		[[ $C =~ $c ]] && ((Ccount++))
	done < ckrandoms #win
	
done < randoms

echo "
A=$Acount
B=$Bcount
C=$Ccount"

results

Code:

[userx@FreeBSD64ssd test]$ ./complaire

A=342
B=267
C=381

something like that perhaps using sub-string checking.

control loop gets the 3 in separate vars. a b c
then using the other file it reads in and checks it by looking at the line of win (file) to check to see if it is within that entire line, if yes, then add count of 1.

36 =~ 32 24 36 = yes add to count

315,000 lines to generate ... its still generating them... it takes time..

Code:

[userx@FreeBSD64ssd test]$ for i in {1..315000} ; do  shuf -i 1-1000 -n 3 | tr '\n' " " >> randoms ; c=$((c>=3?0:c++)) ;  echo >> randoms ;  done

wikigeek · 09-05-2019, 09:01 AM

Firerat · 09-05-2019, 09:07 AM

yeah, I figured out what your awk was doing
so, your random numbers are min-max 00..99

I was lazy and just generated
like

Code:

for i in {1..4};do echo ${RANDOM} ${RANDOM} ${RANDOM};done 
30291 21720 21445
20208 30131 26014
7642 821 4056
17302 23788 17210

what was missing is the win file

and it still is
I see 2 sets of 5

how many lines are in your win file?

BW-userx · 09-05-2019, 09:24 AM

Quote:

Originally Posted by wikigeek

@ BW-userx

Looks complicated. Will this speed up the processing?

I'll give it a try.

not sure,it is still reading that one file 3 times, as I am generating larger files, and thinking up a different algorithm while waiting.

the three inner loops are just reading in the "win" file as I do not even know what the win files data looks like, each inner loop is using one of the numbers from the main file to check each line of "win" to see if that number is within the line before checking the next line in the "win" file.

I do suppose one can consolidate the conditional and read in "win" only once.

Code:

#!/usr/bin/env bash
#set -xv 

Acount=0
Bcount=0
Ccount=0

while read a b c 
do
	while read win
	do
		[[ "$win" =~ "$a" ]] && ((Acount++))
		[[ "$win" =~ "$b" ]] && ((Bcount++))
		[[ "$win" =~ "$c" ]] && ((Ccount++))
		
	done < ckrandoms
	
	
done < randoms

echo "
A=$Acount
B=$Bcount
C=$Ccount"

even awk is only reading it one line at a time.

bigearsbilly · 09-12-2019, 04:02 AM

Take a look at this, saved hours in my working day

https://www.gnu.org/software/parallel/

syg00 · 09-12-2019, 04:37 AM

So the OP has removed all posts. Time op was added to ignore list methinks.

NevemTeve · 09-12-2019, 04:46 AM

Mayhaps this topic should be deleted then. (Also the Original Poster should be hugged.)

Firerat · 09-12-2019, 07:52 AM

they maybe smart,
covering their tracks so teacher doesn't catch them cheating at homework.

BW-userx · 09-12-2019, 07:57 AM

that border lines on paranoia.
think about it...