LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Closed Thread
  Search this Thread
Old 09-05-2019, 06:43 AM   #1
wikigeek
LQ Newbie
 
Registered: Aug 2019
Posts: 6

Rep: Reputation: Disabled
How can I execute a shell script with parallel command or other way to speed up processing


...

Last edited by wikigeek; 09-05-2019 at 11:33 PM.
 
Old 09-05-2019, 06:59 AM   #2
dc.901
Member
 
Registered: Aug 2018
Location: Atlanta, GA - USA
Distribution: CentOS 6-7; SuSE 8-12
Posts: 599

Rep: Reputation: 173Reputation: 173
Please forgive me; it's early and perhaps my tea hasn't kicked in yet.

Quote:
Originally Posted by wikigeek View Post
Code:
#!/bin/bash


while read line; do
         read a b c <<< $line
         echo "Checking for patterns...$a, $b, and $c"
         awk -v b1="$a" -v b2="$b" -v b3="$c" '$0~b1 && $0~b2 && $0~b3' win
         match=$(awk -v b1="$a" -v b2="$b" -v b3="$c" '$0~b1 && $0~b2 && $0~b3' win | wc -l)
         echo "Found $match games(s) with patterns $a, $b, and $c"
         done < mypatterns
Is that running awk twice? Second time to get the count?
Maybe, redirect output to a file first time then do wc -l <file>?
 
2 members found this post helpful.
Old 09-05-2019, 07:11 AM   #3
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: FreeBSD/Slackware-14.2+/ArcoLinux
Posts: 9,078

Rep: Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903
Code:
while read line; do
         read a b c <<< $line
         echo "Checking for patterns...$a, $b, and $c"
         awk -v b1="$a" -v b2="$b" -v b3="$c" '$0~b1 && $0~b2 && $0~b3' win
         match=$(awk -v b1="$a" -v b2="$b" -v b3="$c" '$0~b1 && $0~b2 && $0~b3' win | wc -l)
         echo "Found $match games(s) with patterns $a, $b, and $c"
         done < mypatterns
could be
Code:
while read a b c
echo "checking patterns now...
I do not know awk that well to know what that line is doing exactly. 
but you're repeating the reading in of data. you can split it on the first read.
done < mypatterns
example mypatterns

Code:
$ cat >> mypatterns
dobe bob sally
silly joey bakeoff

$ while read a b c ; do   echo "$a"; echo "$b"; echo "$c"; done < mypatterns
dobe
bob
sally
silly
joey
bakeoff
removing one read of the same data will cut some time off.

Last edited by BW-userx; 09-05-2019 at 07:16 AM.
 
2 members found this post helpful.
Old 09-05-2019, 07:23 AM   #4
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 1,991

Rep: Reputation: 549Reputation: 549Reputation: 549Reputation: 549Reputation: 549Reputation: 549
I'll have to work out what your awk is doing ( the game rules )
and then come up with something else

awk is probably a waste here, since you are only comparing strings


and yeah, running it twice just to get the count is also a waste
so you can half the time right there
 
1 members found this post helpful.
Old 09-05-2019, 07:57 AM   #5
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 1,991

Rep: Reputation: 549Reputation: 549Reputation: 549Reputation: 549Reputation: 549Reputation: 549
what is in win?

just 3 random numbers?
or say like 30 random numbers 3 per line

all you are doing is seeing if 3 random numbers match in any order as fast a possible


So you should probably do it the other way round
i.e read the wins and check if they are in the big list
but you put that fist one in the background and go on to the next
and so on
after they have all finished you can then total the results

you may want to keep count of the ones you are putting in background and limit them idk so only four are running at anyone time

This is homework yes?
 
1 members found this post helpful.
Old 09-05-2019, 08:39 AM   #6
wikigeek
LQ Newbie
 
Registered: Aug 2019
Posts: 6

Original Poster
Rep: Reputation: Disabled
...

Last edited by wikigeek; 09-05-2019 at 11:34 PM.
 
Old 09-05-2019, 08:55 AM   #7
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: FreeBSD/Slackware-14.2+/ArcoLinux
Posts: 9,078

Rep: Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903
create a bunch of randoms in sets of three per line
Code:
 for i in {1..100} ; do  shuf -i 1-1000 -n 3 | tr '\n' " " >> randoms ; c=$((c>=3?0:c++)) ;  echo >> randoms ;  done
##just make a bigger list to compar againt
 for i in {1..200} ; do  shuf -i 1-10000 -n 3 | tr '\n' " " >> ckrandoms ; c=$((c>=3?0:c++)) ;  echo >> ckrandoms ;  done
you can run them line of commands in the terminal then check the output files to see what the samples looks like.


Code:
#!/usr/bin/env bash
#set -xv 

Acount=0
Bcount=0
Ccount=0

while read a b c 
do
	while read A
	do
		[[ "$A" =~ "$a" ]] && ((Acount++))
	done < ckrandoms #win
	
	while read B 
	do
		[[ $B =~ $b ]] && ((Bcount++))
	done < ckrandoms #win
	
	while read C
	do
		[[ $C =~ $c ]] && ((Ccount++))
	done < ckrandoms #win
	
done < randoms

echo "
A=$Acount
B=$Bcount
C=$Ccount"
results
Code:
[userx@FreeBSD64ssd test]$ ./complaire

A=342
B=267
C=381
something like that perhaps using sub-string checking.

control loop gets the 3 in separate vars. a b c
then using the other file it reads in and checks it by looking at the line of win (file) to check to see if it is within that entire line, if yes, then add count of 1.

36 =~ 32 24 36 = yes add to count

315,000 lines to generate ... its still generating them... it takes time..
Code:
[userx@FreeBSD64ssd test]$ for i in {1..315000} ; do  shuf -i 1-1000 -n 3 | tr '\n' " " >> randoms ; c=$((c>=3?0:c++)) ;  echo >> randoms ;  done

Last edited by BW-userx; 09-05-2019 at 09:18 AM.
 
1 members found this post helpful.
Old 09-05-2019, 09:01 AM   #8
wikigeek
LQ Newbie
 
Registered: Aug 2019
Posts: 6

Original Poster
Rep: Reputation: Disabled
...

Last edited by wikigeek; 09-05-2019 at 11:35 PM.
 
Old 09-05-2019, 09:07 AM   #9
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 1,991

Rep: Reputation: 549Reputation: 549Reputation: 549Reputation: 549Reputation: 549Reputation: 549
yeah, I figured out what your awk was doing
so, your random numbers are min-max 00..99

I was lazy and just generated
like
Code:
for i in {1..4};do echo ${RANDOM} ${RANDOM} ${RANDOM};done 
30291 21720 21445
20208 30131 26014
7642 821 4056
17302 23788 17210
what was missing is the win file

and it still is
I see 2 sets of 5

how many lines are in your win file?
 
1 members found this post helpful.
Old 09-05-2019, 09:24 AM   #10
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: FreeBSD/Slackware-14.2+/ArcoLinux
Posts: 9,078

Rep: Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903
Quote:
Originally Posted by wikigeek View Post
@ BW-userx

Looks complicated. Will this speed up the processing?

I'll give it a try.
not sure,it is still reading that one file 3 times, as I am generating larger files, and thinking up a different algorithm while waiting.

the three inner loops are just reading in the "win" file as I do not even know what the win files data looks like, each inner loop is using one of the numbers from the main file to check each line of "win" to see if that number is within the line before checking the next line in the "win" file.

I do suppose one can consolidate the conditional and read in "win" only once.

Code:
#!/usr/bin/env bash
#set -xv 

Acount=0
Bcount=0
Ccount=0

while read a b c 
do
	while read win
	do
		[[ "$win" =~ "$a" ]] && ((Acount++))
		[[ "$win" =~ "$b" ]] && ((Bcount++))
		[[ "$win" =~ "$c" ]] && ((Ccount++))
		
	done < ckrandoms
	
	
done < randoms

echo "
A=$Acount
B=$Bcount
C=$Ccount"
even awk is only reading it one line at a time.

Last edited by BW-userx; 09-05-2019 at 09:33 AM.
 
1 members found this post helpful.
Old 09-12-2019, 04:02 AM   #11
bigearsbilly
Senior Member
 
Registered: Mar 2004
Location: england
Distribution: Slack, Debian, Mint, Puppy, Raspbian
Posts: 3,462

Rep: Reputation: 218Reputation: 218Reputation: 218
Take a look at this, saved hours in my working day

https://www.gnu.org/software/parallel/
 
1 members found this post helpful.
Old 09-12-2019, 04:37 AM   #12
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 18,078

Rep: Reputation: 2900Reputation: 2900Reputation: 2900Reputation: 2900Reputation: 2900Reputation: 2900Reputation: 2900Reputation: 2900Reputation: 2900Reputation: 2900Reputation: 2900
So the OP has removed all posts. Time op was added to ignore list methinks.
 
1 members found this post helpful.
Old 09-12-2019, 04:46 AM   #13
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 3,844

Rep: Reputation: 1320Reputation: 1320Reputation: 1320Reputation: 1320Reputation: 1320Reputation: 1320Reputation: 1320Reputation: 1320Reputation: 1320Reputation: 1320
Mayhaps this topic should be deleted then. (Also the Original Poster should be hugged.)
 
1 members found this post helpful.
Old 09-12-2019, 07:52 AM   #14
Firerat
Senior Member
 
Registered: Oct 2008
Distribution: Debian sid
Posts: 1,991

Rep: Reputation: 549Reputation: 549Reputation: 549Reputation: 549Reputation: 549Reputation: 549
they maybe smart,
covering their tracks so teacher doesn't catch them cheating at homework.
 
1 members found this post helpful.
Old 09-12-2019, 07:57 AM   #15
BW-userx
LQ Guru
 
Registered: Sep 2013
Location: Somewhere in my head.
Distribution: FreeBSD/Slackware-14.2+/ArcoLinux
Posts: 9,078

Rep: Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903Reputation: 1903
that border lines on paranoia.
think about it...
 
1 members found this post helpful.
  


Closed Thread


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Shell script with Menu and sub menu and in sub menu execute another shell script SHWE Linux - Newbie 9 11-03-2018 06:19 PM
LXer: Nvidia and ARM: It's a parallel, parallel, parallel world LXer Syndicated Linux News 0 03-21-2013 06:10 PM
[SOLVED] Executing a command in parallel | GNU parallel or xargs the_gripmaster AIX 3 05-08-2012 07:41 AM
help with execute mulitple shell script within shell script ufmale Programming 6 09-13-2008 12:21 AM
How to execute two shell scripts in parallel Pavitra.v Linux - General 4 09-16-2006 05:59 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 06:11 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration