[SOLVED] Not Sure how to use parallel command with awk

linux2021 · 08-02-2020, 04:58 AM

Greetings All,

Hi, looking for a way to speed up the processing of this code.

Code:

while read line; do

read a b c d  <<< "$line"
 awk -v v1=$a -v v2=$b -v v3=$c -v v4=$d '$0~v1 && $0~v2 && $0~v3 && $0~v4 {print v1, v2, v3, v4}' gamenums.txt 

done <  patterns.txt

The script works fine but is very slow processing many lines. Can parallel or other code speed up the process?

I appreciate any suggestions. Thanks

shruggy · 08-02-2020, 05:21 AM

Is it the GNU parallel or the parallel from moreutils? Because they have different syntax.

But in both cases you have to specify the list of input files parallel will be run against. If you only have one file gamenums.txt I don't see how you can use parallel.

You should think of parallel as of xargs that runs the specified command against each file in the list, but different from xargs does it, well, in parallel.

Instead of different input files those could be different sets of arguments to the command, too.

syg00 · 08-02-2020, 05:33 AM

Every time I look at GNU parallel I get a headache. The authors think it's a cinch, but I find it arcane to wrestle into shape.

linux2021 · 08-02-2020, 05:45 AM

Quote:

Originally Posted by shruggy

Is it the GNU parallel or the parallel from moreutils? Because they have different syntax.

But in both cases you have to specify the list of input files parallel will be run against. If you only have one file gamenums.txt I don't see how you can use parallel.

You should think of parallel as of xargs that runs the specified command against each file in the list, but different from xargs does it, well, in parallel.

Instead of different input files those could be different sets of arguments to the command, too.

It is GNU parallel from ubuntu

Quote:

Originally Posted by syg00

Every time I look at GNU parallel I get a headache. The authors think it's a cinch, but I find it arcane to wrestle into shape.

I agree. It's very complicated to understand unless you're a kernel programmer.

linux2021 · 08-02-2020, 06:35 AM

I updated my question. If anyone knows any bash tricks in speeding up while loops or reading from files, I would like your suggestions to try. Thanks

shruggy · 08-02-2020, 07:06 AM

I'd think on something like

Code:

parallel -C' ' -q awk -v v1={1} -v v2={2} -v v3={3} -v v4={4} '$0~v1 && $0~v2 && $0~v3 && $0~v4 {print v1, v2, v3, v4}' gamenums.txt <patterns.txt

linux2021 · 08-02-2020, 07:45 AM

Quote:

Originally Posted by shruggy

I'd think on something like

Code:

parallel -C' ' -q awk -v v1={1} -v v2={2} -v v3={3} -v v4={4} '$0~v1 && $0~v2 && $0~v3 && $0~v4 {print v1, v2, v3, v4}' gamenums.txt <patterns.txt

Thanks for this suggestion. I am a bit confused tho. I already have a redirect from patterns.txt file in the done line.

If I paste your code, I will have two redirects of patterns.txt file. Which one do I omit?

The one in your code or the one in the done line. Thanks

shruggy · 08-02-2020, 08:07 AM

No, you don't need any loops. This is a one-liner that completely replaces your script.

linux2021 · 08-02-2020, 08:16 AM

Quote:

Originally Posted by shruggy

No, you don't need any loops. This is a one-liner that completely replaces your script.

Thanks, it's much better

smallpond · 08-13-2020, 09:23 AM

For each line of patterns.txt you are splitting the line and starting a new awk and reading all of gamenums.txt, looking for a similar line. Better would be to write a small program (for example in python) that reads the smaller of the two files into memory, sorts it, and then runs through the other one line-by-line to search for the matches with something better than linear time. This converts your problem from O(n^2) to O(n log n).