awk speed optimisation

ghostdog74 · 11-20-2009, 04:34 AM

Quote:

Originally Posted by H_TeXMeX_H

From my tests it's about 4x faster than awk,

hmm, because your program is smaller than (m)awk??

H_TeXMeX_H · 11-20-2009, 04:50 AM

Ok, I got mawk and here's my test results:

Code:

bash-3.1$ time mawk 'NR >= 300 && NR <= 1500' out > tt

real	0m0.003s
user	0m0.001s
sys	0m0.003s
bash-3.1$ time mm 300 1500 out > ttttt

real	0m0.001s
user	0m0.000s
sys	0m0.001s
bash-3.1$ time awk 'NR >= 300 && NR <= 1500' out > tt

real	0m0.004s
user	0m0.002s
sys	0m0.001s
bash-3.1$ time sed -n '300,1500 p' out > tttt

real	0m0.003s
user	0m0.002s
sys	0m0.001s

ghostdog74 · 11-20-2009, 05:02 AM

Quote:

Originally Posted by H_TeXMeX_H

Ok, I got mawk and here's my test results:

Code:

bash-3.1$ time awk 'NR >= 300 && NR <= 1500' out > tt

change the awk code to

Code:

awk 'NR>=300{print}NR==1500{exit}' out

Same as sed.

Code:

sed -n '300,1500{p;q}' out

i think you will see big difference.

H_TeXMeX_H · 11-20-2009, 05:20 AM

Well, you're right about sed, it's just as fast as the C program:

Code:

bash-3.1$ time sed -n '300,1500{p;q}' out > ttt

real	0m0.001s
user	0m0.001s
sys	0m0.000s
bash-3.1$ time sed -n '300,1500{p;q}' out > ttt

real	0m0.001s
user	0m0.000s
sys	0m0.001s

But awk and mawk still don't do any better:

Code:

bash-3.1$ time awk 'NR>=300{print}NR==1500{exit}' out > tt

real	0m0.004s
user	0m0.001s
sys	0m0.003s
bash-3.1$ time mawk 'NR>=300{print}NR==1500{exit}' out > tt

real	0m0.003s
user	0m0.000s
sys	0m0.003s

Strange ...

Well, at least you can use sed and don't need the program I posted.

Rusty2727 · 11-20-2009, 11:37 AM

Very constructive comments!

In the end I also found that 'sed' was the simplest and fastest way to go.

I actually also used 'tail' to remove data from the input as and when it wasn't needed anymore.

This allowed the input to shrink as the while loop progressed making it progressively quicker with each iteration.

Still a bit slow though for my requirement.

Many thanks everybody for all your input.

Have a good weekend!!!

H_TeXMeX_H · 11-20-2009, 01:21 PM

In that case I would try to put it all in a C program if you can.

cyent · 11-22-2009, 03:22 PM

Quote:

Originally Posted by H_TeXMeX_H

oops, sorry, I was tired, here's the correct one

Ok, that's better!

Quote:

run 'strip --strip-unneeded' on it)

Does that really make any difference to the speed? I thought it just stripped out things like debug symbols. ie. Took up less disk space, but didn't make it faster.

H_TeXMeX_H · 11-23-2009, 03:37 AM

Quote:

Originally Posted by cyent

Ok, that's better!

Does that really make any difference to the speed? I thought it just stripped out things like debug symbols. ie. Took up less disk space, but didn't make it faster.

well, maybe just a bit faster, because it takes less time to load it into RAM and it can skip over the unneeded symbols ... I'm not sure, but it does at least save space.