recursive grep speed

allasso · 06-05-2010, 08:40 AM

Hello,

When I grep recursively using:

grep -r '' big/hairy/file_hierarchy/ | grep 'expression'

it is about 10 times faster (no exagerration) than using:

grep -r 'expression' big/hairy/file_hierarchy/

Anyone know why?

pixellany · 06-05-2010, 09:08 AM

I get the opposite result:

Code:

[root@Ath etc]# time grep -r '' .|grep comment
<<output snipped>>
real    0m0.161s
user    0m0.053s
sys     0m0.040s
[root@Ath etc]# time grep -r '' .|grep comment > /dev/null

real    0m0.108s
user    0m0.043s
sys     0m0.047s
[root@Ath etc]# time grep -r comment . > /dev/null

real    0m0.044s
user    0m0.013s
sys     0m0.020s
[root@Ath etc]#

This is what I would expect. (The difference between the first 2 cases is presumably the time to write to the terminal.)

unSpawn · 06-05-2010, 10:10 AM

Also note the very first op that returns a set of items like 'find' or 'grep -r' also serves to fill kernel caches meaning subsequent similar ops on the same set will be faster so you always have to run timing tests a few times. And most of the time using 'cat|less', 'cat|grep', 'ps|grep' (pgrep) or 'ls|grep' ('grep' or 'find') results in inefficiency, rarely it's used for compatibility reasons: one grep good, two greps baaa-aaahhhd!

pixellany · 06-05-2010, 10:15 AM

hmmmm---caching had occured to me. How can you clear the cache to get a valid test?

unSpawn · 06-05-2010, 10:54 AM

http://www.linuxquestions.org/questi...2/#post3084228 (/proc/sys/vm/drop_caches) but IIGC that sysctl only frees memory from clean caches, it does not write out dirty objects. I don't know what else there is to free cached objects apart from 'sync'.

pixellany · 06-05-2010, 11:30 AM

Code:

[root@Ath log]# time grep -r '' . | grep lizard

real    0m1.630s
user    0m0.040s
sys     0m0.057s


[root@Ath log]# time grep -r lizard .

real    0m0.118s
user    0m0.027s
sys     0m0.020s


[root@Ath log]# time grep -r lizard .

real    0m0.054s
user    0m0.030s
sys     0m0.013s


[root@Ath log]# time grep -r '' . | grep lizard

real    0m0.087s
user    0m0.033s
sys     0m0.043s


[root@Ath log]# time grep -r lizard .

real    0m0.054s
user    0m0.017s
sys     0m0.027s


[root@Ath log]# time grep -r '' . | grep lizard

real    0m0.086s
user    0m0.043s
sys     0m0.030s
[root@Ath log]#

Yes--caching is obviously working....How to explain all the details?--I'm not sure.

unSpawn · 06-06-2010, 08:55 AM

If you run 'sync; slabtop -d1 -sa; sync' and perform a system-wide 'find' you should see the amount of objects in caches growing quickly.

allasso · 06-08-2010, 12:14 PM

wow, you guys are talking pretty far over my head.

The results I get are grepping 5 GB hierarchies with about 15000 files. I am looking at times like 72s versus 7s (timing with my watch), rather than msec ranges.

I started observing this about 2 years ago and it has been absolutely consistent, so much so that I just got in the habit of doing it that way for big trees. Every once in a while I check to see if it is still true, and it is (on my computer anyway.)

I use a mac, but doesn't seem like that would make any difference.

allasso · 06-08-2010, 12:19 PM

--error--

Revery · 06-09-2010, 04:43 PM

First off, shame on you guys for trying to derive performance measurements out of tiny greps that don't even take a second. :-)
I'm not sure about the reason behind that time different using those two different syntaxes, but I can offer you the advice of always using the -I flag to ignore binary files. This drastically cuts down the time that my recursive greps take.

pixellany · 06-09-2010, 07:22 PM

Excuse me?? something approaching 1/10th of a second is a VERY LONG TIME for a modern computer.

I wonder if bypassing the binary files will reduce the **difference** between the 2 methods.