correct usage of grep to search a disk

Completely Clueless · 03-26-2009, 10:17 AM

Hi guys,

I've been trying to identify all files on my cut-down version of Damn Small which contain the text string "User Agent:" in them. Because it's only 120Mb in its entirety, I'm quite happy to have grep search the whole system. I'm using this command, but it just generates errors as you can see:

[root@localhost ~]# grep -R 'User Agent:' /*

grep: /dev/dri/card0: Invalid argument
grep: /dev/fuse: Operation not permitted
grep: /dev/log: No such device or address
grep: /dev/bsg/3:0:0:0: Invalid argument
grep: /dev/usbdev2.5_ep00: No such device or address
grep: /dev/usbdev2.5_ep05: No such device or address
grep: /dev/usbdev2.5_ep85: No such device or address

So what am I doing wrong?

Thanks,

CC.

mjp053000 · 03-26-2009, 10:24 AM

Hi,

Try the following commands
find . -name yourFileName
or
find . -exec grep -ls "yourFileName" {} \;

good luck

Completely Clueless · 03-26-2009, 10:57 AM

Quote:

Originally Posted by mjp053000

Hi,

Try the following commands
find . -name yourFileName
or
find . -exec grep -ls "yourFileName" {} \;

good luck

Thanks for trying, but I don't see how this can work. I don't know what the filenames are! I'm simply trying to locate all files on the system which contain the string "User Agent:" and the command I need will then giv me the relevant filenames.

pwc101 · 03-26-2009, 11:08 AM

Code:

find / -type f -exec grep "User Agent:" {} +

Will find all regular files (-type f) in the root directory (/) and pass those results ({}) to grep (-exec grep) with a search string of "User Agent:". The output should include the name of the matching file too.

Completely Clueless · 03-26-2009, 11:18 AM

Quote:

Originally Posted by pwc101

Code:

find / -type f -exec grep "User Agent:" {} +

Will find all regular files (-type f) in the root directory (/) and pass those results ({}) to grep (-exec grep) with a search string of "User Agent:". The output should include the name of the matching file too.

Thanks, it's working now. So this {} is effectively the same as the single vertical line we call a pipe? I can't find the right key on this keyboard right now; it's an unfamiliar computer I'm using.

Mogget · 03-26-2009, 11:22 AM

If I'm looking for a file or folder on the disk where i only know parts of the name i will go to the folder where i know the file I'm looking for is in or under. If you have no clue you go to /

Here's the exact way i do it

Code:

cd /
find . print | grep "the string"

Depending on the size of your drive and how many files there are this could take a while.
Anyway this would result in showing any files that has "the string" in it

/home/user/backup.the string.cpio
/usr/local/bin/the string.sh

pwc101 · 03-26-2009, 11:27 AM

Quote:

Originally Posted by Completely Clueless

Thanks, it's working now. So this {} is effectively the same as the single vertical line we call a pipe?

In a way, yes. You're sending the filenames from the find command to the grep command. find in conjunction with xargs might give you a more familiar syntax:

Code:

find / -type f -print0 | xargs -0 grep "User Agent:"

That should output the same information as the original find command I posted.

pwc101 · 03-26-2009, 11:28 AM

Quote:

Originally Posted by Mogget

If I'm looking for a file or folder on the disk where i only know parts of the name i will go to the folder where i know the file I'm looking for is in or under. If you have no clue you go to /

Here's the exact way i do it

Code:

cd /
find . print | grep "the string"

Depending on the size of your drive and how many files there are this could take a while.
Anyway this would result in showing any files that has "the string" in it

/home/user/backup.the string.cpio
/usr/local/bin/the string.sh

This will only output files that have "the string" in their name, not in their contents.

anonguy9 · 03-26-2009, 11:30 AM

This was tested and works:

Code:

grep -ri 'background' /etc/X11/xdm

It's odd if this wouldn't work:

Code:

grep -r 'User Agent' /

BUT, those messages you reported:

Code:

grep: /dev/dri/card0: Invalid argument
grep: /dev/fuse: Operation not permitted
grep: /dev/log: No such device or address
grep: /dev/bsg/3:0:0:0: Invalid argument
grep: /dev/usbdev2.5_ep00: No such device or address
grep: /dev/usbdev2.5_ep05: No such device or address
grep: /dev/usbdev2.5_ep85: No such device or address

Those are all normal and expected messages!

Just think about it. /dev has all kinds of special stuff in it. Grep is trying to treat files in /dev like text files when they really aren't.

You could string together 'find' to make your script smarter. I took some time to research this because it's an itch I had to scratch. I think this will work for you:

Code:

find / -path /dev -prune -o -type f -name '*' -print0 | xargs -r0 grep -Fi 'User Agent'

-type f for files-only
-some funky stuff I don't understand to omit /dev =)

For others that find this. Finding contents but omitting multiple directories:

Code:

 find / \
    -path '/dev' -prune \
 -o -path '/mnt' -prune \
 -o -type f -name '*' -print0 | xargs -r0 grep -Fi 'contents'

Notice how this uses -o multiple times.

Also, some examples have -wholename. That's the same as -prune from what I can see. Also, I'm told that the item after -path (like '/dev') can be a simple regular expression, so some pretty heavy stuff becomes possible.

Completely Clueless · 03-26-2009, 12:30 PM

[quote=anonguy9;3488512]

You could string together 'find' to make your script smarter. I took some time to research this because it's an itch I had to scratch. I think this will work for you:

Code:

find / -path /dev -prune -o -type f -name '*' -print0 | xargs -r0 grep -Fi 'User Agent'

Well, I've had a good few suggestions and have started up several shells to test them all. It's rather time consuming running these, but the one above is the current front-runner. It shows up just about every instance of the string including places where I wouldn't have expected to see it. Some newbie you are, anonguy9! I've given you a thanks after only your 5th posting so you have a satisfaction rating of 20%. Remarkable! I predict a stellar future for you here on LQ. :-)

anonguy9 · 03-26-2009, 06:15 PM

Thanks, after years of frustration I've switched my research style to make every problem into a reproducible test case and build documentation explaining a proper solution. This happens to make every post thorough. I'm so sick of crappy Linux documentation and terrible explanations with no common-use examples.

Do check back in and let us know how things went, especially if you find the perfect commandline search tool or string of commands.

Btw, when you said "It shows up just about every instance of the string", this makes me wonder.. are there some cases where you're expecting but not seeing the string?

----

As an aside, I implemented some rough helper procedures in my ~/.bashrc and included this new find-contents snippet:

Code:

findfile() { find -type f -name \*"$1"\*|grep --colour=always -i "$1" ; }
finddir()  { find -type d -name \*"$1"\*|grep --colour=always -i "$1" ; }
findin() { find ./ -type f -name '*' -print0 | xargs -r0 grep --colour=always -Fi "$1" ; }

It seems to work ok so far.

I added colouring with grep.

Added for thoroughness: If grep --colour=always doesn't work for you, then try this:

Code:

export GREP_COLOR='1;32'

For the other colour possibilities, here's a table:

http://www.linux.com/howtos/Bash-Pro...WTO/c327.shtml

Code:

Black       0;30     Dark Gray     1;30
Blue        0;34     Light Blue    1;34
Green       0;32     Light Green   1;32
Cyan        0;36     Light Cyan    1;36
Red         0;31     Light Red     1;31
Purple      0;35     Light Purple  1;35
Brown       0;33     Yellow        1;33
Light Gray  0;37     White         1;37

I think the next step would be to have the output also tell you what line the text was found on. That would be sweet.

syg00 · 03-26-2009, 06:27 PM

You might want to prune /proc as well.
Or do it from a liveCD or such, and avoid the pseudo filesystem issue altogether.

Completely Clueless · 03-26-2009, 07:53 PM

Quote:

Originally Posted by anonguy9

Thanks, after years of frustration I've switched my research style to make every problem into a reproducible test case and build documentation explaining a proper solution. This happens to make every post thorough. I'm so sick of crappy Linux documentation and terrible explanations with no common-use examples.

Do check back in and let us know how things went, especially if you find the perfect commandline search tool or string of commands.

Btw, when you said "It shows up just about every instance of the string", this makes me wonder.. are there some cases where you're expecting but not seeing the string?

I think the next step would be to have the output also tell you what line the text was found on. That would be sweet.

You are SO right about the standard of explanations in computer documentation in general, but many of the man pages in particular.

Well, after all the suggested commands have now been run, yours was the ONLY one that delivered the goods and exited without error. I believe there IS a flag you can add to grep to show the corresponding line number but can't recall what it is off hand.

I found other instances of the string in various Firefox cache files from searches I'd made. This is a real nuisance security issue as far as I'm concerned. Few things give me a bug up my ass like browers leaving data behind on a hard drive.

I'll try out your latest suggestions some other time, but for now, the job is done and thanks again for your efforts.

CC.