[SOLVED] Help: Searching multiple files and returning multiple lines matching querry

thealmightyos · 09-12-2014, 01:07 AM

So, I consulted the googles, and what I found did not help my situation. I am confused and not sure what angle to tackle this from first. awk, sed, grep.... Wat do?

I have 111 directories and 768 files that I need to search though and find which files I have to change and what I need to change in them and hopefully I can use this information to create a script to do so. But I don't know how to find those files and print the information from them I need. I need a command/string of commands/script file/ program to:

1: Find first file that contains the string "foo"
2: Print out the filename
3: Print out the line that contains "foo" (grep, basically)
4: Print out the line that contains "foo2"
5: same as 4 only with foo3-9
6: find the next file that contains the string "foo"
Goto 2

Any suggestions?

j-ray · 09-12-2014, 02:55 AM

grep -n foo /ParentDirectory/* > logfile
grep -n foo /ParentDirectory/*/* >> logfile
grep -n foo2 /ParentDirectory/* >> logfile
grep -n foo2 /ParentDirectory/*/* >> logfile

Am I getting you wrong?

rhoekstra · 09-12-2014, 03:26 AM

So only when a file contains 'foo' it needs to be processed for foo2-9 ? Files without foo don't need processing for foo2 ?

Code:

for i in `grep -rl foo /ParentDirectory/`; do # Process 'parentdirectory' and all its subdirectories
  echo $i >>log.txt #print the filenumber
  grep -n foo >>log.txt #print linenumbers with foo
  grep -n foo2-9 >>log.txt #print linenumbers with foo2-9
done

thealmightyos · 09-12-2014, 06:41 PM

Quote:

Originally Posted by rhoekstra

So only when a file contains 'foo' it needs to be processed for foo2-9 ? Files without foo don't need processing for foo2 ?

Code:

for i in `grep -rl foo /ParentDirectory/`; do # Process 'parentdirectory' and all its subdirectories
  echo $i >>log.txt #print the filenumber
  grep -n foo >>log.txt #print linenumbers with foo
  grep -n foo2-9 >>log.txt #print linenumbers with foo2-9
done

yes. sort of. maybe. Oh hell with it, DO OVER

God I write things up horribly when I am tired. I apologize for that. Sleeping meds + linux shell = OS gets sloppy

Back to the topic at hand. I am abandoning the first post example. can not be saved. Starting over.
I can ALMOST get what I want with

Code:

grep -r 'foo\|foo2\|foo3\|foo4\|foo5\|foo6' ./
#there are more "foo's" in there but you get the point

Problem is, now that I found the file, got all the config info I need one more thing. Not even going to use foo for this one: "name =". It is so flipping generic that it prints multiple lines from EVERY SINGLE FILE. So, this is what I need:

For files that match foo, I want to print foo#'s AND the first instance of "name ="

There, see what being awake can do?

thealmightyos · 09-14-2014, 02:27 AM

an even BETTER solution to this is if I could print all lines between and including foo1 to foo2 and then the first instance of "name =". Would use the grep function to print n number of lines after finding a string however these files were made by many different people and they are all formatted differently with random unneeded comments allover the place. Thus, why this is so flipping difficult. If you want something done right and all that.

rhoekstra · 09-14-2014, 02:53 PM

Hmm this doesn't add to the clarity of the question for me I'm afraid.

If you could set up an example situation with example output desired, that would help a lot.

Grepping with '-i' will make it case insensitive, with -A<number> or -B<number> you'll show that <number> of lines (A)fter or (B)efore the matched lines.

With a for loop you can find files with names as you desire with the 'find' tool. you then can make your routine within 'do' and 'done' to determine with every matching file, like echoing the filename and append that to a file. then the grepping on that particular file with all needed parameters (like -i, -A# and -B#) to show the lines you are looking for.

I guess that about is what I can provide for you. Other than that.. try to cut your problem/situation in pieces you want to solve (first, find a file, then, figure out what to do on such file, then, how to output the results, et cetera).

This should add to the resolvability of the problem at hand.

thealmightyos · 09-14-2014, 04:33 PM

ok. let's try this. Those 700+ files I need to go though? This is how they are formatted:

Code:

//description of config
//could be many lines

Name = name of config. usually does not match filename. Need this

MODULE // there could be several modules in a file. This one I don't want
{
     name = name of module
     variable = setting for module
     variable = setting for module
     variable = setting for module
     variable = setting for module
}
-------

Other code and comments

-------
Module
{
     name = foo //this is the one I want to see
     foo2 = setting for foo
     foo3 = setting for foo
     foo4 = setting for foo
     foo5 = setting for foo
     foo6 = setting for foo
     foo7 = setting for foo
     foo8 = setting for foo
     foo9 = setting for foo
}

------

More code I don't need

------
End of File

The output I would like to see is this:

Code:

./dir1/filename1.conf:		name = configname
./dir1/filename1.conf:		foo = name of module
./dir1/filename1.conf:		foo2 = settings
./dir1/filename1.conf:		foo3 = settings
./dir1/filename1.conf:		foo4 = settings
./dir1/filename1.conf:		foo5 = settings
./dir1/filename1.conf:		foo6 = settings
./dir1/filename1.conf:		foo7 = settings
./dir1/filename1.conf:		foo8 = settings
./dir1/filename1.conf:		foo9 = settings
./dir1/filename3.conf:		name = configname
./dir1/filename3.conf:		foo = name of module
./dir1/filename3.conf:		foo2 = settings
./dir1/filename3.conf:		foo3 = settings
./dir1/filename3.conf:		foo4 = settings
./dir1/filename3.conf:		foo5 = settings
./dir1/filename3.conf:		foo6 = settings
./dir1/filename3.conf:		foo7 = settings
./dir1/filename3.conf:		foo8 = settings
./dir1/filename3.conf:		foo9 = settings
./dir5/filename2.conf:		name = configname
./dir5/filename2.conf:		foo = name of module
./dir5/filename2.conf:		foo2 = settings
./dir5/filename2.conf:		foo3 = settings
./dir5/filename2.conf:		foo4 = settings
./dir5/filename2.conf:		foo5 = settings
./dir5/filename2.conf:		foo6 = settings
./dir5/filename2.conf:		foo7 = settings
./dir5/filename2.conf:		foo8 = settings
./dir5/filename2.conf:		foo9 = settings
./dir6/filename9.conf:		name = configname
./dir6/filename9.conf:		foo = name of module
./dir6/filename9.conf:		foo2 = settings
./dir6/filename9.conf:		foo3 = settings
./dir6/filename9.conf:		foo4 = settings
./dir6/filename9.conf:		foo5 = settings
./dir6/filename9.conf:		foo6 = settings
./dir6/filename9.conf:		foo7 = settings
./dir6/filename9.conf:		foo8 = settings
./dir6/filename9.conf:		foo9 = settings

rhoekstra · 09-14-2014, 05:24 PM

Ah.. like that..

There's really not an easy way to get through this other than creating some sort of perl/python/ruby/advanced BASH script that iterates over files, then over modules, to record the module name, and only when foo2-9 variables are found, to report firstly the file, then the file's module name, and thereafter the needed variables, because the lines with relevant info are already passed once you get to know whether you need to report them.

thealmightyos · 09-14-2014, 08:10 PM

Quote:

Originally Posted by rhoekstra

Ah.. like that..

There's really not an easy way to get through this other than creating some sort of perl/python/ruby/advanced BASH script that iterates over files, then over modules, to record the module name, and only when foo2-9 variables are found, to report firstly the file, then the file's module name, and thereafter the needed variables, because the lines with relevant info are already passed once you get to know whether you need to report them.

YES. Thus my frustration.

So there is no easy way to do this, any takers on the hard way? I am still working on some scripts right now but so far I am getting nothing

Right now I feel a little like this: http://xkcd.com/1319/

This was supposed to be the EASY WAY

Anyway, if I run

Code:

grep -r -l "foo" ./ > step1.txt

with foo being the unique module I am targeting, I get a nice clean list of files that have that module which reduces the 750+ files to 86. Not bad. But I am unsure how to feed this list back into grep for another sweep to get the other information. Even if I have to get all the information in chunks and then organize it in a spreadsheet it would be much easier than going into each file and figuring out what they used for variables.

rhoekstra · 09-15-2014, 03:49 AM

You could try to use it in a for loop..

Code:

for i in `cat step1.txt`; do
  grep -n -e 'name =' -e 'foo2 =' -e 'foo3' $i >>step2.txt
done

This will provide you lots of 'name = ' lines which don't have any foo* variables, so it shows little more overhead than needed, but it might help you in narrowing down your problem.

thealmightyos · 09-15-2014, 05:21 PM

Quote:

Originally Posted by rhoekstra

You could try to use it in a for loop..

Code:

for i in `cat step1.txt`; do
  grep -n -e 'name =' -e 'foo2 =' -e 'foo3' $i >>step2.txt
done

This will provide you lots of 'name = ' lines which don't have any foo* variables, so it shows little more overhead than needed, but it might help you in narrowing down your problem.

No, this could be very useful. Since the files in step1.txt are files that only contain the foo element I am looking for I could first run a grep -l -m 1 'name =' to get the name of each file then move on to find the other variables. Store it all in a spreadsheet and edit as needed.

Problem though, when I run that script I get a "grep: cat: No such file or directory" error.

rhoekstra · 09-15-2014, 06:43 PM

Don't forget the back ticks

thealmightyos · 09-15-2014, 08:45 PM

OH! Those are back ticks. I was using apostrophes. Did not look close enough.

Alright, now we are getting somewhere! It isn't pretty or even ideal, but with notepad++ and a spreadsheet I can work with this. Thanks!

rhoekstra · 09-16-2014, 02:24 AM

Thank me by clicking 'yes' at 'did you find this post helpful'

... tx

Glad you've got something to work with.. good luck

thealmightyos · 09-17-2014, 01:00 PM

I have managed to extract the data I need. With just a little more work I can put it into a format that I can create a script for. Thank you for your help