LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 09-12-2014, 12:07 AM   #1
thealmightyos
Member
 
Registered: Mar 2009
Distribution: CentOS 6.5 / 7
Posts: 119

Rep: Reputation: 1
Help: Searching multiple files and returning multiple lines matching querry


So, I consulted the googles, and what I found did not help my situation. I am confused and not sure what angle to tackle this from first. awk, sed, grep.... Wat do?

I have 111 directories and 768 files that I need to search though and find which files I have to change and what I need to change in them and hopefully I can use this information to create a script to do so. But I don't know how to find those files and print the information from them I need. I need a command/string of commands/script file/ program to:

1: Find first file that contains the string "foo"
2: Print out the filename
3: Print out the line that contains "foo" (grep, basically)
4: Print out the line that contains "foo2"
5: same as 4 only with foo3-9
6: find the next file that contains the string "foo"
Goto 2

Any suggestions?

Last edited by thealmightyos; 09-12-2014 at 12:08 AM.
 
Old 09-12-2014, 01:55 AM   #2
j-ray
Senior Member
 
Registered: Jan 2002
Location: germany
Distribution: ubuntu, mint, suse
Posts: 1,591

Rep: Reputation: 145Reputation: 145
grep -n foo /ParentDirectory/* > logfile
grep -n foo /ParentDirectory/*/* >> logfile
grep -n foo2 /ParentDirectory/* >> logfile
grep -n foo2 /ParentDirectory/*/* >> logfile

Am I getting you wrong?
 
Old 09-12-2014, 02:26 AM   #3
rhoekstra
Member
 
Registered: Aug 2004
Location: The Netherlands
Distribution: RedHat 2, 3, 4, 5, Fedora, SuSE, Gentoo
Posts: 372

Rep: Reputation: 42
So only when a file contains 'foo' it needs to be processed for foo2-9 ? Files without foo don't need processing for foo2 ?
Code:
for i in `grep -rl foo /ParentDirectory/`; do # Process 'parentdirectory' and all its subdirectories
  echo $i >>log.txt #print the filenumber
  grep -n foo >>log.txt #print linenumbers with foo
  grep -n foo2-9 >>log.txt #print linenumbers with foo2-9
done

Last edited by rhoekstra; 09-12-2014 at 02:27 AM. Reason: formatted code
 
Old 09-12-2014, 05:41 PM   #4
thealmightyos
Member
 
Registered: Mar 2009
Distribution: CentOS 6.5 / 7
Posts: 119

Original Poster
Rep: Reputation: 1
Quote:
Originally Posted by rhoekstra View Post
So only when a file contains 'foo' it needs to be processed for foo2-9 ? Files without foo don't need processing for foo2 ?
Code:
for i in `grep -rl foo /ParentDirectory/`; do # Process 'parentdirectory' and all its subdirectories
  echo $i >>log.txt #print the filenumber
  grep -n foo >>log.txt #print linenumbers with foo
  grep -n foo2-9 >>log.txt #print linenumbers with foo2-9
done
yes. sort of. maybe. Oh hell with it, DO OVER

God I write things up horribly when I am tired. I apologize for that. Sleeping meds + linux shell = OS gets sloppy

Back to the topic at hand. I am abandoning the first post example. can not be saved. Starting over.
I can ALMOST get what I want with
Code:
grep -r 'foo\|foo2\|foo3\|foo4\|foo5\|foo6' ./
#there are more "foo's" in there but you get the point
Problem is, now that I found the file, got all the config info I need one more thing. Not even going to use foo for this one: "name =". It is so flipping generic that it prints multiple lines from EVERY SINGLE FILE. So, this is what I need:

For files that match foo, I want to print foo#'s AND the first instance of "name ="

There, see what being awake can do?
 
Old 09-14-2014, 01:27 AM   #5
thealmightyos
Member
 
Registered: Mar 2009
Distribution: CentOS 6.5 / 7
Posts: 119

Original Poster
Rep: Reputation: 1
an even BETTER solution to this is if I could print all lines between and including foo1 to foo2 and then the first instance of "name =". Would use the grep function to print n number of lines after finding a string however these files were made by many different people and they are all formatted differently with random unneeded comments allover the place. Thus, why this is so flipping difficult. If you want something done right and all that.
 
Old 09-14-2014, 01:53 PM   #6
rhoekstra
Member
 
Registered: Aug 2004
Location: The Netherlands
Distribution: RedHat 2, 3, 4, 5, Fedora, SuSE, Gentoo
Posts: 372

Rep: Reputation: 42
Hmm this doesn't add to the clarity of the question for me I'm afraid.

If you could set up an example situation with example output desired, that would help a lot.

Grepping with '-i' will make it case insensitive, with -A<number> or -B<number> you'll show that <number> of lines (A)fter or (B)efore the matched lines.

With a for loop you can find files with names as you desire with the 'find' tool. you then can make your routine within 'do' and 'done' to determine with every matching file, like echoing the filename and append that to a file. then the grepping on that particular file with all needed parameters (like -i, -A# and -B#) to show the lines you are looking for.

I guess that about is what I can provide for you. Other than that.. try to cut your problem/situation in pieces you want to solve (first, find a file, then, figure out what to do on such file, then, how to output the results, et cetera).

This should add to the resolvability of the problem at hand.
 
Old 09-14-2014, 03:33 PM   #7
thealmightyos
Member
 
Registered: Mar 2009
Distribution: CentOS 6.5 / 7
Posts: 119

Original Poster
Rep: Reputation: 1
ok. let's try this. Those 700+ files I need to go though? This is how they are formatted:

Code:
//description of config
//could be many lines

Name = name of config. usually does not match filename. Need this

MODULE // there could be several modules in a file. This one I don't want
{
     name = name of module
     variable = setting for module
     variable = setting for module
     variable = setting for module
     variable = setting for module
}
-------

Other code and comments

-------
Module
{
     name = foo //this is the one I want to see
     foo2 = setting for foo
     foo3 = setting for foo
     foo4 = setting for foo
     foo5 = setting for foo
     foo6 = setting for foo
     foo7 = setting for foo
     foo8 = setting for foo
     foo9 = setting for foo
}

------

More code I don't need

------
End of File

The output I would like to see is this:
Code:
./dir1/filename1.conf:		name = configname
./dir1/filename1.conf:		foo = name of module
./dir1/filename1.conf:		foo2 = settings
./dir1/filename1.conf:		foo3 = settings
./dir1/filename1.conf:		foo4 = settings
./dir1/filename1.conf:		foo5 = settings
./dir1/filename1.conf:		foo6 = settings
./dir1/filename1.conf:		foo7 = settings
./dir1/filename1.conf:		foo8 = settings
./dir1/filename1.conf:		foo9 = settings
./dir1/filename3.conf:		name = configname
./dir1/filename3.conf:		foo = name of module
./dir1/filename3.conf:		foo2 = settings
./dir1/filename3.conf:		foo3 = settings
./dir1/filename3.conf:		foo4 = settings
./dir1/filename3.conf:		foo5 = settings
./dir1/filename3.conf:		foo6 = settings
./dir1/filename3.conf:		foo7 = settings
./dir1/filename3.conf:		foo8 = settings
./dir1/filename3.conf:		foo9 = settings
./dir5/filename2.conf:		name = configname
./dir5/filename2.conf:		foo = name of module
./dir5/filename2.conf:		foo2 = settings
./dir5/filename2.conf:		foo3 = settings
./dir5/filename2.conf:		foo4 = settings
./dir5/filename2.conf:		foo5 = settings
./dir5/filename2.conf:		foo6 = settings
./dir5/filename2.conf:		foo7 = settings
./dir5/filename2.conf:		foo8 = settings
./dir5/filename2.conf:		foo9 = settings
./dir6/filename9.conf:		name = configname
./dir6/filename9.conf:		foo = name of module
./dir6/filename9.conf:		foo2 = settings
./dir6/filename9.conf:		foo3 = settings
./dir6/filename9.conf:		foo4 = settings
./dir6/filename9.conf:		foo5 = settings
./dir6/filename9.conf:		foo6 = settings
./dir6/filename9.conf:		foo7 = settings
./dir6/filename9.conf:		foo8 = settings
./dir6/filename9.conf:		foo9 = settings
 
Old 09-14-2014, 04:24 PM   #8
rhoekstra
Member
 
Registered: Aug 2004
Location: The Netherlands
Distribution: RedHat 2, 3, 4, 5, Fedora, SuSE, Gentoo
Posts: 372

Rep: Reputation: 42
Ah.. like that..

There's really not an easy way to get through this other than creating some sort of perl/python/ruby/advanced BASH script that iterates over files, then over modules, to record the module name, and only when foo2-9 variables are found, to report firstly the file, then the file's module name, and thereafter the needed variables, because the lines with relevant info are already passed once you get to know whether you need to report them.
 
Old 09-14-2014, 07:10 PM   #9
thealmightyos
Member
 
Registered: Mar 2009
Distribution: CentOS 6.5 / 7
Posts: 119

Original Poster
Rep: Reputation: 1
Quote:
Originally Posted by rhoekstra View Post
Ah.. like that..

There's really not an easy way to get through this other than creating some sort of perl/python/ruby/advanced BASH script that iterates over files, then over modules, to record the module name, and only when foo2-9 variables are found, to report firstly the file, then the file's module name, and thereafter the needed variables, because the lines with relevant info are already passed once you get to know whether you need to report them.
YES. Thus my frustration.

So there is no easy way to do this, any takers on the hard way? I am still working on some scripts right now but so far I am getting nothing

Right now I feel a little like this: http://xkcd.com/1319/

This was supposed to be the EASY WAY


Anyway, if I run

Code:
grep -r -l "foo" ./ > step1.txt
with foo being the unique module I am targeting, I get a nice clean list of files that have that module which reduces the 750+ files to 86. Not bad. But I am unsure how to feed this list back into grep for another sweep to get the other information. Even if I have to get all the information in chunks and then organize it in a spreadsheet it would be much easier than going into each file and figuring out what they used for variables.

Last edited by thealmightyos; 09-14-2014 at 08:35 PM.
 
Old 09-15-2014, 02:49 AM   #10
rhoekstra
Member
 
Registered: Aug 2004
Location: The Netherlands
Distribution: RedHat 2, 3, 4, 5, Fedora, SuSE, Gentoo
Posts: 372

Rep: Reputation: 42
You could try to use it in a for loop..

Code:
for i in `cat step1.txt`; do
  grep -n -e 'name =' -e 'foo2 =' -e 'foo3' $i >>step2.txt
done
This will provide you lots of 'name = ' lines which don't have any foo* variables, so it shows little more overhead than needed, but it might help you in narrowing down your problem.

Last edited by rhoekstra; 09-15-2014 at 02:49 AM. Reason: formatting
 
Old 09-15-2014, 04:21 PM   #11
thealmightyos
Member
 
Registered: Mar 2009
Distribution: CentOS 6.5 / 7
Posts: 119

Original Poster
Rep: Reputation: 1
Quote:
Originally Posted by rhoekstra View Post
You could try to use it in a for loop..

Code:
for i in `cat step1.txt`; do
  grep -n -e 'name =' -e 'foo2 =' -e 'foo3' $i >>step2.txt
done
This will provide you lots of 'name = ' lines which don't have any foo* variables, so it shows little more overhead than needed, but it might help you in narrowing down your problem.
No, this could be very useful. Since the files in step1.txt are files that only contain the foo element I am looking for I could first run a grep -l -m 1 'name =' to get the name of each file then move on to find the other variables. Store it all in a spreadsheet and edit as needed.

Problem though, when I run that script I get a "grep: cat: No such file or directory" error.
 
Old 09-15-2014, 05:43 PM   #12
rhoekstra
Member
 
Registered: Aug 2004
Location: The Netherlands
Distribution: RedHat 2, 3, 4, 5, Fedora, SuSE, Gentoo
Posts: 372

Rep: Reputation: 42
Don't forget the back ticks
 
1 members found this post helpful.
Old 09-15-2014, 07:45 PM   #13
thealmightyos
Member
 
Registered: Mar 2009
Distribution: CentOS 6.5 / 7
Posts: 119

Original Poster
Rep: Reputation: 1
OH! Those are back ticks. I was using apostrophes. Did not look close enough.

Alright, now we are getting somewhere! It isn't pretty or even ideal, but with notepad++ and a spreadsheet I can work with this. Thanks!
 
Old 09-16-2014, 01:24 AM   #14
rhoekstra
Member
 
Registered: Aug 2004
Location: The Netherlands
Distribution: RedHat 2, 3, 4, 5, Fedora, SuSE, Gentoo
Posts: 372

Rep: Reputation: 42
Thank me by clicking 'yes' at 'did you find this post helpful' ... tx

Glad you've got something to work with.. good luck
 
Old 09-17-2014, 12:00 PM   #15
thealmightyos
Member
 
Registered: Mar 2009
Distribution: CentOS 6.5 / 7
Posts: 119

Original Poster
Rep: Reputation: 1
I have managed to extract the data I need. With just a little more work I can put it into a format that I can create a script for. Thank you for your help
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Searching through Multiple Files Meowser Linux - General 3 06-02-2012 11:18 AM
[SOLVED] Perl regex not matching across multiple lines despite ms flags gfarrell Programming 30 08-18-2010 04:10 AM
Script / two files and matching multiple columes sharky Programming 6 10-21-2008 05:19 PM
AWK/SED Multiple pattern matching over multiple lines issue GigerMalmensteen Programming 15 12-03-2006 05:08 PM
searching for multiple files ryedunn Linux - Newbie 4 09-27-2004 03:21 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 01:43 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration