Excluding multiple patterns from grep

flamingo_l · 12-13-2010, 05:36 AM

hi ,

I have to write a script which would search the IP adesseses in a given directory.

Below is my command.

Code:

grep -HwrnI --exclude=*.log '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' *|grep -v '/\.svn/'

I have to exclude the following from search resluts.

1. Comments
a. Can be starting with /, * or #...
b. Cane be between a line
EX: some text... #comment1

Cananybody please help me on this.

Tinkster · 12-13-2010, 10:56 PM

I don't think you can achieve that in a one-step process. While grep with -E
happily supports extended regular expressions you can't have the -v feature
of grep for a non-match combined w/ a match.

You'll be looking at running your grep, and then suppressing things that
match comments in a 2nd grep.

Code:

grep -HwrnI --exclude=*.log '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' *|grep -v '/\.svn/'|grep -Ev '/|#|\*'

Cheers,
Tink

flamingo_l · 12-29-2010, 03:01 AM

Sorry for the late reply ... Tinkster...

Your idea is not working since the result of the below command contains path name of the file, so every row is getting excluded since the path name contians '/'

Quote:

grep -HwrnI --exclude=*.log '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' *|

Can you please suggest any other idea...

makyo · 01-01-2011, 12:11 PM

Hi.

My understanding of this problem is that you wish exclude lines when any one of a number of patterns is matched on a line. I'm assuming that the part involving IP patterns is solved. A bit of awk can address a situation where a matching any of a number of strings must be excluded. For example, in the script below, the results are from section 3 (after the context and data file are displayed). The first 2 sections detail how the script was run and what the data looked like; those contain some non-standard elements, but the solution is standard:

Code:

#!/usr/bin/env bash

# @(#) s1	Demonstrate line exclusion with awk.

# Section 1, setup, pre-solution.
# Infrastructure details, environment, commands for forum posts. 
# Uncomment export command to test script as external user.
# export PATH="/usr/local/bin:/usr/bin:/bin"
set +o nounset
pe() { for i;do printf "%s" "$i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
C=$HOME/bin/context && [ -f $C ] && . $C specimen awk
set -o nounset
pe

FILE=${1-data1}

# Section 2, display input file.
# Display sample of data file, with head & tail as a last resort.
pe " || start [ first:middle:last ]"
specimen 7 $FILE \
|| { pe "(head/tail)"; head -n 5 $FILE; pe " ||"; tail -n 5 $FILE; }
pe " || end"

# Section 3, solution.
pl " Results:"
awk '
!/[#]|[*]|[/]|apple/
' $FILE

exit 0

producing:

Code:

% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0 (lenny) 
GNU bash 3.2.39
specimen (local) 1.17
GNU Awk 3.1.5

 || start [ first:middle:last ]
Whole: 7:0:7 of 13 lines in file "data1"
Line 1
# at the beginning
* at the beginning
/ at the beginning
A line with # in the middle
A line with * in the middle
A line with / in the middle
Line someplace in the file
We like apples
Finally, symbol at the end #
Finally, symbol at the end *
Finally, symbol at the end /
Last line
 || end

-----
 Results:
Line 1
Line someplace in the file
Last line

The awk script attempts to match any of the alternatives separated by "|". The "[]" symbols serve to effectively quote the enclosed characters, often a help in reading the patterns. The leading "!" is to invert the sense of the match so that if any pattern matches, the implicit print action is avoided. Only when none of the patterns match and the sense is inverted (to true), will the line be printed.

I prefer single passes over data files if it can be done easily.

Best wishes ... cheers, makyo

( Edit 1 )
I just noticed the issue with "/" in the pathname. Do you need the pathname for this problem? If not, then changing the "H" to "h" will eliminate the pathname. If you do need it, then another step will be needed.

flamingo_l · 01-06-2011, 01:02 AM

It is taking me lot of time to understand the code...

Would be great if you can breif a little on the code.

Am using cygwin to execute the code, but it is not working....
Throwing the following error

Quote:

line 32: specimen: command not found

jschiwal · 01-06-2011, 01:09 AM

Quote:

--exclude=*.log

You need to escape the asterisk, or put the glob pattern in quotes. Otherwise it will expand to a list of all log files before the command is executed:
--exclude=filename1.log filename2.log ...l

flamingo_l · 01-06-2011, 01:12 AM

Thanks... jschiswal for the comment, i would put *.log in double quotes...

Can you give me any inputs for my requirement described in post#3.

makyo · 01-06-2011, 04:04 AM

Hi.

Quote:

Originally Posted by flamingo_l

Code:

line 32: specimen: command not found

That section is intended to display the input file, and is supposed to execute head-tail commands if specimen is not available. Perhaps cygwin behaves differently, so replace the lines:

Code:

specimen 7 $FILE \
|| { pe "(head/tail)"; head -n 5 $FILE; pe " ||"; tail -n 5 $FILE; }
pe " || end"

with:

Code:

cat $FILE

Re-run the code, and you should get the results I did ... cheers, makyo

flamingo_l · 01-06-2011, 06:01 AM

hi Makyo,

It is not working.. please find the ouput...

Quote:

$ ./forum1.sh

|| start [ first:middle:last ]
lfjdlfjgj //ldfdlkfkf
#fjdkslfjskj
/* llllllllllllllllllllllllllllllllllllllllll
jjjjjjjjjjjjjjjjjjjjjjj */
-----
Results:

Also, what is C in the script?

makyo · 01-06-2011, 08:05 AM

Hi.

The first part of that script uses some local code to display the computing environment in which that code was run. Let us omit that for now. Please copy and run this:

Code:

#!/usr/bin/env bash

# @(#) s5	Demonstrate line exclusion with awk, minimal version.

FILE=${1-data1}

# Section 2, display input file.
# Display sample of data file, with head & tail as a last resort.
echo
echo " Data file $FILE:"
cat $FILE

# Section 3, solution.
echo
echo " Results:"
awk '
!/[#]|[*]|[/]|apple/
' $FILE

exit 0

which produces this on my system:

Code:

% ./s5

 Data file data1:
Line 1
# at the beginning
* at the beginning
/ at the beginning
A line with # in the middle
A line with * in the middle
A line with / in the middle
Line someplace in the file
We like apples
Finally, symbol at the end #
Finally, symbol at the end *
Finally, symbol at the end /
Last line

 Results:
Line 1
Line someplace in the file
Last line

cheers, makyo