LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 05-11-2011, 08:52 AM   #1
rhbegin
Member
 
Registered: Oct 2003
Location: Arkansas, NWA
Distribution: Fedora/CentOS/SL6
Posts: 381

Rep: Reputation: 23
Question Using grep -v to sort out a log


I have logs from email servers and I am trying to sort out our IP ranges from the log files.

I am using the following:

grep -v "192.168.0.25" s20110511serv.log > s2011_25_removed.log
grep -v "192.168.0.27" s2011_25_removed.log > s2011_2527_gone.log

How can I issue this on one line without having to output to multiple files, also can I pattern match an IP range like 192.168.4.0/18 from the same line???

grep -v "192.168.0.25" "192.168.0.27" "192.168.4.0" the /18 range from the log file?

Any help would be great, I am still learning on grep and I am a bit confused on the syntax to pull out data in one line.

Thanks
 
Old 05-11-2011, 09:40 AM   #2
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,395
Blog Entries: 2

Rep: Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903
You can pipe the output of grep to the standard input of any other command, including grep:
Code:
grep -v "192.168.0.25" s20110511serv.log | grep -v "192.168.0.27" > s2011_2527_gone.log
Grep doesn't know anything about the notation used to describe IP subnetting.
--- rod.

Last edited by theNbomr; 05-11-2011 at 09:42 AM.
 
Old 05-11-2011, 03:02 PM   #3
rhbegin
Member
 
Registered: Oct 2003
Location: Arkansas, NWA
Distribution: Fedora/CentOS/SL6
Posts: 381

Original Poster
Rep: Reputation: 23
Quote:
Originally Posted by theNbomr View Post
You can pipe the output of grep to the standard input of any other command, including grep:
Code:
grep -v "192.168.0.25" s20110511serv.log | grep -v "192.168.0.27" > s2011_2527_gone.log
Grep doesn't know anything about the notation used to describe IP subnetting.
--- rod.
So you can pipe your source ip then again then output to 1 file, that will work.

Is it possible to take out an ip range in a subnet like 192.168.4.0/24 out of the log file then with a different method?

Thank you for the quick response!

 
Old 05-11-2011, 03:27 PM   #4
jcmlq
Member
 
Registered: Aug 2009
Posts: 32

Rep: Reputation: 19
Your first command is saying 'give me everything without 192.168.0.25', and your second line is saying 'give me everything without 192.168.0.27'. You can say in a single line 'give me everything without 192.168.0.2' followed by 5 or 7 with a regular expression.

Code:
grep -v '192.168.0.2[57]'
You could remove an entire /24 with an extended regex like
Code:
grep -v -E '192\.168\.0\.[0-9]{1,3}'
The \. is used to match the '.' characters because '.' is itself a special regular expression character.

The -E means extended grep syntax is enabled.

The meaning of the match expression is '192.168.0.', followed by 1 to 3 instances of the characters 0-9.
 
Old 05-11-2011, 04:21 PM   #5
rhbegin
Member
 
Registered: Oct 2003
Location: Arkansas, NWA
Distribution: Fedora/CentOS/SL6
Posts: 381

Original Poster
Rep: Reputation: 23
Quote:
Originally Posted by jcmlq View Post
Your first command is saying 'give me everything without 192.168.0.25', and your second line is saying 'give me everything without 192.168.0.27'. You can say in a single line 'give me everything without 192.168.0.2' followed by 5 or 7 with a regular expression.

Code:
grep -v '192.168.0.2[57]'
You could remove an entire /24 with an extended regex like
Code:
grep -v -E '192\.168\.0\.[0-9]{1,3}'
The \. is used to match the '.' characters because '.' is itself a special regular expression character.

The -E means extended grep syntax is enabled.

The meaning of the match expression is '192.168.0.', followed by 1 to 3 instances of the characters 0-9.
Thanks a million!

I need to write this down and study it to get a good understanding, I learn something new everything with the command-line.

 
Old 05-11-2011, 04:31 PM   #6
jcmlq
Member
 
Registered: Aug 2009
Posts: 32

Rep: Reputation: 19
One thing I should make absolutely clear about the example to match a /24 that I provided - it works, but it isn't actually correct. It is just as happy to match 192.168.0.999 as it is to match 192.168.0.255

Much (most) of the time that kind of sloppiness is just fine, but if you really need to make sure and only match valid ip addresses then you probably need to use a full blown scripting language and not just grep.

EDIT: the proper ip octet match scheme is something like

Code:
grep -E -v '192\.168\.0\.(1*[0-9]{1,2}|2[0-4][0-9]|25[0-5])'
Since you seem to be pretty new to regex I hesitate to dump that mess in your lap when something much easier to understand does pretty much what you need.

Last edited by jcmlq; 05-11-2011 at 04:52 PM. Reason: added proper ip address match regex
 
Old 05-11-2011, 05:13 PM   #7
rhbegin
Member
 
Registered: Oct 2003
Location: Arkansas, NWA
Distribution: Fedora/CentOS/SL6
Posts: 381

Original Poster
Rep: Reputation: 23
Quote:
Originally Posted by jcmlq View Post
One thing I should make absolutely clear about the example to match a /24 that I provided - it works, but it isn't actually correct. It is just as happy to match 192.168.0.999 as it is to match 192.168.0.255

Much (most) of the time that kind of sloppiness is just fine, but if you really need to make sure and only match valid ip addresses then you probably need to use a full blown scripting language and not just grep.

EDIT: the proper ip octet match scheme is something like

Code:
grep -E -v '192\.168\.0\.(1*[0-9]{1,2}|2[0-4][0-9]|25[0-5])'
Since you seem to be pretty new to regex I hesitate to dump that mess in your lap when something much easier to understand does pretty much what you need.
Thank you again, I am still learning, once I get the basic concepts I will understand it. It takes me a little while on the front end but once I understand it, I got it.



Thank you again!

 
Old 05-11-2011, 05:29 PM   #8
jcmlq
Member
 
Registered: Aug 2009
Posts: 32

Rep: Reputation: 19
You're welcome, good luck in cracking into regular expressions - they are an essential tool for admin and operations work in my opinion.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] parsing out squid access log with awk and grep druisgod Programming 8 04-26-2011 11:40 AM
netstat - grep - tail -f output log manwithaplan Linux - Networking 4 08-21-2009 06:36 PM
grep+awk+sed+paste+sort in one script? mchriste Linux - Software 13 03-05-2009 02:57 PM
Shell script to grep log for a specific error sfcg Programming 8 06-28-2008 10:12 AM
sort grep output wijnands Linux - Newbie 4 10-09-2004 08:14 AM


All times are GMT -5. The time now is 06:46 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration