Go Job Hunting at the LQ Job Marketplace
Go Back > Forums > Non-*NIX Forums > Programming
User Name
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.


  Search this Thread
Old 11-13-2010, 05:04 PM   #1
LQ Newbie
Registered: Nov 2010
Posts: 1

Rep: Reputation: 0
grep multiple values in single pass through log file.

Help needed
I have a huge binary log file.
There are lets say 4 id's that I want to find in a log file.
I know that those 4 id's will be present in the log file and I also know in what order they will be present.
I want to find 1st id from the log then 2nd id and then third id and so on..

Simple/inefficient solution is: Loop through the id's and then grep in the log file. Problem with this solution is for each id grep will search from the beginning of the file.

Better/efficient solution would be: Sine I know the order in which id's will be present in the log file. Loop through id's, grep 1st id and then move on to grep 2nd id and so on...this way I can grep all id's in one pass.
Is this solution possible ?

I have 500000 + values to find in log files and I have to find efficient solution for it. Thanks in advance
Old 11-13-2010, 05:24 PM   #2
Senior Member
Registered: Jan 2003
Posts: 2,786

Rep: Reputation: 370Reputation: 370Reputation: 370Reputation: 370
You should look into regular expressions. Something like this might get what you want:
egrep "1234|5678|9012|3456" binary_log_file
  • egrep launches grep with support for basic regular expressions
  • "1234|5678|9012|3456" means to match any line that contains one or more of 1234, 5678, 9012, and 3456.
  • binary_log_file is, of course, the file you want to grep against
Replace the four-digit numbers I used with the IDs you're looking for.

Keep in mind, this will cause grep to display all matching lines in whatever order they occur in the file.

EDIT: My mistake, you do not need the escape character if the grep string is in double quotes

Last edited by Dark_Helmet; 11-13-2010 at 05:29 PM.
Old 11-13-2010, 05:45 PM   #3
Rogue Class
Registered: Sep 2006
Location: OR, USA
Distribution: Slackware 14.1
Posts: 201
Blog Entries: 3

Rep: Reputation: 78
gzip -dc log.gz | grep -e 1234 -e 5678 -e 90ab -e cdef >results.txt


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Multiple grep outputs appended to single row of CSV file findme.krish Linux - Newbie 5 09-12-2010 07:56 PM
grep multiple words any order (AND, not OR) single line, from many files cedardoc Linux - Newbie 7 07-29-2010 11:23 AM
Replace values in log file laki47 Linux - Newbie 5 05-14-2010 10:41 AM
grep all values other than encrypted password from /etc/shadow file tastie Linux - General 7 01-15-2010 10:03 AM
Pass the contents of firewall log to another file arubala1 Linux - Networking 2 03-16-2006 04:50 AM

All times are GMT -5. The time now is 04:03 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration