[SOLVED] Select lines based on key match

danielbmartin · 03-07-2012, 02:12 PM

I want select lines from a large file based on matching a key value in a smaller file. The key values in the small file are unique; the matching values in the large file are not unique. Both files are sorted.

Sample small file ...

Code:

Cole
Phillips

Sample large file ...

Code:

Bergeron Denise
Bergeron Terrence
Cole Carlton
Cole Donald
Cole Martha
Davis Michelle
Davis Joel
High Alice
High Robert
Phillips Edgar
Phillips Suzanne

Sample output file ...

Code:

Cole Carlton
Cole Donald
Cole Martha
Phillips Edgar
Phillips Suzanne

These samples are representative but the actual files are large so performance is a consideration.

Daniel B. Martin

anomie · 03-07-2012, 02:23 PM

grep(1) can read patterns from a file, a la:

Code:

$ grep -f keys.txt people.txt

danielbmartin · 03-07-2012, 03:41 PM

Quote:

Originally Posted by anomie

Code:

$ grep -f keys.txt people.txt

Thank you, anomie, we are on the right track. It is necessary to match the key strings to the left-most blank-delimited field only. That will make the grep run faster and, more importantly, avoid false matches. If the keys file contains "Martin" I don't want matches on lines in the people file such as "Davidson Martin."

How may we limit the scope of the grep?

Daniel B. Martin

anomie · 03-07-2012, 03:54 PM

There may be a more efficient means for solving this, but I'd simply put the patterns in the keys.txt file.

Input files:

Code:

$ cat keys.txt 
^Cole\>
^Phillips\>

Code:

$ cat people.txt 
Bergeron Denise
Bergeron Terrence
Cole Carlton
Cole Donald
Cole Martha
Davis Michelle
Davis Joel
High Alice
High Robert
Phillips Edgar
Phillips Suzanne
Jo Cole
Coleen Hsu

Result from grep(1):

Code:

$ grep -f keys.txt people.txt 
Cole Carlton
Cole Donald
Cole Martha
Phillips Edgar
Phillips Suzanne

Nominal Animal · 03-07-2012, 05:02 PM

How about

Code:

awk -v 'keyfile=path/to/small/file' 'BEGIN {
    while ((getline < keyfile) > 0) key[$1]
    close(keyfile)
  }
  ($1 in key)' 'path/to/large/file'

The BEGIN rule reads the key file first. First field (word) on each line is saved to the key array as a key, with a null value. (In awk, just referencing an array member will create it. Assume there is a = NULL at the end of the second line.)

The actual rule on the last line reads: If first field matches a key in key array, then print the record. (You can omit the implicit { print } for the last rule.)

Essentially, the above reads the first fields in the small file, then outputs the records (lines) of the large file only if the first field matches one of the ones read from the small file.

Reuti · 03-08-2012, 06:40 AM

Code:

$ join keys.txt people.txt

By default it will match on the first column.

danielbmartin · 03-08-2012, 10:52 AM

Quote:

Originally Posted by anomie

There may be a more efficient means for solving this, but I'd simply put the patterns in the keys.txt file.

Code:

$ cat keys.txt 
^Cole\>
^Phillips\>

The ^ means "starting in column 1" as desired. I don't understand what the \> does for us. I tried this method using only the ^ prefix and it seemed to work.

Daniel B. Martin

anomie · 03-08-2012, 11:11 AM

Those are regular expressions (anchors). The meanings are:

^ -- match beginning of line
\> -- match end of word

If you do not use the latter, you'll also match names like "Coleman Butler".

danielbmartin · 03-08-2012, 06:13 PM

Quote:

Originally Posted by anomie

[*] \> -- match end of word[*]

Thank you, this is something I haven't seen before.

I try to learn from tutorials and Google searches. Even knowing \> I found no mention of it anywhere. Help me to help myself -- where could I have found this on my own?

Daniel B. Martin

anomie · 03-08-2012, 08:26 PM

In this case, you can view the manpages for grep(1):

Code:

Anchoring
   The caret ^ and the dollar sign $ are meta-characters that respectively
   match the empty string at the beginning and end of a line.

The Backslash Character and Special Expressions
   The  symbols  \<  and  \>  respectively  match  the empty string at the
   beginning and end of a word...

danielbmartin · 03-09-2012, 11:03 AM

Quote:

Originally Posted by anomie

In this case, you can view the manpages for grep(1) ...

Yes, found it, thank you. As I climb the Linux learning curve, it becomes apparent that mastering grep and sed requires a proficiency in writing Regular Expressions.

Daniel B. Martin