Bash script to suppress matches in two column list

xsyntax · 09-02-2010, 03:19 AM

I have been using comm to compare two simple column lists, and suppress items that were contained in the second list (suppression list). This was extremely simple and basic, however now list1 has two columns, and I must compare the second column in list1 with my suppression list.

Previously my user list and suppression list looked like:
user list:
user1@domain.com
user2@domain.com

suppression list:
user2@domain.com

And the command I used in my bash script:

Code:

comm -3 user_list suppression_list

Now my user list looks like:
user1@domain.com 3bc81bc52e7f209c3455af320abeee00
user2@domain.com ed076488b22b5359d7ffb16b8e30caed

and my suppression list looks like:
ed076488b22b5359d7ffb16b8e30caed

Basically I need to compare my user list and suppression list to suppress any users that exist in the suppression list, then remove the second column (md5).

I wasn't sure the fastest way to make comparisons if there was a similar command like comm, or if I needed to create an array of users and see if any of them matched the suppression list one by one. This seemed like it would be pretty process intensive. Anyone have any less cumbersome ideas?

Thanks guys, you are always a great resource.

druuna · 09-02-2010, 05:02 AM

Hi,

Try this: grep -vf file2 file1 | sed 's/ .*//'

Example run:

Code:

$ cat file1
user1@domain.com 3bc81bc52e7f209c3455af320abeee00
user2@domain.com ed076488b22b5359d7ffb16b8e30caed
user3@domain.com XXX76488b22b5359d7ffb16b8e30caed
user4@domain.com YYY76488b22b5359d7ffb16b8e30caed

$ cat file2
ed076488b22b5359d7ffb16b8e30caed
YYY76488b22b5359d7ffb16b8e30caed

$ grep -vf file2 file1 | sed 's/ .*//'
user1@domain.com
user3@domain.com

Hope this helps.

xsyntax · 09-02-2010, 11:16 AM

druuna,

Thank you very much, that works perfectly. I was way over complicating things the way I was thinking. Your solution is simple and works perfectly.

Take it easy

druuna · 09-02-2010, 11:17 AM

You're welcome

xsyntax · 09-02-2010, 05:16 PM

I tested the solution above and it works fine with small files, however when I start working with large files the script bogs down to a crawl. Sometimes the suppression file can be up to 2GB.

I think the old script using comm was so much faster because comm requires the lists to be sorted, therefore it can compare the tow lists line by line much faster. I don't know for a fact, but I am assuming the grep is storing as much in memory as possible because it has to scroll through the files multiple times.

Is there anyone out there that has any ideas of ways that I may be able make this perform better? Any ideas are welcome!

druuna · 09-03-2010, 04:21 AM

Hi,

Large files (several Gb's) do tend to put a strain on your computer (depending on the specs of that computer) and not all tools are adequate when it comes to large files.

Perl might be a better choice. Give the following a try:

Code:

#!/usr/bin/perl
use strict ;
use warnings ;

# -------------------------------------------------------------------------- #
# Variables
# ------------------------------------------------------------------ #
my $usermd5file = "file1" ;
my $md5suppfile = "file2" ;

my $user ;
my $md5key ;
my %keyuserpair ;

# -------------------------------------------------------------------------- #
# Subroutines
# ------------------------------------------------------------------ #
# fill hash with md5 -> user pairs
sub processUserMd5File {

   # open user/md5 file
   open( USERMD5FILE, $usermd5file )
     or die "Can't open $usermd5file : $!\n" ;

   # parse user/md5 file
   while ( <USERMD5FILE> ) {
      chomp() ;
      ( $user, $md5key ) = split ;
      $keyuserpair{$md5key} = $user ;
   }

   # close file
   close USERMD5FILE ;
}

# ------------------------------------------------------------------ #
# delete hash pair if in suppression list
sub fetchNotSuppressed {

   # open suppression file
   open( MD5SUPP, $md5suppfile )
     or die "Can't open $md5suppfile : $!\n" ;

   # parse suppression file
   while ( <MD5SUPP> ) {
      chomp() ;
      delete $keyuserpair{$_} if exists $keyuserpair{$_} ;
   }

   # close file
   close MD5SUPP ;
}

# -------------------------------------------------------------------------- #
# Main
# ------------------------------------------------------------------ #

processUserMd5File() ;

fetchNotSuppressed() ;

foreach $md5key ( keys( %keyuserpair ) ) {
   print "$keyuserpair{$md5key}\n" ;
}

exit 0 ;

# -------------------------------------------------------------------------- #
# End

You might need to change the bold parts to your set up:
- /usr/bin/perl => path to perl,
- file1 => file with user-md5 combo's,
- file2 => suppression list (md5 keys)

Hope this helps.

ghostdog74 · 09-03-2010, 05:08 AM

Code:

awk 'FNR==NR{md5[$1];next}(!($2 in md5)){print $1}' suppression userlist