remove records which have 2 same fields

oreka18 · 05-31-2012, 04:19 AM

how can i remove records which have 2 same fields?
my file:

Code:

saeed 1 2 sa
vahid 2 3 45
reza 212 33 sa
amir 1 1 ui
reza 21 33 sa

i want to remove records which first and 3rd field of that are as the same, here line 3 and 5 must be removed.

CTM · 05-31-2012, 07:13 AM

edit: code removed, as it doesn't do exactly what the OP asked (thanks danielbmartin for pointing that out). See grail's solution.

danielbmartin · 05-31-2012, 07:52 AM

Quote:

Originally Posted by CTM

Here's a Perl solution ...

With due respect, OP said lines 3 and 5 must be removed. Your solution removed only line 5.

Daniel B. Martin

grail · 05-31-2012, 09:02 AM

Is the duplicate entry only likely to occur in pairs or is there no certainty behind the repeats?

Below works only for even repetition:

Code:

awk '$1$3 in a{delete a[$1$3];next}{a[$1$3]=$0}END{for(i in a)print a[i]}' file

EDIT: Just came to me, simple change should work for any number of repeats:

Code:

awk '$1$3 in a{a[$1$3]="";next}{a[$1$3]=$0}END{for(i in a)if(a[i])print a[i]}' file

colucix · 05-31-2012, 10:16 AM

Another (similar) way:

Code:

awk 'FNR==NR {_[$1$3]++} FNR<NR && !(_[$1$3]-1)' file file

danielbmartin · 05-31-2012, 10:24 AM

Quote:

Originally Posted by grail

Code:

awk '$1$3 in a{a[$1$3]="";next}{a[$1$3]=$0}END{for(i in a)if(a[i])print a[i]}' file

Love this solution... but I put on my "tester" hat and found a flaw. I constructed an input file which is an extension of that provided by OP.

Code:

saeed 1 2 sa
vahid 2 3 45
reza 212 33 sa (must toss)
amir 1 1 ui
reza 21 33 sa (must toss)
phil19 34 37 p3
jack22 41 47 jk
dave7 28 31 de
phil1 34 937 p4
dick7 42 55 c9

Note that there are two fellows named "phil" but they are distinct. One is phil19 (on line 6); the other is phil1 (on line 9). Lines 3 and 5 are the only lines which should be tossed. Your solution uses fields 1 and 3 (catenated) as an array subscript. My test input file shows how that can lead to trouble.

Perhaps an array subscript which inserts "^" or some other special character will avoid this minor hazard.

Daniel B. Martin

danielbmartin · 05-31-2012, 11:12 AM

Quote:

Originally Posted by colucix

Code:

awk 'FNR==NR {_[$1$3]++} FNR<NR && !(_[$1$3]-1)' file file

A minor change avoids the "two phils" pitfall cited previously.

Code:

awk 'FNR==NR {_[$1 $3]++} FNR<NR && !(_[$1 $3]-1)' $InFile $InFile

Daniel B. Martin

grail · 05-31-2012, 01:13 PM

I would probably use SUBSEP as it won't then get mixed up with another such curve ball allowing for extraneous insertions:

Code:

awk '$1SUBSEP$3 in a{a[$1,$3]="";next}{a[$1,$3]=$0}END{for(i in a)if(a[i])print a[i]}' file