hi all ,

I have a dilemma I hope you can help me solve.

I have a largish text file that I want to process.

The file has 4 colums separated by tab. so it looks like this.


fred john dave pete
dave pete terry phil
john dave pete fred

I would like to remove all lines where there are more than one duplicate entry in column 4.

I am not looking to remove duplicates, I want to completely remove the lines that have more than one entry in column 4.

So if 2 or more entries in colum 4 are the same, remove those two rows. I want to be left with only rows who only ever had a single entry in column 4.

Could you help with this? If so, thanks in advance.

seabro

ps. I am using Centos so I guess tools like grep and awk might do it I just dont know how.

As you describe the file and as you show by example, column 4 consists of one word. Consequently it is impossible to have a duplicate entry in column 4. Please elaborate or correct your problem statement.

It is always helpful to give two sample files, a Before and an After. You gave a Before but no After.

Daniel B. Martin

hi Daniel,

thanks for your reply.

I guess I forgot to add that the test of duplication should be vertically on column 4

before

bob jon fred pete
fred john phil dave
mike phil john pete
fred jack bob seabro

after

fred john phil dave
fred jack bob seabro


I hope that makes more sense!
thanks again,
Seabro

The double argument is not an error: it causes awk to process the file twice: the first time it counts how many repetitions of the name in field 4 are there, the second time it prints out only lines with unique names. Hope this helps.

3 members found this post helpful.

Use sort with the selection of the fourth field and remove with uniq the lines with the duplicates.

3 members found this post helpful.

Thanks Guys, that worked a treat!