deleting duplicate lines without deleting first instance of the duplicated line

jkeertir · 02-07-2011, 03:31 AM

Dear All,

I have a file "test.txt" with following data

#1
aaa
#2
bbb
#3
aaa
#4
ddd

I wanted it to be displayed as

#1
aaa
#2
bbb
#4
ddd

I used awk "'!x[$0]++' test.txt > file.new"
,but it deleted #1 also.I tried using uniq command but i didn't work.
Can anyone Please let me know is there any way to do this using shell script.

With regards,
Keerti

druuna · 02-07-2011, 04:17 AM

Hi,

Is this what you are looking for:

Code:

awk 'BEGIN { RS="#" ; FS="\n" ; ORS="\r" ; OFS=" " } !($2 in array) { array[$2]; print "#" $0 }' infile

Hope this helps.

grail · 02-07-2011, 06:55 AM

and:

Code:

awk '!(NR%2) && !_[$0]++{print line,$0}{line=$0}' OFS="\n" file