LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
LinkBack Search this Thread
Old 02-07-2011, 03:31 AM   #1
jkeertir
Member
 
Registered: Mar 2008
Posts: 70

Rep: Reputation: 15
deleting duplicate lines without deleting first instance of the duplicated line


Dear All,

I have a file "test.txt" with following data

#1
aaa
#2
bbb
#3
aaa
#4
ddd


I wanted it to be displayed as

#1
aaa
#2
bbb
#4
ddd

I used awk "'!x[$0]++' test.txt > file.new"
,but it deleted #1 also.I tried using uniq command but i didn't work.
Can anyone Please let me know is there any way to do this using shell script.

With regards,
Keerti
 
Old 02-07-2011, 04:17 AM   #2
druuna
LQ Veteran
 
Registered: Sep 2003
Posts: 10,532
Blog Entries: 7

Rep: Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371Reputation: 2371
Hi,

Is this what you are looking for:
Code:
awk 'BEGIN { RS="#" ; FS="\n" ; ORS="\r" ; OFS=" " } !($2 in array) { array[$2]; print "#" $0 }' infile
Hope this helps.
 
Old 02-07-2011, 06:55 AM   #3
grail
Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 7,202

Rep: Reputation: 1796Reputation: 1796Reputation: 1796Reputation: 1796Reputation: 1796Reputation: 1796Reputation: 1796Reputation: 1796Reputation: 1796Reputation: 1796Reputation: 1796
and:
Code:
awk '!(NR%2) && !_[$0]++{print line,$0}{line=$0}' OFS="\n" file
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Deleting first line with unique string then all subsequent lines thereafter chriscbe Linux - Newbie 5 01-21-2011 11:58 PM
deleting duplicate files cs-cam Linux - General 3 11-14-2006 11:27 PM
Bash - Deleting duplicate records Wire323 Programming 5 12-04-2005 08:51 AM
Deleting duplicate messages essdeeay Linux - General 1 11-20-2005 07:58 AM


All times are GMT -5. The time now is 10:22 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration