LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 02-04-2012, 04:11 PM   #1
pwd
LQ Newbie
 
Registered: Feb 2012
Posts: 2

Rep: Reputation: Disabled
Post Remove duplicate lines alias


Hello all,
This is a great forum and my first time posting on it.
I would like to have a bash alias like this
Code:
alias dup="{ rm \$* && awk '!(\$0 in t){t[\$0];print}' > \$*; } < \$*"
which would remove duplicate lines from a file without sorting it first. I have tried many different combinations, but always seem to end with this error.
Code:
bash: syntax error near unexpected token `authorized_keys'
Is there a way to have an alias which you type in
Code:
dup authorized_keys
and it successfully removes the duplicate lines without sorting?
I have also replaced
Code:
awk '!(\$0 in t){t[\$0];print}'
with
Code:
perl -ne 'print unless $a{$_}++'
and there was no change. Any ideas? Thanks!
 
Old 02-04-2012, 04:39 PM   #2
Cedrik
Senior Member
 
Registered: Jul 2004
Distribution: Slackware
Posts: 2,140

Rep: Reputation: 242Reputation: 242Reputation: 242
For the perl onliner, try:
Code:
alias dup="perl -i -ne 'print unless \$a{\$_}++' \$*"
 
1 members found this post helpful.
Old 02-05-2012, 12:37 AM   #3
pwd
LQ Newbie
 
Registered: Feb 2012
Posts: 2

Original Poster
Rep: Reputation: Disabled
Thank you, Cedrik. The perl oneliner works perfectly!
 
Old 02-05-2012, 05:00 AM   #4
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957Reputation: 1957
Aliases are designed generally for simple command substitutions (if the command name matches the alias, the designated string is substituted before the line is executed), and are not really appropriate for more complex commands like this. I'd recommend setting it up as a shell function instead.

Code:
dup(){
	{ rm -f "$1" && awk '!( $0 in t ){ t[$0] ; print }' > "$1" ;} <"$1"
}
Edit: You could also simply set your command up as a stand-alone script, and either alias your command name to the script location, or just put/symlink the script somewhere in your path. I see no real reason why you have to clutter up your shell environment with a command like this.

Last edited by David the H.; 02-05-2012 at 05:21 AM. Reason: as stated
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
remove duplicate lines from shell script himu3118 Programming 16 04-21-2012 08:37 AM
[SOLVED] Using sed to remove lines with duplicate ID's, but different endings... wapitismith Linux - Newbie 4 05-08-2010 01:30 PM
Delete Duplicate Lines in a file, leaving only the unique lines left xmrkite Linux - Software 6 01-14-2010 07:18 PM
script to remove the lines which are having the duplicate value in 2 fields ajcapri Linux - Newbie 10 11-29-2009 10:38 PM
[SOLVED] uniq -u : does not seem to remove duplicate lines boxb29 Linux - General 7 08-15-2009 07:34 PM


All times are GMT -5. The time now is 06:47 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration