LinuxQuestions.org
Go Job Hunting at the LQ Job Marketplace
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices



Reply
 
Search this Thread
Old 11-27-2009, 02:11 AM   #1
ajcapri
LQ Newbie
 
Registered: Nov 2009
Posts: 8

Rep: Reputation: 0
script to remove the lines which are having the duplicate value in 2 fields


How to write a script to remove the lines which are having the duplicate value in 1st field and last field and redirect to another file, the file name should have the current time stamp.
 
Old 11-27-2009, 02:45 AM   #2
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,563
Blog Entries: 29

Rep: Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179
Please post a sample of the data, the desired output and what you have tried so far.
 
Old 11-27-2009, 03:30 AM   #3
ajcapri
LQ Newbie
 
Registered: Nov 2009
Posts: 8

Original Poster
Rep: Reputation: 0
I dint know how to proceed with writing the script as i am new to unix.

the sample data could be something like

1|a|aaaa|11
2|b|bbbb|222
3|c|cccc|333
2|d|dddd|333
3|e|aaaa|222


and the o/p i wish to get wud be

1|a|aaaa|11
2|b|bbbb|222
3|c|cccc|333
 
Old 11-27-2009, 04:07 AM   #4
ajcapri
LQ Newbie
 
Registered: Nov 2009
Posts: 8

Original Poster
Rep: Reputation: 0
also I need to preserve the order...and what if the fields i need to be checked(1st and last) for duplication are alpha numeric
 
Old 11-27-2009, 04:17 AM   #5
vinaytp
Member
 
Registered: Apr 2009
Location: Bengaluru, India
Distribution: RHEL 5.4, 6.0, Ubuntu 10.04
Posts: 704

Rep: Reputation: 55
Hello ajcapri,

I have just tried out a code....Hope its working just check

Code:
#!/bin/bash
file=$1
while read line
do
a=`echo $line | cut -d '|' -f 1`
b=`echo $line | cut -d '|' -f 4`
if [ `echo $b | grep $a` ]
then
echo $line >> temp
touch temp
fi
done < $file
Execute it this way...

Code:
sh filename.sh /path/to/the/file/where/pattern/is/stored
Hope it helps....

Cheers !!!
 
Old 11-27-2009, 06:11 AM   #6
ajcapri
LQ Newbie
 
Registered: Nov 2009
Posts: 8

Original Poster
Rep: Reputation: 0
how do i do de same in case i dont know de number of fields...in that situation how do i find de $n value for last field??
 
Old 11-27-2009, 06:51 AM   #7
nagendrar
Member
 
Registered: Apr 2008
Location: HYD, INDIA.
Posts: 146

Rep: Reputation: 15
Write below code in filename.sh

#!/bin/bash
file=$1
while read line
do
len=`echo ${#line}`
name1=`echo "${line:0:1}"`
name2=`echo "${line:$len-1:$len}"`
if [ $name1 == $name2 ]
then
echo $line >> temp
touch temp
fi
done < $file

Run this as follows :
sh filename.sh /path/to/the/file/where/pattern/is/store


Regards,
Nagendra Rednam
 
Old 11-27-2009, 08:11 AM   #8
ajcapri
LQ Newbie
 
Registered: Nov 2009
Posts: 8

Original Poster
Rep: Reputation: 0
i guess the codes u ppl have posted is fore comparing values in de first n last fields...i wat to delete records based on duplicate values in de first field and also based on last field...many thanks for ppl who ve been helping
 
Old 11-27-2009, 08:15 AM   #9
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Servers: Debian Squeeze and Wheezy. Desktop: Slackware64 14.0. Netbook: Slackware 13.37
Posts: 8,563
Blog Entries: 29

Rep: Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179Reputation: 1179
Quote:
Originally Posted by ajcapri View Post
i guess the codes u ppl have posted is fore comparing values in de first n last fields...i wat to delete records based on duplicate values in de first field and also based on last field...many thanks for ppl who ve been helping
Wot u "duplicate values in de first field"? Field wi XX wd hv dup X. K?
 
Old 11-28-2009, 03:36 AM   #10
bsat
Member
 
Registered: Feb 2009
Posts: 347

Rep: Reputation: 72
Try this awk script
you can put it into a file, say "awk_script" and your data in a file say "data".

BEGIN{FS="|"}
{
str=substr($NF,1,1)
if($1==str)
{
print $0
}
}

run it as follows

awk -f awk_script data
 
Old 11-29-2009, 10:38 PM   #11
ajcapri
LQ Newbie
 
Registered: Nov 2009
Posts: 8

Original Poster
Rep: Reputation: 0
by duplicates i mean..

abc|aaa|bbb|ccc
def|ddd|eee|fff
ghi|ddd|eee|ccc
abc|ggg|hhh|iii
ghi|rrr|sss|fff


for this sample data, i need the last 2 rows removed as the value in the first field ("abc" and "ghi") are already present in the previous scanned rows. Further more, i need the 3rd and lst row removed as the values present in the last field ("ccc" and "fff") are duplicates.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] uniq -u : does not seem to remove duplicate lines boxb29 Linux - General 7 08-15-2009 07:34 PM
bash script to remove the blank lines in the file naveensankineni Programming 7 03-25-2008 09:34 PM
How to remove first 2 lines of a file in a script nazs Programming 16 02-19-2007 08:08 AM
awk to remove first 3 lines and print remaining $1, $2 fields phyx Linux - General 1 01-10-2007 06:21 PM
Make a script remove lines from a file? spiffytech Linux - Software 5 12-29-2005 12:50 PM


All times are GMT -5. The time now is 05:22 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration