LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 08-05-2020, 04:37 AM   #1
ghpradeep
Member
 
Registered: Mar 2011
Posts: 32

Rep: Reputation: 0
need to remove duplicate row from file


Hello,

I have below file output. I need to remove duplicate row starting with CIS

each cis setting will come from different server. I

CIS:1.1.2;PASS;CIS:1.1.2;PASS;CIS:1.1.2;PASS
CIS:1.1.3;PASS;CIS:1.1.3;PASS;CIS:1.1.3;PASS
CIS:1.1.4;PASS;CIS:1.1.4;PASS;CIS:1.1.4;PASS
CIS:1.1.5;PASS;CIS:1.1.5;PASS;CIS:1.1.5;PASS
CIS:1.1.6;PASS;CIS:1.1.6;PASS;CIS:1.1.6;PASS

i want output like this

CIS:1.1.2;PASS;PASS;PASS
CIS:1.1.3;PASS;PASS;PASS
CIS:1.1.4;PASS;PASS;PASS
CIS:1.1.5;PASS;PASS;PASS
CIS:1.1.6;PASS;PASS;PASS
 
Old 08-05-2020, 05:03 AM   #2
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,068
Blog Entries: 3

Rep: Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523
It's rather easy with AWK. What have you tried so far. Please post the code and say where you are stuck.
 
1 members found this post helpful.
Old 08-05-2020, 05:04 AM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 19,104

Rep: Reputation: 3327Reputation: 3327Reputation: 3327Reputation: 3327Reputation: 3327Reputation: 3327Reputation: 3327Reputation: 3327Reputation: 3327Reputation: 3327Reputation: 3327
Not to mention "cut" - online search didn't return any candidates ?.
 
1 members found this post helpful.
Old 08-05-2020, 05:59 AM   #4
ghpradeep
Member
 
Registered: Mar 2011
Posts: 32

Original Poster
Rep: Reputation: 0
Hello,

I tried to split content into multiple file like below.
1.txt
CIS:1.1.2;PASS
CIS:1.1.3;PASS
CIS:1.1.4;PASS
CIS:1.1.6;PASS

2.txt
CIS:1.1.2;FAIL
CIS:1.1.3;FAIL
CIS:1.1.4;FAIL
CIS:1.1.6;FAIL


awk -F ';' 'FNR==NR{a[$1]=$2;next} {print $1, $2, a[$1]}' 2.txt 1.txt

i am able to achieved below

CIS:1.1.2 PASS FAIL
CIS:1.1.3 PASS FAIL
CIS:1.1.4 PASS FAIL
CIS:1.1.6 PASS FAIL


but when i am trying to work on single file i am not able to get require output. in future my main file will contain multiple server CIS details.
 
Old 08-05-2020, 06:26 AM   #5
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,068
Blog Entries: 3

Rep: Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523Reputation: 2523
That's a different question, but one that would be easily solvable with join if you look at the -t option.
 
Old 08-06-2020, 10:33 AM   #6
MadeInGermany
Senior Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 1,457

Rep: Reputation: 664Reputation: 664Reputation: 664Reputation: 664Reputation: 664Reputation: 664
Quote:
Originally Posted by ghpradeep View Post
Hello,

I have below file output. I need to remove duplicate row starting with CIS

each cis setting will come from different server. I

CIS:1.1.2;PASS;CIS:1.1.2;PASS;CIS:1.1.2;PASS
CIS:1.1.3;PASS;CIS:1.1.3;PASS;CIS:1.1.3;PASS
CIS:1.1.4;PASS;CIS:1.1.4;PASS;CIS:1.1.4;PASS
CIS:1.1.5;PASS;CIS:1.1.5;PASS;CIS:1.1.5;PASS
CIS:1.1.6;PASS;CIS:1.1.6;PASS;CIS:1.1.6;PASS

i want output like this

CIS:1.1.2;PASS;PASS;PASS
CIS:1.1.3;PASS;PASS;PASS
CIS:1.1.4;PASS;PASS;PASS
CIS:1.1.5;PASS;PASS;PASS
CIS:1.1.6;PASS;PASS;PASS
With sed:
Code:
sed 's/;CIS:[^;]*//g' file
 
1 members found this post helpful.
Old 08-06-2020, 11:26 PM   #7
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: distro hopper
Posts: 9,784

Rep: Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411Reputation: 4411
I only read the top post:
Code:
❯ cat cis.txt 
CIS:1.1.2;PASS;CIS:1.1.2;PASS;CIS:1.1.2;PASS
CIS:1.1.3;PASS;CIS:1.1.3;PASS;CIS:1.1.3;PASS
CIS:1.1.4;PASS;CIS:1.1.4;PASS;CIS:1.1.4;PASS
CIS:1.1.5;PASS;CIS:1.1.5;PASS;CIS:1.1.5;PASS
CIS:1.1.6;PASS;CIS:1.1.6;PASS;CIS:1.1.6;PASS

❯ cat cis.txt | cut -d\; -f1,2,4,6
CIS:1.1.2;PASS;PASS;PASS
CIS:1.1.3;PASS;PASS;PASS
CIS:1.1.4;PASS;PASS;PASS
CIS:1.1.5;PASS;PASS;PASS
CIS:1.1.6;PASS;PASS;PASS

Last edited by dugan; 08-06-2020 at 11:27 PM.
 
Old 08-09-2020, 12:15 AM   #8
grail
LQ Guru
 
Registered: Sep 2009
Location: Perth
Distribution: Manjaro
Posts: 9,796

Rep: Reputation: 3065Reputation: 3065Reputation: 3065Reputation: 3065Reputation: 3065Reputation: 3065Reputation: 3065Reputation: 3065Reputation: 3065Reputation: 3065Reputation: 3065
Better get an awk in there now we are posting solutions:
Code:
awk -F';CIS[^;]*' '$1=$1' OFS="" file
awk 'gsub(/;CIS[^;]*/,"")' file
 
Old 08-10-2020, 08:19 PM   #9
individual
Member
 
Registered: Jul 2018
Posts: 292
Blog Entries: 1

Rep: Reputation: 223Reputation: 223Reputation: 223
Here's my solution in Perl.
Code:
perl -F';' -E 'print "$F[0];" . join(";", grep {$_ ne $F[0]} @F[1..$#F])' data.txt
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
how to make the output in multiple row become single row? ahpin Linux - Newbie 4 07-19-2013 07:49 AM
[SOLVED] find a null value in a row/column and delete entire row umix Linux - Newbie 10 10-13-2011 01:26 AM
Remove duplicate entries on a row sebelk Programming 2 11-01-2010 09:43 AM
gnome-terminal is missing one pixel row and its an important row rednuht Linux - General 1 12-24-2009 10:30 AM
Shell script to parse csv-like output, row by row utahnix Linux - General 8 12-08-2007 05:03 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 07:26 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration