need to remove duplicate row from file

Hello,

I have below file output. I need to remove duplicate row starting with CIS

each cis setting will come from different server. I

CIS:1.1.2;PASS;CIS:1.1.2;PASS;CIS:1.1.2;PASS
CIS:1.1.3;PASS;CIS:1.1.3;PASS;CIS:1.1.3;PASS
CIS:1.1.4;PASS;CIS:1.1.4;PASS;CIS:1.1.4;PASS
CIS:1.1.5;PASS;CIS:1.1.5;PASS;CIS:1.1.5;PASS
CIS:1.1.6;PASS;CIS:1.1.6;PASS;CIS:1.1.6;PASS

i want output like this

CIS:1.1.2;PASS;PASS;PASS
CIS:1.1.3;PASS;PASS;PASS
CIS:1.1.4;PASS;PASS;PASS
CIS:1.1.5;PASS;PASS;PASS
CIS:1.1.6;PASS;PASS;PASS

It's rather easy with AWK. What have you tried so far. Please post the code and say where you are stuck.

Not to mention "cut" - online search didn't return any candidates ?.

Hello,

I tried to split content into multiple file like below.
1.txt
CIS:1.1.2;PASS
CIS:1.1.3;PASS
CIS:1.1.4;PASS
CIS:1.1.6;PASS

2.txt
CIS:1.1.2;FAIL
CIS:1.1.3;FAIL
CIS:1.1.4;FAIL
CIS:1.1.6;FAIL


awk -F ';' 'FNR==NR{a[$1]=$2;next} {print $1, $2, a[$1]}' 2.txt 1.txt

i am able to achieved below

CIS:1.1.2 PASS FAIL
CIS:1.1.3 PASS FAIL
CIS:1.1.4 PASS FAIL
CIS:1.1.6 PASS FAIL


but when i am trying to work on single file i am not able to get require output. in future my main file will contain multiple server CIS details.

That's a different question, but one that would be easily solvable with join if you look at the -t option.

Originally Posted by ghpradeep Hello, I have below file output. I need to remove duplicate row starting with CIS each cis setting will come from different server. I CIS:1.1.2;PASS;CIS:1.1.2;PASS;CIS:1.1.2;PASS CIS:1.1.3;PASS;CIS:1.1.3;PASS;CIS:1.1.3;PASS CIS:1.1.4;PASS;CIS:1.1.4;PASS;CIS:1.1.4;PASS CIS:1.1.5;PASS;CIS:1.1.5;PASS;CIS:1.1.5;PASS CIS:1.1.6;PASS;CIS:1.1.6;PASS;CIS:1.1.6;PASS i want output like this CIS:1.1.2;PASS;PASS;PASS CIS:1.1.3;PASS;PASS;PASS CIS:1.1.4;PASS;PASS;PASS CIS:1.1.5;PASS;PASS;PASS CIS:1.1.6;PASS;PASS;PASS

With sed:

sed 's/;CIS:[^;]*//g' file

I only read the top post:

❯ cat cis.txt 
CIS:1.1.2;PASS;CIS:1.1.2;PASS;CIS:1.1.2;PASS
CIS:1.1.3;PASS;CIS:1.1.3;PASS;CIS:1.1.3;PASS
CIS:1.1.4;PASS;CIS:1.1.4;PASS;CIS:1.1.4;PASS
CIS:1.1.5;PASS;CIS:1.1.5;PASS;CIS:1.1.5;PASS
CIS:1.1.6;PASS;CIS:1.1.6;PASS;CIS:1.1.6;PASS

❯ cat cis.txt | cut -d\; -f1,2,4,6
CIS:1.1.2;PASS;PASS;PASS
CIS:1.1.3;PASS;PASS;PASS
CIS:1.1.4;PASS;PASS;PASS
CIS:1.1.5;PASS;PASS;PASS
CIS:1.1.6;PASS;PASS;PASS

Better get an awk in there now we are posting solutions:

awk -F';CIS[^;]*' '$1=$1' OFS="" file
awk 'gsub(/;CIS[^;]*/,"")' file

Here's my solution in Perl.

perl -F';' -E 'print "$F[0];" . join(";", grep {$_ ne $F[0]} @F[1..$#F])' data.txt