LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Programming (https://www.linuxquestions.org/questions/programming-9/)
-   -   need to remove duplicate row from file (https://www.linuxquestions.org/questions/programming-9/need-to-remove-duplicate-row-from-file-4175679929/)

ghpradeep 08-05-2020 04:37 AM

need to remove duplicate row from file
 
Hello,

I have below file output. I need to remove duplicate row starting with CIS

each cis setting will come from different server. I

CIS:1.1.2;PASS;CIS:1.1.2;PASS;CIS:1.1.2;PASS
CIS:1.1.3;PASS;CIS:1.1.3;PASS;CIS:1.1.3;PASS
CIS:1.1.4;PASS;CIS:1.1.4;PASS;CIS:1.1.4;PASS
CIS:1.1.5;PASS;CIS:1.1.5;PASS;CIS:1.1.5;PASS
CIS:1.1.6;PASS;CIS:1.1.6;PASS;CIS:1.1.6;PASS

i want output like this

CIS:1.1.2;PASS;PASS;PASS
CIS:1.1.3;PASS;PASS;PASS
CIS:1.1.4;PASS;PASS;PASS
CIS:1.1.5;PASS;PASS;PASS
CIS:1.1.6;PASS;PASS;PASS

Turbocapitalist 08-05-2020 05:03 AM

It's rather easy with AWK. What have you tried so far. Please post the code and say where you are stuck.

syg00 08-05-2020 05:04 AM

Not to mention "cut" - online search didn't return any candidates ?.

ghpradeep 08-05-2020 05:59 AM

Hello,

I tried to split content into multiple file like below.
1.txt
CIS:1.1.2;PASS
CIS:1.1.3;PASS
CIS:1.1.4;PASS
CIS:1.1.6;PASS

2.txt
CIS:1.1.2;FAIL
CIS:1.1.3;FAIL
CIS:1.1.4;FAIL
CIS:1.1.6;FAIL


awk -F ';' 'FNR==NR{a[$1]=$2;next} {print $1, $2, a[$1]}' 2.txt 1.txt

i am able to achieved below

CIS:1.1.2 PASS FAIL
CIS:1.1.3 PASS FAIL
CIS:1.1.4 PASS FAIL
CIS:1.1.6 PASS FAIL


but when i am trying to work on single file i am not able to get require output. in future my main file will contain multiple server CIS details.

Turbocapitalist 08-05-2020 06:26 AM

That's a different question, but one that would be easily solvable with join if you look at the -t option.

MadeInGermany 08-06-2020 10:33 AM

Quote:

Originally Posted by ghpradeep (Post 6152825)
Hello,

I have below file output. I need to remove duplicate row starting with CIS

each cis setting will come from different server. I

CIS:1.1.2;PASS;CIS:1.1.2;PASS;CIS:1.1.2;PASS
CIS:1.1.3;PASS;CIS:1.1.3;PASS;CIS:1.1.3;PASS
CIS:1.1.4;PASS;CIS:1.1.4;PASS;CIS:1.1.4;PASS
CIS:1.1.5;PASS;CIS:1.1.5;PASS;CIS:1.1.5;PASS
CIS:1.1.6;PASS;CIS:1.1.6;PASS;CIS:1.1.6;PASS

i want output like this

CIS:1.1.2;PASS;PASS;PASS
CIS:1.1.3;PASS;PASS;PASS
CIS:1.1.4;PASS;PASS;PASS
CIS:1.1.5;PASS;PASS;PASS
CIS:1.1.6;PASS;PASS;PASS

With sed:
Code:

sed 's/;CIS:[^;]*//g' file

dugan 08-06-2020 11:26 PM

I only read the top post:
Code:

❯ cat cis.txt
CIS:1.1.2;PASS;CIS:1.1.2;PASS;CIS:1.1.2;PASS
CIS:1.1.3;PASS;CIS:1.1.3;PASS;CIS:1.1.3;PASS
CIS:1.1.4;PASS;CIS:1.1.4;PASS;CIS:1.1.4;PASS
CIS:1.1.5;PASS;CIS:1.1.5;PASS;CIS:1.1.5;PASS
CIS:1.1.6;PASS;CIS:1.1.6;PASS;CIS:1.1.6;PASS

❯ cat cis.txt | cut -d\; -f1,2,4,6
CIS:1.1.2;PASS;PASS;PASS
CIS:1.1.3;PASS;PASS;PASS
CIS:1.1.4;PASS;PASS;PASS
CIS:1.1.5;PASS;PASS;PASS
CIS:1.1.6;PASS;PASS;PASS


grail 08-09-2020 12:15 AM

Better get an awk in there now we are posting solutions:
Code:

awk -F';CIS[^;]*' '$1=$1' OFS="" file
awk 'gsub(/;CIS[^;]*/,"")' file


individual 08-10-2020 08:19 PM

Here's my solution in Perl.
Code:

perl -F';' -E 'print "$F[0];" . join(";", grep {$_ ne $F[0]} @F[1..$#F])' data.txt


All times are GMT -5. The time now is 07:59 PM.