Welcome to the most active Linux Forum on the web.
Go Back > Forums > Non-*NIX Forums > Programming
User Name
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.


  Search this Thread
Old 08-07-2019, 07:17 AM   #1
LQ Newbie
Registered: Apr 2019
Posts: 12

Rep: Reputation: Disabled
How to put the command to remove duplicate lines in my awk script ?

I create a CGI in bash/html.

My awk script looks like :
echo "<table>"
for fn in /var/www/cgi-bin/LPAR_MAP/*;
echo "<td>"
echo "<PRE>"

awk -F',|;' -v test="$test" '
     NR==1 { 
        split(FILENAME ,a,"[-.]");
      $0 ~ test {
              print "DATE ========================== : " a[4] 
          print ""
          print "LPARS :" $2
          print "RAM : " $5
          print "CPU 1 : " $6
          print "CPU 2 : " $7
          print "" 
          print ""
      }' $fn;

echo "</PRE>"
echo "</td>"
echo "</table>"

This script allow to analyze 276 csv files that looks like :
MO2PPC21;mplaix0311;Running;AIX 7.1 7100-05-02-1832;35;0.6;4;DefaultPool;shared;uncap;64
MO2PPC21;miaibv194;Running;AIX 6.1 6100-09-11-1810;11;0.2;1;DefaultPool;shared;uncap;64
MO2PPC21;mplaix0032;Running;AIX 6.1 6100-09-11-1810;105;4.0;11;DefaultPool;shared;uncap;128
MO2PPC21;miaibv238;Running;AIX 7.1 7100-05-02-1810;10;0.5;1;DefaultPool;shared;uncap;64
MO2PPC21;miaibv193;Running;AIX 6.1 6100-09-11-1810;12;0.2;1;DefaultPool;shared;uncap;64
MO1PPC17;miaibe03;Running;AIX 5.2 5200-10-08-0930;25;null;3;null;ded;share_idle_procs;null
MO1PPC17;miaiba12;Running;AIX 5.2 5200-10-08-0930;17;null;2;null;ded;share_idle_procs;null
MO1PPC17;miaibf03;Running;AIX 5.2 5200-10-08-0930;30;null;3;null;ded;share_idle_procs;null
MO1PPC17;miaibc05;Running;AIX 5.2 5200-10-08-0930;40;null;2;null;ded;share_idle_procs;null
And to display them in my CGI like this :

The numbers of columns is equal at the number of csv to analyze.

As you can see in the screenshot, some lines are sometimes the same in each csv files

The idea is to delete the lines that are the same in all my csv files.

I know the awk command :
awk '!a[$0]++'
But this command needs a file to be achieved.

Someone helped me and tell me to do that :
or fn in /var/www/cgi-bin/LPAR_MAP/*; do awk -F',|;' -v test="$test" '
     NR==1 { 
        split(FILENAME ,a,"[-.]");
      ($0 ~ test) && (!dup[$0]++) {
              print "DATE ========================== : " a[4] 
          print ""
          print "LPARS :" $2
          print "RAM : " $5
          print "CPU 1 : " $6
          print "CPU 2 : " $7
          print "" 
          print ""
      }' $fn; done
But it doesn't work... It's last thing that I need to complete this script...

Do you have any idea to do that ?
Old 08-07-2019, 07:41 AM   #2
LQ Guru
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,795
Blog Entries: 15

Rep: Reputation: 1645Reputation: 1645Reputation: 1645Reputation: 1645Reputation: 1645Reputation: 1645Reputation: 1645Reputation: 1645Reputation: 1645Reputation: 1645Reputation: 1645
You could just pipe the output into "sort -u" so it displays only unique lines.


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: How To Empty a File, Delete N Lines From a File, Remove Matching String From a File, And Remove Empty/Blank Lines From a File In Linux LXer Syndicated Linux News 0 11-22-2017 12:30 PM
remove duplicate lines from shell script himu3118 Programming 16 04-21-2012 07:37 AM
Delete Duplicate Lines in a file, leaving only the unique lines left xmrkite Linux - Software 6 01-14-2010 06:18 PM
script to remove the lines which are having the duplicate value in 2 fields ajcapri Linux - Newbie 10 11-29-2009 09:38 PM
[SOLVED] uniq -u : does not seem to remove duplicate lines boxb29 Linux - General 7 08-15-2009 06:34 PM > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 11:59 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration