Organize Log File

0.o · 05-19-2011, 02:35 PM

Guys/Girls,

I am trying to organize a log file by user id and source IP. The data I have is in the form of the following:

Code:

user1,127.0.0.1
user2,127.0.0.2
user2,127.0.0.3
user1,127.0.0.5
...

There can be multiple source IPs associated to a single user. The way that I need the data organized is in the following format:

Code:

user1:127.0.0.1,127.0.0.5
user2:127.0.0.2,127.0.0.3
...

I have the following perl script to insert the data into a hash of arrays, but I can't seem to figure out how to put the source IPs all on one line and eliminate duplicate IPs.

Code:

#!/usr/bin/perl

use warnings;
use strict;
our %HoA = ();

open(LOG, '<', 'names_ips.csv');

while( my $line = <LOG>)
{
        my ($who, $where) = split /:\s*/, $line;
        chomp($where,$who);
        $HoA{$who} = [ $where ];
}
for my $items ( keys %HoA )
{
        print "$items,@{ $HoA{$items} }\n";
}

If someone could point me in the right direction, I would appreciate it!

Thanks!!

Nominal Animal · 05-19-2011, 04:15 PM

Perhaps someone else can help you with Perl. I'd personally just use awk for this:

Code:

awk '
BEGIN {
    RS = "[\t\n\v\f\r ]*[\n\r][\t\n\v\f\r ]*"
    FS = ",[\t\v\f ]*"
    users = 0
}

(NF > 1) {
    if (!($1 in list)) {
        user[++users] = $1
        list[$1] = ","
    }
    for (i = 2; i <= NF; i++)
        if (length($i) > 0)
            if (!index(list[$1], "," $i ","))
                list[$1] = list[$1] $i ","
}

END {
    for (u = 1; u <= users; u++) {
        thisuser = user[u]
        thislist = list[thisuser]
        sub(/^,+/, "", thislist)
        sub(/,+$/, "", thislist)
        printf("%s:%s\n", thisuser, thislist)
    }
}' logfile

This script saves user names in the user array, IP address list for each user (keyed by the user name) as a string in list array, with a comma before and after each IP address. An address is added only if it does not exist in the list; the commas make sure partial matches are not considered.

Also, note that colon (:) as a separator is a very bad idea. It makes things very difficult when you start using IPv6 addresses -- and you will, sooner or later. I recommend using either a comma or a pipe (|) instead.

theNbomr · 05-20-2011, 01:55 PM

I haven't tested this, but I think it looks better. The HoA needs elements added as they are found, for each user. When they are printed, they can be first turned into a scalar, using join.

Code:

#!/usr/bin/perl

use warnings;
use strict;
our %HoA = ();

open(LOG, '<', 'names_ips.csv');

while( my $line = <LOG>)
{
        my ($who, $where) = split /:\s*/, $line;
        chomp($where,$who);
        push @{$HoA{$who}}, $where;
}
for my $user ( keys %HoA )
{
        my $ipList = join ",", $HoA{$user};
        print "$user:$ipList\n";
}

--- rod.

Birei · 05-21-2011, 05:59 PM

Hi,

Another solution using 'Perl':

Code:

$ cat script.pl
use warnings;
use strict;
use autodie;

@ARGV == 1 or die "Usage: perl $0 file\n";

my (%user, @fields);
open my $fh, "<", $ARGV[0];

while ( <$fh> ) {
	chomp;
	@fields = split /,/;
	push @{$user{ $fields[0] }}, $fields[1] if @fields == 2 && ! ($fields[1] ~~ @{$user{ $fields[0] }});
}

for ( sort keys %user ) {
	print $_, ":", (join ",", @{$user{ $_ }}), "\n";
}

close $fh;
$ perl script.pl infile
(Output supressed)

Regards,
Birei