Need a perl regexp master

sal_paradise42 · 10-12-2007, 11:12 AM

perl noob hoping to write a script to add values to a sql db at work.
Here is the relevant part of code

Code:

while (<file2>) {
 40 if (my ($mainbit) =
 41   m{
 42 ^
 43   \s*
 44   (?:\b(?:\d{1,3}\.){3}\d{1,3}\b)
 45   /
 46   (\d{1,2})
 47   \s is \s subnetted, \s
 48   \d+
 49   \s subnets \s*
 50   .*
 51 $
 52 }x) {
 53 $lastbit = $mainbit;
 54 } elsif (my ($type, $default, $subtype, $netip, $subnetbits, $ad, $metric, $destip, $uptime, $int) =
 55 m{
 56  ^
 57  ([CSIRMBDOE])          # type
 58  (\*)? \s+              # is default?
 59   ((?:EX)|(?:IA)|(?:N1)|(?:N2)|(?:E1)|(?:E2)|(?:L1)|(?:L2)|(?:ia))? \s*   #sub type
 60   (\b(?:\d{1,3}\.){3}\d{1,3}\b) #Network address
 61   (?:
 62     / 
 63    (\d{1,2})   # subnet bits
 64    )?
 65    \s
 66    \[(\d{1,3}) # adminsitrative distance
 67    /        
 68   (\d{1,10})\]  # metric
 69   \s via \s
 70   (\b(?:\d{1,3}\.){3}\d{1,3}\b)  #destaddr
 71   (?:
 72    , \s
 73                 ([^,]+)         # uptime                                                                      
 74                 , \s                                                                                          
 75                 ([^\s]+)  # Exit interface                                                                    
 76               )?                                                                                              
 77   .*                                                                                                          
 78            $                                                                                                  
 79            }x) {                                                                                              
 80 $default = "" unless defined $default;     

 81           $subtype = "" unless defined $subtype;                                                              
 82           $uptime = "" unless defined $uptime;                                                                
 83           $int = "" unless defined $int;                                                                      
 84           $subnetbits = $lastbit unless defined $subnetbits;
 85           $count++;                                                                                           
 86           push(@networks, "$netip/$subnetbits");                                                              
 87           #print "$type\t$netip\t$destip\n";                                                                  
 88 }                                                                                                             
 89 }

Basically im trying to extract individual values from the following output

Code:

 4 7600-UT01#show ip route vrf vrfData ospf
  5      69.0.0.0/8 is variably subnetted, 184 subnets, 7 masks
  6 O E2    69.4.191.252/30 
  7            [110/20] via 10.136.217.2, 00:28:33, Serial1/1.7/26:0.1
  8 O E2    69.4.191.244/30 
  9            [110/20] via 10.136.215.2, 00:28:33, Serial1/1.7/24:0.1
 10 O E2    69.4.191.240/30 
 11            [110/20] via 10.136.214.2, 00:28:33, Serial1/1.7/23:0.1
 12 O E2    69.4.191.220/30 
 13            [110/20] via 10.136.236.2, 00:28:33, Serial1/1.8/13:0.1
 14 O E2    69.4.191.216/30 [110/20] via 10.136.232.2, 00:28:33, Serial1/1.8/9:0.1
 15 O E2    69.4.191.212/30 
 16            [110/20] via 10.136.203.2, 00:28:33, Serial1/1.7/12:0.1
 17 O E2    69.4.191.208/30 [110/20] via 10.136.225.2, 00:28:33, Serial1/1.8/2:0.1
 18 O E2    69.4.191.192/28 [110/20] via 10.137.8.2, 00:28:33, Serial1/1.3/9:0.1
 19 O E2    69.4.191.176/28 [110/20] via 10.136.226.2, 00:28:33, Serial1/1.8/3:0.1
 20 O E2    69.4.191.168/29 
 21            [110/20] via 10.136.205.2, 00:28:33, Serial1/1.7/14:0.1

It works great for lines where the output is only one line like line 18 of the previous output, but it doesn't match line 20 and 21 because they are in two different lines. So I am looking for one or two solutions.
Have $_ =~ match on a multiline like that and make it one line, or change my regexp to also match on those lines somehow.
Thanks in advance.

ghostdog74 · 10-12-2007, 01:30 PM

Code:

while (<>) {
    chomp;      # strip record separator
    if ($. > 1) {
        $Last = $line;
        $line = $_;
    }
    if (!/O E2/) {
        $get = sprintf('%s %s', $Last, $line);        
        @l = split(" ", $get);
        print $l[2] . $l[5]; #change as necesary
    }
}

sal_paradise42 · 10-13-2007, 01:05 AM

pretty sure this is over my head.
What changes do I need to make exactly?
When I use that piece of code alone I don't match on any of the stuff I was matching on before, but it does change the desired lines the way I want to.
Is there a way I can make a change to *only* those lines right after "while <file2>" ?
i.e s/^O (?:E2)? etc\n\s+/$1 $2/ and have the rest of my code work with that?

Sorry this is like my first major undertaking into perl.

ghostdog74 · 10-13-2007, 03:42 AM

use it on the command line with that file you are parsing

eg

# perl myscript.pl "file2"

to see the outcome at various points of the code, you can insert some print statements

Code:

while (<>) {
    chomp;      # strip record separator
    if ($. > 1) { #skip first line 69.0.0.0/8 is variably subnetted, 184 subnets, 7 masks
        $Last = $line;
        $line = $_;
    }
    if (!/O E2/) {
        $get = sprintf('%s %s', $Last, $line);       
        print $get; # see what's going on..
        @l = split(" ", $get); #you know what split is don't you?
        print $l[2] . $l[5]; #change as necesary as to the fields you want to get.
    }
}

sal_paradise42 · 10-13-2007, 02:48 PM

thanks for the help ghostdog. That clears up a few things, but I feel like im full circle again. It seems that I ran into two new issues.
In your recommended script it only prints out the lines that match !/^O E2/, so I figure to put a "print $_\n" after the if statement, but it prints out the lines twice.
so if I have the following line:

Code:

O E2    69.4.191.252/30 
           [110/20] via 10.136.217.2, 01:06:46, Serial1/1.7/26:0.1

It will print out like this:

Code:

O E2    69.4.191.244/30 
O E2 69.4.191.244/30 [110/20] via 10.136.215.2, 00:13:50, Serial1/1.7/24:0.1 #exactly how I need it but can't get rid of the entry above. Tried with a regex but no luck

The other problem is that there is lines like the following:

Code:

O E2    69.4.190.64/30 [110/20] via 10.136.10.2, 01:06:46, Serial1/1.1/11:0.1
                       [110/20] via 10.136.248.2, 01:06:46, Serial1/1.8/25:0.1

IT will print them out like this:

Code:

O E2 69.4.190.64/30 [110/20] via 10.136.10.2, 00:13:50, Serial1/1.1/11:0.1 [110/20] via 10.136.248.2, 00:13:50, Serial1/1.8/25:0.1

So the array positions are different.
This is probably fairly trivial stuff for an experience programmer, but I am having a hard time figuring ways around it.
Regardless Im excited about the possibilities at work as I am figuring some of this stuff out.
Thanks again for your help.

angrybanana · 10-13-2007, 06:14 PM

Please, try to post a full explanation of your problem the first time. Not knowing what you want to do, or what the expected inputs and outputs are really hinders us in answering your question.

That being said, here's my best guess at what you're trying to do.

Code:

use strict;
use warnings;

my ($last, $line, @lines, @fields);

while(<>){
        chomp;
        s/^\s+//; s/\s+$//;
        $line = $_;
        if (/^O E2/){
                push @lines, $line if ( do{ my @t = split " ", $line;@t} > 3);
                $last = $1 if $line =~ /(^O E2\s+.*?)(\s|$)/;
        } else {
                next unless $last;
                push @lines, join " ", $last, $line;
        }
}

for $line (@lines){
        @fields = split " ", $line;
        print "$line\n";
}

Here's an example of it in action.
Input:

Code:

     69.0.0.0/8 is variably subnetted, 184 subnets, 7 masks
O E2    69.4.191.252/30 
           [110/20] via 10.136.217.2, 00:28:33, Serial1/1.7/26:0.1
O E2    69.4.191.244/30 
           [110/20] via 10.136.215.2, 00:28:33, Serial1/1.7/24:0.1
O E2    69.4.191.240/30 
           [110/20] via 10.136.214.2, 00:28:33, Serial1/1.7/23:0.1
O E2    69.4.191.220/30 
           [110/20] via 10.136.236.2, 00:28:33, Serial1/1.8/13:0.1
O E2    69.4.191.216/30 [110/20] via 10.136.232.2, 00:28:33, Serial1/1.8/9:0.1
O E2    69.4.191.212/30 
           [110/20] via 10.136.203.2, 00:28:33, Serial1/1.7/12:0.1
O E2    69.4.191.208/30 [110/20] via 10.136.225.2, 00:28:33, Serial1/1.8/2:0.1
O E2    69.4.191.192/28 [110/20] via 10.137.8.2, 00:28:33, Serial1/1.3/9:0.1
O E2    69.4.191.176/28 [110/20] via 10.136.226.2, 00:28:33, Serial1/1.8/3:0.1
O E2    69.4.191.168/29 
           [110/20] via 10.136.205.2, 00:28:33, Serial1/1.7/14:0.1
O E2    69.4.190.64/30 [110/20] via 10.136.10.2, 01:06:46, Serial1/1.1/11:0.1
                       [110/20] via 10.136.248.2, 01:06:46, Serial1/1.8/25:0.1

Output

Code:

O E2    69.4.191.252/30 [110/20] via 10.136.217.2, 00:28:33, Serial1/1.7/26:0.1
O E2    69.4.191.244/30 [110/20] via 10.136.215.2, 00:28:33, Serial1/1.7/24:0.1
O E2    69.4.191.240/30 [110/20] via 10.136.214.2, 00:28:33, Serial1/1.7/23:0.1
O E2    69.4.191.220/30 [110/20] via 10.136.236.2, 00:28:33, Serial1/1.8/13:0.1
O E2    69.4.191.216/30 [110/20] via 10.136.232.2, 00:28:33, Serial1/1.8/9:0.1
O E2    69.4.191.212/30 [110/20] via 10.136.203.2, 00:28:33, Serial1/1.7/12:0.1
O E2    69.4.191.208/30 [110/20] via 10.136.225.2, 00:28:33, Serial1/1.8/2:0.1
O E2    69.4.191.192/28 [110/20] via 10.137.8.2, 00:28:33, Serial1/1.3/9:0.1
O E2    69.4.191.176/28 [110/20] via 10.136.226.2, 00:28:33, Serial1/1.8/3:0.1
O E2    69.4.191.168/29 [110/20] via 10.136.205.2, 00:28:33, Serial1/1.7/14:0.1
O E2    69.4.190.64/30 [110/20] via 10.136.10.2, 01:06:46, Serial1/1.1/11:0.1
O E2    69.4.190.64/30 [110/20] via 10.136.248.2, 01:06:46, Serial1/1.8/25:0.1

Notice that the last IP is repeated twice. I did this because you didn't quite explain how you wanted to deal with IP addresses that had more then one line associated with them. Use the second loop to parse your data it should be straight forward. I have @fields set up for you and ready to go, $fields[number] will get you the specific field.

If you don't understand the script and need me to explain some parts of it, just ask. I'm just too tired to put comments on it right now.

PS: Perl makes my head hurt

sal_paradise42 · 10-14-2007, 02:05 AM

hey angry,
I didn't think I was going to end up with a completely new piece of code, but I guess I was totally over my head, and probably why I lacked in my explanation.
That was a great guess to what I actually need.
Here is the input and here is the output
Your assumption was correct, if the IP has two lines associated I wanted two entries for it. I also changed your line from

Code:

#$last = $1 if $line =~ /(^O E2\s+.*?)(\s|$)/;

to

Code:

$last = $1 if $line =~ /(^O (?:E2){0,1}\s+.*?)(\s|$)/;

since there is instances where the E2 is not present after ^O.

Anyway, as you can see the problem that I am running into is that I need to strip out the lines that contain

Code:

^.*variably subnetted.*$

and

Code:

 7600-UT01#exit
Connection closed by foreign host.

I try to do this with a regex after chomp. A s/blah// removes it, but the ip addresses right before that line get printed twice. Anyway I am really close but just need to strip out the lines that match that.
Thanks for taking the time, and I apologize for a lack of understanding.
P.S can you explain what this lines are doing?

Code:

push @lines, $line if ( do{ my @t = split " ", $line;@t} > 3);
                $last = $1 if $line =~ /(^O E2\s+.*?)(\s|$)/;

angrybanana · 10-14-2007, 04:30 AM

Here's the changes, this should work now:

Code:

use strict;
use warnings;

my ($last, $line, $limit, @lines, @fields);

while(<>){
        next if /variably subnetted/;
        last if /#exit/;
        chomp;
        s/^\s+//; s/\s+$//;
        $line = $_;
        if (/^O (E2)?/){ 
                if ($1){ $limit = 3 } else { $limit = 2 }
                push @lines, $line if ( do{ my @t = split " ", $line;@t} > $limit);
                $last = $1 if $line =~ /(^O (?:E2)?\s+.*?)(\s|$)/;
        } else {
                next unless $last;
                push @lines, join " ", $last, $line;
        }
}

for $line (@lines){
        @fields = split " ", $line;
        print "$line\n";
}

What I added where two new lines:

Code:

next if /variably subnetted/; #skip the line completely if it contains match
last if /#exit/; #exit the loop once we see "#exit"

as far as what this is doing:

Code:

push @lines, $line if ( do{ my @t = split " ", $line;@t} > 3);

splits the line on the spaces, if there's more then 3 fields then we assume that It's a full line. We add it to @lines. Now that I know that the E2 is optional, I added an if statement that checks if E2 was there or not, and sets the $limit to either 3 or 2 accordingly.
The do block is there because I couldn't figure out a cleaner way to do this (I'm new to perl myself). Basically it stores the split values in a temporary array @t, then it returns that array. If I don't do that it gives me a warning, which is annoying.

Code:

$last = $1 if $line =~ /(^O E2\s+.*?)(\s|$)/;

Here we store the first part of the line in $last. We do this so that we can join $last with line later on if the IP has multiple lines associated with it. $1 = group one in the regex match (parenthesis)

Hope that helps, best of luck.

sal_paradise42 · 10-14-2007, 06:17 PM

thanks angry and ghostdog. Working like a charm.