LinuxQuestions.org - Perl breaking my balls....

- Programming (https://www.linuxquestions.org/questions/programming-9/)

- - Perl breaking my balls.... (https://www.linuxquestions.org/questions/programming-9/perl-breaking-my-balls-278266/)

bulliver

01-15-2005 09:16 PM

Perl breaking my balls....

Hello all,

Because I want to be a good sysadmin I figured I should wrap my head around perl sooner or later. Now, I don't know if this is because I learned python first or what, but I am having a hell of a time figuring anything out at all.

I have been wanting to write a script that parses my exim logs and uses it to create a (sort-of) real time ip ban list from the results, so I decided to try in perl. It has taken me the better part of today, and all I have to show is about 20 lines of working code.

What I am stuck on presently is dealing with arrays. I have two arrays, lets say @already_banned and @to_be_banned, both which are made up of 1 or more ip addresses. I need to loop over the values of @to_be_banned, and remove them if they also exist in @already_banned.

I went and bought both Learning Perl, and Programming Perl, and neither book has helped with what I want (or anything else for that matter...). Everything I have tried has either bombed with cryptic error messages, or worse, run successfully, but not done what I want....

To tell the truth I haven't even been able to figure out how to check an array for membership? Why is this seemingly simple task so difficult?

I found a section in the perl cookbook titled "Finding Elements in One Array but Not Another" which appears to be what I want...but when I tried to integrate it...the program ran but did nothing (ie: the target ips were not removed...)

I have also tried "if ( $_ in @array) {" type of construct but perl tells me that "'in' is uninitialized"?!?!? I just don't get it.

Does anyone feel like taking the time to help me out? I don't want you to write my code, but I have been stuck at this spot for several hours now, and I'm about to say to hell with perl and do it in python (which if I had I would have been done hours ago)

Again, I want to learn perl...there must be something basic I am missing....

btmiller

01-15-2005 11:11 PM

This is what foreach is good for, e.g.:

Code:

$found = 0;

foreach $elt (@array) {

  if($elt == $newelt) { # use eq if comparing strings == for numbers

    # new element already in @array)

    $found = 1;

    last;

  }

}

if(!$found) {

  # add new element to the array

  push(@array, $newelt);

}

If you'll be doing this a lot, it's probably more efficient to use hashes, but I'm not sure and on a fast machine it probably doesn't really make a difference.

bulliver

01-15-2005 11:38 PM

I'm so lost that I am not sure how your example relates to what I want to do here. I'm going to post my code so you can see what I have so far:

Code:

#!/usr/bin/perl



open LOG, "/var/log/exim/exim_reject.log";



# this returns an array of ip addresses

# 'chump' is a string in my log that indicates an IP trying to relay

@lines = <LOG>;

foreach $line (@lines) {

    if ($line =~ m/chump/) { 

        @ip = split / /, $line;

        my $ip = "$ip[3]\n";

        $ip =~ s/[\[]//;

        $ip =~ s/[\]]//;

        push (@rbl, $ip);

        }

    }



@rbl = sort @rbl;



# this I stole from Perl Cookbook, it acts like unix 'uniq' command

# but I don't even understand how it works...

%seen = ();

foreach $item (@rbl) {

    push(@rbls, $item) unless $seen{$item}++;

    }



# returns an array of IP addresses I already banned

foreach $rule (`iptables -L banned -n`) {

    if ($rule =~ m/[0-255]\.[0-255]\.[0-255]\.[0-255]/) {

        $rule =~ s/\s+/ /g;

        @_ = split / /, $rule;

        push (@banned, $_[3]);

        }

    }



# this is where I'm having trouble...



foreach $bl (@rbls) {

    foreach $ip (@banned) {

        unless ( $bl eq $ip ) {

            print "To be banned: "; # this is placeholder for 'system' iptables command

            print $bl;  # if I ever get this to work...

            }

        }

    }

As it stands this last stanza is not working at all. My problem is that I can't figure out how to search for a $value in an @array.
So for this last code stanza, I guess the psuedo code is:

Code:

foreach $ip_address (@ip_to_ban) {

    unless ($ip_address in @already_banned) {

        <iptables command on $ip_address>;

        }

    }

sasho

01-16-2005 02:19 PM

You probably solved this, but here it goes.

I started off from where you said you have a problem.

You should recognise most of the code as yours, it only has small changes:

Code:

$i = 0; #increment to keep track of  index in @rbls

$already_banned = 0; #use as a flag,  indicates if $bl is to be banned



foreach $bl (@rbls) {

    foreach $ip (@banned) {

        if ($bl eq $ip) {

            $already_banned = 1;

            delete @rbls[$i];

        }

    }

    #make iptables rule:

    if (!$already_banned) {

        print "$bl should be banned...\n";

        #iptables rule here;

    }

    else {

        #reset the already_banned flag:

        $already_banned  = 0;

    }

    $i++;

}

Basically, if an IP from @rbls is found in the @banned list, it is deleted by using the index $i. At that time a flag that $bl is already banned is set ($already_banned). After exiting the inner foreach loop, if $bl is not already banned, you put your iptables rule there, otherwise you reset the flag for the next item in the @rbls list.

Try it, it should do what you want.

bulliver

01-16-2005 03:09 PM

Hey thanks for the help. I understand your code quite well, unfortunetly It is still not removing the banned ips from @rbls.

I have two banned ips on the system I'm testing this on...when I add:

Code:

foreach $b (@banned) {

    print "$b\n";

    }

To make sure @banned is working ok, it prints each ip no prob. Here is the output of a run: (I realize it's not cool to post all these ips, but they all tried using my MTA to relay spam, if anyone is upset I will edit them out)

Code:

24.66.229.173 # output of the print @banned array 

4.4.67.71        # these top two ips are already banned...

210.182.169.41

 should be banned...

211.105.125.102

 should be banned...

211.217.232.112

 should be banned...

211.219.147.8

 should be banned...

211.227.47.165

 should be banned...

219.91.110.220

 should be banned...

220.163.68.232

 should be banned...

221.140.55.94

 should be banned...

222.101.92.244

 should be banned...

24.66.229.173        # still here

 should be banned...

24.66.229.176

 should be banned...

24.66.229.187

 should be banned...

4.4.67.71                # still here

 should be banned...

61.84.38.254

 should be banned...

See that delete statement was what I was looking for. In chapt 3 "lists and arrays" in learning perl there is _no_ mention whatsoever about delete. The only thing even close to removing entries from an array is pop, which isn't what I want.

This is my biggest frustration with perl so far...that so many times the code will run without error but still not work properly. This gives you no clue as to why it isn't working.

sasho

01-16-2005 04:37 PM

Hmm I have it working flawlessly with your list. Save and run the code below (perl some_name.pl):

Code:

@rbls=(

"210.182.169.41",

"211.105.125.102",

"211.217.232.112",

"211.219.147.8",

"211.227.47.165",

"219.91.110.220",

"220.163.68.232",

"221.140.55.94",

"222.101.92.244",

"24.66.229.173","24.66.229.176",

"24.66.229.187",

"4.4.67.71",

"61.84.38.254");



@banned=("24.66.229.173", "4.4.67.71");



$i = 0;

$already_banned = 0;

foreach $bl (@rbls) {

    foreach $ip (@banned) {

        if ($bl eq $ip) {

            $already_banned = 1;

            delete @rbls[$i];

        }

    }



    #make iptables rule:

    if (!$already_banned) {

        #iptables rule here;

        print "$bl should be banned...\n";

    }

    else {

        #reset the already_banned flag:

        $already_banned  = 0;

    }

    $i++;

}

this is the output:

Quote:

perl test.pl
210.182.169.41 should be banned...
211.105.125.102 should be banned...
211.217.232.112 should be banned...
211.219.147.8 should be banned...
211.227.47.165 should be banned...
219.91.110.220 should be banned...
220.163.68.232 should be banned...
221.140.55.94 should be banned...
222.101.92.244 should be banned...
24.66.229.176 should be banned...
24.66.229.187 should be banned...
61.84.38.254 should be banned...

I didn't know about delete either... i ran across it while trying to help you.

sasho

01-16-2005 04:57 PM

I think I know what your problem is. It looks like each IP you push onto the array carries a newline character at the end. Strip that with the chomp function rpior to pushing onto @rbls. Then there will not be a break like you have in the output:

Quote:

220.163.68.232
should be banned...

It will be:

Quote:

220.163.68.232 should be banned...

Note there's no newline character in your output from @banned (from your example).

bulliver

01-16-2005 06:13 PM

Yes, thank you so much....

I changed:

Code:

@lines = <LOG>; # to...

chomp(@lines = <LOG>);

and added:

chomp($bl); # to the foreach loop....

Works perfect now....

Thanks again for all the help

MetaPhase

01-17-2005 12:54 PM

Re: Perl breaking my balls....

Quote:

Originally posted by bulliver
Hello all,

Because I want to be a good sysadmin I figured I should wrap my head around perl sooner or later. Now, I don't know if this is because I learned python first or what, but I am having a hell of a time figuring anything out at all.

I have been wanting to write a script that parses my exim logs and uses it to create a (sort-of) real time ip ban list from the results, so I decided to try in perl. It has taken me the better part of today, and all I have to show is about 20 lines of working code.

What I am stuck on presently is dealing with arrays. I have two arrays, lets say @already_banned and @to_be_banned, both which are made up of 1 or more ip addresses. I need to loop over the values of @to_be_banned, and remove them if they also exist in @already_banned.

Hi Bulliver,
First of all, don't despair - with a little help from a good book, such as "Learning Perl", you should be up and running fairly quickly. My suggestion is to simply start reading it, from the beginning. Don't jump to chapter 2 before reading chapter 1... If you follow the book as it is laid out, your confusion should quickly disappear.

As for your question, to iterate over an array, one way is to use the foreach function. So for example:

Code:

my @new_ip_array;

foreach my $to_be_ip (@to_be_banned) {

  my $found_flag = 0;

  foreach my $banned_ip (@already_banned) {

      if ($to_be_ip eq $banned_ip) {

        $found_flag = 1;

      }

  }

  if ($found_flag == 0) {

      push @new_ip_array , $to_be_ip;

  }

}

Obviously this is not the most efficient method of doing this, as I am looping over every item of @already_banned for each item of @to_be_banned - which if the lists are very large, may take a while... however you need to walk before you can run, so I chose the simplest method I could, without using more advanced features of Perl such as hashes.

At the end of this code, the @new_ip_array will contain those IPs that were in @to_be_banned but not in @already_banned. If you have any questions about the code, feel free to ask.

Quote:

I have also tried "if ( $_ in @array) {" type of construct but perl tells me that "'in' is uninitialized"?!?!? I just don't get it.

There is no function, operator or keyword in Perl called "in". Perhaps you're confusing your languages?

MetaPhase

01-17-2005 12:59 PM

Quote:

Originally posted by bulliver
This is my biggest frustration with perl so far...that so many times the code will run without error but still not work properly. This gives you no clue as to why it isn't working.

According to "perldoc perl", one of the bugs in perl is that "The -w switch is not mandatory."

I couldn't agree more :-)

As a Perl beginner, always use the "-w" switch. I would also suggest to use "use strict;" i.e.:

Code:

#! /usr/bin/perl -w

use strict;

This will help you catch a wide range of problems.

bulliver

01-17-2005 02:14 PM

Quote:

First of all, don't despair - with a little help from a good book, such as "Learning Perl", you should be up and running fairly quickly. My suggestion is to simply start reading it, from the beginning. Don't jump to chapter 2 before reading chapter 1... If you follow the book as it is laid out, your confusion should quickly disappear.

I do have this book, but I am not finding it too helpful. The books examples are typically 1 or 2 line snippets that are very hard for me to put together into working code. I will read it a few more times until I get this though...

Quote:

This fact was not lost on me, especially since in a language I know (python) the last stanza could be as simple as:

Code:

for ip in to_be_banned:

    if not ip in banned_already:

        os.system( "iptables -A banned -p tcp -s %s -j DROP" % ip )

I will, however, read up on hashes, and try to make my rbl.pl script more efficient.

Quote:

There is no function, operator or keyword in Perl called "in". Perhaps you're confusing your languages?

I guess so. This is kind of amusing. I was googling for "perl array membership" and happened across this page, part of an article called "A taste of Perl 6" which contains this snippet:

Code:

# NOT Perl 6 code...

if ( $value in @list ) {...}

Not Perl 6 code right, so I figured it was Perl 5 code, and was therefore valid :rolleyes:

As for -w and use strict that is very good advice and I will use them from now on....

All times are GMT -5. The time now is 06:19 AM.