LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 01-15-2005, 09:16 PM   #1
bulliver
Senior Member
 
Registered: Nov 2002
Location: Edmonton AB, Canada
Distribution: Gentoo x86_64; Gentoo PPC; FreeBSD; OS X 10.9.4
Posts: 3,760
Blog Entries: 4

Rep: Reputation: 78
Perl breaking my balls....


Hello all,

Because I want to be a good sysadmin I figured I should wrap my head around perl sooner or later. Now, I don't know if this is because I learned python first or what, but I am having a hell of a time figuring anything out at all.

I have been wanting to write a script that parses my exim logs and uses it to create a (sort-of) real time ip ban list from the results, so I decided to try in perl. It has taken me the better part of today, and all I have to show is about 20 lines of working code.

What I am stuck on presently is dealing with arrays. I have two arrays, lets say @already_banned and @to_be_banned, both which are made up of 1 or more ip addresses. I need to loop over the values of @to_be_banned, and remove them if they also exist in @already_banned.

I went and bought both Learning Perl, and Programming Perl, and neither book has helped with what I want (or anything else for that matter...). Everything I have tried has either bombed with cryptic error messages, or worse, run successfully, but not done what I want....

To tell the truth I haven't even been able to figure out how to check an array for membership? Why is this seemingly simple task so difficult?

I found a section in the perl cookbook titled "Finding Elements in One Array but Not Another" which appears to be what I want...but when I tried to integrate it...the program ran but did nothing (ie: the target ips were not removed...)

I have also tried "if ( $_ in @array) {" type of construct but perl tells me that "'in' is uninitialized"?!?!? I just don't get it.

Does anyone feel like taking the time to help me out? I don't want you to write my code, but I have been stuck at this spot for several hours now, and I'm about to say to hell with perl and do it in python (which if I had I would have been done hours ago)

Again, I want to learn perl...there must be something basic I am missing....
 
Old 01-15-2005, 11:11 PM   #2
btmiller
Senior Member
 
Registered: May 2004
Location: In the DC 'burbs
Distribution: Arch, Scientific Linux, Debian, Ubuntu
Posts: 4,289

Rep: Reputation: 378Reputation: 378Reputation: 378Reputation: 378
This is what foreach is good for, e.g.:

Code:
$found = 0;
foreach $elt (@array) {
  if($elt == $newelt) { # use eq if comparing strings == for numbers
    # new element already in @array)
     $found = 1;
     last;
   }
}
if(!$found) {
  # add new element to the array
  push(@array, $newelt);
}
If you'll be doing this a lot, it's probably more efficient to use hashes, but I'm not sure and on a fast machine it probably doesn't really make a difference.
 
Old 01-15-2005, 11:38 PM   #3
bulliver
Senior Member
 
Registered: Nov 2002
Location: Edmonton AB, Canada
Distribution: Gentoo x86_64; Gentoo PPC; FreeBSD; OS X 10.9.4
Posts: 3,760

Original Poster
Blog Entries: 4

Rep: Reputation: 78
I'm so lost that I am not sure how your example relates to what I want to do here. I'm going to post my code so you can see what I have so far:
Code:
#!/usr/bin/perl

open LOG, "/var/log/exim/exim_reject.log";

# this returns an array of ip addresses
# 'chump' is a string in my log that indicates an IP trying to relay
@lines = <LOG>;
foreach $line (@lines) {
    if ($line =~ m/chump/) { 
        @ip = split / /, $line;
        my $ip = "$ip[3]\n";
        $ip =~ s/[\[]//;
        $ip =~ s/[\]]//;
        push (@rbl, $ip);
        }
    }

@rbl = sort @rbl;

# this I stole from Perl Cookbook, it acts like unix 'uniq' command
# but I don't even understand how it works...
%seen = ();
foreach $item (@rbl) {
    push(@rbls, $item) unless $seen{$item}++;
    }

# returns an array of IP addresses I already banned
foreach $rule (`iptables -L banned -n`) {
    if ($rule =~ m/[0-255]\.[0-255]\.[0-255]\.[0-255]/) {
        $rule =~ s/\s+/ /g;
        @_ = split / /, $rule;
        push (@banned, $_[3]);
        }
    }

# this is where I'm having trouble...

foreach $bl (@rbls) {
    foreach $ip (@banned) {
        unless ( $bl eq $ip ) {
            print "To be banned: "; # this is placeholder for 'system' iptables command
            print $bl;  # if I ever get this to work...
            }
        }
    }
As it stands this last stanza is not working at all. My problem is that I can't figure out how to search for a $value in an @array.
So for this last code stanza, I guess the psuedo code is:
Code:
foreach $ip_address (@ip_to_ban) {
    unless ($ip_address in @already_banned) {
        <iptables command on $ip_address>;
        }
    }
 
Old 01-16-2005, 02:19 PM   #4
sasho
Member
 
Registered: Jan 2005
Distribution: Arch
Posts: 120

Rep: Reputation: 17
You probably solved this, but here it goes.

I started off from where you said you have a problem.

You should recognise most of the code as yours, it only has small changes:


Code:
$i = 0; #increment to keep track of  index in @rbls
$already_banned = 0; #use as a flag,  indicates if $bl is to be banned

foreach $bl (@rbls) {
    foreach $ip (@banned) {
        if ($bl eq $ip) {
            $already_banned = 1;
            delete @rbls[$i];
        }
    }
    #make iptables rule:
    if (!$already_banned) {
         print "$bl should be banned...\n";
         #iptables rule here;
    }
     else {
         #reset the already_banned flag:
         $already_banned  = 0;
     }
    $i++;
}
Basically, if an IP from @rbls is found in the @banned list, it is deleted by using the index $i. At that time a flag that $bl is already banned is set ($already_banned). After exiting the inner foreach loop, if $bl is not already banned, you put your iptables rule there, otherwise you reset the flag for the next item in the @rbls list.

Try it, it should do what you want.
 
Old 01-16-2005, 03:09 PM   #5
bulliver
Senior Member
 
Registered: Nov 2002
Location: Edmonton AB, Canada
Distribution: Gentoo x86_64; Gentoo PPC; FreeBSD; OS X 10.9.4
Posts: 3,760

Original Poster
Blog Entries: 4

Rep: Reputation: 78
Hey thanks for the help. I understand your code quite well, unfortunetly It is still not removing the banned ips from @rbls.

I have two banned ips on the system I'm testing this on...when I add:
Code:
foreach $b (@banned) {
    print "$b\n";
    }
To make sure @banned is working ok, it prints each ip no prob. Here is the output of a run: (I realize it's not cool to post all these ips, but they all tried using my MTA to relay spam, if anyone is upset I will edit them out)
Code:
24.66.229.173 # output of the print @banned array 
4.4.67.71        # these top two ips are already banned...
210.182.169.41
 should be banned...
211.105.125.102
 should be banned...
211.217.232.112
 should be banned...
211.219.147.8
 should be banned...
211.227.47.165
 should be banned...
219.91.110.220
 should be banned...
220.163.68.232
 should be banned...
221.140.55.94
 should be banned...
222.101.92.244
 should be banned...
24.66.229.173        # still here
 should be banned...
24.66.229.176
 should be banned...
24.66.229.187
 should be banned...
4.4.67.71                # still here
 should be banned...
61.84.38.254
 should be banned...
See that delete statement was what I was looking for. In chapt 3 "lists and arrays" in learning perl there is _no_ mention whatsoever about delete. The only thing even close to removing entries from an array is pop, which isn't what I want.

This is my biggest frustration with perl so far...that so many times the code will run without error but still not work properly. This gives you no clue as to why it isn't working.
 
Old 01-16-2005, 04:37 PM   #6
sasho
Member
 
Registered: Jan 2005
Distribution: Arch
Posts: 120

Rep: Reputation: 17
Hmm I have it working flawlessly with your list. Save and run the code below (perl some_name.pl):

Code:
@rbls=(
"210.182.169.41",
"211.105.125.102",
"211.217.232.112",
"211.219.147.8",
"211.227.47.165",
"219.91.110.220",
"220.163.68.232",
"221.140.55.94",
"222.101.92.244",
"24.66.229.173","24.66.229.176",
"24.66.229.187",
"4.4.67.71",
"61.84.38.254");

@banned=("24.66.229.173", "4.4.67.71");

$i = 0;
$already_banned = 0;
foreach $bl (@rbls) {
    foreach $ip (@banned) {
        if ($bl eq $ip) {
            $already_banned = 1;
            delete @rbls[$i];
        }
    }

    #make iptables rule:
    if (!$already_banned) {
         #iptables rule here;
         print "$bl should be banned...\n";
    }
    else {
         #reset the already_banned flag:
         $already_banned  = 0;
     }
    $i++;
}
this is the output:

Quote:
perl test.pl
210.182.169.41 should be banned...
211.105.125.102 should be banned...
211.217.232.112 should be banned...
211.219.147.8 should be banned...
211.227.47.165 should be banned...
219.91.110.220 should be banned...
220.163.68.232 should be banned...
221.140.55.94 should be banned...
222.101.92.244 should be banned...
24.66.229.176 should be banned...
24.66.229.187 should be banned...
61.84.38.254 should be banned...
I didn't know about delete either... i ran across it while trying to help you.

Last edited by sasho; 01-16-2005 at 04:40 PM.
 
Old 01-16-2005, 04:57 PM   #7
sasho
Member
 
Registered: Jan 2005
Distribution: Arch
Posts: 120

Rep: Reputation: 17
I think I know what your problem is. It looks like each IP you push onto the array carries a newline character at the end. Strip that with the chomp function rpior to pushing onto @rbls. Then there will not be a break like you have in the output:

Quote:
220.163.68.232
should be banned...
It will be:
Quote:
220.163.68.232 should be banned...
Note there's no newline character in your output from @banned (from your example).

Last edited by sasho; 01-16-2005 at 05:02 PM.
 
Old 01-16-2005, 06:13 PM   #8
bulliver
Senior Member
 
Registered: Nov 2002
Location: Edmonton AB, Canada
Distribution: Gentoo x86_64; Gentoo PPC; FreeBSD; OS X 10.9.4
Posts: 3,760

Original Poster
Blog Entries: 4

Rep: Reputation: 78
Yes, thank you so much....

I changed:
Code:
@lines = <LOG>; # to...
chomp(@lines = <LOG>);
and added:
chomp($bl); # to the foreach loop....
Works perfect now....

Thanks again for all the help
 
Old 01-17-2005, 12:54 PM   #9
MetaPhase
LQ Newbie
 
Registered: Jan 2005
Posts: 8

Rep: Reputation: 0
Re: Perl breaking my balls....

Quote:
Originally posted by bulliver
Hello all,

Because I want to be a good sysadmin I figured I should wrap my head around perl sooner or later. Now, I don't know if this is because I learned python first or what, but I am having a hell of a time figuring anything out at all.

I have been wanting to write a script that parses my exim logs and uses it to create a (sort-of) real time ip ban list from the results, so I decided to try in perl. It has taken me the better part of today, and all I have to show is about 20 lines of working code.

What I am stuck on presently is dealing with arrays. I have two arrays, lets say @already_banned and @to_be_banned, both which are made up of 1 or more ip addresses. I need to loop over the values of @to_be_banned, and remove them if they also exist in @already_banned.
Hi Bulliver,
First of all, don't despair - with a little help from a good book, such as "Learning Perl", you should be up and running fairly quickly. My suggestion is to simply start reading it, from the beginning. Don't jump to chapter 2 before reading chapter 1... If you follow the book as it is laid out, your confusion should quickly disappear.

As for your question, to iterate over an array, one way is to use the foreach function. So for example:
Code:
my @new_ip_array;
foreach my $to_be_ip (@to_be_banned) {
   my $found_flag = 0;
   foreach my $banned_ip (@already_banned) {
      if ($to_be_ip eq $banned_ip) {
         $found_flag = 1;
      }
   }
   if ($found_flag == 0) {
      push @new_ip_array , $to_be_ip;
   }
}
Obviously this is not the most efficient method of doing this, as I am looping over every item of @already_banned for each item of @to_be_banned - which if the lists are very large, may take a while... however you need to walk before you can run, so I chose the simplest method I could, without using more advanced features of Perl such as hashes.

At the end of this code, the @new_ip_array will contain those IPs that were in @to_be_banned but not in @already_banned. If you have any questions about the code, feel free to ask.

Quote:

I have also tried "if ( $_ in @array) {" type of construct but perl tells me that "'in' is uninitialized"?!?!? I just don't get it.
There is no function, operator or keyword in Perl called "in". Perhaps you're confusing your languages?
 
Old 01-17-2005, 12:59 PM   #10
MetaPhase
LQ Newbie
 
Registered: Jan 2005
Posts: 8

Rep: Reputation: 0
Quote:
Originally posted by bulliver
This is my biggest frustration with perl so far...that so many times the code will run without error but still not work properly. This gives you no clue as to why it isn't working.
According to "perldoc perl", one of the bugs in perl is that "The -w switch is not mandatory."

I couldn't agree more :-)

As a Perl beginner, always use the "-w" switch. I would also suggest to use "use strict;" i.e.:
Code:
#! /usr/bin/perl -w
use strict;
This will help you catch a wide range of problems.
 
Old 01-17-2005, 02:14 PM   #11
bulliver
Senior Member
 
Registered: Nov 2002
Location: Edmonton AB, Canada
Distribution: Gentoo x86_64; Gentoo PPC; FreeBSD; OS X 10.9.4
Posts: 3,760

Original Poster
Blog Entries: 4

Rep: Reputation: 78
Quote:
First of all, don't despair - with a little help from a good book, such as "Learning Perl", you should be up and running fairly quickly. My suggestion is to simply start reading it, from the beginning. Don't jump to chapter 2 before reading chapter 1... If you follow the book as it is laid out, your confusion should quickly disappear.
I do have this book, but I am not finding it too helpful. The books examples are typically 1 or 2 line snippets that are very hard for me to put together into working code. I will read it a few more times until I get this though...

Quote:
Obviously this is not the most efficient method of doing this, as I am looping over every item of @already_banned for each item of @to_be_banned - which if the lists are very large, may take a while... however you need to walk before you can run, so I chose the simplest method I could, without using more advanced features of Perl such as hashes.
This fact was not lost on me, especially since in a language I know (python) the last stanza could be as simple as:
Code:
for ip in to_be_banned:
    if not ip in banned_already:
        os.system( "iptables -A banned -p tcp -s %s -j DROP" % ip )
I will, however, read up on hashes, and try to make my rbl.pl script more efficient.

Quote:
There is no function, operator or keyword in Perl called "in". Perhaps you're confusing your languages?
I guess so. This is kind of amusing. I was googling for "perl array membership" and happened across this page, part of an article called "A taste of Perl 6" which contains this snippet:
Code:
# NOT Perl 6 code...
if ( $value in @list ) {...}
Not Perl 6 code right, so I figured it was Perl 5 code, and was therefore valid

As for -w and use strict that is very good advice and I will use them from now on....
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Tar Balls Sharkscott Linux - Newbie 11 01-11-2006 10:55 AM
breaking waves... marsques Linux - Software 4 05-18-2004 01:25 AM
perl(Cwd) perl(File::Basename) perl(File::Copy) perl(strict)....What are those? Baldorg Linux - Software 1 11-09-2003 08:09 PM
AcidRip is breaking my balls! Need help!! Hammett Linux - Software 1 10-31-2003 12:01 PM
can you use rsync on tar balls wedgeworth Linux - Hardware 1 09-24-2003 02:14 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 03:31 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration