[SOLVED] /var/log/secure parse with perl script

Habitual · 10-16-2012, 09:55 AM

Let's dispense with the foreplay, I don't know jack about perl, however...In a desire to increase my personal skill-set and Linux SysAdmin value to the community,
I decided to undertake the exercises in Perl for System Administration. I'll admit I jumped to Chapter 9 after reading the intro. and (because it mentioned the log files all us admins have to manipulate).

I am trying to massage some important data from /var/log/secure using a perl script as a template I found here....

It processes just fine but doesn't return any records and this is where I need help.

The source document was written less than a year ago and didn't run as "copied" from that site, I had to add 3 lines of code to even get this far.

I suspect this line to be the culprit:

Code:

elsif($line =~ /[Ff]ailed\spassword\sfor\s(w+)\sfroms(d+.d+.d+.d+)/)

to which I have tried the simple to the not-so-simple strings, all without any result.

I've tried:

Code:

elsif($line =~ /Failed password for /)
elsif($line =~ /Failed /)
elsif($line =~ /[Ss]shd /)
elsif($line =~ /sshd /)

but nothing "hits".

My current reading material is
http://perldoc.perl.org/perlrequick.html
http://perldoc.perl.org/perlre.html#Regular-Expressions

There may be some shortcuts I have taken as I assumed this would be a good learning experience and a "do-able" exercise with immediate benefit.

Anybody willing to show me the error of my ways? If it's not the RE portion as I suspect, please point me in the right direction.

Thank you for your time.

frederickbear · 10-16-2012, 11:04 AM

Do you have an example of what you're expecting the RE to match?

also the example in the link has \s after from so the RE matches from followed by space rather than froms

grail · 10-16-2012, 11:47 AM

You could also try a simple test first, I ran:

Code:

grep -E '[Ff]ailed\spassword\sfor\s(\w+)\sfrom\s(\d+\.\d+\.\d+\.\d+)' /var/log/secure

Now this returned nothing for me, and you may well need to remove some of the round brackets as they are there to store data for later, but it could well be, as it is
for me, that this data does not exist in the secure log you are checking.

Habitual · 10-16-2012, 12:24 PM

Quote:

Originally Posted by frederickbear

Do you have an example of what you're expecting the RE to match?

Sure, sorry I didn't include a sample:

Code:

Oct 16 13:55:13 domain.com sshd[21141]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33279 ssh2
Oct 16 13:55:14 domain.com sshd[21149]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 33894 ssh2
Oct 16 13:55:14 domain.com sshd[21153]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33406 ssh2
Oct 16 13:55:15 domain.com sshd[21168]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 33965 ssh2
Oct 16 13:55:15 domain.com sshd[21169]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33427 ssh2
Oct 16 13:55:15 domain.com sshd[21189]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33475 ssh2
Oct 16 13:55:16 domain.com sshd[21188]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 34058 ssh2
Oct 16 13:55:16 domain.com sshd[21195]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33494 ssh2
Oct 16 13:55:20 domain.com sshd[21205]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 34328 ssh2
Oct 16 13:55:20 domain.com sshd[21209]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33670 ssh2
Oct 16 13:55:32 domain.com sshd[21221]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 35322 ssh2
Oct 16 13:55:34 domain.com sshd[21225]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 34595 ssh2

"also the example in the link has \s after from so the RE matches from followed by space rather than froms"

fredrickbear: Thanks, that should be

Code:

elsif($line =~ /[Ff]ailed\spassword\sfor\s(\w+)\sfrom\s(\d+\.\d+\.\d+\.\d+)/)

grail: I am working on some out using your code snippet, which also returns nothing here on /var/log/secure as well. I am not concerned with my localhost atm, these are mixed CentOS and Ubuntu flavored remote hosts

Work In Progress!!

Thanks.

frederickbear · 10-16-2012, 01:20 PM

It looks as though the log file format has changed from that when the original regex was written

This part for\s(\w+)\sfrom\s(\d+\.\d+\.\d+\.\d+) would match 'for username from 123.123.123.123'

to cope with 'invalid user postgres' and also the ::ffff: which is causing the regex to fail you could try:

Code:

elsif($line =~ /[Ff]ailed\spassword\sfor\s(.+)\sfrom\s::ffff:(\d+\.\d+\.\d+\.\d+)/)

the (.+) is a little crude but with the other stuff around it should match correctly.

Habitual · 10-16-2012, 02:18 PM

frederickbear:

Thank you, I can accept your estimate that the log file format may have changed, but "crude"? How can I tell?

Alas, your snippet doesn't work on the target.

I did find my bash script that has a one-liner in that does process the "Failed passwords" on 9 hosts' /var/log/secure (after I scp them down) and that line is

Code:

perl -ne 'if (/Failed password for.* from (.*?) /)  {print $1, "\n"}' "$CONFDIR"/*" | sed -e 's/::ffff://g'

Now how I can turn that into a working replacement for

Code:

elsif($line =~  /[Ff]ailed\spassword\sfor\s(.+)\sfrom\s::ffff:(\d+\.\d+\.\d+\.\d+)/)

is where I guess I need some help.

Thank you for your time.

frederickbear · 10-16-2012, 02:36 PM

Quote:

Originally Posted by Habitual

Thank you, I can accept your estimate that the log file format may have changed, but "crude"? How can I tell?

"crude" was a reference to my hack of the line. Here's a variation on the theme.

Code:

elsif($line =~  /[Ff]ailed\spassword\sfor\s(.*?)\sfrom\s.*?(\d+\.\d+\.\d+\.\d+)/)

I copied some of your log file output into a file called 'text' and tried this:

Code:

# perl -ne '/[Ff]ailed\spassword\sfor\s(.*?)\sfrom\s.*?(\d+\.\d+\.\d+\.\d+)/; print "$1 $2\n";'  text
invalid user postgres 74.86.64.194
invalid user postgres 67.228.13.82
invalid user postgres 74.86.64.194
invalid user postgres 67.228.13.82
invalid user postgres 74.86.64.194
invalid user postgres 74.86.64.194

althought the previous version looked to have worked as well. It may be the regex isn't actually getting used in the script.

Habitual · 10-16-2012, 03:07 PM

frederickbear,

I am a chop and hack (machete method) script builder. I love bash and I can craft some 'stuff'.
This is where I am reading now....How will my regular expression match? a sub-section of Pattern Matching, Regular Expressions, and Parsing over at perlmonks

It's crazy mad on the documentation and tutorials, it's easy to get overwhelmed by it. So I am trying to avoid getting lost in it.

Code:

perldoc perlrequick

First example...

Code:

"Hello World" =~ /World/;  # matches

so

Code:

"line" =~ /Failed/;  #  should match?

...If the documentation "didn't work" then I'd have to suppose 20,000 of Larry's friends would know it, so it must be my RE expression.

Have a Great Day!

frederickbear · 10-16-2012, 05:01 PM

Quote:

Originally Posted by Habitual

Code:

"line" =~ /Failed/;  #  should match?

...If the documentation "didn't work" then I'd have to suppose 20,000 of Larry's friends would know it, so it must be my RE expression.

you are trying to find the string 'Failed' inside of string 'line' which won't match. Earlier you used $line so I'm assuming that's a typo. Did you mean to try something more like:

Code:

"Oct 16 13:55:13 domain.com sshd[21141]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33279 ssh2" =~ /Failed/; # which would be a match

Habitual · 10-17-2012, 07:00 AM

frederickbear: (and others too!)

Thanks. Yes, it was/is a typo missing the "$".
I am still having trouble associating what the perl code does (and what people are saying it does)

"if ($blah =~ /stuff/) { # this code gets run if $blah contains 'stuff' }'"
says this 2008 reference.
and
a 2000 dated article titled "Common RegEx Gotchas" in the "Specifying Too Much" section says...
"...Suppose that you want to find the word Serial, followed by a colon and then a nine-digit number. The data lines might look like this: my $line = "Name: Some Soldier, Rank: Leftenant, Serial: 426879824, Boots: black";

A regex novice might bite off more than he could chew with the following:

Code:

$line =~ /^\w*: \w*\s*\w*, \w*: \w*, \w+: (\d)*, \w*: \w*$/;

If all you're interested in is the Serial number, only ask for that. It'll make your regex simpler, and it will handle deviations from what you think the line ought to look like. (That happens more often than you want to think.)

Code:

$line =~ /[Ss]erial: (\d{9})/;

Seems pretty consistent from 2000 to 2008.
So, what is not consistent here would be me and my code.

I am focused today on my line of code that processes lines (missing from the source, remember...) "my $line = 0
This appears to be an error for it is not a numeric integer I am after in "$line". Isn't that correct?

It very well could be from what I read

Code:

my $line =~ /Failed;

which also runs but report is empty.

but from the Gotchas link (same section) it seems like "my $line = " should be more like

Code:

my $line = "Month: some month, Day: some day, Time: time stamp, Host: some host, Service: sshd, Failed: 's/Failed password for ', User: user, From: '/s::ffffff:'";

based on this expected line(s) content(s)...

Code:

Oct 16 13:22:46 hostname sshd[17289]: Failed password for root from ::ffff:217.18.242.235 port 48207 ssh2

I hope you know how to fix this and are allowing me the courtesy of finding it on my "own".

Thank you and Have a Great Day!

grail · 10-17-2012, 10:18 AM

As you are on the path to enlightenment

May I suggest that you start your regex at the beginning and build from there to see where the error lie?

Code:

$ cat Habitual.in
Oct 16 13:55:13 domain.com sshd[21141]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33279 ssh2
Oct 16 13:55:14 domain.com sshd[21149]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 33894 ssh2
Oct 16 13:55:14 domain.com sshd[21153]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33406 ssh2
Oct 16 13:55:15 domain.com sshd[21168]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 33965 ssh2
Oct 16 13:55:15 domain.com sshd[21169]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33427 ssh2
Oct 16 13:55:15 domain.com sshd[21189]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33475 ssh2
Oct 16 13:55:16 domain.com sshd[21188]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 34058 ssh2
Oct 16 13:55:16 domain.com sshd[21195]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33494 ssh2
Oct 16 13:55:20 domain.com sshd[21205]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 34328 ssh2
Oct 16 13:55:20 domain.com sshd[21209]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33670 ssh2
Oct 16 13:55:32 domain.com sshd[21221]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 35322 ssh2
Oct 16 13:55:34 domain.com sshd[21225]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 34595 ssh2
$ perl -ne '$line = $_;if($line =~ /[Ff]ailed/){print $_}' Habitual.in
Oct 16 13:55:13 domain.com sshd[21141]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33279 ssh2
Oct 16 13:55:14 domain.com sshd[21149]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 33894 ssh2
Oct 16 13:55:14 domain.com sshd[21153]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33406 ssh2
Oct 16 13:55:15 domain.com sshd[21168]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 33965 ssh2
Oct 16 13:55:15 domain.com sshd[21169]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33427 ssh2
Oct 16 13:55:15 domain.com sshd[21189]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33475 ssh2
Oct 16 13:55:16 domain.com sshd[21188]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 34058 ssh2
Oct 16 13:55:16 domain.com sshd[21195]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33494 ssh2
Oct 16 13:55:20 domain.com sshd[21205]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 34328 ssh2
Oct 16 13:55:20 domain.com sshd[21209]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33670 ssh2
Oct 16 13:55:32 domain.com sshd[21221]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 35322 ssh2
Oct 16 13:55:34 domain.com sshd[21225]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 34595 ssh2
$ perl -ne 'chomp;$line = $_;if($line =~ /[Ff]ailed\spassword.*:(\d+\.\d+\.\d+\.\d+)/){print $_," ",$1,"\n"}' Habitual.in
Oct 16 13:55:13 domain.com sshd[21141]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33279 ssh2 74.86.64.194
Oct 16 13:55:14 domain.com sshd[21149]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 33894 ssh2 67.228.13.82
Oct 16 13:55:14 domain.com sshd[21153]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33406 ssh2 74.86.64.194
Oct 16 13:55:15 domain.com sshd[21168]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 33965 ssh2 67.228.13.82
Oct 16 13:55:15 domain.com sshd[21169]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33427 ssh2 74.86.64.194
Oct 16 13:55:15 domain.com sshd[21189]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33475 ssh2 74.86.64.194
Oct 16 13:55:16 domain.com sshd[21188]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 34058 ssh2 67.228.13.82
Oct 16 13:55:16 domain.com sshd[21195]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33494 ssh2 74.86.64.194
Oct 16 13:55:20 domain.com sshd[21205]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 34328 ssh2 67.228.13.82
Oct 16 13:55:20 domain.com sshd[21209]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 33670 ssh2 74.86.64.194
Oct 16 13:55:32 domain.com sshd[21221]: Failed password for invalid user postgres from ::ffff:67.228.13.82 port 35322 ssh2 67.228.13.82
Oct 16 13:55:34 domain.com sshd[21225]: Failed password for invalid user postgres from ::ffff:74.86.64.194 port 34595 ssh2 74.86.64.194

So if you build it as you see fit you can experiment on where it may be going wrong for you

Habitual · 10-17-2012, 03:24 PM

Quote:

Originally Posted by grail

As you are on the path to enlightenment

May I suggest that you start your regex at the beginning and build from there to see where the error lie?
...

God, I hope that is the case, Thanks grail.
However, I am trying to migrate away from the perl one-liner(s) that I have and know work and incorporate it into a proper perl.pl script.

Where I am getting burnt is...
These one-liners work (as does yours)

Code:

perl -ne 'if (/Failed (.*?) port/) {print}' /home/jj/Documents/cirrhus9/Accounts/donorschoose/Security/*secure*
perl -ne 'if (/Failed password for.* from (.*?) port/) {print}' /home/jj/Documents/cirrhus9/Accounts/donorschoose/Security/*secure*

but I remain stuck at or near this line...:

Code:

elsif($line=~ /Failed password for (.*?) from .*?(\d+\.\d+\.\d+\.\d+)/s)

Thank you for your time.

grail · 10-18-2012, 11:43 AM

Yeah sorry for the confusion there, I was using the one-liner just to show the code. Simply place it into a file and run to see how you go.

Code:

#!/usr/bin/env perl
#
#
use strict;
use warnings;

my $line;

open(FILE, $ARGV[0]) or die "Can't open file: $!\n";

while (<FILE>){
    chomp;

    $line = $_;

    if($line =~ /[Ff]ailed\spassword.*:(\d+\.\d+\.\d+\.\d+)/){
        print "$line $1\n"
    }
}

Habitual · 10-19-2012, 10:42 AM

grail:

Thanks much. I bought a book.
and ordered another...

chrism01 · 10-23-2012, 06:55 PM

See my post on your other thread here https://www.linuxquestions.org/quest...30#post4813430