Advanced VI Question

tbeehler · 06-18-2007, 12:01 AM

Hello,

I have a lot of people slamming my postfix server and I want to ban them completely with iptables. What I'd like to do is grab one of my log files and I'm quite sure that VI has the ability to grab the ip addresses out of the log file and pipe them into another one. I'm just unsure of the syntax. I have tried a ton of ways and I'm just not getting it, so I thought I'd post my question here. Here's an example of what I'd like to do.

Jun 17 08:04:28 hostname postfix/smtpd[12919]: NOQUEUE: reject: RCPT from mail.skyeassociates.com[64.105.201.130]: 450 <+._-xfllbhq@mydomain.com>: Recipient address rejected: User unknown in local reci pient table; from=<> to=<+._-xfllbhq@mydomain.com> proto=ESMTP helo=<mail.skyeassociates.com>

The domain names have been changed to protect the guilty.

But what I'd like to know is, is there a way for vi to grab just the ip addresses out of this one line and pipe it into another file? Of course, I have thousands of entries like this one, so whatever VI command that needs to happen, needs to happen on a global scale.

Thanks in advance!

blackhole54 · 06-18-2007, 01:33 AM

This doesn't exactly answer your question, but I think fail2ban already does (among other things) exactly what you want to do.

If you want to do it yourself anyway, I would think the tool you want would be sed rather than vi, but I haven't put any thought into what regular expression you would need. I quick google search led me to believe that fail2ban harvests its info from log files with regular expressions, so even if you want to roll your own, it might be instructive to see what regular expression they use.

FMC · 06-18-2007, 06:38 AM

Honestly, vi is not the tool that you need. vi is a visual interactive text editor, if you want automation you have to use a non interactive text editor, this is the case of sed, for example. grep can be used as well, it have regular expression suport.

If you need a regex to find an IP address just say, I can right it for you, thats something not so complicated!

[]´s, FMC!

archtoad6 · 06-18-2007, 09:37 AM

Here's some test/sample code to get you started:

Code:

$ X='Jun 17 08:04:28 hostname postfix/smtpd[12919]: \
NOQUEUE: reject: RCPT from mail.skyeassociates.com[64.105.201.130]: \
450 <+._-xfllbhq@mydomain.com>: Recipient address rejected: \
User unknown in local reci pient table; \
from=<> to=<+._-xfllbhq@mydomain.com> \
proto=ESMTP helo=<mail.skyeassociates.com>'

$ echo $X  | egrep -o '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
64.105.201.130

This may do, as all I did was pipe your extract through my regex. However, I tried a bullet-proofing test:

Code:

$ Z='1.2.3.4 1234.35.67.89===999.0.23.4444-713.555.1234))\
)1.2222.3.4 1.2.3333.4. 1.2.3.4444,,,11.22.33.44'

$ R='[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'

$ echo $Z |egrep -o "${R}"
1.2.3.4
234.35.67.89
999.0.23.444
1.2.3.444
11.22.33.44

Lines 2 - 4 are bogus, which shows that a more complicated regex set might be better:

Code:

$ echo $Z |egrep -o "(^|[^0-9])${R}($|[^0-9])" |egrep -o "${R}"
1.2.3.4
11.22.33.44

Note: I have folded the input variable definitions to avoid hor. scrolling in Konqueror.

archtoad6 · 06-26-2007, 01:07 PM

So, what happened?

tbeehler · 08-24-2007, 10:39 AM

Quote:

Originally Posted by archtoad6

So, what happened?

I terribly apologize for me not getting back with you. But your code did set me on the right path. Also, the above mentioned fail2ban worked REALLY well. It banned more spammers then I knew were hitting me. We now ban approximately 800 spam ip's a day. It's REALLY great! Thank you guys for all your help! I appreciate it!

tbeehler · 08-24-2007, 10:44 AM

Quote:

Originally Posted by blackhole54

This doesn't exactly answer your question, but I think fail2ban already does (among other things) exactly what you want to do.

If you want to do it yourself anyway, I would think the tool you want would be sed rather than vi, but I haven't put any thought into what regular expression you would need. I quick google search led me to believe that fail2ban harvests its info from log files with regular expressions, so even if you want to roll your own, it might be instructive to see what regular expression they use.

Thank you very much for the suggestion. Fail2ban worked better then I could have imagined.

tbeehler · 08-24-2007, 10:50 AM

echo $X | egrep -o '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | trash.txt

was the line that worked flawlessly for me. Just thought I'd let you know what worked for me.

I use fail2ban along side of the above mentioned code to weed out continual spammers. I have my mail server to automatically ban ip's that hit our mail server with false email addresses like jeff1112@mydomain.com which doesn't exist, 5 times in 10 minutes. It will ban that ip for 24 hours. However, I have continual spammers who I see every 24 hours, so I do a reverse lookup to see who they are and if it isn't someone important that we usually deal with (maybe someone who's machine is unknowingly being a spam zombie, they are permanently banned via "iptables -I INPUT -s ipaddress -j DROP"

I just thought I'd share how I'm set up with everyone else in case it was helpful to them as well. Thank you all very much for your help!

Poetics · 08-24-2007, 07:53 PM

As an addition (I also use fail2ban to great effect) the grepping out of IP addresses is fairly simple in perl and works very flawlessly in helping set up other scripts -- such as banning blocked IPs that recur more than 3x a week or the like. Might want to take a look at it.

tbeehler · 08-27-2007, 09:47 AM

Quote:

Originally Posted by Poetics

As an addition (I also use fail2ban to great effect) the grepping out of IP addresses is fairly simple in perl and works very flawlessly in helping set up other scripts -- such as banning blocked IPs that recur more than 3x a week or the like. Might want to take a look at it.

Do you have something already set up in Perl? Care to share it? My Perl skills leave a lot to be desired.

Thanks for the tip too!