shell script to monitor log file
 

I have a log file (log.tst) that I would like to do what i think is a nested loop. ? or not.

So, let's say I have the below lines in the file.

10:20:13 192.35.5.22 refused no auth
10:21:00 53.34.553.23 refused invalid
10:21:30 192.35.5.22 refused no auth
10:22:00 192.35.5.22 refused no auth
10:23:00 192.35.5.22 refused no auth
10:24:40 192.35.5.22 refused no auth
10:24:50 192.35.5.22 refused no auth
10:25:00 53.34.553.23 refused no auth

I have a threshold that's set at 5. So when an IP gets refused more than five times in five minutes, I want to send myself an email with the IP.

Any suggestions? Or, any pointers of where I can fumble my way through examples? Thank You.

See if logcheck can do this or similar for you (http://logcheck.org/).

What are you ultimately trying to do? Somehow I get the feeling you want to take the next step with is auto-blocking of such events. If so, fail2ban might be what you are looking for. But there still may be other solutions too.

further information...
 

Ultimately, I want to add the offending IP to hosts.deny. I would love to just use a ready made utility, but I cannot on this particular device. It has to be a script that is run via cron that monitors and then simply puts the IP in hosts.deny. It cannot be a daemon or any other running service. That being the case, got any suggestions of how to go about such a script or know of anywhere I can find such info? The OS is SLES 9 version 3. Thank You.

Go check out fail2ban then.

The problem is a little more complex then perhaps you are thinking (avoid blacklisting your own sites, duration of blacklist, hit interval and rate, etc.).

For example, I tried the following today.. but it didn't get the result I wanted.

grep refused ./log.tst | /bin/awk '{print $2}' | /bin/awk '{count[$2]++}END{for(j in count) print j,count[j]}'

that got me something like
192.35.5.22 6
53.34.553.23 2

Not quite what I'm looking for. From what I read online, maybe I need some sort of nested loop, but I don't know where to start.

http://denyhosts.sourceforge.net/
- Appends /etc/hosts.deny and adds the newly banned hosts
- /etc/hosts.deny entries can be expired (purge) at a user specified time (new in 0.8)


http://www.aczoom.com/cms/blockhosts

and this one updates the firewall rulkes, I know that's not what you want, but I'll list it for your future reference. http://www.fail2ban.org/wiki/index.php/Main_Page


(This place is brutal.. take a couple minutes to look something up and you get relegated to repeating things that have already been posted... :) )

I checked out fail2ban, but it's not quite what I'm looking for. I don't want to permanently ban the IP's, just temporarily deny them until they can be looked into. I cannot add any additional rpm's on the device, it's locked down that way. The solution I'm looking for is via a script, the help I'm requesting is how to format that script, or what syntax to use, or perhaps even a pointer in where to find similar scripts or examples. Any such information or tips would be greatly appreciated. I have go the rest of the script figured out, except for how to get a variable with the offending IP in it, and only the offending IP. Thank You.

I currently am running denyhosts on other servers, but I cannot run any daemons or services on this device. It has to be via script. Frustrating, but challenging. Thoughts, ideas?

You do Know a daemon is just a script/prog in an infinite loop (detached from terminal)?
eg

nohup ./some_loop.sh &

Can you clarify what you think you mean please?

calipryss, it is difficult to help when requirements and restrictions are withheld, and only come out piece by piece. It wastes peoples time. I sensed correctly that you actually wanted sometime different that you asked.

For the future, please provide a clearly stated goal, and as many of your requirements as possible.

Like chrism01, I wonder about your distinction between "script" and "daemon or services". These are two separate concepts: one a language, the other a mode of operation.

I'm still trying to figure out if you will be able to modify the hosts.deny file, since you [don't have rights/aren't allowed] to install anything on the machine.. which is it ? modifying the hosts.deny file requires root privileges.

Let me clarify. Although I have root privileges, I cannot run any app's per our agreement with a vendor who supplied this device. I can create scripts run by cron. I'm not withholding requirements but I don't feel a need to get into the nitty gritty details when in the end, I'm asking specifically about a script to do what I reflected in my original emails. I appreciate your alternative suggestions, but that wasn't my question.

For those of you that are trying to help me accomplish this script - thank you. For those of you that are more interested in changing or challenging my approach, don't bother responding as a script is the only solution I can implement.

To further clarify, this script would not be continuously running as any daemon, it would run via cron maybe every 15 minutes or something to that affect. It's a gray area, I recognize that and trust me, I realize that in the end, it's essentially the same concept but as long as it's not a daemon, I can implement it.

Thank You.

That's fine. But please understand when you ask for other peoples time, for free, you must do more upfront legwork and spend time to be as clear as possible.

Your requirements are still lacking:

How are previous log reads managed? Does the script need to manage and maintain state? Or does the script simply re-read the entire log and resend previously sent alerts?

You say "shell script". Does that mean it must all be shell? Which shell? Other scripting languages make the job much faster and far easier (awk, perl). Can those be used? If you have to call external utilities for each line of a large log file, this is very expensive, and very slow. Compare:

That was 50000+ processes created.

I have a partially written perl script if you want that.

Thank you for your response. I finally figured it out today. I wrote a script that gets the last five minutes worth of log data and greps for any refusals. The output of that is basically two fields, time and ip of which I then took a count of times the ip was refused and compared that to a threshold I had already specified. If the count is greater than my threshold, I send myself an email and put the ip in hosts.deny for all protocols. The time was a little tricky with how the application (that creates the log) posts time in the file.

As far as expense is concerned, I dummied up a file and ran it a few times and with how powerful this device is and how little CPU, Memory, etc it uses, the expense was at the utmost minimal.

While I understand the nature of forum is to provide free help and assistance, I had only intended on asking my original question: how to get a script to read a log file and obtain an ip for any IP getting refused more than five times in five minutes. If I post again in the future, I will be more clear as to the specifics as well as try to be more clear with my goal so the post stays on track. I appreciate your feedback and in the future, if I have scripting questions I will post more accurately, maybe in the programming forum?

To answer your question - it's a bash shell. If you're interested I can post the script. I'm sure it's very novice, but in learning something new, you have to start somewhere right?

Because so many questions in forums do not represent the underlying goal, and instead focus on implementation specifics, there are occasional "false positives" in the assumption that there is a yet a larger, more fundamental problem.

No worries. Good to hear you have your solution.