LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 08-04-2008, 10:30 PM   #1
calipryss
LQ Newbie
 
Registered: Jun 2007
Posts: 9

Rep: Reputation: 0
shell script to monitor log file


I have a log file (log.tst) that I would like to do what i think is a nested loop. ? or not.

So, let's say I have the below lines in the file.

10:20:13 192.35.5.22 refused no auth
10:21:00 53.34.553.23 refused invalid
10:21:30 192.35.5.22 refused no auth
10:22:00 192.35.5.22 refused no auth
10:23:00 192.35.5.22 refused no auth
10:24:40 192.35.5.22 refused no auth
10:24:50 192.35.5.22 refused no auth
10:25:00 53.34.553.23 refused no auth

I have a threshold that's set at 5. So when an IP gets refused more than five times in five minutes, I want to send myself an email with the IP.

Any suggestions? Or, any pointers of where I can fumble my way through examples? Thank You.
 
Old 08-04-2008, 10:35 PM   #2
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 59
See if logcheck can do this or similar for you (http://logcheck.org/).

What are you ultimately trying to do? Somehow I get the feeling you want to take the next step with is auto-blocking of such events. If so, fail2ban might be what you are looking for. But there still may be other solutions too.
 
Old 08-04-2008, 10:45 PM   #3
calipryss
LQ Newbie
 
Registered: Jun 2007
Posts: 9

Original Poster
Rep: Reputation: 0
further information...

Ultimately, I want to add the offending IP to hosts.deny. I would love to just use a ready made utility, but I cannot on this particular device. It has to be a script that is run via cron that monitors and then simply puts the IP in hosts.deny. It cannot be a daemon or any other running service. That being the case, got any suggestions of how to go about such a script or know of anywhere I can find such info? The OS is SLES 9 version 3. Thank You.

Last edited by calipryss; 08-04-2008 at 10:46 PM.
 
Old 08-04-2008, 10:49 PM   #4
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 59
Go check out fail2ban then.

The problem is a little more complex then perhaps you are thinking (avoid blacklisting your own sites, duration of blacklist, hit interval and rate, etc.).
 
Old 08-04-2008, 10:55 PM   #5
calipryss
LQ Newbie
 
Registered: Jun 2007
Posts: 9

Original Poster
Rep: Reputation: 0
For example, I tried the following today.. but it didn't get the result I wanted.

grep refused ./log.tst | /bin/awk '{print $2}' | /bin/awk '{count[$2]++}END{for(j in count) print j,count[j]}'

that got me something like
192.35.5.22 6
53.34.553.23 2

Not quite what I'm looking for. From what I read online, maybe I need some sort of nested loop, but I don't know where to start.
 
Old 08-04-2008, 10:56 PM   #6
farslayer
Guru
 
Registered: Oct 2005
Location: Willoughby, Ohio
Distribution: linuxdebian
Posts: 7,231
Blog Entries: 5

Rep: Reputation: 189Reputation: 189
http://denyhosts.sourceforge.net/
- Appends /etc/hosts.deny and adds the newly banned hosts
- /etc/hosts.deny entries can be expired (purge) at a user specified time (new in 0.8)



http://www.aczoom.com/cms/blockhosts

and this one updates the firewall rulkes, I know that's not what you want, but I'll list it for your future reference. http://www.fail2ban.org/wiki/index.php/Main_Page


(This place is brutal.. take a couple minutes to look something up and you get relegated to repeating things that have already been posted... )

Last edited by farslayer; 08-04-2008 at 11:03 PM.
 
Old 08-04-2008, 10:59 PM   #7
calipryss
LQ Newbie
 
Registered: Jun 2007
Posts: 9

Original Poster
Rep: Reputation: 0
I checked out fail2ban, but it's not quite what I'm looking for. I don't want to permanently ban the IP's, just temporarily deny them until they can be looked into. I cannot add any additional rpm's on the device, it's locked down that way. The solution I'm looking for is via a script, the help I'm requesting is how to format that script, or what syntax to use, or perhaps even a pointer in where to find similar scripts or examples. Any such information or tips would be greatly appreciated. I have go the rest of the script figured out, except for how to get a variable with the offending IP in it, and only the offending IP. Thank You.
 
Old 08-04-2008, 11:04 PM   #8
calipryss
LQ Newbie
 
Registered: Jun 2007
Posts: 9

Original Poster
Rep: Reputation: 0
I currently am running denyhosts on other servers, but I cannot run any daemons or services on this device. It has to be via script. Frustrating, but challenging. Thoughts, ideas?
 
Old 08-05-2008, 01:36 AM   #9
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,261

Rep: Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028Reputation: 2028
You do Know a daemon is just a script/prog in an infinite loop (detached from terminal)?
eg

nohup ./some_loop.sh &

Can you clarify what you think you mean please?
 
Old 08-05-2008, 01:49 AM   #10
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 59
calipryss, it is difficult to help when requirements and restrictions are withheld, and only come out piece by piece. It wastes peoples time. I sensed correctly that you actually wanted sometime different that you asked.

For the future, please provide a clearly stated goal, and as many of your requirements as possible.

Like chrism01, I wonder about your distinction between "script" and "daemon or services". These are two separate concepts: one a language, the other a mode of operation.
 
Old 08-05-2008, 10:22 AM   #11
farslayer
Guru
 
Registered: Oct 2005
Location: Willoughby, Ohio
Distribution: linuxdebian
Posts: 7,231
Blog Entries: 5

Rep: Reputation: 189Reputation: 189
I'm still trying to figure out if you will be able to modify the hosts.deny file, since you [don't have rights/aren't allowed] to install anything on the machine.. which is it ? modifying the hosts.deny file requires root privileges.
 
Old 08-05-2008, 12:35 PM   #12
calipryss
LQ Newbie
 
Registered: Jun 2007
Posts: 9

Original Poster
Rep: Reputation: 0
Let me clarify. Although I have root privileges, I cannot run any app's per our agreement with a vendor who supplied this device. I can create scripts run by cron. I'm not withholding requirements but I don't feel a need to get into the nitty gritty details when in the end, I'm asking specifically about a script to do what I reflected in my original emails. I appreciate your alternative suggestions, but that wasn't my question.

For those of you that are trying to help me accomplish this script - thank you. For those of you that are more interested in changing or challenging my approach, don't bother responding as a script is the only solution I can implement.

To further clarify, this script would not be continuously running as any daemon, it would run via cron maybe every 15 minutes or something to that affect. It's a gray area, I recognize that and trust me, I realize that in the end, it's essentially the same concept but as long as it's not a daemon, I can implement it.

Thank You.
 
Old 08-05-2008, 01:50 PM   #13
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 59
That's fine. But please understand when you ask for other peoples time, for free, you must do more upfront legwork and spend time to be as clear as possible.

Your requirements are still lacking:

How are previous log reads managed? Does the script need to manage and maintain state? Or does the script simply re-read the entire log and resend previously sent alerts?

You say "shell script". Does that mean it must all be shell? Which shell? Other scripting languages make the job much faster and far easier (awk, perl). Can those be used? If you have to call external utilities for each line of a large log file, this is very expensive, and very slow. Compare:

Code:
$ wc /var/log/somelog
   50363  856414 8961876 /var/log/somelog
$ time while read line; do (( i++)) ; done < /var/log/somelog

real    0m1.555s
user    0m1.130s
sys     0m0.422s

$ time while read line; do /bin/echo > /dev/null ; done < /var/log/somelog

real    2m2.810s
user    1m1.696s
sys     1m29.438s
That was 50000+ processes created.

I have a partially written perl script if you want that.
 
Old 08-05-2008, 09:52 PM   #14
calipryss
LQ Newbie
 
Registered: Jun 2007
Posts: 9

Original Poster
Rep: Reputation: 0
Thank you for your response. I finally figured it out today. I wrote a script that gets the last five minutes worth of log data and greps for any refusals. The output of that is basically two fields, time and ip of which I then took a count of times the ip was refused and compared that to a threshold I had already specified. If the count is greater than my threshold, I send myself an email and put the ip in hosts.deny for all protocols. The time was a little tricky with how the application (that creates the log) posts time in the file.

As far as expense is concerned, I dummied up a file and ran it a few times and with how powerful this device is and how little CPU, Memory, etc it uses, the expense was at the utmost minimal.

While I understand the nature of forum is to provide free help and assistance, I had only intended on asking my original question: how to get a script to read a log file and obtain an ip for any IP getting refused more than five times in five minutes. If I post again in the future, I will be more clear as to the specifics as well as try to be more clear with my goal so the post stays on track. I appreciate your feedback and in the future, if I have scripting questions I will post more accurately, maybe in the programming forum?

To answer your question - it's a bash shell. If you're interested I can post the script. I'm sure it's very novice, but in learning something new, you have to start somewhere right?
 
Old 08-05-2008, 10:46 PM   #15
Mr. C.
Senior Member
 
Registered: Jun 2008
Posts: 2,529

Rep: Reputation: 59
Because so many questions in forums do not represent the underlying goal, and instead focus on implementation specifics, there are occasional "false positives" in the assumption that there is a yet a larger, more fundamental problem.

No worries. Good to hear you have your solution.
 
  


Reply

Tags
count, monitoring, script, shell


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
log monitor bash script tronica Programming 5 05-14-2008 02:53 AM
Filtering a CSV file from web log with shell script? Micro420 Programming 8 08-22-2007 03:13 AM
Shell Script For Myspace Log in condonm Programming 1 02-08-2007 06:34 AM
A live log monitor script ? ifm Linux - Software 1 07-19-2002 02:54 AM
Shell File Upload Monitor Script noodle123 Programming 0 04-22-2002 11:36 AM


All times are GMT -5. The time now is 05:30 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration