Script to Conditionally Send Mail Based on Logs

zokken · 02-25-2015, 08:26 AM

Hey All,

So what I'm attempting to do is search for a particular string in a log file that appears after a certain time. Pull the usernames from the results and count the number of times each username appears.

What I'm having trouble with is coming up with a way to trigger an email if any of the usernames appear more than a certain number of times.

Here's a simplified snipped of code of what I've done so far.

Code:

#!/bin/bash


checktime () {
   while read line; do
      # code that checks if the timestamp of each line is after
      # a certain time; if it is, echo the line
   done
}

# grep logfile for search_string; pass the results to the checktime function.
# use awk to pull usernames from the results and count number of occurrences
# for each user
grep search_string logfile | checktime | awk '{print $9}' sort | uniq -c

This part works. What I'd like to do next is send an email with the usernames and their count if the count exceeds a certain amount. I can come up with ways to do it, but it doesn't seem very efficient/elegant. For example:

Code:

#!/bin/bash

results=""

checktime () {
   while read line; do
      # code that checks if the timestamp of each line is after
      # a certain time; if it is, echo the line
   done
}

# print username if it appears more than 50 times in the results
checklogs () {
   while read line; do
      if [[ $(echo $line | awk '{print $1}') -ge 50 ]]; then
         echo $line
      fi
   done
}

# grep logfile for search_string; pass the results to the checktime function.
# use awk to pull usernames from the results and count number of occurrences
# for each user. pass this to the checklogs function.
results=$(grep search_string logfile | checktime | awk '{print $9}' sort | uniq -c | checklogs

# if $results isn't empty, at least one user exceeded the count. email results.
[[ "$results" != "" ]] && echo $results | /bin/mail -s "results" addy@domain.com

This seems to work (though I'm left with the formatting problem when echo'ing $results to an email), but I'm looking for suggestions on how to better do it.

unSpawn · 02-25-2015, 09:02 AM

Code:

checktime () {
   while read line; do
      # code that checks if the timestamp of each line is after
      # a certain time; if it is, echo the line
}

The problem with this is that anything passing through syslog has a human readable time stamp set. That is good but not efficient to work with so you'll better internally convert time stamps to epoch (as in date '+%s' --date=something)' for easy comparison...

Code:

# print username if it appears more than 50 times in the results
checklogs () {
   while read line; do
      if [[ $(echo $line | awk '{print $1}') -ge 50 ]]; then
         echo $line
      fi
   done
}

If the output is always a value string pair then you could use an array and use ${ARRAY[0]}.

Code:

# if $results isn't empty, at least one user exceeded the count. email results.
[[ "$results" != "" ]] && echo $results | /bin/mail -s "results" addy@domain.com

Same here: RESULTS can be an array so if '[ ${#RESULTS[@]} =! 0 ]' that's the trigger to send an email. Personally I'd start by cutting the log file at the time stamp first (use 'mktemp' to hold temporary files?). Unless you have a very fast growing log file and can't afford to miss entries while processing this ensures you only have to check time stamps once at the beginning of the script. In pseudo code:

Code:

define THRESHOLD
define LOG_FILE
define TIME_STAMP
define STRING

test -s LOG_FILE && exit 0
grep -q -m1 TIME_STAMP LOG_FILE || exit 0

mktemp TEMPDIR && { cut LOG_FILE at TIME_STAMP > TEMPDIR/TEMPFILE
awk '/STRING/ {print STRING_FIELD USERNAME_FIELD} TEMPDIR/TEMPFILE | sort -u | while read STRING_VALUE USER_VALUE; do
 [ `grep -m50 -c "STRING_VALUE.*USER_VALUE" (or something)` -ge THRESHOLD ] && echo USER_VALUE
done > TEMPDIR/REPORTFILE; test -s TEMPDIR/REPORTFILE && mail TEMPDIR/REPORTFILE
} && CLEANUP

exit 0

HTH

pan64 · 02-25-2015, 09:52 AM

my usual comment is: do not use pipe chains like grep|awk|sort|uniq, usually it can be solved within a single perl/awk/python/whatever script. It would be more efficient, and every requirement can be implemented - not to speak about the while cycle containing a $(... | awk)!
Especially the function checklogs can be replaced by:
awk '$1>=50'
Also checkline will look quite similar:
awk ' <condition> '
(but it is a bit harder, you need to specify input line format and how/what do you want to check)
Next, grep pattern file usually can be replaced by awk '/pattern/' file
sort and uniq were also already implemented in awk - and you only need to combine all of it into one single script

zokken · 02-26-2015, 09:26 AM

Thanks for the replies; they're both definitely helpful.