[SOLVED] Nagios notifies about RECOVERY but not PROBLEM via email.

4Paul4 · 07-10-2012, 09:26 PM

I am using check_logfiles on Windows Server 2008 R2 over NRPE via NSClient++ to monitor a stack-trace logfile using Nagios.

When I cause an exception to be logged, nagios picks it up and displays it in the web interface, but no notification is actually sent out.

If I check again, a recovery is detected (because no exceptions have occurred since) and I receive a notification about the recovery.

If I click on 'Notifications' in the web interface, I can see that both OK and CRITICAL are logged there. Both have contact set as 'nagiosadmin' and both have Notification Command set as 'servicenotify'.

So why aren't I actually getting the notifications about the problem?

Could there be escape characters out of the log file destroying the logging?

EDIT:

It's not a problem with the emailing. I tested this out by adding an echo >> /tmp/file.log to servicenotify. This log contains the RECOVERY message, but not the PROBLEM message.

angel115 · 07-11-2012, 02:32 AM

Hi 4Paul4,

What does the nagios log say?
Can you see the notification being send in the log?

Correct the path with your configuration accordingly:

Code:

#grep "notify" /usr/local/nagios/var/nagios.log

Angel

4Paul4 · 07-11-2012, 06:18 PM

Hi Angel,

Yes, I can see the notifications using

Code:

grep "notify" /var/nagios/nagios.log"

I'm going to try writing a wrapper script for this check.

angel115 · 07-12-2012, 02:21 AM

To help you troubleshooting this issue, you can add "| tee -a $nagios_log_path/notification.log" at the end of the command_name "notify-service-by-email" and "notify-host-by-email"

So you can see the command output and see if you got an error message.

4Paul4 · 07-12-2012, 06:43 PM

Thanks for the reply. Something I failed to mention before; swapped out the default servicenotify and hostnotify scripts for a bash script. I am sending these notifications through amazon's SNS service. This is the servicenotify file I created:

Code:

sns-publish arn:aws:sns:us-west-1:xxxxxxxxxxxx:NAGIOS --message "There is a $1 with service $2 on $3 at $4. State is: $5.  ADDITIONALINFO: $6 $7 $8 $9 $10 $11 $12 $13 $14 $15 $16 $17 $18 $19 $20 $21 $22 $23" |tee -a /tmp/nagios.log

Adding this to the end did not result in a file:

Code:

echo "$1 $2 $3 $4 $5 $6 $7 $8 $9 $10 $11 $12 $13 $14 $15" >> /tmp/test

also resulted in no output for [CRITICAL]] but worked for the [OK]
I think there is a problem receiving the log file as passed parameters in this bash script.

This is the output of check_logfiles for an example exception:

Code:

CRITICAL - (2 errors in check_logfiles.protocol-2012-07-12-03-53-56) - Message : System.Exception: Registration of new users is disabled. RelatedId 

xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx ...|TestException2_lines=118 TestException2_warnings=0 TestException2_criticals=2 TestException2_unknowns=0
tag TestException2 CRITICAL
Message : testexceptionRegistration of new users is disabled.
Message : System.Exception: Registration of new users is disabled. RelatedId xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx

Is there a way around this?

4Paul4 · 07-12-2012, 08:15 PM

I have it working now. I needed to sanitize the input.
I found this thread http://stackoverflow.com/questions/8...ize-user-input; more specifically, I used this command:

Code:

export CLEANSTRING="`echo -n "${STRING}" | tr -cd '[:alnum:] [:space:]' | tr '[:space:]' '-'  | tr '[:upper:]' '[:lower:]'`"

Exceptions are coming through via SNS now. Thanks for the help.