Need Help with SNMPTT and NAGIOS config

kcleveland · 11-23-2010, 06:46 PM

Okay so I am officially stuck- I've tried everything I can think of, so I'm hoping you guys can jump in and provide some help.

I have setup a service in nagios that acts as an snmp trap receiver for a switch on our network. It shows up in the web interface as a service running under the switch that is being monitored.

I can send a trap from the switch to my nagios server, and here is what I get:

Code:

tail /var/log/snmptt/snmptt.log

Tue Nov 23 17:04:50 2010 .1.3.6.1.6.3.1.1.5.3 Normal "Status Events" extreme400 - Link down on interface 12.  Admin state: up.  Operational state: down 
Tue Nov 23 17:04:55 2010 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" extreme400 - Link up on interface 12.  Admin state: up.  Operational state: up

The exact same output also shows up in /var/log/messages

However, I never get the email alert or see the color change in the web interface?

Also, it should be noted before moving on, that my nagios setup does indeed send mail correctly for other monitored services when they alarm, and that the other services running under this host also send email correctly when they alarm- just not this one service, so I'm thinking SNMPTT is not actually ever passing it to nagios correctly.

Here is my host template:

Code:

define host{
        name                    generic-switch
        use                     generic-host
        check_period            24x7
        check_interval          5
        retry_interval          1
        max_check_attempts      10
        check_command           check-host-alive
        notification_period     24x7
        notification_interval   30
        notification_options    d,u,r,f,s
        contact_groups          admins
        register                0
        }

And service template:

Code:

define service{
        name                            snmptrap-service
        use                             generic-service
        service_description             TRAP
        is_volatile                     1
        flap_detection_enabled          0
        check_command                   check-host-alive
        max_check_attempts              1
        normal_check_interval           1
        retry_check_interval            1
        passive_checks_enabled          1
        notification_options            d,u,r,f,s
        contact_groups                  admins
        register                        0
        }

And the actual host definition:

Code:

define host{
        use             generic-switch
        host_name       extreme400#2
        alias           Extreme x450a 
        address         XXX.XXX.XXX.XXX
        contact_groups  admins
        hostgroups      switches
        }

And the actual service definition:

Code:

define service{
        use                     snmptrap-service
        host_name               extreme400#2
        service_description     TRAP
        contact_groups          admins
        }

The /etc/snmp/snmptt.conf file:

Code:

EVENT enterpriseSpecific .1.3.6.* "Status Events Normal"
FORMAT Wildcard Test Trap from $r.
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "Wildcard trap received"
SDESC
A generic wildcard test trap received
EDESC

Does anyone have any idea why the trap is not making that last short leg of the journey to nagios?

Thanks!

kcleveland · 11-24-2010, 08:02 AM

Okay, so I think I've found what is causing the problem, and have also fixed it:

After running the command in my snmptt.conf file straight from the command line (this command can be found in several tutorials on the net, but it doesn't seem to work as is):

Code:

/usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "Wildcard received"

And the command runs successfully, but here is what /var/log/messages has to say about it:

Code:

Nov 24 05:49:37 ubuntu nagios: EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;TRAP;1;Wildcard received;
Nov 24 05:49:37 ubuntu nagios: Warning:  Passive check result was received for service '1' on host 'TRAP', but the host could not be found!

So I'm thinking it was a problem with the way the snmptt.conf command is being run.

It says a check was received for service 1 on host TRAP, which is not correct, so I changed the command to this and ran it again:

Code:

/usr/local/nagios/libexec/eventhandlers/submit_check_result extreme400#2 TRAP 1 "Wildcard received"

And now it works like a charm- email notification came through right away

However, I am still very curious as to why the $r variable is not holding my hostname correctly- could it be the fact that my hostname has a pound sign in it (extreme400#2) ? Any ideas or easy ways to check? I already echoed the "$r" variable after running the command to see its contents and it is indeed an empty variable- which is why the original command erroneously though my hostname was TRAP.

kcleveland · 11-24-2010, 09:31 AM

Okay, so maybe I didn't fix the problem completely-

I've setup snmptt to translate any traps that I receive into Nagios alerts. I get the traps just fine, and when I run /usr/local/nagios/libexec/eventhandlers/submit_check_result extreme400-2 TRAP 1 "Wildcard trap received" (same command from my snmptt.conf file) at the command line, I soon after get the email notification (so I know the .cfg's are good), and when I cat /var/log/messages everything looks good:

Nov 24 06:38:00 ubuntu nagios: EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;extreme400-2;TRAP;1;Wildcard received
Nov 24 06:38:05 ubuntu nagios: PASSIVE SERVICE CHECK: extreme400-2;TRAP;1;Wildcard received
Nov 24 06:38:05 ubuntu nagios: SERVICE ALERT: extreme400-2;TRAP;WARNING;HARD;1;Wildcard received
Nov 24 06:38:05 ubuntu nagios: SERVICE NOTIFICATION: nagiosadmin;extreme400-2;TRAP;WARNING;notify-service-by-email;Wildcard received
Nov 24 06:38:05 ubuntu nagios: SERVICE ALERT: extreme400-2;TRAP;OK;HARD;1;PING OK - Packet loss = 0%, RTA = 0.68 ms
Nov 24 06:38:05 ubuntu nagios: SERVICE NOTIFICATION: nagiosadmin;extreme400-2;TRAP;OK;notify-service-by-email;PING OK - Packet loss = 0%, RTA = 0.68 ms

The problem comes when I don't issue the above command locally- instead I generate a trap from the extreme400-2 to the Nagios server, and I can see the trap arrive at the Nagios server from cat /var/log/messages:

Nov 24 06:48:06 ubuntu snmptt[0]: .1.3.6.1.6.3.1.1.5.3 Normal "Status Events" extreme400-2 - Link down on interface 12. Admin state: down. Operational state: down
Nov 24 06:48:12 ubuntu snmptt[0]: .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" extreme400-2 - Link up on interface 12. Admin state: up. Operational state: up

However, that's as far as the trap ever gets- it doesn't go on to process the trap and send the process service check result up to Nagios so it can generate an alert or send out an email notification.

My snmptt.conf file is setup to execute the submit_check_result eventhandler command whenever it receives a trap from .1.3.6.* so I'm not really sure why its not passing it up to nagios and generating an alert. Here is my snmptt.conf file:

EVENT enterpriseSpecific .1.3.6.* "Status Events Normal"
FORMAT Wildcard Test Trap from $r.
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result extreme400-2 TRAP 1 "Wildcard test trap received"
SDESC
A generic wildcard test trap received
EDESC

Any advice or direction will be greatly appreciated.

kcleveland · 11-25-2010, 11:52 AM

Okay, so more progress has been made- I turned on debugging in the snmptt.ini file, sent a trap from the switch to the nagios server, the looked at the snmptt.debug log - here is what it said:

Code:

tail /var/log/snmptt/snmptt.debug 


OID of received trap: .1.3.6.1.6.3.1.1.5.4.  Will attempt to translate to text
  Translated to linkUp
Link up on interface 12.  Admin state: up.  Operational state: up

.1.3.6.1.6.3.1.1.5.4 Normal "Status Events" extreme400-2 - Link up on interface 12.  Admin state: up.  Operational state: up


EXEC line(s):
  EXEC line not defined

So snmptt is translating the trap correctly, and it is trying to run the EXEC command in snmptt.conf which notifies Nagios, but it says that the EXEC line is not defined?? Here is my snmptt.conf file -

Code:

EVENT enterpriseSpecific .1.3.6.* Normal "Status Events"
FORMAT Wildcard Test Trap from $R.
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $R TRAP 1 "Wildcard test trap received"
SDESC
A generic wildcard test trap received
EDESC

I can run submit_check_result manually, so I know it is there and working - does anybody know why it cannot find the EXEC statement in my snmptt.conf file?

kcleveland · 11-29-2010, 08:19 AM

Guys, I really need some helping out on this one - I know someone out there must have encountered this before, please jump in!!

loganseth · 03-18-2011, 09:55 AM

I have a similar issue - did you ever solve this?

I am at the point where my debug output shows the exec command it ran:

EXEC command:/usr/local/nagios/libexec/eventhandlers/submit_check_result [MyHostName] TRAP 1 "[MyMessage]"

and when I manually run that (with sudo first) it works, hands it off to Nagios, et voila.

But on it's own. nada.

I'm thinking a permissions issue but can't track it down!

drwxr-xr-x 2 nagios nagios 4096 2011-03-18 09:26 eventhandlers
-rwxr-xr-x 1 nagios nagios 1182 2009-11-07 09:03 submit_check_result

where /etc/group:
nagios:x:1001:snmptt

What could be missing!? SO CLOSE!

loganseth · 03-18-2011, 01:10 PM

For my particular post, this resolved it:

http://forums.meulie.net/viewtopic.php?f=45&t=4616

(changed #!/bin/sh to #!/bin/bash in submit_check_result)

ARGH!!

ashish_chandra · 06-13-2013, 06:00 AM

I was facing the same issue ; and the issue was very trivial .
You need to disable the selinux in /etc/sysconfig/selinux .

This resolves the issue .

Thanks

kcleveland · 06-14-2013, 08:11 AM

No, selinux is not the problem. Please do not troll looking to dig up old threads to post on.

dd-linux · 10-30-2014, 04:48 AM

For me I think it is a permission issue, though I am not sure and really cannot work around the problem. Any help will be appreciated.

While trying to run the following submit_check_result command in my CLI

Code:

/usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "A psAlarmNoDirectoryRequesterProtocolSpecified trap $*"

I get the following result,

Code:

bash: /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "A psAlarmNoDirectoryRequesterProtocolSpecified trap $*" : Permission denied

Any clues on how to set the permission right?

Thanks.
-D

robcar · 05-30-2015, 02:30 AM

I post in this old thread because I found a solution for my similar case which, hopefully, could help others and spare some days in troubleshooting.

In my case I successfully had managed to have some hosts (HP switches) service TRAPs end up 'in red' in Nagios, but not other hosts (a Websense appliance and some VMWare ESX hosts).

What lead me to the solution was the log:

Quote:

/var/log/nagios3/nagios.log

I sent a couple of test traps from the Websense appliance and took a look at the log:

Code:

[1432897516] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;websense-wcg.domain.loc.domain.loc;TRAP;2; Websense Alert: this event is for testing use
[1432897516] Warning:  Passive check result was received for service 'TRAP' on host 'websense-wcg.domain.loc.domain.loc', but the host could not be found!
[1432897526] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;ws02faiv-wcg.domain.loc;TRAP;2; Websense Alert: this event is for testing use
[1432897526] Warning:  Passive check result was received for service 'TRAP' on host 'websense-wcg.domain.loc', but the host could not be found!

Where did it took those hostnames, websense-wcg.domain.loc.domain.loc and websense-wcg.domain.loc, from?
Of course from DNS! The Websense test trap was doing a reverse query to my DNS and using the hostnames in the traps instead of IPs.
So I deleted both RRs in DNS and added a single RR for the IP of the appliance: proxy.domain.loc

Then I had to define a Nagios host definition matching the exact FQDN hostname that the trap was sending:

Code:

define host{
        host_name               proxy.domain.loc
        address                 10.10.50.57
        parents                 core-switch
        use                     generic-host
        notes                   Websense V10000 Appliance
}

And finally I got the notification in Nagios:

Code:

[1432897531] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;proxy.domain.loc;TRAP;2; Websense Alert: this event is for testing use
[1432897536] PASSIVE SERVICE CHECK: proxy.domain.loc;TRAP;2;Websense Alert: this event is for testing use
[1432897536] SERVICE ALERT: proxy.domain.loc;TRAP;CRITICAL;SOFT;1; Websense Alert: this event is for testing use

Hope that above could help someone.

--
rob