Using SNMP Traps
 

For a client, and at the client's behest, I am configuring a module that collects real-time data to send updates about that data via SNMP traps to a specified SNMP Trap daemon, which then invokes a command (which I also have written) to collect the data in the trap, open a socket to an application that is located someplace else (might be localhost, might not), and forward that data.

While I think this is roundabout, it is what the client has asked for.

I have encountered a peculiar problem though, which appears to be a show-stopper.

My application may send out Trap messages as frequently as 20 times per second, depending on what it sees in the data it collects. I have built the trap message correctly (at least, it is being recognized by snmptrapd correctly) and in testing of individual messages, everything works. Snmptrapd invokes my command, which collects the data and ships it to the next module via the TCP socket.

When I run my data collection app full-speed, things go to hell. Specifically, the message handling rate is abysmally slow.

Snmptrapd seems to receive the trap, wait 7 seconds, then forward the trap to the command that processes it. After another 7 seconds, snmptrapd accepts another trap and the process repeats.

I have eliminated my command as the delay; when I replace the command with a sample BASH script the same thing happens. It certainly appears that snmptrapd is taking its good old time about processing trap messages if there are multiple messages pending.

I realize that snmptrapd is supposed to be processing extraordinary conditions, but as I say this is what my client has asked me to do. And in any case, the time lag involved is absurd. There must be something wrong.

Can anyone here shed some light on this?

Attached is an excerpt from /var/log/messages that shows what is happening.

Interesting. Does tracing snmptrapd show any leads wrt waiting?

total guess, but is snmptrapd trying to do name lookups? as in, not using '-n' ?

To this point I have diagnosed far enough to establish that the problem specifically has to do with snmptrapd. I have not performed any diagnostics particularly on snmptrapd to try to identify the problem. When I was satisfied that snmptrapd was the problem, I had reached a stopping point anyway and I decided to mull it over and do some research before proceeding. Asking here is part of my research.

So I have not yet tried to trace it, and it is running with no options set; just (per ps -ax | grep snmptrapd)

16750 ? Ss 0:01 /usr/sbin/snmptrapd

However, while mulling it over, something did occur to me. The time of several seconds *could* be a timeout occurring.

A bit more background. In this project, I have written the handler for the card that I am using, and I have written part of the driver. I have written all the signal processing code and am now writing all the signal management code (which is where snmp comes in). For snmp, I have written the MIB, and I have written a plugin module for snmpd to handle the data for the system I am building. That plugin module manages several pieces of data that are being given to SNMP, but it specifically does not include any code for the OID that is the subject of the trap; this OID is identified in the MIB but I do nothing with it in snmpd.

So what I am wondering is whether for some reason snmptrapd is talking to snmpd and trying to get snmpd to do something with this OID, that snmpd knows about but has no code to handle. I have learned along the way that snmpd will pass things to the plugin module then wait until the plugin module responds. The module should be responding immediately if it has nothing to do with the OID, but could perhaps snmpd not be happy with that? And perhaps retrying, or waiting or something?

I have found the specific behavior of snmpd to be rather badly documented and I have learned a lot by just exercising the system. Could it be that I need to write some code to do something with this OID in my plugin module?

Good guess. I added the -n option to snmptrapd and now things are working the way you'd expect them to work.