For a client, and at the client's behest, I am configuring a module that collects real-time data to send updates about that data via SNMP traps to a specified SNMP Trap daemon, which then invokes a command (which I also have written) to collect the data in the trap, open a socket to an application that is located someplace else (might be localhost, might not), and forward that data.
While I think this is roundabout, it is what the client has asked for.
I have encountered a peculiar problem though, which appears to be a show-stopper.
My application may send out Trap messages as frequently as 20 times per second, depending on what it sees in the data it collects. I have built the trap message correctly (at least, it is being recognized by snmptrapd correctly) and in testing of individual messages, everything works. Snmptrapd invokes my command, which collects the data and ships it to the next module via the TCP socket.
When I run my data collection app full-speed, things go to hell. Specifically, the message handling rate is abysmally slow.
Snmptrapd seems to receive the trap, wait 7 seconds, then forward the trap to the command that processes it. After another 7 seconds, snmptrapd accepts another trap and the process repeats.
I have eliminated my command as the delay; when I replace the command with a sample BASH script the same thing happens. It certainly appears that snmptrapd is taking its good old time about processing trap messages if there are multiple messages pending.
I realize that snmptrapd is supposed to be processing extraordinary conditions, but as I say this is what my client has asked me to do. And in any case, the time lag involved is absurd. There must be something wrong.
Can anyone here shed some light on this?
Attached is an excerpt from /var/log/messages that shows what is happening.
Code:
Apr 13 14:18:31 Dadsbox snmptrapd[16750]: 192.168.0.10 [UDP: [192.168.0.10]:32769]: Trap , DISMAN-EVENT-MIB::sysUpTimeInstance 6512948, SNMPv2-MIB::snmpTrapOID.0 ExacqSCAN-MIB::exacqnotiftrap, ExacqSCAN-MIB::scanLastData.1 "dsize=100@drops:1441951500,90000@adds:1441974000,63000,65129484,-43.202725,0@colls:"
Apr 13 14:18:38 Dadsbox scan_traphandler[17445]: scan_traphandler is processing a trap
Apr 13 14:18:45 Dadsbox snmptrapd[16750]: 192.168.0.10 [UDP: [192.168.0.10]:32769]: Trap , DISMAN-EVENT-MIB::sysUpTimeInstance 6512950, SNMPv2-MIB::snmpTrapOID.0 ExacqSCAN-MIB::exacqnotiftrap, ExacqSCAN-MIB::scanLastData.1 "dsize=50@drops@adds@colls:1441974000,63000,65129484;"
Apr 13 14:18:52 Dadsbox scan_traphandler[17483]: scan_traphandler is processing a trap
Apr 13 14:19:00 Dadsbox snmptrapd[16750]: 192.168.0.10 [UDP: [192.168.0.10]:32769]: Trap , DISMAN-EVENT-MIB::sysUpTimeInstance 6512952, SNMPv2-MIB::snmpTrapOID.0 ExacqSCAN-MIB::exacqnotiftrap, ExacqSCAN-MIB::scanLastData.1 "dsize=100@drops:1441974000,63000@adds:1441893000,184500,65129518,-67.087541,0@colls:"
Apr 13 14:19:09 Dadsbox scan_traphandler[17542]: scan_traphandler is processing a trap