LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 04-13-2009, 05:47 PM   #1
jiml8
Senior Member
 
Registered: Sep 2003
Posts: 3,171

Rep: Reputation: 116Reputation: 116
Using SNMP Traps


For a client, and at the client's behest, I am configuring a module that collects real-time data to send updates about that data via SNMP traps to a specified SNMP Trap daemon, which then invokes a command (which I also have written) to collect the data in the trap, open a socket to an application that is located someplace else (might be localhost, might not), and forward that data.

While I think this is roundabout, it is what the client has asked for.

I have encountered a peculiar problem though, which appears to be a show-stopper.

My application may send out Trap messages as frequently as 20 times per second, depending on what it sees in the data it collects. I have built the trap message correctly (at least, it is being recognized by snmptrapd correctly) and in testing of individual messages, everything works. Snmptrapd invokes my command, which collects the data and ships it to the next module via the TCP socket.

When I run my data collection app full-speed, things go to hell. Specifically, the message handling rate is abysmally slow.

Snmptrapd seems to receive the trap, wait 7 seconds, then forward the trap to the command that processes it. After another 7 seconds, snmptrapd accepts another trap and the process repeats.

I have eliminated my command as the delay; when I replace the command with a sample BASH script the same thing happens. It certainly appears that snmptrapd is taking its good old time about processing trap messages if there are multiple messages pending.

I realize that snmptrapd is supposed to be processing extraordinary conditions, but as I say this is what my client has asked me to do. And in any case, the time lag involved is absurd. There must be something wrong.

Can anyone here shed some light on this?

Attached is an excerpt from /var/log/messages that shows what is happening.

Code:
Apr 13 14:18:31 Dadsbox snmptrapd[16750]: 192.168.0.10 [UDP: [192.168.0.10]:32769]: Trap , DISMAN-EVENT-MIB::sysUpTimeInstance 6512948, SNMPv2-MIB::snmpTrapOID.0 ExacqSCAN-MIB::exacqnotiftrap, ExacqSCAN-MIB::scanLastData.1 "dsize=100@drops:1441951500,90000@adds:1441974000,63000,65129484,-43.202725,0@colls:"
Apr 13 14:18:38 Dadsbox scan_traphandler[17445]: scan_traphandler is processing a trap
Apr 13 14:18:45 Dadsbox snmptrapd[16750]: 192.168.0.10 [UDP: [192.168.0.10]:32769]: Trap , DISMAN-EVENT-MIB::sysUpTimeInstance 6512950, SNMPv2-MIB::snmpTrapOID.0 ExacqSCAN-MIB::exacqnotiftrap, ExacqSCAN-MIB::scanLastData.1 "dsize=50@drops@adds@colls:1441974000,63000,65129484;"
Apr 13 14:18:52 Dadsbox scan_traphandler[17483]: scan_traphandler is processing a trap
Apr 13 14:19:00 Dadsbox snmptrapd[16750]: 192.168.0.10 [UDP: [192.168.0.10]:32769]: Trap , DISMAN-EVENT-MIB::sysUpTimeInstance 6512952, SNMPv2-MIB::snmpTrapOID.0 ExacqSCAN-MIB::exacqnotiftrap, ExacqSCAN-MIB::scanLastData.1 "dsize=100@drops:1441974000,63000@adds:1441893000,184500,65129518,-67.087541,0@colls:"
Apr 13 14:19:09 Dadsbox scan_traphandler[17542]: scan_traphandler is processing a trap

Last edited by jiml8; 04-13-2009 at 05:51 PM.
 
Old 04-13-2009, 07:30 PM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594Reputation: 3594
Interesting. Does tracing snmptrapd show any leads wrt waiting?
 
Old 04-14-2009, 01:04 AM   #3
grepmasterd
Member
 
Registered: Aug 2003
Location: Seattle
Distribution: ubuntu, lately
Posts: 182
Blog Entries: 1

Rep: Reputation: 35
total guess, but is snmptrapd trying to do name lookups? as in, not using '-n' ?
 
Old 04-14-2009, 02:55 AM   #4
jiml8
Senior Member
 
Registered: Sep 2003
Posts: 3,171

Original Poster
Rep: Reputation: 116Reputation: 116
To this point I have diagnosed far enough to establish that the problem specifically has to do with snmptrapd. I have not performed any diagnostics particularly on snmptrapd to try to identify the problem. When I was satisfied that snmptrapd was the problem, I had reached a stopping point anyway and I decided to mull it over and do some research before proceeding. Asking here is part of my research.

So I have not yet tried to trace it, and it is running with no options set; just (per ps -ax | grep snmptrapd)

16750 ? Ss 0:01 /usr/sbin/snmptrapd

However, while mulling it over, something did occur to me. The time of several seconds *could* be a timeout occurring.

A bit more background. In this project, I have written the handler for the card that I am using, and I have written part of the driver. I have written all the signal processing code and am now writing all the signal management code (which is where snmp comes in). For snmp, I have written the MIB, and I have written a plugin module for snmpd to handle the data for the system I am building. That plugin module manages several pieces of data that are being given to SNMP, but it specifically does not include any code for the OID that is the subject of the trap; this OID is identified in the MIB but I do nothing with it in snmpd.

So what I am wondering is whether for some reason snmptrapd is talking to snmpd and trying to get snmpd to do something with this OID, that snmpd knows about but has no code to handle. I have learned along the way that snmpd will pass things to the plugin module then wait until the plugin module responds. The module should be responding immediately if it has nothing to do with the OID, but could perhaps snmpd not be happy with that? And perhaps retrying, or waiting or something?

I have found the specific behavior of snmpd to be rather badly documented and I have learned a lot by just exercising the system. Could it be that I need to write some code to do something with this OID in my plugin module?
 
Old 04-15-2009, 12:29 PM   #5
jiml8
Senior Member
 
Registered: Sep 2003
Posts: 3,171

Original Poster
Rep: Reputation: 116Reputation: 116
Quote:
Originally Posted by grepmasterd View Post
total guess, but is snmptrapd trying to do name lookups? as in, not using '-n' ?
Good guess. I added the -n option to snmptrapd and now things are working the way you'd expect them to work.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
SNMP Authentication Failure Traps mstasinski Linux - Server 1 08-22-2014 04:09 PM
SNMP traps receiving noir911 Linux - Server 1 10-11-2008 07:32 AM
How to configure traps with net-snmp fancylad Linux - Server 1 05-25-2008 02:19 PM
Snmp Traps magin Linux - Networking 0 07-13-2007 06:13 AM
snmp traps to syslog jackpal Linux - Networking 2 08-30-2005 04:43 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 10:39 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration