Tricky debugging problem - threads + dynamically linked driver with fixed interface
I'm looking for some ideas on how to tackle this tricky debugging problem.
Quick description of problem: Very occasional lockups in SQLBindCol, from UnixODBC using FreeTDS driver.
Constraints: The problem occurs rarely (many days can pass without incident), on a performance-critical system and cannot be replicated on a system where verbose debugging can be left turned on. I can't do run-time debugging of the program when the freeze occurs.
I can detect the problem the program which calls the SQLBindCol function. The way I did this was to create an array of messages through the critical code, adding timestamped "got to part 1" kind of things, and then when the problem happens (an exception is raised) I dump out this structure.
I would like to do something similar with the internals of the SQLBindCol function.
However, I don't know how to get a message to the driver to dump out recent timing information.
My first attempt was to add a new ODBC call in unixodbc and implement this in the driver - something like SQLBindColDebug, which would accept an additional parameter - a pointer to a structure in the calling program which would be populated with timing information while SQLBindColDebug runs... I could then dump the contents when I get an exception.
However, it seems that adding new calls to UnixODBC is a real pain - I'm basically changing the ODBC API and then having to implement my new API calls in the FreeTDS layer. It's a lot of changes, and basically I failed to get it working after several days of trying.
Can someone suggest another approach without having to change the interface to the driver?
One more complication - the program is multi-threaded, with multiple threads make the same SQLBindCol ODBC call... else I might consider some sort of scheme with a static variable to store debug information, and think up some mechanism of triggering this when the exception is raised.... not sure about that either.