Advice sought on Profiling Tools
 

I am looking for preferably Open Source tools to aid in profiling an application running under Linux, either SUSE or Red Hat, preferably 64-bit but possibly 32-bit. I am interested in performance issues, especially where and how long the app is spending its time. I have looked at OProfile however its shortcoming is that it only samples "occasionally". I am interested in something that might have fuller coverage. But I would also prefer not to have to recompile to instrument the tool/code. There are pieces that seem like they should all fit together nicely if you use bits from each of them, e.g., strace, ltrace, or lsstack, and then parsing those outputs and looking at more particulars.
But is there some "magic" tool out there that "does it all"?

All advice will be greatly appreciated, and hopefully someone will point me in the right direction to find the tool of my dreams.

Thank you very much, Coach

It would help if you could be more specific about things. It depends on what the application does, performance on *what* plane you're talking about (if you already narrowed that down) and how slash where you're testing this. If you already tried a wide array of tools and none of the standard ones gave you satisfactory output then (depending on what you need to see) you might conclude that non-invasive profiling just isn't going to cut it and (provided it's not a stable production environment) you might have to resort to an instrumented application or an instrumented kernel like Linux Kernel State Tracer (LKST), LTTng / LTTV or SystemTap or use both. As said before: being specific about things usually yields more satisfactory replies.

The only one I know of is gprof, see whether it helps.

Thanks unSpawn:
I appreciate that being vague can make it more difficult to get better and more complete answers. It is a tough decision: what is enough information vs what is too much. And it is probably better to err on the side of too much.
And I do appreciate the possibles you were able to provide.
Our team is responsible for performance testing and analysis of primarily a low latency messaging suite running at the plumbing level of the PC (interfacing directly to the network stack, at the TCP/UDP layer), along with higher level applications that leverage that messaging. Our task is two-fold really; part 1 is to find and isolate bugs in the code developed by others, and part B is to analyze the performance of the various components. Finding bugs is never too hard; but figuring out where they are actually infesting can be difficult. The analysis part is more interesting, but the challenge is avoiding the Heisenberg principle. Hence the desire for something that is not too obtrusive.
There seem to be a number of different options out there. It is surprising, however, that when you look at what they are actually doing the number of them that are all using some other tool (like fill-in-the-blank-for-the-first-letter-PROF) and are only providing a wrapper to supply a "nice" report. We were hoping to find some additional or different perspective/insight to available tools, to find things that others had had some success with, to make our research into available tools a little quicker and less painful.
Again, thanks to you and others for your helpful responses!
Coach

I've previously looked at LKST and wasn't enthralled. LTTng might be the go - I have pulled the patchset, but will (probably) have to revert back to a 2.6.25 kernel to apply and test it. Might have a go tomorrow. (*)
More tools are appearing all the time - I've tried latencytop and systemtap for cruising around (the edges of) the system. I don't use either to any great depth it must be said, as I don't have a particular issue to chase.

(*): oops sorry, make that perfmon2. aghhh ...
Returning you to your regular thread, whilst I go back to sleep ...