ALL,

I am trying to get the correct CPU usage using top. I ran collectD and I got different results. This especially happens when the CPU usage is close to 100%.

Has anyone experienced having the top command report different results than other CPU usage utilities?

All the tools I know use sampling - by definition this is only as accurate as the data collected.
Also helps if you measure the same thing. Have a look at the collectD FAQ for a brief discussion.

I agree that the data sampling must be good. Both collectD and top use the /proc/stat file to calculate its results.

The test between top and collectD were running the same data on the same system. But the results were different. The frequency of every ten seconds was the same too.

The only thing I can think of is that collectD has a daemon collect the data from /proc/stat and sends it to another client program it has. Now top is just one process and it does all of the calculations and display by itself. So it is doing much more work on the fly.

I feel that top has its limitations. It looks like top has a few bugs in it.

The only other difference I could think of is priorities on top vs other daemons or tasks in the system.

How were you running top - with "-d 10", or "-n 1" in a timed loop ?. Makes a big difference; the manpage warns about using a single invocation.

I looked into this a while back too for similar reasons I suspect. As it happens, I have an old strace of top laying around. It probes /proc/stat at the beginning, processes all the pids (regardless of whether you limit it to specific pid/user), then probes /proc/stat again, presumably to work out the summary numbers (I haven't looked at the code).
This time lag skews all the numbers somewhat.

You call it a bug, maybe it's just working as designed. All code has design decisions, you just have to somehow figure out what they were and how that might affect you.
I haven't tried collectD - thanks for that, I'll give it a try-out. You might also want to have a look at collectl.

I am running it with the -d 10 option. The exact command is top -b -d 10. So that is what I was entering at the command line.

Did you notice how much of a lag did you have?

syg00 - interesting comment about the way top calculates the CPU time - in other words it's actually measuring its own contribution to the load, something I guess I hadn't thought too much about with collectl, but when doing multiple things there's no getting around it. If you report CPU and process metrics you'd have the same result. On the other hand if you ONLY measure CPU with collectl that's all you'll get as it reads /proc/stats and that's all. Reading though all the /proc/pid structures to get the top processes IS very heavyweight and a reason collectl defaults to only reporting these stats once a minute.

The other thing I'd comment on is if you really want to see what the CPU is doing, I'd run collectl with a sampling interval of 1 second or even 0.1 seconds. The system will barely notice the load. And to get even more data use --verbose. In other words:

collectl -sc --verbose [default=-i1]

-mark