hmm, just tried sar with a 1 second sample just to see what it does since I always use collectl
Look at this from sar:
sar -u 1 8640
Linux 2.6.18-128.el5 (hpdc3dmgt1) 08/10/2010
06:24:43 PM CPU %user %nice %system %iowait %steal %idle
06:24:44 PM all 0.75 0.00 1.00 0.00 0.00 98.25
06:24:45 PM all 0.12 0.00 0.37 0.50 0.00 99.00
06:24:46 PM all 0.12 0.00 0.12 0.50 0.00 99.25
06:24:47 PM all 0.38 0.00 0.00 0.00 0.00 99.62
and then this from collectl
collectl -sc --verbose -oT
# CPU SUMMARY (INTR, CTXSW & PROC /sec)
# User Nice Sys Wait IRQ Soft Steal Idle Intr Ctxsw Proc RunQ Run Avg1 Avg5 Avg15
18:25:07 1 0 0 0 0 0 0 98 1048 363 2 588 0 1.54 1.27 1.17
18:25:08 0 0 0 1 0 0 0 98 1032 295 2 588 0 1.50 1.27 1.17
collectl reports CPU consumed for IRQ processing which is pretty important if you want the complete picture. The other thing about collectl is you can look at the summary data level like I showed above and if there is a particular time of interest play the data back again and show utilization by individual CPU since averages can be misleading! You might have a cpu summary that says 25% when in fact you have 1 cpu pegged at 100% processing interrupts and the other 3 sitting idle.
also, I always suggest looking at a lot more than just cpu because maybe something else is contributing to the problem. What if some device is generating a ton if interrupts and you want to know which device?
-mark