I'm trying to analyse the performance of a file processing setup. Client is given files, sends to server for processing, records result and moves files into two folders depending on whether the file is wanted or not. The server's job is more intensive, so the bottleneck is on the server, but the load profiles look odd. It's multi-stage, so some files need analysing at only level1, some need level1 then level2, and some go all the way to level3. At level3, processing speed is directly related to number of times it's running 'interesting process'. Sometimes during the processing of a batch we see a smaller number of these than we expect.

The aim is to get a combination of cpu statistics, number of times 'interesting process' is running and log this out to a file so that we can a) graph what's going on to get a feel of the thing and b) cross-reference this to the logs on the client to find what the client is up to in the slow bits of the batch cycle.

I've tried , and it's great for most of the metrics, but I can't find a way to count the instances of 'interesting process' with it.

Currently I'm running: This gives me a nice text file with a row every 2 seconds containing

• a timestamp for the record
• load averages
• number of times the machine is running the process I'm interested in

Load average isn't accurate enough, or rather more specifically, it lags too much. I need to include instantaneous CPU usage. I could do with the something like the output from the top row of or , but just appended to the line in the output file, and unlabeled, so that I can then run analysis on it and x-ref to the client logs.

I've also tried adding into the watch, but it doesn't work. On its own the command works fine, but when you wrap it into the watch command, you get idle time for each cpu, on separate lines - not funny with multiple cores.

Ideal output would be Timestamp, Number of times 'interesting process' is running, cpu %usr %sys %ni %idle %wait %hi %si

so the data would be

10:04:42, 16, 58.0 us, 26.0 sy, 0.0 ni, 12.0 id, 0.0 wa, 0.0 hi, 0.0 si
10:04:44, 18, 64.0 us, 29.0 sy, 0.0 ni, 7.0 id, 0.0 wa, 0.0 hi, 0.0 si

As a bonus section, is there a way I can add onto the end of the row "percentage of I/O channel to disk used" to help work out if I'm actually I/O bound?

Am I heading the right way or is there a better way? - any ideas appreciated.

What do you want to achieve at all? What is your goal?

I do not really understand it.
So I guess the goal is to speed things up and probably you want to find a bottleneck.

sar would be a good tool to see the load of the host and to check if you have enough resources.
To analyze a process you need to use different tools, but it depends on the process itself (which language, libraries and other things. do you have sources...).

The load itself is not accurate and actually I would say that is completely useless.

Hmmm - so many misconceptions.
loadavg is not CPU% - go read this - be warned it will make your eyes water.
As for "instantaneous CPU usage", that has precisely 2 values - 0% and 100%. Either a CPU/core/whatever is active or it ain't. The values you see (and want) are time based averages. Usually averages of averages - can you see where this is heading ?.

Piping commands like that add enormous (in CPU terms) latency into the stream - especially the pointless tee. The only way to get semi-sensible metrics is to use a tool that generates sane historical records that you can parse later. sar is too coarse - have a look at collectl or collectd. The interactive commands might be useful for indicative ideas, but post processing is the real answer.

1 members found this post helpful.

Cross-posted: https://serverfault.com/questions/10...ll-to-single-l

Thanks for the pointers.
I'm aware of the difference between loadavg and current usage, and I accept that at a particular instance in time the CPU is either at 100% or 0%.
Yes, what I need is an average, but over a considerably shorter timescale than 1 minute.
Thanks again, I'll have a look at collectl and collectd.