High system CPU usage
 

Hi all,
I have a quad-processor, (12x4) 48-core server, with a scientific calculation software currently used to compute chemical simulations.
I'm running a simulation now, which uses 12 cores, with mpi parallelization. I noticed that when the system is idle, or when I run single-core simulations, the sys% of CPU usage reported by vmstat is always close to 0. When I run the simulation, instead, the user% and sys% seem to "steal" one another the cpu time: sys stays close to 0 for a minute or so, then it drastically lowers and at the same time the sys% grows.
Is this behaviour normal? Is the kernel, for some reason, stealing cpu time from useful calculations? How could I assess the if this is a bottleneck for my calculation speeds?
I already found out that those spikes of system cpu usage do not derive from disk writes, and that IO waiting time is always 0.

Thank you

Hi -

Q: When the trend changes, is your simulation process still the only process using most of the CPU?
"top" is a good way to check.

Q: When the trend changes, is your process swapping?
Again, "top" is a good way to check.

Q: Assuming your simulation process is the only factor, and assuming you have enough RAM at all times, then you might want to profile your app in search of potential "hot spots".

Yes this is normal. Some of the activity caused by the program can be executed in user space while other activity is executed in kernel space. All programs have these characteristics. For example doing arithmetic can be executed in user space while allocating memory must be done in kernel space.

As paulsm4 mentioned you can use several tools to profile your system workload. top, atop, htop and sar are my favorites. atop and sar can collect data over some period of time and provide reports of activity and resource usage. top and htop are interactive and will show you what is happening at the time that you run them.