High system CPU usage
I have a quad-processor, (12x4) 48-core server, with a scientific calculation software currently used to compute chemical simulations.
I'm running a simulation now, which uses 12 cores, with mpi parallelization. I noticed that when the system is idle, or when I run single-core simulations, the sys% of CPU usage reported by vmstat is always close to 0. When I run the simulation, instead, the user% and sys% seem to "steal" one another the cpu time: sys stays close to 0 for a minute or so, then it drastically lowers and at the same time the sys% grows.
Is this behaviour normal? Is the kernel, for some reason, stealing cpu time from useful calculations? How could I assess the if this is a bottleneck for my calculation speeds?
I already found out that those spikes of system cpu usage do not derive from disk writes, and that IO waiting time is always 0.