I am doing some computation with 24 identical processes on a dual 6 core xeon with hypertheading - more than 24 jobs are scheduled by scons to fully utilise the machine.
For a few minutes the processes run fine and the user time is close to 100%, but then suddenly the system time is around 30% (by looking at top or "System Load Indictator").
If I then kill and rerun the same job the system time is immediately 30%.
After a reboot the process runs at 100% user time for a few minutes again.

What could this be caused by ?
What could I do to discover what the problem is ?

cat /proc/version
Linux version 4.2.0-21-generic (buildd@lgw01-50) (gcc version 5.2.1 20151010 (Ubuntu 5.2.1-22ubuntu2) ) #25-Ubuntu SMP Wed Dec 2 18:42:25 UTC 2015

Did you buy or download the distro?
What computation are you running?
Hardware info?

The problem is fixed by doing
sudo rmmod intel_powerclamp.

Apparently intel_powerclamp injects CPU doing nothing time to reduce the temperature of the CPU. So intel_powerclamp thinks the computer CPU is too hot.

And yes it is hot here - problem solved.

ACPI fan control not working on Dell Precision T7500 for some reason, the CPU was being throttled by this intel_powerclamp system because it was getting too hot.

The fan control had never been working.

I could not find a fix for the fan control, it has the latest BIOS, tried acpi_osi=Linux kernel parameter, fancontrol program did not work.

I wrote a python script to control the fans based on sensor temps, see attached.

Attached Files

control_fans.py.txt (952 Bytes, 8 views)

1 members found this post helpful.

And it probably is "too hot!"

Generally speaking, today's consumer microprocessors and motherboards are not designed to run at near-100 utilization ... nor even close. They overheat because they are not equipped with heat sinks sufficient to get rid of it. If the chip overheats, it will drop dead. For good.

Probably the best thing to do is to "back off." Don't attempt to run the thing at 100 utilization.

There is also another possibility that can cause performance to seriously drop, and that is virtual-memory paging. Be sure that the machine has adequate "real RAM," and that you are not doing anything that would over-commit memory. Paging causes unpredictable delays on the order of milliseconds, while also competing for use of the (probably, "only one?") disk drive.

Thanks @alexibu - gotta get back into python properly. That'd be after some time looking at lua, ... and golang, er, and scala.

Damn, I wish these folks wouldn't invent a new language every time I go to bed ...