[SOLVED] Tracking down high loadavg

bucovaina78 · 07-22-2014, 10:12 AM

Hi there,

I've got a bit of a strange situation here. I've got a server running about 30 virtual desktops. It has a loadavg of about 18 with spikes to 25 on a quad core machine and I don't know where it comes from.

What I've checked so far:

The swap partition (5GB) is nearly full but no swap in/swap out activity is taking place so I guess this is a left over from some process in the past but causing no harm.
free shows I've got 38GB of 94GB RAM free
top shows the CPU load as follows: Cpu(s): 9.9%us, 2.6%sy, 0.0%ni, 87.4%id, 0.1%wa, 0.0%hi, 0.1%si, 0.0%st
iostat shows virtually no read or write activity on any disk.
I've checked IRQ activity and by far the highest IRQ activity I can see is on the ethernet port

Anyone an idea what else can cause such a high loadavg on a seemingly unloaded machine?

T3RM1NVT0R · 07-22-2014, 03:18 PM

Can you share the output of the following:

1. grep proc /proc/cpuinfo
2. lscpu
3. vmstat 1 10
4. uptime
5. free -m

Put the output of above commands in "code" so that it will be easy to read. Also, which linux you are using?

syg00 · 07-22-2014, 07:37 PM

If it affecting your SLAs ?. If not why do you care.
Oftentimes it's some crappy software (Oracle, Apache come to mind) that drop threads into uninterruptible sleep and forgets about them. Try this and post the output as well

Code:

top -b -n 1 | awk '{if (NR <=7) print; else if ($8 ~ /[RD]/) {print; count++} } END {print "Total: "count}'

bucovaina78 · 07-23-2014, 12:40 AM

I guess I have found what's going on with the command above. There are some processes in uninterruptible sleep. NFSv4 mounts, authenticated with Kerberos. Somehow I guess the ticket has expired and the process is now trying to access the disk until, ... forever.

syg00 · 07-23-2014, 07:05 PM

Nope - when in state "D" they are doing nothing - except contributing to loadavg. This is different to Unix loadavg calculation (which is what you read of most on the web).

bucovaina78 · 07-24-2014, 12:39 AM

Thanks for clarifying!