LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 10-10-2017, 12:05 AM   #1
vasya_petrovich
LQ Newbie
 
Registered: Oct 2017
Posts: 1

Rep: Reputation: Disabled
High CPU load by native_queued_spin_lock_slowpath


I'm using Debian 9(stretch edition) kernel 4.9., hp dl385 g7 server with 32 cpu cores. NIC queues are tied to processor cores. Server is shaping traffic (iproute2 and htb discipline + skbinfo + ipset + ifb) and filtering some rules by iptables.

At that moment, when traffic goes up about 1gbit/s cpu is very high loaded. Perf tool tells me that kernel module native_queued_spin_lock_slowpath loading cpu about 40%.

Who knows how to find the problem?
 
Old 10-10-2017, 01:02 AM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,328

Rep: Reputation: 4176Reputation: 4176Reputation: 4176Reputation: 4176Reputation: 4176Reputation: 4176Reputation: 4176Reputation: 4176Reputation: 4176Reputation: 4176Reputation: 4176
A quick seach found this - makes interesting reading.
 
1 members found this post helpful.
Old 10-11-2017, 09:46 AM   #3
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 11,089
Blog Entries: 4

Rep: Reputation: 4085Reputation: 4085Reputation: 4085Reputation: 4085Reputation: 4085Reputation: 4085Reputation: 4085Reputation: 4085Reputation: 4085Reputation: 4085Reputation: 4085
You are quite-possibly exceeding the ability of the hardware to move packets in-and-out. There are too many CPUs simultaneously trying to queue and to handle work ... more than the physical hardware can actually do. Therefore, they are "Ferraris stuck in traffic on a two-lane highway."

A spin-lock should actually "spin" only briefly and only occasionally. If you find a significant amount of time being spent "spinning," realize that this time is also "100% wasted."

You only need to dedicate enough CPUs to this task to handle "1 gigabit per second." You accomplish nothing of value by dedicating more CPUs to the task such that they merely wait for one another, especially given that they are literally-wasting time in a spin-lock when they should be finding something else to do. (Other important work might not be being done, because everyone's spinning their wheels.)

You might well find that only one CPU is actually needed to marshal the I/O requests. (Or, one CPU per physical NIC.) The others don't need to be concerned with it at all. You can use "affinity" rules to distribute computing resources among competing priorities.

Last edited by sundialsvcs; 10-11-2017 at 09:48 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] High CPU load, top not showing any high CPU process angel115 Linux - Server 8 10-12-2015 01:58 AM
mysql server not responding with high cpu usage and high load avgs landysaccount Linux - Server 2 09-15-2013 04:46 AM
[SOLVED] High CPU load, but low CPU usage (high idle CPU) baffy Linux - Newbie 5 03-13-2013 10:24 AM
apache high cpu load on high traffic awww Linux - Newbie 9 10-04-2011 03:22 PM
High load and high cpu kernel usage enid Linux - Server 8 09-30-2010 04:33 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 08:58 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration