LinuxQuestions.org
Help answer threads with 0 replies.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 01-14-2015, 10:37 AM   #1
ASTRAPI
Member
 
Registered: Feb 2007
Posts: 210

Rep: Reputation: 16
Question Troubleshoot system load 90 on Centos 6.6 64bit


Hi

I am getting a system load of 90 and i have a lot of free ram swap i not used at all and cpu is high about 90% but i am sure that the problem is from a background system command that can't be found using top

Using top all are normal and i have very low numbers as always....

I am not also under attack for sure as i don't have many connections on the server...

Is there anyway to check with more details what is causing this system load?

Maybe an application task is running or maybe a task crash?

After restarting the server is fine now but just to know for the next time...

Thanks
 
Old 01-14-2015, 02:56 PM   #2
John VV
LQ Muse
 
Registered: Aug 2005
Location: A2 area Mi.
Posts: 17,624

Rep: Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651Reputation: 2651
if top is telling you that nothing is using the cores in the CPU

this is a SERVER ? right ?
so gnome is NOT installed

read/ write bottleneck ? maybe

what dose "iotop" say

( might not be installed )
Code:
su -
yum install iotop
 
Old 01-14-2015, 05:12 PM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,128

Rep: Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120
There are not many places where CPU usage is unaccounted. Is 90% CPU a normal usage for you ?. Or do you normally run less than this and it increases as loadavg increases (note these are not the same thing) ?.

Does this high loadavg affect your ability to service user requests ?. Why do you reboot ?.

Without hard data not much we can do to help.
 
Old 01-14-2015, 07:38 PM   #4
ASTRAPI
Member
 
Registered: Feb 2007
Posts: 210

Original Poster
Rep: Reputation: 16
Quote:
this is a SERVER ? right ?
Yes

Quote:
read/ write bottleneck ?
I don't think there is a problem with I/O but i will run next time the iotop and i will post here.

Quote:
Is 90% CPU a normal usage for you ?
No as i always get maximum 40% usage daily.

Quote:
Does this high loadavg affect your ability to service user requests ?
From 2 seconds my page loads at 5 seconds.

Quote:
Why do you reboot ?
I was thinking that something is running on the system that is not visible on top or something crash or maybe i got a loop on something and restart solve my problem 100% for today at least...
 
Old 01-14-2015, 07:55 PM   #5
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,128

Rep: Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120Reputation: 4120
Next time you have the problem, run the following, and post the output here
Quote:
top -b -n 1 | awk '{if (NR <=7) print; else if ($8 ~ /[RD]/) {print; count++} } END {print "Total: "count}'
 
Old 01-15-2015, 09:36 AM   #6
dt64
Member
 
Registered: Sep 2012
Distribution: RHEL5/6, CentOS5/6
Posts: 218

Rep: Reputation: 38
is this a VPS or bare-metal server?
If it was a VPS it could be influenced by other VMs on the same host or over-committing system ressources.
 
Old 01-15-2015, 01:14 PM   #7
ASTRAPI
Member
 
Registered: Feb 2007
Posts: 210

Original Poster
Rep: Reputation: 16
Dedicated:

CPU: Intel Xeon E5-1650v2 6c/12t 3,5 GHz+/3,9 GHz+
RAM: 128 GB DDR3 ECC 1600MHz
Disks: 3 x 160 GB SSD Intel DC S3500 SATA3 6 Gbps SoftRAID 5
Public network card: 1x 1 Gbps

Only one forum inside using Nginx phpfpm mariadb memcache zendopcache
 
Old 01-15-2015, 04:02 PM   #8
dt64
Member
 
Registered: Sep 2012
Distribution: RHEL5/6, CentOS5/6
Posts: 218

Rep: Reputation: 38
Have you checked system processor usage with top?
Are you running some kind of JVM, e.g. tomcat?
How did the resource usage increase, from one minute to the next or slowly over a few days/weeks?
Just asking since I've seen something similar recently.
 
Old 01-16-2015, 03:00 PM   #9
ASTRAPI
Member
 
Registered: Feb 2007
Posts: 210

Original Poster
Rep: Reputation: 16
No i do not have tomcat o anything else related....

I have the problem exactly at 16:45 but yesterday was a small spike of 20% more load than normal for 2 minutes so i am waiting to get more load so i can run a few commands to check it....
 
Old 01-16-2015, 05:02 PM   #10
asc_henry
LQ Newbie
 
Registered: Nov 2012
Posts: 1

Rep: Reputation: Disabled
I have experienced similar issues. my system uses two xeon quad core E5450 processors and I have isolated issue to video drivers or card hardware.
In my case all 8 cores spike to above 90% capacity and wireless will show far above normal transmit volume vs rx volume, ie tx buffers are very high.

I can change video attributes and get temp relief! Thermal issues are suspect.

Good luck!!!!

Henry
Semper Fidelis
Sir/Mam

Last edited by asc_henry; 01-17-2015 at 10:37 AM. Reason: spelling corection
 
Old 01-17-2015, 06:08 AM   #11
dt64
Member
 
Registered: Sep 2012
Distribution: RHEL5/6, CentOS5/6
Posts: 218

Rep: Reputation: 38
Just another idea:
if you get another high load situation, check the amount of open files and file opening rate.
I've recently seen some problem in this area with an "unmatured" peace of software where the kernel had to open files on a very high rate but never clodsed them again, which caused system ressources being eaten by the kernel until the machine froze.
 
Old 01-17-2015, 08:45 AM   #12
ASTRAPI
Member
 
Registered: Feb 2007
Posts: 210

Original Poster
Rep: Reputation: 16
Ok thanks i will check it
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
centos 6 fails to load file system while booting khhuff Linux - Kernel 5 12-24-2012 12:05 PM
CentOS 5.5 VPS web server has 'fallen over' - how to troubleshoot? jsp_1983 Linux - Newbie 5 01-29-2011 09:00 PM
New Install of Debian Failed to Load Grub Troubleshoot options Gutsy_Iain_08 Linux - Newbie 1 03-21-2010 01:37 AM
Centos Server Troubleshoot Tips LGX Linux - Newbie 4 12-17-2009 10:47 PM
Cannot load DSO (32bit) in 64bit CentOS jasperphang Linux - Newbie 4 07-27-2006 04:56 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 07:49 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration