LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 07-11-2012, 09:03 PM   #1
ravindert
Member
 
Registered: Oct 2011
Posts: 68

Rep: Reputation: Disabled
High average load on the server


Hi,

I have 24 processor cpu server and it is under the high load of 39 on it . Below are the out put of the top command. Can you please let me know how i can find that which causing the high load on the server and how i can out of this problem.

And due to high the application and database on the server is suffering.


top - 11:02:10 up 223 days, 9:25, 2 users, load average: 39.21, 35.10, 33.79
Tasks: 1029 total, 14 running, 1012 sleeping, 0 stopped, 3 zombie
Cpu(s): 7.0%us, 15.4%sy, 0.0%ni, 34.2%id, 43.1%wa, 0.1%hi, 0.2%si, 0.0%st
Mem: 98992208k total, 98757200k used, 235008k free, 1084k buffers
Swap: 29786104k total, 23562400k used, 6223704k free, 56454664k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1320 root 20 -5 0 0 0 R 99.8 0.0 2252:24 kswapd0
12062 oracle 18 0 70.1g 129m 124m R 76.1 0.1 1:03.91 oracle
1321 root 11 -5 0 0 0 R 69.2 0.0 2828:25 kswapd1
19889 oracle 16 0 70.6g 21g 21g D 28.7 22.6 210:59.11 oracle
11548 oracle 16 0 70.1g 486m 480m D 13.8 0.5 0:04.61 oracle
1706 oracle 16 0 70.1g 5.0g 5.0g D 12.5 5.3 0:25.56 oracle
3592 oracle 16 0 70.2g 4.8g 4.7g R 11.5 5.0 20:42.09 oracle
10157 oracle 15 0 70.1g 96m 90m S 11.5 0.1 0:01.11 oracle
8755 oracle 15 0 70.3g 7.5g 7.4g S 10.9 8.0 28:02.40 oracle
15226 oracle 16 0 70.2g 8.8g 8.8g D 10.9 9.3 7:27.35 oracle
3594 oracle 16 0 70.2g 3.0g 3.0g R 10.2 3.2 1:42.09 oracle
27801 oracle 18 0 70.2g 2.4g 2.4g R 9.9 2.6 0:15.16 oracle
1704 oracle 16 0 70.1g 5.0g 5.0g R 8.9 5.3 0:28.74 oracle
1712 oracle 16 0 70.1g 5.0g 5.0g R 8.9 5.3 0:27.23 oracle
7947 oracle 16 0 70.2g 2.0g 2.0g R 8.9 2.1 0:20.47 oracle
5767 oracle 18 0 1685m 99m 93m D 8.6 0.1 0:03.46 oracle
15253 oracle 16 0 70.1g 9.3g 9.3g D 8.6 9.8 8:17.51 oracle
19244 oracle 15 0 70.2g 4.2g 4.2g S 8.6 4.5 75:49.58 oracle
13249 oracle 16 0 71.2g 10g 9.8g R 7.9 11.3 6:34.37 oracle
29821 oracle 16 0 70.2g 26g 26g R 7.9 28.1 80:38.52 oracle
16404 oracle 16 0 1685m 55m 54m R 7.2 0.1 0:03.35 oracle
28809 oracle 15 0 70.6g 12g 12g S 7.2 13.7 71:05.26 oracle
1710 oracle 16 0 70.1g 5.0g 5.0g R 6.9 5.3 0:28.60 oracle
25713 oracle 18 0 70.2g 2.4g 2.4g D 6.9 2.6 0:17.56 oracle
13854 oracle 18 0 1687m 72m 65m D 6.3 0.1 0:02.50 oracle
15230 oracle 15 0 70.2g 8.3g 8.3g D 5.9 8.8 6:38.32 oracle
8693 root 16 0 51452 2076 1332 S 4.9 0.0 52:52.95 Xvnc
15249 oracle 16 0 70.1g 8.6g 8.6g D 4.9 9.1 9:07.09 oracle
13892 oracle 18 0 1687m 41m 35m D 4.6 0.0 0:02.51 oracle
16366 oracle 15 0 1687m 581m 580m S 4.6 0.6 19:10.80 oracle
29811 oracle 15 0 1688m 640m 639m S 4.6 0.7 8495:03 oracle
1708 oracle 16 0 70.1g 5.1g 5.1g D 3.6 5.4 0:29.05 oracle
3514 oracle 15 0 70.2g 4.2g 4.2g S 3.3 4.5 10:40.16 oracle
13343 oracle 16 0 70.4g 6.7g 6.4g S 3.3 7.0 2:16.85 oracle
15569 oracle 15 0 70.1g 1.8g 1.8g S 3.3 1.9 1:16.47 oracle
29628 oracle 16 0 70.1g 10g 10g D 2.6 10.8 3:31.36 oracle
21109 oracle 15 0 70.2g 5.4g 5.4g D 2.3 5.7 1:05.45 oracle


Thanks
Ravinder
 
Old 07-11-2012, 09:23 PM   #2
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.9, Centos 7.3
Posts: 17,411

Rep: Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397
1. try to use code tags when posting for ease of reading https://www.linuxquestions.org/quest...do=bbcode#code

2. Very bad swapping ie not enough RAM, although its probably a memory leak or eg DB overload
Code:
Cpu(s): 7.0%us, 15.4%sy, 0.0%ni, 34.2%id, 43.1%wa, 0.1%hi, 0.2%si, 0.0%st

Swap: 29786104k total, 23562400k used, 6223704k free, 56454664k cached

PID  USER  PR NI VIRT RES SHR S %CPU %MEM TIME+   COMMAND
1320 root  20 -5 0    0    0  R 99.8  0.0 2252:24 kswapd0
a) Note the very high wait percent in Cpu measure.

b) High usage of swap space

c) kernel swap (kswapd) using 99.8% of cpu

d) A lot of Processes in State 'D' http://slack-linux.blogspot.com.au/2...ate-codes.html
HTH
 
Old 07-11-2012, 09:44 PM   #3
ravindert
Member
 
Registered: Oct 2011
Posts: 68

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by chrism01 View Post
1. try to use code tags when posting for ease of reading https://www.linuxquestions.org/quest...do=bbcode#code

2. Very bad swapping ie not enough RAM, although its probably a memory leak or eg DB overload
Code:
Cpu(s): 7.0%us, 15.4%sy, 0.0%ni, 34.2%id, 43.1%wa, 0.1%hi, 0.2%si, 0.0%st

Swap: 29786104k total, 23562400k used, 6223704k free, 56454664k cached

PID  USER  PR NI VIRT RES SHR S %CPU %MEM TIME+   COMMAND
1320 root  20 -5 0    0    0  R 99.8  0.0 2252:24 kswapd0
a) Note the very high wait percent in Cpu measure.

b) High usage of swap space

c) kernel swap (kswapd) using 99.8% of cpu

d) A lot of Processes in State 'D' http://slack-linux.blogspot.com.au/2...ate-codes.html
HTH


Thanks Chris for your reply.

Can you please let me know the steps by which i can find what cause of the high load on the server (as i have to show this to our client ) and what are the process in the Uninterpretable D state.
Secondly how to get rid of kswapd issue.
Third ,How to get rid of the high kernel swap space.

Thanks Again
 
Old 07-11-2012, 09:50 PM   #4
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.9, Centos 7.3
Posts: 17,411

Rep: Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397
Forget your last 2 qns; they are just symptoms, not causes.

You have a lot of Oracle processes running and most of the ones shown there are in States D or S. As per my link, that means they are waiting in one form or another.
You need a good Oracle DBA to find out what its doing.
 
Old 07-11-2012, 09:53 PM   #5
ravindert
Member
 
Registered: Oct 2011
Posts: 68

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by chrism01 View Post
Forget your last 2 qns; they are just symptoms, not causes.

You have a lot of Oracle processes running and most of the ones shown there are in States D or S. As per my link, that means they are waiting in one form or another.
You need a good Oracle DBA to find out what its doing.

Thanks Chris,

Can you please confirm me that this in any way is OS or hardware issue which causing high average load on the server.

Thanks
Ravinder
 
Old 07-11-2012, 10:11 PM   #6
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.9, Centos 7.3
Posts: 17,411

Rep: Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397Reputation: 2397
Like I said, you'll need to investigate; no way to guess from here.
Obviously check disk space for a start, but seriously, get an Oracle DBA in the first instance.
 
Old 07-11-2012, 11:46 PM   #7
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 15,950

Rep: Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210
This is one of the (very) few cases where some-one has queried a high loadavg, and I could honestly say - "don't worry".
On a 24 processor system that's only a 1.5 ratio.

And in truth, loadavg is again only a symptom anyway.

Looks like some process(es) is/are acquiring memory, and your disk storage system can't cope with the load.
As a quick fix, I'd make sure I added some swap (for that much memory), and make sure *all* the swap is isolated from other I/O - especially userland I/O from Oracle. That means put it on it's own disk(s), and own controllers/HBAs (whatever) if possible.
Then hopefully the Oracle I/O will have a chance to complete in reasonable time, and those tasks will drop out of (long) uninterruptible sleep, and the loadavg will consequently drop as well.

Then you need to go find the cause. If the tasks really need more memory, give it to them.

Last edited by syg00; 07-11-2012 at 11:47 PM.
 
Old 07-12-2012, 12:04 AM   #8
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 15,950

Rep: Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210Reputation: 2210
That system has been up for a while - let's see the output from these commands. And use tags around it.
Code:
uname -a
cat /proc/meminfo
slabtop -o -s c | head -n 20
 
Old 07-12-2012, 01:10 AM   #9
ravindert
Member
 
Registered: Oct 2011
Posts: 68

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by syg00 View Post
That system has been up for a while - let's see the output from these commands. And use tags around it.
Code:
uname -a
cat /proc/meminfo
slabtop -o -s c | head -n 20


Thanks for the reply,

Below are the output of the slabtop command and /proc/meminfo
It means i have to increase the system swap memory which is currently 29GB and that will help to bring the load average down and also solve the performance issue for the oracle database.

Secondly, can you please let me know how much swap space should be on this server.

[root@adc3bs19 ~]# slabtop -o -s c | head -n 20
Active / Total Objects (% used) : 742878 / 976607 (76.1%)
Active / Total Slabs (% used) : 86117 / 86262 (99.8%)
Active / Total Caches (% used) : 111 / 164 (67.7%)
Active / Total Size (% used) : 280889.38K / 324263.68K (86.6%)
Minimum / Average / Maximum Object : 0.02K / 0.33K / 128.00K

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
377566 365341 96% 0.52K 53938 7 215752K radix_tree_node
172414 102108 59% 0.17K 7837 22 31348K vm_area_struct
17180 11716 68% 0.74K 3436 5 13744K ext3_inode_cache
45345 22000 48% 0.25K 3023 15 12092K filp
50742 33927 66% 0.21K 2819 18 11276K dentry_cache
14196 12242 86% 0.58K 2366 6 9464K proc_inode_cache
2426 2334 96% 2.00K 1213 2 4852K size-2048
3304 3277 99% 1.04K 472 7 3776K dm_mpath
3588 3385 94% 1.00K 897 4 3588K size-1024
1293 1069 82% 2.06K 431 3 3448K sighand_cache
108288 69368 64% 0.02K 752 144 3008K anon_vma
1384 1235 89% 1.84K 692 2 2768K task_struct
670 641 95% 4.00K 670 1 2680K size-4096
[root@adc3bs19 ~]#

[root@adc3bs19 ~]# cat /proc/meminfo
MemTotal: 98992208 kB
MemFree: 430936 kB
Buffers: 8304 kB
Cached: 60578144 kB
SwapCached: 3489112 kB
Active: 63771036 kB
Inactive: 2804084 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 98992208 kB
LowFree: 430936 kB
SwapTotal: 29786104 kB
SwapFree: 10258864 kB
Dirty: 32 kB
Writeback: 0 kB
AnonPages: 2686840 kB
Mapped: 59711432 kB
Slab: 349132 kB
PageTables: 31305596 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 79282208 kB
Committed_AS: 87304376 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 280664 kB
VmallocChunk: 34359457499 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
[root@adc3bs19 ~]#


Thanks
Ravinder
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Load average too high chobong Linux - Newbie 2 08-11-2011 02:38 AM
load average high in Suse 10.3 and 11.3 deepaksoni SUSE / openSUSE 0 10-06-2010 12:55 AM
High load average with no reason wmasry Linux - Server 2 01-13-2010 04:21 PM
load average high sang_froid Linux - Server 3 12-14-2009 11:18 AM
Server with high load average and no obvious reason. DotHQ Linux - Server 15 03-06-2009 04:53 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 06:02 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration