Hi,

I have 24 processor cpu server and it is under the high load of 39 on it . Below are the out put of the top command. Can you please let me know how i can find that which causing the high load on the server and how i can out of this problem.

And due to high the application and database on the server is suffering.


top - 11:02:10 up 223 days, 9:25, 2 users, load average: 39.21, 35.10, 33.79
Tasks: 1029 total, 14 running, 1012 sleeping, 0 stopped, 3 zombie
Cpu(s): 7.0%us, 15.4%sy, 0.0%ni, 34.2%id, 43.1%wa, 0.1%hi, 0.2%si, 0.0%st
Mem: 98992208k total, 98757200k used, 235008k free, 1084k buffers
Swap: 29786104k total, 23562400k used, 6223704k free, 56454664k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1320 root 20 -5 0 0 0 R 99.8 0.0 2252:24 kswapd0
12062 oracle 18 0 70.1g 129m 124m R 76.1 0.1 1:03.91 oracle
1321 root 11 -5 0 0 0 R 69.2 0.0 2828:25 kswapd1
19889 oracle 16 0 70.6g 21g 21g D 28.7 22.6 210:59.11 oracle
11548 oracle 16 0 70.1g 486m 480m D 13.8 0.5 0:04.61 oracle
1706 oracle 16 0 70.1g 5.0g 5.0g D 12.5 5.3 0:25.56 oracle
3592 oracle 16 0 70.2g 4.8g 4.7g R 11.5 5.0 20:42.09 oracle
10157 oracle 15 0 70.1g 96m 90m S 11.5 0.1 0:01.11 oracle
8755 oracle 15 0 70.3g 7.5g 7.4g S 10.9 8.0 28:02.40 oracle
15226 oracle 16 0 70.2g 8.8g 8.8g D 10.9 9.3 7:27.35 oracle
3594 oracle 16 0 70.2g 3.0g 3.0g R 10.2 3.2 1:42.09 oracle
27801 oracle 18 0 70.2g 2.4g 2.4g R 9.9 2.6 0:15.16 oracle
1704 oracle 16 0 70.1g 5.0g 5.0g R 8.9 5.3 0:28.74 oracle
1712 oracle 16 0 70.1g 5.0g 5.0g R 8.9 5.3 0:27.23 oracle
7947 oracle 16 0 70.2g 2.0g 2.0g R 8.9 2.1 0:20.47 oracle
5767 oracle 18 0 1685m 99m 93m D 8.6 0.1 0:03.46 oracle
15253 oracle 16 0 70.1g 9.3g 9.3g D 8.6 9.8 8:17.51 oracle
19244 oracle 15 0 70.2g 4.2g 4.2g S 8.6 4.5 75:49.58 oracle
13249 oracle 16 0 71.2g 10g 9.8g R 7.9 11.3 6:34.37 oracle
29821 oracle 16 0 70.2g 26g 26g R 7.9 28.1 80:38.52 oracle
16404 oracle 16 0 1685m 55m 54m R 7.2 0.1 0:03.35 oracle
28809 oracle 15 0 70.6g 12g 12g S 7.2 13.7 71:05.26 oracle
1710 oracle 16 0 70.1g 5.0g 5.0g R 6.9 5.3 0:28.60 oracle
25713 oracle 18 0 70.2g 2.4g 2.4g D 6.9 2.6 0:17.56 oracle
13854 oracle 18 0 1687m 72m 65m D 6.3 0.1 0:02.50 oracle
15230 oracle 15 0 70.2g 8.3g 8.3g D 5.9 8.8 6:38.32 oracle
8693 root 16 0 51452 2076 1332 S 4.9 0.0 52:52.95 Xvnc
15249 oracle 16 0 70.1g 8.6g 8.6g D 4.9 9.1 9:07.09 oracle
13892 oracle 18 0 1687m 41m 35m D 4.6 0.0 0:02.51 oracle
16366 oracle 15 0 1687m 581m 580m S 4.6 0.6 19:10.80 oracle
29811 oracle 15 0 1688m 640m 639m S 4.6 0.7 8495:03 oracle
1708 oracle 16 0 70.1g 5.1g 5.1g D 3.6 5.4 0:29.05 oracle
3514 oracle 15 0 70.2g 4.2g 4.2g S 3.3 4.5 10:40.16 oracle
13343 oracle 16 0 70.4g 6.7g 6.4g S 3.3 7.0 2:16.85 oracle
15569 oracle 15 0 70.1g 1.8g 1.8g S 3.3 1.9 1:16.47 oracle
29628 oracle 16 0 70.1g 10g 10g D 2.6 10.8 3:31.36 oracle
21109 oracle 15 0 70.2g 5.4g 5.4g D 2.3 5.7 1:05.45 oracle


Thanks
Ravinder

1. try to use code tags when posting for ease of reading https://www.linuxquestions.org/quest...do=bbcode#code

2. Very bad swapping ie not enough RAM, although its probably a memory leak or eg DB overload
a) Note the very high wait percent in Cpu measure.

b) High usage of swap space

c) kernel swap (kswapd) using 99.8% of cpu

d) A lot of Processes in State 'D' http://slack-linux.blogspot.com.au/2...ate-codes.html
HTH

Thanks Chris for your reply.

Can you please let me know the steps by which i can find what cause of the high load on the server (as i have to show this to our client ) and what are the process in the Uninterpretable D state.
Secondly how to get rid of kswapd issue.
Third ,How to get rid of the high kernel swap space.

Thanks Again

Forget your last 2 qns; they are just symptoms, not causes.

You have a lot of Oracle processes running and most of the ones shown there are in States D or S. As per my link, that means they are waiting in one form or another.
You need a good Oracle DBA to find out what its doing.

Thanks Chris,

Can you please confirm me that this in any way is OS or hardware issue which causing high average load on the server.

Thanks
Ravinder

Like I said, you'll need to investigate; no way to guess from here.
Obviously check disk space for a start, but seriously, get an Oracle DBA in the first instance.

This is one of the (very) few cases where some-one has queried a high loadavg, and I could honestly say - "don't worry".
On a 24 processor system that's only a 1.5 ratio.

And in truth, loadavg is again only a symptom anyway.

Looks like some process(es) is/are acquiring memory, and your disk storage system can't cope with the load.
As a quick fix, I'd make sure I added some swap (for that much memory), and make sure *all* the swap is isolated from other I/O - especially userland I/O from Oracle. That means put it on it's own disk(s), and own controllers/HBAs (whatever) if possible.
Then hopefully the Oracle I/O will have a chance to complete in reasonable time, and those tasks will drop out of (long) uninterruptible sleep, and the loadavg will consequently drop as well.

Then you need to go find the cause. If the tasks really need more memory, give it to them.

That system has been up for a while - let's see the output from these commands. And use tags around it.

Thanks for the reply,

Below are the output of the slabtop command and /proc/meminfo
It means i have to increase the system swap memory which is currently 29GB and that will help to bring the load average down and also solve the performance issue for the oracle database.

Secondly, can you please let me know how much swap space should be on this server.

[root@adc3bs19 ~]# slabtop -o -s c | head -n 20
Active / Total Objects (% used) : 742878 / 976607 (76.1%)
Active / Total Slabs (% used) : 86117 / 86262 (99.8%)
Active / Total Caches (% used) : 111 / 164 (67.7%)
Active / Total Size (% used) : 280889.38K / 324263.68K (86.6%)
Minimum / Average / Maximum Object : 0.02K / 0.33K / 128.00K

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
377566 365341 96% 0.52K 53938 7 215752K radix_tree_node
172414 102108 59% 0.17K 7837 22 31348K vm_area_struct
17180 11716 68% 0.74K 3436 5 13744K ext3_inode_cache
45345 22000 48% 0.25K 3023 15 12092K filp
50742 33927 66% 0.21K 2819 18 11276K dentry_cache
14196 12242 86% 0.58K 2366 6 9464K proc_inode_cache
2426 2334 96% 2.00K 1213 2 4852K size-2048
3304 3277 99% 1.04K 472 7 3776K dm_mpath
3588 3385 94% 1.00K 897 4 3588K size-1024
1293 1069 82% 2.06K 431 3 3448K sighand_cache
108288 69368 64% 0.02K 752 144 3008K anon_vma
1384 1235 89% 1.84K 692 2 2768K task_struct
670 641 95% 4.00K 670 1 2680K size-4096
[root@adc3bs19 ~]#

[root@adc3bs19 ~]# cat /proc/meminfo
MemTotal: 98992208 kB
MemFree: 430936 kB
Buffers: 8304 kB
Cached: 60578144 kB
SwapCached: 3489112 kB
Active: 63771036 kB
Inactive: 2804084 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 98992208 kB
LowFree: 430936 kB
SwapTotal: 29786104 kB
SwapFree: 10258864 kB
Dirty: 32 kB
Writeback: 0 kB
AnonPages: 2686840 kB
Mapped: 59711432 kB
Slab: 349132 kB
PageTables: 31305596 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 79282208 kB
Committed_AS: 87304376 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 280664 kB
VmallocChunk: 34359457499 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
[root@adc3bs19 ~]#


Thanks
Ravinder