[SOLVED] tc-server high load

zhjim · 09-25-2012, 04:01 AM

Hi folks,

I have to tackle a problem with a tc-server that produces high load also there are no calls to the webapps themself. We have the insight plugin running.

As I never got my head around how to performance test a tomcat server I hope someone around here can give me some info where I have to look. I will gladly provide any info needed. Starting with the output of top and ps auxww.

top

Code:

top - 10:56:50 up 95 days, 18:27,  1 user,  load average: 19.73, 20.23, 17.02
Tasks: 102 total,   4 running,  98 sleeping,   0 stopped,   0 zombie
Cpu0  : 30.9%us, 68.3%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.8%si,  0.0%st
Cpu1  : 27.4%us, 71.8%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.8%si,  0.0%st
Cpu2  : 36.3%us, 60.7%sy,  0.0%ni,  0.0%id,  0.0%wa,  1.1%hi,  1.9%si,  0.0%st
Mem:   5089300k total,  4693260k used,   396040k free,   212948k buffers
Swap:  3069944k total,      280k used,  3069664k free,  2197428k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                         
30855 user   20   0 5146m 1.9g  10m S 250.4 38.3 116:21.47 /home/user/java/current/bin/java -Djava.util.logging.config.file=/home/auser/tc-user/conf/logging.properties -Xss192K -Xms2358m -Xmx2358m -XX:NewSize=128m -XX:MaxNewSize=7
    9 root      20   0     0    0    0 R 19.3  0.0  28030:51 [ksoftirqd/1]

ps auxww

Code:

user  30855  193 38.3 5269524 1950676 ?     Sl   09:56 117:13 /home/user/java/current/bin/java -Djava.util.logging.config.file=/home/user/tc-user/conf/logging.properties -Xss192K -Xms2358m -Xmx2358m -XX:NewSize=128m -XX:MaxNewSize=768m -XX:SurvivorRatio=6 -XX:PermSize=512m -XX:MaxPermSize=1024m -Dde.company.name=dev-user -Dde.company.port=29000,29001 -Dde.company.env=production -Duser.timezone=Europe/Berlin -Dsun.net.inetaddr.ttl=60 -Dsun.net.inetaddr.negative.ttl=60 -Dinsight.base=/home/user/tc-user/insight -Dinsight.logs=/home/user/tc-user/logs -Dinsight.max.frames=900000 -Dinsight-max-frames=900000 -Djava.library.path=/home/user/tc-user/insight/sigar-lib -Djava.util.logging.manager=com.springsource.tcserver.serviceability.logging.TcServerLogManager -Djava.endorsed.dirs=/home/user/bin/vfabric/vfabric-tc-server-standard-2.7.0.RELEASE/tomcat-7.0.27.A.RELEASE/endorsed -classpath /home/user/tc-user/bin/insight-bootstrap-tcserver-1.5.1.SR3.jar:/home/user/tc-user/lib/aspectjweaver-1.6.11.SR1.jar:/home/user/bin/vfabric/vfabric-tc-server-standard-2.7.0.RELEASE/tomcat-7.0.27.A.RELEASE/bin/bootstrap.jar:/home/user/bin/vfabric/vfabric-tc-server-standard-2.7.0.RELEASE/tomcat-7.0.27.A.RELEASE/bin/tomcat-juli.jar -Dcatalina.base=/home/user/tc-user -Dcatalina.home=/home/user/bin/vfabric/vfabric-tc-server-standard-2.7.0.RELEASE/tomcat-7.0.27.A.RELEASE -Djava.io.tmpdir=/home/user/tc-user/temp org.apache.catalina.startup.Bootstrap start

technicalthug · 09-25-2012, 04:34 AM

can you also show us a

Code:

vmstat 1 25

zhjim · 09-25-2012, 04:41 AM

here we go:

Code:

[user@userapp01 ~]$ vmstat 1 25
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
16  0    256 330712 212784 2229492    0    0     0     1    1    1 23 43 33  0  0	
16  0    256 330704 212784 2229512    0    0     0    12 5008 684601 35 65  0  0  0	
20  0    256 330704 212792 2229508    0    0     0    32 2862 769904 38 62  0  0  0	
14  0    256 330728 212792 2229512    0    0     0     0 2937 765131 36 64  0  0  0	
19  0    256 330728 212792 2229512    0    0     0     0 3610 715461 38 62  0  0  0	
16  0    256 330728 212792 2229512    0    0     0     0 5228 627261 36 64  0  0  0	
21  0    256 330728 212800 2229508    0    0     0    12 7656 482532 31 69  0  0  0	
123  0    256 330728 212800 2229512    0    0     0     0 5906 518505 32 68  0  0  0	
18  0    256 330728 212800 2229512    0    0     0     0 9525 484026 33 67  0  0  0	
18  0    256 330720 212800 2229512    0    0     0     0 10693 457198 33 67  0  0  0	
18  0    256 330720 212800 2229512    0    0     0     0 13598 467842 31 69  0  0  0	
18  0    256 330720 212800 2229512    0    0     0     0 4575 649533 35 65  0  0  0	
17  0    256 330720 212800 2229512    0    0     0     0 3580 758566 36 64  0  0  0	
19  0    256 330720 212800 2229516    0    0     0     0 7543 567891 32 68  0  0  0	
24  0    256 330720 212800 2229516    0    0     0     0 10903 605057 32 68  0  0  0	
19  0    256 330720 212800 2229516    0    0     0     0 9826 622429 33 67  0  0  0	
19  0    256 330720 212800 2229516    0    0     0     0 7746 640672 35 65  0  0  0	
17  0    256 330720 212800 2229516    0    0     0     0 3477 756612 38 62  0  0  0	
20  0    256 330720 212808 2229516    0    0     0    16 3570 728278 36 64  0  0  0	
16  0    256 330728 212808 2229516    0    0     0     0 3361 734041 37 63  0  0  0	
18  0    256 330728 212808 2229516    0    0     0     0 3031 758183 33 67  0  0  0	
15  0    256 330728 212808 2229516    0    0     0     0 3797 696663 36 64  0  0  0	
24  0    256 330728 212808 2229516    0    0     0     0 6859 616710 33 67  0  0  0	
17  0    256 330728 212808 2229516    0    0     0     0 3879 672685 35 65  0  0  0	
18  0    256 330728 212808 2229516    0    0     0     0 4760 651995 40 60  0  0  0

technicalthug · 09-25-2012, 05:07 AM

It looks like something is just spinning, the vmstat shows you have anywhere from 10-100 processes waiting in the CPU's runqueue and you have a large amount on context switching going on.

Based on that information and the fact your system has been on for 3 solid months, i found the following: http://www.wired.com/wiredenterprise...th-java-linux/

Possible fix: resync your network time daemon

zhjim · 09-25-2012, 05:10 AM

Definetly a point there. I was aware of the leap second bug but did not make the connection. I'll give it a go after lunch. Thank you.

technicalthug · 09-25-2012, 05:16 AM

Good luck mate, report back how it goes!

zhjim · 09-25-2012, 07:23 AM

You're the man

Doing a date -s "`date`" and a restart of the app and where back to "normal" tc cpu usage.

Thanks YOU.