Multithreaded application, CPU and memory usage

mohitanchlia · 02-14-2009, 11:44 AM

We run Jboss app server that of course is all multithreaded under one JVM. I have couple of question regarding monitoring on per thread basis:

1. Is there a way to see which thread is bound to which CPU core?
2. Is there a way to see the CPU, Memory usage per thread? Something like prstat on Sun box which is real time and gives detailed information about threads per CPU

paulsm4 · 02-14-2009, 06:04 PM

Hi -

To quote from your other post (http://www.linuxquestions.org/questi...of-8-704643/):

Quote:

Hi -

My own experience is that JBoss does a pretty good job of scheduling workloads across multiple CPUs.

The first thing you need to do is figure out *which* process is hogging CPU 0. It's very likely a specific process (rather than a bunch of different programs), but it *isn't* necessarily a JBoss-related process.

Your first task is to figure out what *process* is causing the problem (it might not be JBoss at all, and JBoss typically runs several different apps that use many different JVM processes). Here's a good place to start (you can easily Google for many other "Linux Performance Tuning 101" articles

http://www.linux-mag.com/id/829

'Hope that helps .. PSM

PS:
In case the problem is your app server, JBoss is very rich in performance monitoring tools to help you home in on the precise problem(s) and correct it (them):

http://oreilly.com/catalog/jbossadn/...ch08_part1.pdf

PPS:
What makes you think there's only one JVM for the whole app server? We use JBoss under Windows (it sounds like you're running under Linux), but our (Windows) JBoss definitely spawns multiple JVMs.

PPS:
Please double-check that you have an SMP version of Linux installed (running a uniprocess Linux might account for one CPU getting all the activity :-))

Final PPS:
Did you ever get resolution on your "kill -9" question:
http://www.linuxquestions.org/questi...546/page2.html

mohitanchlia · 02-15-2009, 11:30 AM

May be in your case you are running multiple App servers on one box. We run one app server which creates one Java process but of course it's all multi-threaded.

So going back to my question, could someone please tell me if there is any way in Linux to do this:

1. Is there a way to see which thread is bound to which CPU core?
2. Is there a way to see the CPU, Memory usage per thread? Something like prstat on Sun box which is real time and gives detailed information about threads per CPU

How to check if we are running SMP version of OS.

paulsm4 · 02-15-2009, 01:23 PM

My experience with JBoss (multiple different versions of JBoss, in both development and production environments) is all on Windows (not Linux). Nevertheless, I am quite certain that JBoss should distribute workload well among multiple CPUs (or multiple cores) if your system is configured correctly (and if a JBoss app is even the source of your presumed CPU bottleneck).

Q: Have you looked at any of the performance monitoring links I sent you? Specifically, have you looked at any of "ps", "top" or the JBoss console?

Q: What exactly have you found so far?

Q: Have you tried using Sun's Java Console:
http://java.sun.com/developer/techni.../jconsole.html
<= The JConsole GUI uses MBeans, just like JBoss itself - but you're certain to find more good information here

Q: Whatever happened with the "kill -9" problem you were asking about last August?

Thank you in advance .. PSM

PS:
I haven't even asked about OS, JBoss and Java versions yet. Or your database. Or about any third-party components that might be in the mix...

mohitanchlia · 02-16-2009, 10:49 AM

I am specifically looking for answer to below questions:

1. Is there a way to see which thread is bound to which CPU core?
2. Is there a way to see the CPU, Memory usage per thread? Something like prstat on Sun box which is real time and gives detailed information about threads per CPU

paulsm4 · 02-17-2009, 10:18 AM

Hi, mohitanchlia -

You're a stubborn son-of-a-gun, aren't you ;-)? I sympathize - so am I.

In any case:
1. "prstat", as you know, is a Solaris tool.
You might be able to find a Linux port if you Google hard enough; you might even be able to write one yourself. But AFAIK, there isn't any standard version of "prstat" on non-Solaris platforms.

2. There are many ways to see if your OS is multiprocessor or not.
Two Pop Favorites are:
a) uname -a =>

Quote:

Linux gateway 2.6.22-1-mepis-smp #1 SMP PREEMPT Mon Feb 18 21:44:02 EST 2008 i686 GNU/Linux
<= You can tell by the letters "SMP" that this is a multi-CPU install

b) "top -H", then type "1" (digit "one"):

Quote:

top - 08:14:42 up 32 min, 1 user, load average: 0.10, 0.09, 0.09
Tasks: 178 total, 5 running, 173 sleeping, 0 stopped, 0 zombie
Cpu0 : 2.7%us, 0.7%sy, 0.0%ni, 96.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 2.3%us, 1.3%sy, 0.0%ni, 96.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2065816k total, 610416k used, 1455400k free, 26328k buffers
Swap: 0k total, 0k used, 0k free, 320296k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4144 paulsm 15 0 32388 15m 12m S 3 0.8 0:34.77 kded
...
<= You can see performance statistics for each of the two cores in this laptop PC...

3. Linux (like most *nix, including, of course, Solaris) has a *wealth* of tools for performance monitoring and analysis. JBoss and Sun Java also have their own, rich, toolkits and resources.

I strongly urge you to familiarize yourself with some of them (if you haven't already) - they can definitely help you with this particular question.

In particular, I urge you to play with JBoss "jmx-console", and with Sun's free download "JConsole".

IMHO .. PSM

babaqga · 08-30-2010, 10:13 AM

Did you find out how it is done, I'm basically doing the same thing.