[SOLVED] Please suggest

ankitpandey · 02-01-2012, 08:20 AM

Hey All,

I had a server in which memory utilization can be maximum of 8Gb, but some process utilize very high memory (those are BiBuss process). So i want to kill the BiBus process with maximum execution time when memory utilization for the server is near (say) 7.5Gb. Please help on this.

Regards,
Ankit

hda7 · 02-01-2012, 09:40 AM

Take a look at the "free", "ps", and "kill" commands.

ukiuki · 02-01-2012, 09:43 AM

You can also use some of process monitor programs like top or htop or conky to keep eyes on it, top and htop give you the ability to kill processes.

Regards

ankitpandey · 02-02-2012, 02:38 AM

This would be manual effort, i want to automate this, any help on this.

Thanks,
Ankit

hda7 · 02-02-2012, 08:27 AM

Quote:

Originally Posted by ankitpandey

This would be manual effort, i want to automate this, any help on this.

Thanks,
Ankit

You could schedule a script to run periodically that will
1) use "free" to determine the amount of free memory, and if necessary
2) use "ps" to find the process you want to kill and
3) "kill" it

tommylovell · 02-02-2012, 10:28 AM

ankitpandey, high memory utilization is not necessarily bad. If you are using 100% of your memory but most of it is buffers and cache, and you are not using any swap, that's a good thing.

On the other hand, if you go from that situation to one where your cache has been trimmed down substantially and you are using a lot of swap, it might be worthwhile to find the process that is using a lot of virtual memory and putting pressure on real memory. (It may have a memory leak, it might be looping, it might be too ambitious. Who knows?) While it may be relatively easy to find the culprit or culprits programatically, it may not be so easy to come up with the criteria for which process to kill. It's very situational, so you won't find many examples of this. And you better make sure that the code you develop is at least as smart as you are. (Is it possible to write code that is smarter than you are?)

Sorry I have no code to offer, but look at 'ps -ely' and the "sz" and "rss" columns. RSS is real memory that the process is occupying; SIZE is roughly the virtual amount of memory that the process is using. (See 'man ps'.)

ankitpandey · 02-03-2012, 01:19 AM

tommylovell, thanks for the post, it was really knowledgeable to me.

I am just giving the instance of memory utilization. At this moment i can graphically see utilization to be around 6Gb.

Prstat -a

Code:

  PID  USERNAME SIZE   RSS  STATE  PRI  NICE  TIME    CPU PROCESS/NLWP       
 13660 j2ee-r   1678M 1258M cpu42    3    0  86:13:36 0.6% java/362
 18252 j2ee-r    938M  894M cpu64    9    0   0:28:57 0.5% BIBusTKServerMa/24
  7596 j2ee-r    363M  318M cpu21   32    0   0:02:03 0.3% BIBusTKServerMa/24
 27078 j2ee-r    206M  161M sleep    1    0   0:00:17 0.2% BIBusTKServerMa/24
 21635 j2ee-r    307M  274M cpu5     3    0   0:01:08 0.1% BIBusTKServerMa/24
 17122 j2ee-r   1155M  125M sleep   37    0  65:31:02 0.1% java/143
 22152 j2ee-r    443M  397M sleep   53    0   0:04:06 0.1% BIBusTKServerMa/24

Quote:

NPROC USERNAME SWAP RSS MEMORY TIME CPU
19 j2ee-r 5607M 4779M 3.7% 213:47:46 2.0%
2 user1 2688K 14M 0.0% 0:00:00 0.0%
2 user2 2304K 13M 0.0% 0:00:00 0.0%
1 smmsp 1424K 9016K 0.0% 0:02:41 0.0%
10 sysmon 8160K 14M 0.0% 0:00:16 0.0%
6 daemon 9384K 12M 0.0% 0:46:45 0.0%
32 root 272M 320M 0.2% 83:56:23 0.0%

When the utilization goes to 8Gb then it automatically kill's the j2se process which brings the services down. We had BIBusTKServerMa process which can be killed in the sake to keep the environment up. So i was thinking to kill the BIBusTKServerMa process with maximum run time when memory utilization is near to 8 Gb.

Please suggest.

Thanks,
Ankit

ankitpandey · 02-03-2012, 01:36 AM

ps -ely

Code:

S    UID   PID  PPID   C PRI NI   RSS     SZ    WCHAN TTY         TIME CMD
 S      0  7624  4098   0  40 20 14432  25368        ? ?          40:01 inetd
 S      0  7775  4738   0  48 20   184   2848        ? console     0:00 ttymon
 S      0  4098  4098   0   0 SY     0      0        ? ?           0:00 zsched
 S      0 22056  7694   0  70 20 86888 184432        ? ?        3987:05 java
 S      0  4706  4098   0  40 20   936   3016        ? ?           2:13 init
 S      0  7610  4738   0  40 20   640   2504        ? ?           0:30 sac
 S      0  4748  4098   0  98 20  7696  12552        ? ?          30:14 svc.conf
 S      0  4738  4098   0  40 20 37872  57640        ? ?          23:32 svc.star
 S      0  7694  4098   0  40 20   144   2320        ? ?           0:00 cimomboo
 S      1  7193  4098   0  40 20  1520   3720        ? ?           0:02 statd
 S      0  6963  4098   0  40 20  1136   3304        ? ?           0:05 cron
 S      1  6114  4098   0  40 20  3640   6280        ? ?          33:00 kcfd
 S      0 13048  4098   0  40 20  1240   4264        ? ?           0:01 python
 S      0 19377  4098   0  40 20 35480  42136        ? ?          92:07 nscd
 S      1  7118  4098   0  40 20  2080   3528        ? ?           8:20 rpcbind
 S      0  8343  4098   0  40 20   824   2952        ? ?           0:09 snmpdx
 S      0  7726  4098   0  40 20   832   1720        ? ?          11:10 utmpd
 S      0 13168  4098   0  40 20   184   3016        ? ?           0:00 automoun
 S      1  7469  4098   0  41 20  1328   2840        ? ?           1:35 lockd
 S      0  7671  7610   0  40 20   696   2912        ? ?           0:25 ttymon
 S      0  8469  4098   0  51 20   128   3608        ? ?           0:00 dmispd
 S      0  8125  4098   0  40 20  1792   4760        ? ?         120:41 syslogd
 S      0  8203  4098   0  40 20  6088  12232        ? ?         238:33 snmpd
 S      0  8464  4098   0  52 20   184   4240        ? ?           0:00 snmpXdmi
 S      1 13130  4098   0  40 20  1544   5200        ? ?           3:51 nfsmapid
 S      0 13170 13168   0  40 20  7272   9400        ? ?         377:40 automoun
 S  46347 27576 27557   0  42 20  4656   7616        ? ?           0:02 sshd
 S  46347 27647 27576   0  40 20  1472   1888        ? pts/86      0:01 ksh
 S      1 13128  4098   0  40 20   128   2808        ? ?           0:00 nfs4cbd
 S      0 19316  6684   0  40 20  3016   5704        ? ?           0:00 sshd
 S  46347 19318 19316   0  52 20  5016   7616        ? ?           0:06 sshd
 S     25 13157  4098   0  40 20  1704   8624        ? ?           2:41 sendmail
 S      0  5069  6684   0  40 20  4320   5704        ? ?           0:00 sshd
 S      0 18711  6684   0  40 20  2920   5704        ? ?           0:00 sshd
 S      0  6684  4098   0  40 20  1864   4632        ? ?           0:19 sshd
 S      0 23079  6684   0  40 20  3704   5704        ? ?           0:00 sshd
 S      0 13073 13048   0  40 20 18360  20744        ? ?          93:43 python
 S      0 19446  4098   0  40 20  5840   7976        ? ?           5:07 ldap_cac
 O 12345  5070  5069   0  53 20 10296  11872          ?           0:00 sshd
 S 12346 23080 23079   0  40 20  8832  11872        ? ?           0:00 sshd
 S      0 13158  4098   0  40 20  1856   8624        ? ?          13:10 sendmail
 S  46347 10688 10655   0  40 20  1456   1888        ? pts/54      0:00 ksh
 S  55040 16963  4098   0  98 20  1424   3664        ? ?           9:00 wrapper
 S  55040 17122  4098   0  62 20 127512 1182848        ? ?        3931:34 java
 S  55040  1361  4098   0  40 20  1680   3768        ? ?           0:00 ssh-agen
 S 12346  5091  5070   0  40 20  1624   1888        ? pts/111     0:00 ksh
 O  55040 21635 13660   0  62 20 400608 430296          ?           3:13 BIBusTKS
 O  55040 17295 14069   0  40 20  1640   1920          pts/111     0:00 ps
 S  55040 17121  4098   0  45 20 46160 141640        ? ?        3616:19 rmiregis
 S  55040  2949 13660   0  49 20 421784 463816        ? ?           5:21 BIBusTKS
 S  46347   866   865   0  40 20  5136   7616        ? ?           0:01 sshd
 S  55040 23404 23324   0  40 20  2760   3528        ? pts/90      0:01 bash
 S  46347   875   866   0  40 20  1472   1888        ? pts/16      0:01 ksh
 S  55040 23611  4098   0  43 20  1480   3768        ? ?           0:00 ssh-agen
 S  46347 18712 18711   0  46 20  4672   7616        ? ?           0:04 sshd
 S  55040 27078 13660   0  98 20 271312 309784        ? ?           2:08 BIBusTKS
 S      0 27557  6684   0  40 20  2560   5704        ? ?           0:00 sshd
 S  46347 10655 10654   0  57 20  5144   7616        ? ?           0:01 sshd
 S  55040 22152 13660   0  46 20 545040 596712        ? ?          10:14 BIBusTKS
 S  46347 18723 18712   0  40 20  1472   1888        ? pts/103     0:02 ksh
 O  55040 13660 16963   0  71 20 1289352 1765352          ?        5182:11 java
 S  55040 16861 13660   0  63 20 678384 718808        ? ?          28:29 BIBusTKS
 S  55040 14069  5091   0  40 20  2896   3528        ? pts/111     0:00 bash
 S      0 10654  6684   0  40 20  3120   5704        ? ?           0:00 sshd
 S  46347 19328 19318   0  40 20  1480   1888        ? pts/29      0:03 ksh
 S  55040  7596 13660   0  51 20 446792 490368        ? ?           3:40 BIBusTKS
 S      0   865  6684   0  40 20  3112   5704        ? ?           0:00 sshd
 S 12345 23324 23080   0  40 20  2216   3528        ? pts/90      0:00 bash
 O  55040 21756 13660   1  89 20 388432 434312          ?           4:50 BIBusTKS

tommylovell · 02-03-2012, 08:16 AM

Quote:

Originally Posted by ankitpandey

When the utilization goes to 8Gb then it automatically kill's the j2se process which brings the services down.

What kills the j2se process? Is it something you wrote? Is it the system's OOM Killer?

Quote:

Originally Posted by ankitpandey

We had BIBusTKServerMa process which can be killed in the sake to keep the environment up. So i was thinking to kill the BIBusTKServerMa process with maximum run time when memory utilization is near to 8 Gb.

Please suggest.

The BIBusTKServerMa processes are relatively small. I don't think killing them is going to buy you much. Does it help when you kill them manually?

And I go back to my original statement that your system using 8Gb is normally a good thing and not something to worry about. Under normal conditions as your system runs, it will try to cache all of its I/O data in memory. At some point in time you should have a system using nearly all of real memory with a very large cache, and now able to avoid real I/O if the data is in cache! When memory is needed by your applications, it trims back the cache. But in any event, using all of memory is not necessarily bad.

Having your cache trimmed back due to memory pressure, swapping, and running out of swap are bad things. You shouldn't be keying your actions off of memory use going to 8Gb, you should be looking at the larger picture.

Your high use of swap is a concern. (If virtual memory is swapped out and then infrequently referenced that's no big deal - it's just taking disk space - but is that likely with your app?) In all likelyhood, high virtual storage usage is putting pressure on real memory, the system has had to swap out (and in) a lot of pages, meaning I/O and related poor performance. It has also probably trimmed down cache at the same time meaning you're not finding data in the cache and real I/O has to be performed.

This looks like your system is simply too small for the workload that you are putting on it and trying to kill small processes is not going to buy you much in the long run. It really looks like you need more memory added to this system.

Since this is a java app, have you looked into tuning your java? You may get some relief by adjusting heap size and garbage collection frequency. (Something I am unfamiliar with, but seems to help.)

We have 800+ Linux systems running. Here is the output of a 'free' command run on a heavily used WebSphere (apache/java) system.

Code:

             total       used       free     shared    buffers     cached
Mem:      24682284   24364196     318088          0     560008    7613192
-/+ buffers/cache:   16190996    8491288
Swap:      8388600        204    8388396

Note the nice high value for 'cached', the minimal use of swap. This systems is running just fine and has an uptime of 116 days.

I invite other opinions. I'm not always right.

ankitpandey · 02-03-2012, 09:23 AM

Its is system that kills j2se process when utilization reaches its limit.

You are quite right that BiBuss acquire very less memory, but yes killing the BiBuss with maximum run helps us in a drop of memory utilization. I feel that most of them get hung and hence acquire more memory than they should ( i cannot proof this).

Can you tell me on adjusting heap size and garbage collection frequency.
As i am not the owner to this environment i cannot make any changes but just to manage what i have been provided.

We are using solaris.

tommylovell · 02-04-2012, 08:30 AM

Quote:

Originally Posted by ankitpandey

We are using solaris.

Sorry, I made the assumption that it was Linux. So, OOM Killer and cache related comments don't apply.

But java is java, and I think it's generally considered good practice to review tuning parameters periodically and make tweaks. As I mentioned earlier, this is well outside of my area of expertise, but there is quite a bit of information published on the topic. You can start with http://www.oracle.com/technetwork/ja...-5-138395.html and http://java.sun.com/performance/refe...rs/tuning.html. Both good articles.

onebuck · 02-04-2012, 05:10 PM

Hi,

Welcome to LQ!

Quote:

Originally Posted by ankitpandey

Hey All,

I had a server in which memory utilization can be maximum of 8Gb, but some process utilize very high memory (those are BiBuss process). So i want to kill the BiBus process with maximum execution time when memory utilization for the server is near (say) 7.5Gb. Please help on this.

Regards,
Ankit

In the future, please try to use 'vbcode' tags at the top of the reply window when showing long lists or data. 'vbcode' tag # is for code while the balloon to the right is for quotes.
I enclosed your output in post #8 as an example.

Please try to provide descriptive titles so members can have an idea of thread content.

FYI: I suggest that you look at 'How to Ask Questions the Smart Way' so in the future your queries provide information that will aid us in diagnosis of the problem or query.