LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Other *NIX Forums > Solaris / OpenSolaris
User Name
Password
Solaris / OpenSolaris This forum is for the discussion of Solaris and OpenSolaris.
General Sun, SunOS and Sparc related questions also go here.

Notices

Reply
 
LinkBack Search this Thread
Old 04-09-2009, 02:06 AM   #1
rajaniyer123
Member
 
Registered: Feb 2004
Location: BARODA, GUJARAT
Posts: 254

Rep: Reputation: 30
How to check memory consumed by hidden/defunct/zombie process


Hi

In one of the prodcution server I am facing typical problem, in which I have 4 GB of RAM. Once the application is running there should be 2 GB free memory approx.

In that server only Broadvision is running, I have shutdown the BV still the free memory it shows only 500 MB instaed of 2 GB, I would like to know which is the hidden process which consumes the Physical Memory.

I am not able to track the process which utilizes high memory in top/prstat but not able to figure out.

Also I have checked defunct process and killed it but it does not make any difference.

Please suggest.

I want to aviod the reboot of the server.

Thanks
Rajan
 
Old 04-09-2009, 02:41 AM   #2
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris10, Solaris 11, Mint, OL
Posts: 9,459

Rep: Reputation: 352Reputation: 352Reputation: 352Reputation: 352
You should provide some material to help understanding what is happening on your system.

In any case, free memory is wasted memory so there is no point complaining 1.5 GB of RAM is used as long as 500 MB is still available.
 
Old 05-13-2009, 11:26 AM   #3
rajaniyer123
Member
 
Registered: Feb 2004
Location: BARODA, GUJARAT
Posts: 254

Original Poster
Rep: Reputation: 30
Hi

Sorry, for late update

As I got the access of server just yesterday and I found that

while running prstat -s size, by adding all those memories the sum is approx 2.5 GB, in my case total physical memory is 4 GB.

I would like to know which other hidden process/service are taking this memory.

In case any command/script by which I can get the amount of wasted memory it will helpful.

Thanks
Rajan
 
Old 05-13-2009, 02:41 PM   #4
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris10, Solaris 11, Mint, OL
Posts: 9,459

Rep: Reputation: 352Reputation: 352Reputation: 352Reputation: 352
I'm afraid you are confusing RAM (4 GB installed) and virtual memory (prstat size column).

Can you post some commands output:
Code:
cat /etc/release
df -k /tmp
vmstat 2 2
kstat -n system_pages
kstat -n arcstats
 
Old 05-13-2009, 09:40 PM   #5
rajaniyer123
Member
 
Registered: Feb 2004
Location: BARODA, GUJARAT
Posts: 254

Original Poster
Rep: Reputation: 30
Please find herewith all o/p


cat /etc/release
Solaris 10 3/05 s10_74L2a SPARC
Copyright 2005 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 22 January 2005


df -k /tmp
Filesystem kbytes used avail capacity Mounted on
swap 7708576 673608 7034968 9% /tmp


Code:
vmstat 2 2
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr m0 m1 m2 m3   in   sy   cs us sy id
 0 0 0 2966296 1139336 142 549 1 1 1  0  0  3  3  3  0 1168 3304 1909 16  5 78
 0 0 0 7036632 687560 0  13  4  0  0  0  0  0  0  0  0 2464 6248 4588 42  5 53


kstat -n system_pages
module: unix                            instance: 0
name:   system_pages                    class:    pages
        availrmem                       312511
        crtime                          25.844381975
        desfree                         3995
        desscan                         25
        econtig                         46137344
        fastscan                        130850
        freemem                         85614
        kernelbase                      16777216
        lotsfree                        7990
        minfree                         1997
        nalloc                          15920364
        nalloc_calls                    8346
        nfree                           15507901
        nfree_calls                     5153
        nscan                           0
        pagesfree                       85614
        pageslocked                     198852
        pagestotal                      511363
        physmem                         519475
        pp_kernel                       209737
        slowscan                        100
        snaptime                        33133073.1726136

kstat -n arcstats
Thanks
Rajan

Last edited by rajaniyer123; 05-16-2009 at 09:28 AM.
 
Old 05-14-2009, 02:49 AM   #6
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris10, Solaris 11, Mint, OL
Posts: 9,459

Rep: Reputation: 352Reputation: 352Reputation: 352Reputation: 352
1.6 GB is allocated by the kernel, possibly by shared memory segments.

Please post these commands output and use code tags like I do to help readability:
Code:
prstat -n 100 -Z -s rss -c 1 1
echo ::memstat | mdb -k
 
Old 05-14-2009, 06:35 AM   #7
rajaniyer123
Member
 
Registered: Feb 2004
Location: BARODA, GUJARAT
Posts: 254

Original Poster
Rep: Reputation: 30
Please find herewith the o/p.

prstat -n 100 -Z -s rss -c 1 1
Code:
  PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
 17246 bv1to1    140M  138M sleep   59    0   0:03:19 0.1% dmem_adm/3
 24649 bv1to1     84M   75M sleep   53    0   2:45:13 6.4% cntdb/3
 24601 bv1to1     80M   71M sleep   53    0   2:45:21 6.9% cntdb/3
 24715 bv1to1     80M   71M sleep   49    0   2:45:12 6.4% cntdb/3
 24503 bv1to1     80M   71M cpu0    35    0   2:45:23 6.8% cntdb/3
 24676 bv1to1     80M   71M run     42    0   2:46:01 6.1% cntdb/3
 25636 bv1to1     93M   66M sleep   60    0   0:00:07 0.0% sched_srv/4
  1293 root       50M   46M sleep   59    0   2:12:35 0.0% se.sparcv9/1
 24637 bv1to1     44M   36M sleep   59    0   0:06:28 0.6% cmsdb/3
 24726 bv1to1     44M   36M sleep   59    0   0:06:17 0.4% cmsdb/3
 24591 bv1to1     44M   36M sleep   59    0   0:06:31 0.3% cmsdb/3
 24675 bv1to1     44M   35M sleep   59    0   0:06:24 0.3% cmsdb/3
 24714 bv1to1     44M   35M sleep   59    0   0:06:19 0.3% cmsdb/3
 24590 bv1to1     44M   34M sleep   59    0   0:00:05 0.0% cntdb/3
 28649 bv1to1     42M   34M sleep   59    0   0:00:02 0.0% mr_om_srv/3
 28648 bv1to1     41M   33M sleep   59    0   0:00:02 0.0% mr_genseq_srv/3
 24985 bv1to1     38M   31M sleep   49    0   0:10:09 0.4% genericdb/3
 24987 bv1to1     38M   31M sleep   59    0   0:10:05 0.5% genericdb/3
 24988 bv1to1     38M   31M sleep   59    0   0:10:06 0.4% genericdb/3
 24989 bv1to1     38M   31M sleep   59    0   0:10:10 0.4% genericdb/3
 24986 bv1to1     38M   31M cpu0    59    0   0:10:08 0.4% genericdb/3
 23938 bv1to1     38M   31M sleep   59    0   0:00:07 0.0% sched_poll_d/4
 23956 bv1to1     38M   30M sleep   59    0   0:00:00 0.0% deliv_comp_d/3
 23947 bv1to1     18M   16M sleep   59    0   0:01:43 0.0% deliv_smtp_d/3
   219 daemon     95M   12M sleep   59    0   0:02:29 0.0% nfsmapid/3
 23764 bv1to1     13M   11M sleep   59    0   0:04:58 0.0% bvconf_srv/3
 23748 bv1to1   9432K 8320K sleep   59    0   0:00:21 0.0% orbixd/1
     9 root     9392K 6320K sleep   59    0   0:07:22 0.0% svc.configd/12
     7 root     8424K 4728K sleep   59    0   0:37:41 0.0% svc.startd/12
 13035 root     4472K 4120K cpu1    49    0   0:00:00 0.1% prstat/1
 12947 root     6992K 3360K sleep   59    0   0:00:00 0.0% sshd/1
   370 root     6080K 2608K sleep   59    0   0:10:37 0.0% automountd/3
   423 root     5920K 2552K sleep   59    0   0:05:48 0.0% fmd/13
 15322 bv1to1   3152K 2400K sleep   59    0   0:00:00 0.0% bash/1
  9725 root     3936K 2376K sleep   59    0   0:00:00 0.0% bash/1
  9760 root     3920K 2352K sleep   59    0   0:00:00 0.0% bash/1
  9019 root     3936K 2280K sleep   59    0   0:00:00 0.0% bash/1
 13034 root     3040K 2272K sleep   49    0   0:00:00 0.0% bash/1
 12951 riyer    3008K 2256K sleep   59    0   0:00:00 0.0% bash/1
 12980 riyer    3000K 2232K sleep   59    0   0:00:00 0.0% bash/1
 12949 riyer    6520K 2120K sleep   59    0   0:00:00 0.0% sshd/1
  9258 root     3936K 2000K sleep   59    0   0:00:00 0.0% bash/1
  7568 postfix  3608K 1912K sleep   59    0   0:00:00 0.0% pickup/1
 12925 root     3944K 1840K sleep   59    0   0:00:00 0.0% bash/1
  1341 root     3160K 1816K sleep   59    0   0:00:29 0.0% syslog-ng/3
    60 root     3688K 1784K sleep   59    0   2:06:06 0.0% picld/6
 24495 bv1to1   2736K 1568K sleep   59    0   0:00:00 0.0% bvsystemd/1
 21351 root     3944K 1552K sleep   59    0   0:00:00 0.0% bash/1
    80 root     2768K 1528K sleep   59    0   0:00:00 0.0% devfsadm/7
  8190 root     3944K 1520K sleep   59    0   0:00:00 0.0% bash/1
    78 daemon   4384K 1464K sleep   59    0   0:00:00 0.0% kcfd/3
 24703 root       19M 1440K sleep   59    0   0:00:00 0.0% catrapmuxd/1
 22046 root     3944K 1376K sleep   59    0   0:00:00 0.0% bash/1
  8406 root     3944K 1376K sleep   59    0   0:00:00 0.0% bash/1
 19284 root     3944K 1368K sleep   59    0   0:00:00 0.0% bash/1
 17244 root     3944K 1368K sleep   59    0   0:00:00 0.0% bash/1
  2323 root     3936K 1360K sleep   59    0   0:00:00 0.0% bash/1
 11241 root     3936K 1352K sleep   59    0   0:00:00 0.0% bash/1
  9136 root     3944K 1352K sleep   59    0   0:00:00 0.0% bash/1
 17234 bv1to1   1592K 1352K sleep   59    0   0:00:00 0.0% bvconf/1
 23109 root     3944K 1344K sleep   59    0   0:00:00 0.0% bash/1
  4101 root     3936K 1304K sleep   59    0   0:00:00 0.0% bash/1
  4049 root     2264K 1304K sleep   59    0   0:27:53 0.0% camf/1
 12981 root     1320K 1096K sleep   59    0   0:00:00 0.0% sh/1
  1286 root     3920K 1032K sleep   59    0   0:58:01 0.0% sshd/1
  9724 root     1200K  992K sleep   59    0   0:00:00 0.0% script.hide/1
  9018 root     1200K  968K sleep   59    0   0:00:00 0.0% script.hide/1
  1285 postfix  3800K  904K sleep   59    0   0:18:54 0.0% qmgr/1
    66 root     2560K  896K sleep   59    0   0:00:00 0.0% syseventd/14
  9257 root     1200K  888K sleep   59    0   0:00:00 0.0% script.hide/1
 12924 root     1200K  880K sleep   59    0   0:00:00 0.0% script.hide/1
  2322 root     1200K  856K sleep   59    0   0:00:00 0.0% script.hide/1
   208 root     2752K  824K sleep   59    0   0:42:55 0.0% cron/1
 21350 root     1200K  808K sleep   59    0   0:00:00 0.0% script.hide/1
  8189 root     1200K  800K sleep   59    0   0:00:00 0.0% script.hide/1
   212 daemon   2728K  760K sleep   59    0   8:59:12 0.0% rpcbind/1
  1266 root     3600K  672K sleep   59    0   0:04:29 0.0% master/1
   236 root     2192K  528K sleep   59    0   0:00:00 0.0% ttymon/1
     1 root     2392K  464K sleep   59    0   0:42:23 0.0% init/1
   232 root     2200K  432K sleep   59    0   0:00:13 0.0% ttymon/1
   233 root     1280K  424K sleep   59    0   0:01:00 0.0% utmpd/1
 17243 root     1200K  360K sleep   59    0   0:00:00 0.0% script.hide/1
  9135 root     1200K  360K sleep   59    0   0:00:00 0.0% script.hide/1
 11240 root     1200K  352K sleep   59    0   0:00:05 0.0% script.hide/1
  4100 root     1200K  352K sleep   59    0   0:00:00 0.0% script.hide/1
 23108 root     1200K  344K sleep   59    0   0:00:02 0.0% script.hide/1
 22045 root     1200K  344K sleep   59    0   0:00:00 0.0% script.hide/1
  8405 root     1200K  344K sleep   59    0   0:00:00 0.0% script.hide/1
   226 root     4416K  336K sleep   59    0   0:07:09 0.0% inetd/4
   222 root     2000K  328K sleep   59    0   0:00:14 0.0% sac/1
 19283 root     1200K  320K sleep   59    0   0:01:49 0.0% script.hide/1
    88 root     2344K  192K sleep   59    0   0:00:00 0.0% powerd/3
   229 daemon   2368K  112K sleep   60  -20   0:00:00 0.0% lockd/2
   216 daemon   2344K  112K sleep   60  -20   0:00:00 0.0% nfs4cbd/2
   218 daemon   2808K   64K sleep   59    0   0:00:00 0.0% statd/1
ZONEID    NPROC  SIZE   RSS MEMORY      TIME  CPU ZONE
     0       95 1662M 1243M    31%  33:07:57  37% global
Total: 95 processes, 218 lwps, load averages: 1.07, 1.11, 1.09



 echo ::memstat | mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     198068              1547   38%
Anon                        82833               647   16%
Exec and libs                8809                68    2%
Page cache                 158906              1241   31%
Free (cachelist)            23537               183    5%
Free (freelist)             47322               369    9%

Total                      519475              4058
Physical                   511363              3995
Thanks
Rajan

Last edited by rajaniyer123; 05-15-2009 at 09:57 PM.
 
Old 05-14-2009, 02:49 PM   #8
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris10, Solaris 11, Mint, OL
Posts: 9,459

Rep: Reputation: 352Reputation: 352Reputation: 352Reputation: 352
Again, please edit your posting to use code tags to improve readability of CLI output.

To do that:
click the "Edit" button on the bottom of your posting, then click the "Go advanced" button, then select the CLI transcript with your mouse, click the "#" icon (Wrap [code] tages around selection) and finally click on "Save changes".
 
Old 05-26-2009, 09:33 AM   #9
rajaniyer123
Member
 
Registered: Feb 2004
Location: BARODA, GUJARAT
Posts: 254

Original Poster
Rep: Reputation: 30
Hi

In the same ref. I have compared mdb output of some other good server, it seems that in above server where memory is very less the kernel's memory consumption is high. Any idea how I can identify and kill this processes, without rebooting the server and free up the memory.


Good SErver


Code:
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                      61072               477   12%
Anon                        78303               611   15%
Exec and libs                9053                70    2%
Page cache                  87203               681   17%
Free (cachelist)           235333              1838   45%
Free (freelist)             48511               378    9%

Total                      519475              4058
Physical                   511363              3995
Bad Server

Code:
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     226433              1769   44%
Anon                       145732              1138   28%
Exec and libs                9117                71    2%
Page cache                  57135               446   11%
Free (cachelist)            25120               196    5%
Free (freelist)             55938               437   11%

Total                      519475              4058
Physical                   511363              3995
Thanks
Rajan

Last edited by rajaniyer123; 05-26-2009 at 09:34 AM.
 
Old 05-26-2009, 04:18 PM   #10
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris10, Solaris 11, Mint, OL
Posts: 9,459

Rep: Reputation: 352Reputation: 352Reputation: 352Reputation: 352
If the kernel uses memory, it is unlikely you can free that memory by killing processes unless you count the kernel as a process, which it is not.

Are you using ZFS ?

Are there a lot of or very large shared memory segments used ?
run "ipcs -ma" to figure it out.
 
Old 05-26-2009, 09:43 PM   #11
rajaniyer123
Member
 
Registered: Feb 2004
Location: BARODA, GUJARAT
Posts: 254

Original Poster
Rep: Reputation: 30
The system is using UFS only.

also the o/p of ipcs -ma is

Code:
IPC status from <running system> as of Tue May 26 22:41:19 EDT 2009
T         ID      KEY        MODE        OWNER    GROUP  CREATOR   CGROUP NATTCH      SEGSZ  CPID  LPID   ATIME    DTIME    CTIME
Shared Memory:
m         14   0          --rw-------     root     root     root     root      2     159804 21599  8123  5:15:22 22:40:59  5:15:22

Thanks
Rajan

Last edited by rajaniyer123; 05-26-2009 at 09:45 PM.
 
Old 05-27-2009, 02:11 AM   #12
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris10, Solaris 11, Mint, OL
Posts: 9,459

Rep: Reputation: 352Reputation: 352Reputation: 352Reputation: 352
Okay, no shared memory issue here.

You can get detailed kernel memory usage with this command:
Code:
echo ::kmastat | mdb -k
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to check memory usage on running process/applications rickylim Linux - General 17 05-08-2013 09:09 PM
memory consumed by a process ZAMO Linux - General 1 02-19-2009 09:22 AM
How to find how much memory consumed by Linux Kernel in my system kspradhan Linux - Kernel 3 11-17-2008 11:12 PM
shell script terminates as zombie, ie, <defunct> Kropotkin Linux - Newbie 3 02-20-2007 12:15 AM
kjournald process consumed all CPU time tttx Linux - General 1 05-11-2005 03:28 PM


All times are GMT -5. The time now is 03:54 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration