LinuxQuestions.org - How to check memory consumed by hidden/defunct/zombie process

- Solaris / OpenSolaris (https://www.linuxquestions.org/questions/solaris-opensolaris-20/)

- - How to check memory consumed by hidden/defunct/zombie process (https://www.linuxquestions.org/questions/solaris-opensolaris-20/how-to-check-memory-consumed-by-hidden-defunct-zombie-process-717875/)

How to check memory consumed by hidden/defunct/zombie process

Hi

In one of the prodcution server I am facing typical problem, in which I have 4 GB of RAM. Once the application is running there should be 2 GB free memory approx.

In that server only Broadvision is running, I have shutdown the BV still the free memory it shows only 500 MB instaed of 2 GB, I would like to know which is the hidden process which consumes the Physical Memory.

I am not able to track the process which utilizes high memory in top/prstat but not able to figure out.

Also I have checked defunct process and killed it but it does not make any difference.

Please suggest.

I want to aviod the reboot of the server.

Thanks
Rajan

You should provide some material to help understanding what is happening on your system.

In any case, free memory is wasted memory so there is no point complaining 1.5 GB of RAM is used as long as 500 MB is still available.

Hi

Sorry, for late update

As I got the access of server just yesterday and I found that

while running prstat -s size, by adding all those memories the sum is approx 2.5 GB, in my case total physical memory is 4 GB.

I would like to know which other hidden process/service are taking this memory.

In case any command/script by which I can get the amount of wasted memory it will helpful.

Thanks
Rajan

I'm afraid you are confusing RAM (4 GB installed) and virtual memory (prstat size column).

Can you post some commands output:

Code:

cat /etc/release

df -k /tmp

vmstat 2 2

kstat -n system_pages

kstat -n arcstats

Please find herewith all o/p

cat /etc/release
Solaris 10 3/05 s10_74L2a SPARC
Copyright 2005 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 22 January 2005

df -k /tmp
Filesystem kbytes used avail capacity Mounted on
swap 7708576 673608 7034968 9% /tmp

Code:

vmstat 2 2

 kthr      memory            page            disk          faults      cpu

 r b w  swap  free  re  mf pi po fr de sr m0 m1 m2 m3  in  sy  cs us sy id

 0 0 0 2966296 1139336 142 549 1 1 1  0  0  3  3  3  0 1168 3304 1909 16  5 78

 0 0 0 7036632 687560 0  13  4  0  0  0  0  0  0  0  0 2464 6248 4588 42  5 53





kstat -n system_pages

module: unix                            instance: 0

name:  system_pages                    class:    pages

        availrmem                      312511

        crtime                          25.844381975

        desfree                        3995

        desscan                        25

        econtig                        46137344

        fastscan                        130850

        freemem                        85614

        kernelbase                      16777216

        lotsfree                        7990

        minfree                        1997

        nalloc                          15920364

        nalloc_calls                    8346

        nfree                          15507901

        nfree_calls                    5153

        nscan                          0

        pagesfree                      85614

        pageslocked                    198852

        pagestotal                      511363

        physmem                        519475

        pp_kernel                      209737

        slowscan                        100

        snaptime                        33133073.1726136



kstat -n arcstats

Thanks
Rajan

1.6 GB is allocated by the kernel, possibly by shared memory segments.

Please post these commands output and use code tags like I do to help readability:

Code:

prstat -n 100 -Z -s rss -c 1 1

echo ::memstat | mdb -k

Please find herewith the o/p.

prstat -n 100 -Z -s rss -c 1 1

Code:

  PID USERNAME  SIZE  RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP

 17246 bv1to1    140M  138M sleep  59    0  0:03:19 0.1% dmem_adm/3

 24649 bv1to1    84M  75M sleep  53    0  2:45:13 6.4% cntdb/3

 24601 bv1to1    80M  71M sleep  53    0  2:45:21 6.9% cntdb/3

 24715 bv1to1    80M  71M sleep  49    0  2:45:12 6.4% cntdb/3

 24503 bv1to1    80M  71M cpu0    35    0  2:45:23 6.8% cntdb/3

 24676 bv1to1    80M  71M run    42    0  2:46:01 6.1% cntdb/3

 25636 bv1to1    93M  66M sleep  60    0  0:00:07 0.0% sched_srv/4

  1293 root      50M  46M sleep  59    0  2:12:35 0.0% se.sparcv9/1

 24637 bv1to1    44M  36M sleep  59    0  0:06:28 0.6% cmsdb/3

 24726 bv1to1    44M  36M sleep  59    0  0:06:17 0.4% cmsdb/3

 24591 bv1to1    44M  36M sleep  59    0  0:06:31 0.3% cmsdb/3

 24675 bv1to1    44M  35M sleep  59    0  0:06:24 0.3% cmsdb/3

 24714 bv1to1    44M  35M sleep  59    0  0:06:19 0.3% cmsdb/3

 24590 bv1to1    44M  34M sleep  59    0  0:00:05 0.0% cntdb/3

 28649 bv1to1    42M  34M sleep  59    0  0:00:02 0.0% mr_om_srv/3

 28648 bv1to1    41M  33M sleep  59    0  0:00:02 0.0% mr_genseq_srv/3

 24985 bv1to1    38M  31M sleep  49    0  0:10:09 0.4% genericdb/3

 24987 bv1to1    38M  31M sleep  59    0  0:10:05 0.5% genericdb/3

 24988 bv1to1    38M  31M sleep  59    0  0:10:06 0.4% genericdb/3

 24989 bv1to1    38M  31M sleep  59    0  0:10:10 0.4% genericdb/3

 24986 bv1to1    38M  31M cpu0    59    0  0:10:08 0.4% genericdb/3

 23938 bv1to1    38M  31M sleep  59    0  0:00:07 0.0% sched_poll_d/4

 23956 bv1to1    38M  30M sleep  59    0  0:00:00 0.0% deliv_comp_d/3

 23947 bv1to1    18M  16M sleep  59    0  0:01:43 0.0% deliv_smtp_d/3

  219 daemon    95M  12M sleep  59    0  0:02:29 0.0% nfsmapid/3

 23764 bv1to1    13M  11M sleep  59    0  0:04:58 0.0% bvconf_srv/3

 23748 bv1to1  9432K 8320K sleep  59    0  0:00:21 0.0% orbixd/1

    9 root    9392K 6320K sleep  59    0  0:07:22 0.0% svc.configd/12

    7 root    8424K 4728K sleep  59    0  0:37:41 0.0% svc.startd/12

 13035 root    4472K 4120K cpu1    49    0  0:00:00 0.1% prstat/1

 12947 root    6992K 3360K sleep  59    0  0:00:00 0.0% sshd/1

  370 root    6080K 2608K sleep  59    0  0:10:37 0.0% automountd/3

  423 root    5920K 2552K sleep  59    0  0:05:48 0.0% fmd/13

 15322 bv1to1  3152K 2400K sleep  59    0  0:00:00 0.0% bash/1

  9725 root    3936K 2376K sleep  59    0  0:00:00 0.0% bash/1

  9760 root    3920K 2352K sleep  59    0  0:00:00 0.0% bash/1

  9019 root    3936K 2280K sleep  59    0  0:00:00 0.0% bash/1

 13034 root    3040K 2272K sleep  49    0  0:00:00 0.0% bash/1

 12951 riyer    3008K 2256K sleep  59    0  0:00:00 0.0% bash/1

 12980 riyer    3000K 2232K sleep  59    0  0:00:00 0.0% bash/1

 12949 riyer    6520K 2120K sleep  59    0  0:00:00 0.0% sshd/1

  9258 root    3936K 2000K sleep  59    0  0:00:00 0.0% bash/1

  7568 postfix  3608K 1912K sleep  59    0  0:00:00 0.0% pickup/1

 12925 root    3944K 1840K sleep  59    0  0:00:00 0.0% bash/1

  1341 root    3160K 1816K sleep  59    0  0:00:29 0.0% syslog-ng/3

    60 root    3688K 1784K sleep  59    0  2:06:06 0.0% picld/6

 24495 bv1to1  2736K 1568K sleep  59    0  0:00:00 0.0% bvsystemd/1

 21351 root    3944K 1552K sleep  59    0  0:00:00 0.0% bash/1

    80 root    2768K 1528K sleep  59    0  0:00:00 0.0% devfsadm/7

  8190 root    3944K 1520K sleep  59    0  0:00:00 0.0% bash/1

    78 daemon  4384K 1464K sleep  59    0  0:00:00 0.0% kcfd/3

 24703 root      19M 1440K sleep  59    0  0:00:00 0.0% catrapmuxd/1

 22046 root    3944K 1376K sleep  59    0  0:00:00 0.0% bash/1

  8406 root    3944K 1376K sleep  59    0  0:00:00 0.0% bash/1

 19284 root    3944K 1368K sleep  59    0  0:00:00 0.0% bash/1

 17244 root    3944K 1368K sleep  59    0  0:00:00 0.0% bash/1

  2323 root    3936K 1360K sleep  59    0  0:00:00 0.0% bash/1

 11241 root    3936K 1352K sleep  59    0  0:00:00 0.0% bash/1

  9136 root    3944K 1352K sleep  59    0  0:00:00 0.0% bash/1

 17234 bv1to1  1592K 1352K sleep  59    0  0:00:00 0.0% bvconf/1

 23109 root    3944K 1344K sleep  59    0  0:00:00 0.0% bash/1

  4101 root    3936K 1304K sleep  59    0  0:00:00 0.0% bash/1

  4049 root    2264K 1304K sleep  59    0  0:27:53 0.0% camf/1

 12981 root    1320K 1096K sleep  59    0  0:00:00 0.0% sh/1

  1286 root    3920K 1032K sleep  59    0  0:58:01 0.0% sshd/1

  9724 root    1200K  992K sleep  59    0  0:00:00 0.0% script.hide/1

  9018 root    1200K  968K sleep  59    0  0:00:00 0.0% script.hide/1

  1285 postfix  3800K  904K sleep  59    0  0:18:54 0.0% qmgr/1

    66 root    2560K  896K sleep  59    0  0:00:00 0.0% syseventd/14

  9257 root    1200K  888K sleep  59    0  0:00:00 0.0% script.hide/1

 12924 root    1200K  880K sleep  59    0  0:00:00 0.0% script.hide/1

  2322 root    1200K  856K sleep  59    0  0:00:00 0.0% script.hide/1

  208 root    2752K  824K sleep  59    0  0:42:55 0.0% cron/1

 21350 root    1200K  808K sleep  59    0  0:00:00 0.0% script.hide/1

  8189 root    1200K  800K sleep  59    0  0:00:00 0.0% script.hide/1

  212 daemon  2728K  760K sleep  59    0  8:59:12 0.0% rpcbind/1

  1266 root    3600K  672K sleep  59    0  0:04:29 0.0% master/1

  236 root    2192K  528K sleep  59    0  0:00:00 0.0% ttymon/1

    1 root    2392K  464K sleep  59    0  0:42:23 0.0% init/1

  232 root    2200K  432K sleep  59    0  0:00:13 0.0% ttymon/1

  233 root    1280K  424K sleep  59    0  0:01:00 0.0% utmpd/1

 17243 root    1200K  360K sleep  59    0  0:00:00 0.0% script.hide/1

  9135 root    1200K  360K sleep  59    0  0:00:00 0.0% script.hide/1

 11240 root    1200K  352K sleep  59    0  0:00:05 0.0% script.hide/1

  4100 root    1200K  352K sleep  59    0  0:00:00 0.0% script.hide/1

 23108 root    1200K  344K sleep  59    0  0:00:02 0.0% script.hide/1

 22045 root    1200K  344K sleep  59    0  0:00:00 0.0% script.hide/1

  8405 root    1200K  344K sleep  59    0  0:00:00 0.0% script.hide/1

  226 root    4416K  336K sleep  59    0  0:07:09 0.0% inetd/4

  222 root    2000K  328K sleep  59    0  0:00:14 0.0% sac/1

 19283 root    1200K  320K sleep  59    0  0:01:49 0.0% script.hide/1

    88 root    2344K  192K sleep  59    0  0:00:00 0.0% powerd/3

  229 daemon  2368K  112K sleep  60  -20  0:00:00 0.0% lockd/2

  216 daemon  2344K  112K sleep  60  -20  0:00:00 0.0% nfs4cbd/2

  218 daemon  2808K  64K sleep  59    0  0:00:00 0.0% statd/1

ZONEID    NPROC  SIZE  RSS MEMORY      TIME  CPU ZONE

    0      95 1662M 1243M    31%  33:07:57  37% global

Total: 95 processes, 218 lwps, load averages: 1.07, 1.11, 1.09







 echo ::memstat | mdb -k

Page Summary                Pages                MB  %Tot

------------    ----------------  ----------------  ----

Kernel                    198068              1547  38%

Anon                        82833              647  16%

Exec and libs                8809                68    2%

Page cache                158906              1241  31%

Free (cachelist)            23537              183    5%

Free (freelist)            47322              369    9%



Total                      519475              4058

Physical                  511363              3995

Thanks
Rajan

Again, please edit your posting to use code tags to improve readability of CLI output.

To do that:
click the "Edit" button on the bottom of your posting, then click the "Go advanced" button, then select the CLI transcript with your mouse, click the "#" icon (Wrap [code] tages around selection) and finally click on "Save changes".

Hi

In the same ref. I have compared mdb output of some other good server, it seems that in above server where memory is very less the kernel's memory consumption is high. Any idea how I can identify and kill this processes, without rebooting the server and free up the memory.

Good SErver

Code:

Page Summary                Pages                MB  %Tot

------------    ----------------  ----------------  ----

Kernel                      61072              477  12%

Anon                        78303              611  15%

Exec and libs                9053                70    2%

Page cache                  87203              681  17%

Free (cachelist)          235333              1838  45%

Free (freelist)            48511              378    9%



Total                      519475              4058

Physical                  511363              3995

Bad Server

Code:

Page Summary                Pages                MB  %Tot

------------    ----------------  ----------------  ----

Kernel                    226433              1769  44%

Anon                      145732              1138  28%

Exec and libs                9117                71    2%

Page cache                  57135              446  11%

Free (cachelist)            25120              196    5%

Free (freelist)            55938              437  11%



Total                      519475              4058

Physical                  511363              3995

Thanks
Rajan

If the kernel uses memory, it is unlikely you can free that memory by killing processes unless you count the kernel as a process, which it is not.

Are you using ZFS ?

Are there a lot of or very large shared memory segments used ?
run "ipcs -ma" to figure it out.

The system is using UFS only.

also the o/p of ipcs -ma is

Code:

IPC status from <running system> as of Tue May 26 22:41:19 EDT 2009

T        ID      KEY        MODE        OWNER    GROUP  CREATOR  CGROUP NATTCH      SEGSZ  CPID  LPID  ATIME    DTIME    CTIME

Shared Memory:

m        14  0          --rw-------    root    root    root    root      2    159804 21599  8123  5:15:22 22:40:59  5:15:22

Thanks
Rajan

Okay, no shared memory issue here.

You can get detailed kernel memory usage with this command:

Code:

echo ::kmastat | mdb -k