How to correctly measuring Memory usage per process
I'm on Linux revision 2.6.18-274.el5
I'm using the following basic flow to measure the total resident memory set per process (PID):
- I sum up the VmRSS value from /proc/PID/status files for PID and all its child processes.
Similarly to measure virtual memory:
- I sum up the VmSize value from /proc/PID/status files for PID and all its child processes.
Recently I have seen cases where this total, greatly exceeds the total physical memory (as indicated by 'top'), while no SWAP condition existed.
I understand I'm missing something -
These cases occurred when a process spawned multiple equal child-processes, all with same VmSize,VmRSS pair values. Summing up these values exceeds server memory by 5-10 times, depending on the number of spawned child processes.
My questions are:
1. The 'pmap' solution suggested here (linuxquestions.org/linux/articles/Technical/Understanding_memory_usage_on_Linux) does not seem to clear things up, as it shows under 'Dirty' similar values of this in RSS. How do I interpret pmap results ?
2. I assume that there is memory sharing - perhaps memory mappd files - between these processes. I have used 'lsof -p PID' to list these connection. my question is how to interpret lsof output and connect between the lsof output and the memory shown by VmRSS ?
3. If i have several processes sharing memory, how do I avoid double counting the shared memory ?
4. For the 'top' command output - when a process is shown with state (S) of S (i.e. sleep) what is the meaning of the RSS value ? does the process continue to use that RSS value worth of memory ?
e.g. top output
top - 19:48:15 up 126 days, 51 min, 9 users, load average: 6.36, 5.97, 5.98
Tasks: 317 total, 9 running, 308 sleeping, 0 stopped, 0 zombie
Cpu(s): 47.3%us, 11.3%sy, 0.0%ni, 40.7%id, 0.6%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 132061256k total, 98894476k used, 33166780k free, 640920k buffers
Swap: 51199144k total, 20596k used, 51178548k free, 60546200k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
365 igorb 21 0 13.5g 13g 3728 R 100.0 10.6 53:40.97 fm_shell_exec
306 igorb 25 0 13.5g 13g 3796 R 99.8 10.6 53:53.02 fm_shell_exec
307 igorb 25 0 13.5g 13g 3608 R 99.8 10.6 53:52.78 fm_shell_exec
314 igorb 18 0 13.5g 13g 3672 R 99.8 10.6 53:50.13 fm_shell_exec
366 igorb 25 0 13.5g 13g 812 S 0.0 10.6 0:00.00 fm_shell_exec
373 igorb 25 0 13.5g 13g 812 S 0.0 10.6 0:00.00 fm_shell_exec
482 igorb 25 0 13.5g 13g 812 S 0.0 10.6 0:00.00 fm_shell_exec
588 igorb 25 0 13.5g 13g 812 S 0.0 10.6 0:00.00 fm_shell_exec
589 igorb 25 0 13.5g 13g 812 S 0.0 10.6 0:00.00 fm_shell_exec
590 igorb 25 0 13.5g 13g 812 S 0.0 10.6 0:00.00 fm_shell_exec
591 igorb 25 0 13.5g 13g 812 S 0.0 10.6 0:00.00 fm_shell_exec
592 igorb 25 0 13.5g 13g 812 S 0.0 10.6 0:00.00 fm_shell_exec
Any tips are gladly accepted.
Y.
|