I'm on Linux revision 2.6.18-274.el5

I'm using the following basic flow to measure the total resident memory set per process (PID):
- I sum up the VmRSS value from /proc/PID/status files for PID and all its child processes.

Similarly to measure virtual memory:
- I sum up the VmSize value from /proc/PID/status files for PID and all its child processes.

Recently I have seen cases where this total, greatly exceeds the total physical memory (as indicated by 'top'), while no SWAP condition existed.
I understand I'm missing something -

These cases occurred when a process spawned multiple equal child-processes, all with same VmSize,VmRSS pair values. Summing up these values exceeds server memory by 5-10 times, depending on the number of spawned child processes.

My questions are:
1. The 'pmap' solution suggested here (linuxquestions.org/linux/articles/Technical/Understanding_memory_usage_on_Linux) does not seem to clear things up, as it shows under 'Dirty' similar values of this in RSS. How do I interpret pmap results ?
2. I assume that there is memory sharing - perhaps memory mappd files - between these processes. I have used 'lsof -p PID' to list these connection. my question is how to interpret lsof output and connect between the lsof output and the memory shown by VmRSS ?
3. If i have several processes sharing memory, how do I avoid double counting the shared memory ?
4. For the 'top' command output - when a process is shown with state (S) of S (i.e. sleep) what is the meaning of the RSS value ? does the process continue to use that RSS value worth of memory ?

e.g. top output
top - 19:48:15 up 126 days, 51 min, 9 users, load average: 6.36, 5.97, 5.98
Tasks: 317 total, 9 running, 308 sleeping, 0 stopped, 0 zombie
Cpu(s): 47.3%us, 11.3%sy, 0.0%ni, 40.7%id, 0.6%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 132061256k total, 98894476k used, 33166780k free, 640920k buffers
Swap: 51199144k total, 20596k used, 51178548k free, 60546200k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
365 igorb 21 0 13.5g 13g 3728 R 100.0 10.6 53:40.97 fm_shell_exec
306 igorb 25 0 13.5g 13g 3796 R 99.8 10.6 53:53.02 fm_shell_exec
307 igorb 25 0 13.5g 13g 3608 R 99.8 10.6 53:52.78 fm_shell_exec
314 igorb 18 0 13.5g 13g 3672 R 99.8 10.6 53:50.13 fm_shell_exec
366 igorb 25 0 13.5g 13g 812 S 0.0 10.6 0:00.00 fm_shell_exec
373 igorb 25 0 13.5g 13g 812 S 0.0 10.6 0:00.00 fm_shell_exec
482 igorb 25 0 13.5g 13g 812 S 0.0 10.6 0:00.00 fm_shell_exec
588 igorb 25 0 13.5g 13g 812 S 0.0 10.6 0:00.00 fm_shell_exec
589 igorb 25 0 13.5g 13g 812 S 0.0 10.6 0:00.00 fm_shell_exec
590 igorb 25 0 13.5g 13g 812 S 0.0 10.6 0:00.00 fm_shell_exec
591 igorb 25 0 13.5g 13g 812 S 0.0 10.6 0:00.00 fm_shell_exec
592 igorb 25 0 13.5g 13g 812 S 0.0 10.6 0:00.00 fm_shell_exec


Any tips are gladly accepted.
Y.

I don't think there is any practical way to measure what you want to measure. I know it is something one would like to measure. I know it is something a lot of Linux admins think they are measuring. But I still don't think it is practical to measure.

I have read about some optional kernel features that make it closer to being practical to measure. But none of them looked too great from the crude descriptions I read and beyond that I don't know details.

For your question 4, the CPU state of the process is largely independent of its memory state. The sleep state does not change the meaning of any memory stats.