LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel
User Name
Password
Linux - Kernel This forum is for all discussion relating to the Linux kernel.

Notices

Reply
 
Search this Thread
Old 02-16-2007, 04:18 PM   #1
rhimbo
LQ Newbie
 
Registered: Dec 2004
Posts: 29

Rep: Reputation: 15
How to _accurately_ measure memory usage?


Hello all,

We're running an embedded Linux system as follows:
  • Embedded Linux based on 2.4.29 Kernel
  • Custom distro (created our own by picking and choosing packages and utils that we needed)
  • Using ucblib for our libc
  • Virtual memory - yes
  • No swap (swapping turned off and no swap partitions)
  • Heavy use of shared libraries

We're trying to accurately measure memory usage. Specifically, we need to measure the number of unique pages of memory used by our application software.

The problem we're having, I think, is the inaccurate counting of shared library pages. The per process memory calculation performed by tools such as "vmstat", "free" and "top" counts a shared page m times where m is the number of processes that share the page. The result is a memory usage figure that's higher than the actual physical usage. In fact, for each set of m processes that share n pages, the figure is (m x n), which is [m x (n - 1)] pages too high. The correct figure should be simply n.

Instead, we need to count total number of unique pages used. Moreover, we need two measurements eventually. First and foremost, we need to accurately measure the total memory used when the system is running in a "steady state" (of course this would be an average measurement over time). Secondly, we eventually need to measure the amount of memory used per process.

Can anyone point me to system utilities, tools or even some strategy/approach for writing my own application that would yield the correct measurement?

One problem we have is the lack of support of 3rd party tools. We're running a proprietary system on a MIPS processor (not too popular these days). I think we're going to encounter some difficulty getting support--or even getting a version that runs on our platform.

Many thanks for any suggestions or leads.


Vartan
 
Old 02-16-2007, 05:25 PM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,446

Rep: Reputation: 1068Reputation: 1068Reputation: 1068Reputation: 1068Reputation: 1068Reputation: 1068Reputation: 1068Reputation: 1068
Don't know about a 2.4 kernel - too many years since I used one in anger.
One of the later 2.6 kernels (2.6.14, something like that) introduced smaps. Just what you need.
 
Old 02-16-2007, 06:51 PM   #3
tovis
Member
 
Registered: Jan 2007
Location: Budapest
Distribution: Debian
Posts: 74

Rep: Reputation: 15
What about the /proc file system? There is a precise list of all used memory, is not it?
 
Old 02-19-2007, 12:35 AM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,446

Rep: Reputation: 1068Reputation: 1068Reputation: 1068Reputation: 1068Reputation: 1068Reputation: 1068Reputation: 1068Reputation: 1068
Perhaps have a look at this. Discusses using pmap - not sure if it is available under 2.4; wasn't on the only system I have access to.

Be aware that the “writeable/private” total cited is the mapped total, not the (private) resident size.
As I said, smaps does this nicely, but not where you are.
 
Old 02-19-2007, 10:16 AM   #5
nx5000
Senior Member
 
Registered: Sep 2005
Location: Out
Posts: 3,307

Rep: Reputation: 53
Have a look at memstat
 
Old 02-19-2007, 07:06 PM   #6
rhimbo
LQ Newbie
 
Registered: Dec 2004
Posts: 29

Original Poster
Rep: Reputation: 15
Hi,

Thanks for the reply.... Sorry, one question... Could you explain the term "mapped." Is it the processes' private resident size plus the shared lib space?

Thanks.
 
Old 02-19-2007, 07:10 PM   #7
rhimbo
LQ Newbie
 
Registered: Dec 2004
Posts: 29

Original Poster
Rep: Reputation: 15
memstat source code?

Hi all,

Thanks for the replies.

Can anyone point me to memstat source code? I followed the posted link and did a Google search but had no luck... several dead links.
 
Old 02-19-2007, 07:18 PM   #8
rhimbo
LQ Newbie
 
Registered: Dec 2004
Posts: 29

Original Poster
Rep: Reputation: 15
/proc file system

Hi again,

OK, I was told moving to 2.6 kernel is not an option for the foreseeable future. So... I'm considering the following two options:
  1. Write a shell script to call pmap for each process
  2. Write a C program to access kernel page table and count pages used

In the first approach, I suppose I can write a shell script that pipes ps output to a toolchain that counts each shared lib once. Then, it can count the private memory used for each process and sum the result.

In the second approach, I'm wondering if I can write a C program that counts the total memory pages used in use. This would not give me per process memory usage figures, but it would give em an accurate number of total memory usage. Can anyone get me started on how to do this? Does this require accessing the /proc fiel system or some other part of the kernel? Can anyone point me to docs kernel API docs or anything?

Thanks again.
 
Old 02-19-2007, 11:10 PM   #9
rhimbo
LQ Newbie
 
Registered: Dec 2004
Posts: 29

Original Poster
Rep: Reputation: 15
How to interpret pmap output

I've been playing with pmap. That seems to be my best option given that I can't go to a Linux 2.6 kernel.

Can anyone tell me how I should interpret the lines with [ anon ] and [ stack ] ? I am guessing that the [ stack ] line is the actual process stack? Is that correct?

What about [ anon ] ? What is it; how should I count it?

For example, here's some out put from my system:

Code:
# pmap -d 3789
3789:   /usr/sbin/atd
Address   Kbytes Mode  Offset           Device    Mapping
b7e09000      36 r-x-- 0000000000000000 008:00002 libnss_files-2.3.6.so
b7e12000       4 r-x-- 0000000000008000 008:00002 libnss_files-2.3.6.so
b7e13000       4 rwx-- 0000000000009000 008:00002 libnss_files-2.3.6.so
b7e14000       4 rwx-- 00000000b7e14000 000:00000   [ anon ]
b7e15000       8 r-x-- 0000000000000000 008:00002 libdl-2.3.6.so
b7e17000       4 r-x-- 0000000000001000 008:00002 libdl-2.3.6.so
b7e18000       4 rwx-- 0000000000002000 008:00002 libdl-2.3.6.so
b7e19000    1168 r-x-- 0000000000000000 008:00002 libc-2.3.6.so
b7f3d000       4 ----- 0000000000124000 008:00002 libc-2.3.6.so
b7f3e000       8 r-x-- 0000000000124000 008:00002 libc-2.3.6.so
b7f40000       8 rwx-- 0000000000126000 008:00002 libc-2.3.6.so
b7f42000       8 rwx-- 00000000b7f42000 000:00000   [ anon ]
b7f44000       8 r-x-- 0000000000000000 008:00002 libpam_misc.so.0.77
b7f46000       4 rwx-- 0000000000001000 008:00002 libpam_misc.so.0.77
b7f47000      28 r-x-- 0000000000000000 008:00002 libpam.so.0.77
b7f4e000       4 rwx-- 0000000000006000 008:00002 libpam.so.0.77
b7f4f000      64 r-x-- 0000000000000000 008:00002 libselinux.so.1
b7f5f000       4 rwx-- 0000000000010000 008:00002 libselinux.so.1
b7f60000       4 rwx-- 00000000b7f60000 000:00000   [ anon ]
b7f70000       4 rwx-- 00000000b7f70000 000:00000   [ anon ]
b7f71000      84 r-x-- 0000000000000000 008:00002 ld-2.3.6.so
b7f86000       4 r-x-- 00000000b7f86000 000:00000   [ anon ]
b7f87000       4 r-x-- 0000000000015000 008:00002 ld-2.3.6.so
b7f88000       4 rwx-- 0000000000016000 008:00002 ld-2.3.6.so
b7f89000      20 r-x-- 0000000000000000 008:00002 atd
b7f8e000       4 rwx-- 0000000000004000 008:00002 atd
b8402000     132 rwx-- 00000000b8402000 000:00000   [ anon ]
bfb79000      84 rw--- 00000000bfb79000 000:00000   [ stack ]
mapped: 1716K    writeable/private: 272K    shared: 0K
[root@mandolin server]#


Thanks again.
 
Old 02-20-2007, 06:00 AM   #10
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,446

Rep: Reputation: 1068Reputation: 1068Reputation: 1068Reputation: 1068Reputation: 1068Reputation: 1068Reputation: 1068Reputation: 1068
Add it all in - that big [anon] will be the heap.
Personally I would just run pmap -d and parse out the total line at the bottom, as the referenced article suggested. The writable/private probably is what you are looking for. I maybe read too much into your requirements - I thought you were looking for (true) RSS rather than (true) private (mapped was a poor choice of terminology - think virtual). If your app has malloc'd storage but not referenced it I guess you still want to count it - use the total, and save yourself some grief.
The maths will still be out, but only marginally - for the data segments for shared libs that should probably be booked to you.
 
Old 02-21-2007, 11:00 PM   #11
rhimbo
LQ Newbie
 
Registered: Dec 2004
Posts: 29

Original Poster
Rep: Reputation: 15
Thanks again for your reply syg00. Actually, I think the writable/private totals are wrong, are they not? I understood the article to say that the totals double count the shared libs.

I wrote a shell script to take output from "ps -ef" and then, for each PID, run "pmap -d" on it. I then run it through sed and sort. I noticed the addresses are all unique. Outputting unique lines from sort, I count the shared lib segments only once, and each process's private code and data space once. All the "anon"s are counted once also. I think this is what I need.

I do have a few questions however. :-)

I noticed that for a lot of the processes owned by root, I get output like the following when I run pmap on its PID:

Code:
# ps -ef | more
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Jan08 ?        00:00:01 init [5]
root         2     1  0 Jan08 ?        00:00:02 [migration/0]
root         3     1  0 Jan08 ?        00:00:00 [ksoftirqd/0]
root         4     1  0 Jan08 ?        00:00:04 [migration/1]
root         5     1  0 Jan08 ?        00:00:00 [ksoftirqd/1]
root         6     1  0 Jan08 ?        00:00:01 [events/0]
...
root       107     9  0 Jan08 ?        00:00:00 [kblockd/0]
root       108     9  0 Jan08 ?        00:00:00 [kblockd/1]
root       111     1  0 Jan08 ?        00:00:00 [khubd]
root       165     9  0 Jan08 ?        00:00:24 [pdflush]
root       166     9  0 Jan08 ?        00:00:19 [pdflush]

# pmap -d 5
5:   [ksoftirqd/1]
Address   Kbytes Mode  Offset           Device    Mapping
mapped: 0K    writeable/private: 0K    shared: 0K
# 
# pmap -d 165
165:   [pdflush]
Address   Kbytes Mode  Offset           Device    Mapping
mapped: 0K    writeable/private: 0K    shared: 0K
#
...
#
Does this mean that pmap can't "see" the main memory used by these processes? Is this kernel memory? I'm guessing here.

How can I tell if memory allocated/used by the kernel is accounted for?

How can I tell how much memory the kernel is using?

So now I need to validate my results. I wrote a simple C program to continually malloc() memory (e.g. 1 Mbyte at a time) until failure. I then total the amount of memory successfully malloc()'ed. If I add this number to my shell script output (that summed the figures reported by pmap for all processes), I should get the total memory "seen" by the OS (kernel). Correct or not?

Many thanks again for your replies and patience!
 
  


Reply

Tags
measurement, memory, usage


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to measure bandwidth usage on an IP-VPN setup using MRTG? depam Linux - Hardware 2 02-20-2007 10:38 AM
How do I measure stack usage of a C/C++ program? exit3219 Programming 4 02-02-2007 12:25 PM
need top clone- to measure CPU usage pao Linux - Software 4 10-07-2004 10:46 AM
how to determine cpu usage, memory usage, I/O usage by a particular user logged on li rags2k Programming 4 08-21-2004 05:45 AM
measure memory usage unosoft Linux - Software 1 10-07-2003 07:25 PM


All times are GMT -5. The time now is 11:17 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration