LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices



Reply
 
Search this Thread
Old 02-02-2013, 02:05 PM   #1
mrfenwayfool
LQ Newbie
 
Registered: Feb 2013
Posts: 3

Rep: Reputation: Disabled
process memory usage


Hello,

I am working with an embedded system running Linux. I have a single process that has numerous threads doing the work... so there is really only a single process to worry about for all practical purposes. What I am trying to do is have this process offer IP based services as long as it has enough memory.

So for example, if my system has 8G, I want to keep allowing new connections to the services as long as there is at least 1G of free memory... but if there is less than that I want to reject new services... but continue allowing the existing connections.

Here is my problem... my process currently figures out what the current memory utilization is by looking as RSS. This works... the calculation is close enough for my purposes (yes, I know, shared memory can mess with this)... still, the calculation is fine. The BIG issue is... the RSS value seems to be a "high water mark". That is, it never shrinks since there are no other processes requesting memory... so my system stops accepting new connections even as old ones quit.

What are my options here? Is my only option to mess with malloc hooks or is there a better way to deal with this? Can I somehow "flush" my process so it gives back all unused RAM?

MrFenwayFool

Last edited by mrfenwayfool; 02-02-2013 at 02:06 PM.
 
Old 02-02-2013, 04:55 PM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,491

Rep: Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077
Couple of simple solutions:
- do the sums yourself from /proc/<pid>/smaps
- use drop_caches

Normally I rail against people recommending the latter because it messes with disk caching. Probably not an issue with you.
Code:
cat 3 > /proc/sys/vm/drop_caches
 
Old 02-03-2013, 05:33 PM   #3
mrfenwayfool
LQ Newbie
 
Registered: Feb 2013
Posts: 3

Original Poster
Rep: Reputation: Disabled
Thanks for the advice.

I tried using drop_caches but I don't think that does anything to help me. The issue is that the process heap keeps expanding... sbrk() is never called to shrink the heap under normal circumstances. The result of this is that RSS shows as a "high water mark". For example, even though my process heap grows to 8G in size... that does NOT mean that my process is using ALL of that 8G at any given instant.

What I have found is that calling malloc_trim(0) every now and then from the application does indeed shrink the heap. I am playing with this but it seems calling that frequently really hits performance (there's a good reason the heap normally doesn't shrink!). Instead, I'm looking at mallinfo() which reports not only how large the heap is but also how much of the heap is actively allocated... which is really what I am really interested in.

Again, thanks for the advice!

MrFenwayFool
 
Old 02-05-2013, 10:14 AM   #4
sundialsvcs
Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 5,455

Rep: Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172
The approach that I would take is simply to determine how many connections you think the system should accept, and build a simple but adjustable throttle to that effect.

You can sometimes do simple things like "wrapping" the memory allocation/release calls that your application uses with code that increments or decrements a global running total ... often faster than asking the memory-allocation subsystem for that figure.

You could design a simple servo-mechanism: a thread that wakes up every (configurable... n) seconds and decides whether the throttle should be raised or lowered, within (configurable) boundaries and by some (configurable) increment. This approach would make the system become responsive to conditions as they actually are observed to take place, and reduces the need to "guess correctly."

Yes, there's a reason why the segment doesn't shrink: it forces everything to grind to a stop while a heck of a lot of page-faults take place in quick succession. Instead, the system as-designed justs the virtual memory manager do its thing. Pages that are "part of the heap," but not actively being used, get paged out. Some memory-allocators use a special "zero-fill this block of memory" system call which gives the OS the opportunity to zero the page by removing it from the VM page-tables, but I don't recall if Linux does this or not.

Last edited by sundialsvcs; 02-05-2013 at 10:16 AM.
 
Old 02-09-2013, 03:05 PM   #5
mrfenwayfool
LQ Newbie
 
Registered: Feb 2013
Posts: 3

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by sundialsvcs View Post
The approach that I would take is simply to determine how many connections you think the system should accept, and build a simple but adjustable throttle to that effect.

You can sometimes do simple things like "wrapping" the memory allocation/release calls that your application uses with code that increments or decrements a global running total ... often faster than asking the memory-allocation subsystem for that figure.

You could design a simple servo-mechanism: a thread that wakes up every (configurable... n) seconds and decides whether the throttle should be raised or lowered, within (configurable) boundaries and by some (configurable) increment. This approach would make the system become responsive to conditions as they actually are observed to take place, and reduces the need to "guess correctly."

Yes, there's a reason why the segment doesn't shrink: it forces everything to grind to a stop while a heck of a lot of page-faults take place in quick succession. Instead, the system as-designed justs the virtual memory manager do its thing. Pages that are "part of the heap," but not actively being used, get paged out. Some memory-allocators use a special "zero-fill this block of memory" system call which gives the OS the opportunity to zero the page by removing it from the VM page-tables, but I don't recall if Linux does this or not.
Hmmm... this has turned into a long and winding road.

Looking at malloc_stats output has enlightened me a bit on the memory usage. That is, there are multiple arenas of memory since libc is trying to prevent blocking due to multiple threads running (my particular application has many, many threads due to its long history). Anyway, I thought perhaps I could limit the number of arenas using MALLOC_ARENA_MAX... and while that did limit the arenas... I'm still seeing "in use" memory for each arena far below the "system space"... meaning the arenas are growing even though there *should* be plenty of memory available. I'm wondering if this is fragmentation or fastbins using up memory... then being freed... and never used again. My application is a massive C++ application. I'm going to look into this more closely. Seems like my application is not using as much memory as I thought if you can believe the statistics from malloc_stats. I'm also wondering if I should give tcmalloc a try and see if it has similar issues. Regardless... something is not right here.
 
Old 02-10-2013, 03:18 PM   #6
sundialsvcs
Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 5,455

Rep: Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172Reputation: 1172
Be very alert as to exactly which metric you are observing. When space is released and available for recycling, many metrics do not go down. Also, processes in a virtual memory environment can't see what the operating system's VM manager is doing (although the software is designed to be friendly to it).

Easily the best way to get meaningful statistics is to arrange for the process, itself, to create log datasets (or externally-visible in-memory "trace tables") which describe the workflow characteristics in its own, application specific, therefore this-business-specific terms. Get out of the abstract and into the concrete. If the processing system tracks units-of-work, track those units. Track easily-gathered memory manager statistics and include them in the same or in a separate stream (taking care that none of the stats are expensive to obtain, so as to skew the results by virtue of gathering them). Now, pull these data out to a statistics system: look for statistically significant correlations.

Build "tweakable" controls and knobs, not so much in operating-system terms but rather in "what this software system does for a living" terms. Gather data about the system with various settings and, once again, use a stats package.

I've done this sort of thing many times, and one of the most interesting things that comes from it is: "real surprises."
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Process for investigating memory usage listerthrawn Linux - Server 7 04-23-2012 09:28 AM
Get max memory usage by process ? Alexiia Linux - Newbie 3 09-08-2009 09:06 AM
getrusage and process memory usage gearoid_murphy Programming 2 11-09-2006 05:33 AM
about displaying process memory usage maginotjr Slackware 2 09-13-2005 12:43 PM
Process memory usage wombat53 Linux - Newbie 5 07-21-2005 08:42 PM


All times are GMT -5. The time now is 06:50 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration