What's normal ? Freemem
 

Hey guys,

When should I get worried that my linux system is running out of memory?

I have a couple of Redhat 6.2 ( linux 2.2) machines and a couple of Redhat 7.3 ( linux 2.4 ) machines and I noticed that memory runs consistently at >90%.

In the past I have seen a Rehdat 6.2 machine grind to halt when freemem drops to around 4 megs, regardless of how much memory is installed.

I have a 7.3 machine that has been running totally fine at 2.9 megs free.

This seems quite small.

I am specifically worried about a rather large client who has 2GB installed on a 7.3 box.
These machines, in some respects, are quire loaded , with beteeen 300 and 600 apache and beteeen 300 and 600 tomcat process ( IBM JVM). With all the things that can go wrong with tomcat I naturally zoomed in on this aspact and tried to figure out why I was low on freemem.

Upon closer inspection I found that it was not tomcat (nor apache) at all.

With both apache and tomcat "down" my freemem was still at a chunky 85%.

Free showed 1.5 GB in "cached"

I have examined how memory is [de]allocated during the day and found that a good Gig of memory is freed up when the logs are rotated. I also noticed that relative large chunks ( hundreds of megs) are freed up at relatively few number of times in the day. This observation is based on 10 minute sar data so the granularity is not great. I mention this because I noticed that during the day if I watch a "vmstat 3" I can see "little" increaes in free memory ( 1k ). I have a basic understanding of io buffering and caching. I am busy working through the kernel docs on the vm, So I am doing what I can to understand the system via existing documentation. My concern is that I will reach an abrupt brick wall. My experience under 6.2 has been that at 4 megs free the system starts swaping, interactivity goes way down and web apps grind to a halt. This disturbs me. There doesn't seem to be any sort of smooth degredation of performance as memory usage peaks. Is this just the nature of the beast?

Any insight would be greatly apprecaitated

david

Just out of curiosity (and to hopefully understand
the problem) ... what hardware are we looking at here?

CPU/RAM/HD-IO-system?

As a rule of thumb "free mem" should mean diddly
squat on a Linux-box since it will adjust buffers/cache
on the fly as required ... if it really has to start swapping
out the core processes and revive them soon this would
explain the "getting chunky" part ... the two things to look
at in this situation would be adding more RAM :} or a faster,
more intelligent HD subsystem (a SCSI-RAID 5)? ...

Cheers,
Tink

Yeah I am gonna hafta say that I don't htink has anyting to do with the hardware I'm on. we have 2 GB on the "problem system" and mirrored SCSI 160 disks ( is a proliant ). More to the point I am seeing this bavior on a myriad of hardware platforms.

I don't care if the kernel uses all the memory, as long as that's normal. When does the kernel stop eating up Physical ram for disk cache?

david

When? it stops when there's either nothing left to cache or there's no more space...

Thought I would post some of this thread on other mediums ( CALUG mailing list ):

----- Original Message -----
From: <ecn@metaslash.com>
To: <david@quadratic.net>
Sent: Tuesday, February 04, 2003 6:28 PM
Subject: [CALUG List] Re: What is normal? freemem?


> Call me crazy, but isn't the kernel supposed to cache as much of the
> file system as it can? And shouldn't "free" memory be a short-term
> situation? After all, free memory is a file cache just waiting to
> happen.
>
> You didn't mention any numbers associated with swap. Are you normally
> not using any swap? A big-o-honkin' JVM can typically swap out chunks
> of symbol data without affecting performance.
>
> -Eric


Yeah swap is not being used much at all.

What I want to know is , how does interactivity and performance react when I finally "run out" of free mem.
I understand the idea that you want all resources to be used to do the job better, which may translate into the kernel using a pile of physical ram for disk cache. However I am concern that I will see an immediate and effectivly lethal effect when I run out of ram.

This was true on redhat 6.2 ( linux 2.2) systems.

At the top end of the memory usage scale you would get ( from a web users point of view ) :

the web site is up ( freemem = X +3)
the web site is up ( freemem = X +2)
the web site is up ( freemem = X +1)
the web site is up ( freemem = X )
the web site is down ( freemem = X -1 )
the web site is down ( freemem = X -2 )
the web site is down ( freemem = X -3 )
the web site is down ( freemem = 0

Notice there is no:
the web site is slow

when it comes to free memory: at what point do I panic? should I panic when Freemem = X ? Should I panic when I realize that X exists? and what can I do?

My most common answer is : Install more memory.
I believe that this is like puming more blood into a patient with a sucking chest wound.
It's not going to fix the problem.

The machine is at 2GB of ram at this time.
As an experiemnt with this specific setup we have upgraded a single machine in the cluster ( I have 4 machine behaving this way) to 3 GB. If this "beefed up system" drops to 70 megs over this busy period I can conlued (with some uncertainty) that adding more ram is only going to push this problem off. Now I have to admit that my free mem sky rockets right after the log rotate.
I considered today how the kernel decides to cache any given blocks.
Needless to say I need to read more.

david

Ok well testing and example are the best way to demonstrate the behavor of the kernel.

check out
http://www.quadratic.net/~david/gnuplot/#Sar_Analysis

Based on this I am much more confident that I don't have to worry about reaching "low free memory" states.

Note the "saw tooth". The "restoration" of free memory happened at 4am , when cron.daily executes.
I suspect this is as a result of logrotate doing it's thing.
Still don't understand 100%.
But based on the threads so far I am getting there.

Thanks again guys for ptiching you idea and notes.

david