LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 02-04-2003, 12:36 PM   #1
drthornt
Member
 
Registered: May 2001
Location: Toronto
Distribution: RH 7.2
Posts: 33

Rep: Reputation: 15
What's normal ? Freemem


Hey guys,

When should I get worried that my linux system is running out of memory?

I have a couple of Redhat 6.2 ( linux 2.2) machines and a couple of Redhat 7.3 ( linux 2.4 ) machines and I noticed that memory runs consistently at >90%.

In the past I have seen a Rehdat 6.2 machine grind to halt when freemem drops to around 4 megs, regardless of how much memory is installed.

I have a 7.3 machine that has been running totally fine at 2.9 megs free.

This seems quite small.

I am specifically worried about a rather large client who has 2GB installed on a 7.3 box.
These machines, in some respects, are quire loaded , with beteeen 300 and 600 apache and beteeen 300 and 600 tomcat process ( IBM JVM). With all the things that can go wrong with tomcat I naturally zoomed in on this aspact and tried to figure out why I was low on freemem.

Upon closer inspection I found that it was not tomcat (nor apache) at all.

With both apache and tomcat "down" my freemem was still at a chunky 85%.

Free showed 1.5 GB in "cached"

I have examined how memory is [de]allocated during the day and found that a good Gig of memory is freed up when the logs are rotated. I also noticed that relative large chunks ( hundreds of megs) are freed up at relatively few number of times in the day. This observation is based on 10 minute sar data so the granularity is not great. I mention this because I noticed that during the day if I watch a "vmstat 3" I can see "little" increaes in free memory ( 1k ). I have a basic understanding of io buffering and caching. I am busy working through the kernel docs on the vm, So I am doing what I can to understand the system via existing documentation. My concern is that I will reach an abrupt brick wall. My experience under 6.2 has been that at 4 megs free the system starts swaping, interactivity goes way down and web apps grind to a halt. This disturbs me. There doesn't seem to be any sort of smooth degredation of performance as memory usage peaks. Is this just the nature of the beast?

Any insight would be greatly apprecaitated

david
 
Old 02-04-2003, 12:56 PM   #2
Tinkster
Moderator
 
Registered: Apr 2002
Location: earth
Distribution: slackware by choice, others too :} ... android.
Posts: 23,067
Blog Entries: 11

Rep: Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928Reputation: 928
Just out of curiosity (and to hopefully understand
the problem) ... what hardware are we looking at here?

CPU/RAM/HD-IO-system?

As a rule of thumb "free mem" should mean diddly
squat on a Linux-box since it will adjust buffers/cache
on the fly as required ... if it really has to start swapping
out the core processes and revive them soon this would
explain the "getting chunky" part ... the two things to look
at in this situation would be adding more RAM :} or a faster,
more intelligent HD subsystem (a SCSI-RAID 5)? ...

Cheers,
Tink
 
Old 02-04-2003, 04:24 PM   #3
drthornt
Member
 
Registered: May 2001
Location: Toronto
Distribution: RH 7.2
Posts: 33

Original Poster
Rep: Reputation: 15
Yeah I am gonna hafta say that I don't htink has anyting to do with the hardware I'm on. we have 2 GB on the "problem system" and mirrored SCSI 160 disks ( is a proliant ). More to the point I am seeing this bavior on a myriad of hardware platforms.

I don't care if the kernel uses all the memory, as long as that's normal. When does the kernel stop eating up Physical ram for disk cache?

david
 
Old 02-04-2003, 04:31 PM   #4
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,417

Rep: Reputation: 1985Reputation: 1985Reputation: 1985Reputation: 1985Reputation: 1985Reputation: 1985Reputation: 1985Reputation: 1985Reputation: 1985Reputation: 1985Reputation: 1985
When? it stops when there's either nothing left to cache or there's no more space...
 
Old 02-04-2003, 10:28 PM   #5
drthornt
Member
 
Registered: May 2001
Location: Toronto
Distribution: RH 7.2
Posts: 33

Original Poster
Rep: Reputation: 15
Thought I would post some of this thread on other mediums ( CALUG mailing list ):

----- Original Message -----
From: <ecn@metaslash.com>
To: <david@quadratic.net>
Sent: Tuesday, February 04, 2003 6:28 PM
Subject: [CALUG List] Re: What is normal? freemem?


> Call me crazy, but isn't the kernel supposed to cache as much of the
> file system as it can? And shouldn't "free" memory be a short-term
> situation? After all, free memory is a file cache just waiting to
> happen.
>
> You didn't mention any numbers associated with swap. Are you normally
> not using any swap? A big-o-honkin' JVM can typically swap out chunks
> of symbol data without affecting performance.
>
> -Eric


Yeah swap is not being used much at all.

What I want to know is , how does interactivity and performance react when I finally "run out" of free mem.
I understand the idea that you want all resources to be used to do the job better, which may translate into the kernel using a pile of physical ram for disk cache. However I am concern that I will see an immediate and effectivly lethal effect when I run out of ram.

This was true on redhat 6.2 ( linux 2.2) systems.

At the top end of the memory usage scale you would get ( from a web users point of view ) :

the web site is up ( freemem = X +3)
the web site is up ( freemem = X +2)
the web site is up ( freemem = X +1)
the web site is up ( freemem = X )
the web site is down ( freemem = X -1 )
the web site is down ( freemem = X -2 )
the web site is down ( freemem = X -3 )
the web site is down ( freemem = 0

Notice there is no:
the web site is slow

when it comes to free memory: at what point do I panic? should I panic when Freemem = X ? Should I panic when I realize that X exists? and what can I do?

My most common answer is : Install more memory.
I believe that this is like puming more blood into a patient with a sucking chest wound.
It's not going to fix the problem.

The machine is at 2GB of ram at this time.
As an experiemnt with this specific setup we have upgraded a single machine in the cluster ( I have 4 machine behaving this way) to 3 GB. If this "beefed up system" drops to 70 megs over this busy period I can conlued (with some uncertainty) that adding more ram is only going to push this problem off. Now I have to admit that my free mem sky rockets right after the log rotate.
I considered today how the kernel decides to cache any given blocks.
Needless to say I need to read more.

david
 
Old 02-09-2003, 11:46 AM   #6
drthornt
Member
 
Registered: May 2001
Location: Toronto
Distribution: RH 7.2
Posts: 33

Original Poster
Rep: Reputation: 15
Ok well testing and example are the best way to demonstrate the behavor of the kernel.

check out
http://www.quadratic.net/~david/gnuplot/#Sar_Analysis

Based on this I am much more confident that I don't have to worry about reaching "low free memory" states.

Note the "saw tooth". The "restoration" of free memory happened at 4am , when cron.daily executes.
I suspect this is as a result of logrotate doing it's thing.
Still don't understand 100%.
But based on the threads so far I am getting there.

Thanks again guys for ptiching you idea and notes.

david
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Is this normal? NomadABC Linux - Software 15 02-02-2005 12:13 PM
is that normal ? winxshadi76 Linux - General 3 11-29-2004 08:07 AM
Is this normal? Y3k Linux - Newbie 5 06-04-2004 10:14 AM
is it normal uncle_fungus1 Linux - Software 3 11-22-2003 04:38 PM
is it normal for.. ph33r Linux - Networking 3 05-29-2002 09:12 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 01:46 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration