LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 01-29-2009, 06:22 AM   #1
vikas027
Senior Member
 
Registered: May 2007
Location: Sydney
Distribution: RHEL, CentOS, Debian
Posts: 1,266

Rep: Reputation: 99
Question Server consuming cache memory


Hi All,

I am facing a strange problem in few of my Linux boxes.

The cache is consuming a large amount of memory, while swap is unused. Also, not even a single memory consuming process is running.


These servers are Red Hat Enterprise Linux AS release 4
OR
Red Hat Enterprise Linux AS release 3


Code:
sh-2.05b# free -m
             total       used       free     shared    buffers     cached
Mem:          1001        973         28          0        230        440
-/+ buffers/cache:        302        698
Swap:         1999          0       1999

TOP output
Code:
sh-2.05b# top
 06:21:47  up 12 days, 16:48,  2 users,  load average: 0.00, 0.00, 0.00
135 processes: 134 sleeping, 1 running, 0 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    0.4%    0.0%    5.3%   0.0%     0.0%    0.0%   94.1%
           cpu00    0.0%    0.0%    0.0%   0.0%     0.0%    0.0%  100.0%
           cpu01    0.9%    0.0%   10.6%   0.0%     0.0%    0.0%   88.3%
Mem:  1025376k av,  996684k used,   28692k free,       0k shrd,  235672k buff
                    605932k actv,  113872k in_d,   15640k in_c
Swap: 2047992k av,       0k used, 2047992k free                  450880k cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
 1386 root      15   0   384  384   304 S     2.9  0.0  26:36   1 vmware-guestd
   10 root      15   0     0    0     0 SW    0.4  0.0   0:57   1 kupdated
 1351 root      15   0     0    0     0 SW    0.4  0.0   1:11   1 vmmemctl
21656 root      15   0 10240  10M  2764 S     0.4  0.9   0:33   1 ovcd
21706 root      15   0  8388 8388  4436 S     0.4  0.8   0:51   1 coda
 5883 root      20   0  1148 1148   880 R     0.4  0.1   0:00   1 top
    1 root      15   0   228  228   172 S     0.0  0.0   0:30   0 init
    2 root      RT   0     0    0     0 SW    0.0  0.0   0:00   0 migration/0
    3 root      RT   0     0    0     0 SW    0.0  0.0   0:00   1 migration/1
    4 root      15   0     0    0     0 SW    0.0  0.0   0:00   0 keventd
    5 root      34  19     0    0     0 SWN   0.0  0.0   0:00   0 ksoftirqd/0
    6 root      34  19     0    0     0 SWN   0.0  0.0   0:00   1 ksoftirqd/1
    9 root      25   0     0    0     0 SW    0.0  0.0   0:00   0 bdflush
    7 root      15   0     0    0     0 SW    0.0  0.0   0:07   0 kswapd
    8 root      15   0     0    0     0 SW    0.0  0.0   4:32   1 kscand
   11 root      25   0     0    0     0 SW    0.0  0.0   0:00   0 mdrecoveryd
   22 root      15   0     0    0     0 SW    0.0  0.0   0:40   0 kjournald
  413 root      15   0     0    0     0 SW    0.0  0.0   0:00   0 kjournald
  414 root      15   0     0    0     0 SW    0.0  0.0   0:00   1 kjournald
  415 root      15   0     0    0     0 SW    0.0  0.0   0:10   0 kjournald
  416 root      15   0     0    0     0 SW    0.0  0.0   0:00   0 kjournald
  417 root      15   0     0    0     0 SW    0.0  0.0   0:00   0 kjournald
  418 root      15   0     0    0     0 SW    0.0  0.0   0:32   0 kjournald
  419 root      15   0     0    0     0 SW    0.0  0.0   2:07   1 kjournald
  885 root      15   0   384  384   276 S     0.0  0.0   0:06   0 syslogd
  889 root      18   0   208  208   148 S     0.0  0.0   0:00   0 klogd
  900 root      15   0   260  260   200 S     0.0  0.0   2:28   1 irqbalance
  918 rpc       17   0   196  196   120 S     0.0  0.0   0:00   0 portmap
  938 rpcuser   25   0   240  240   164 S     0.0  0.0   0:00   0 rpc.statd
  951 root      15   0   196  196   140 S     0.0  0.0   0:09   0 mdadm
I did some googling work, but it did not helped. I found that to free system cache we have run these commands
Code:
sync; echo 3 > /proc/sys/vm/drop_caches
But this file is NOT present in any of the servers.

These are the contents of other files in /proc/sys/vm/ folder

Code:
sh-3.00# ls -l /proc/sys/vm/
total 0
-rw-r--r--  1 root root 0 Jan 29 13:17 block_dump
-rw-r--r--  1 root root 0 Jan 29 13:17 dirty_background_ratio
-rw-r--r--  1 root root 0 Jan 29 13:17 dirty_expire_centisecs
-rw-r--r--  1 root root 0 Jan 29 13:17 dirty_ratio
-rw-r--r--  1 root root 0 Jan 29 13:17 dirty_writeback_centisecs
-rw-r--r--  1 root root 0 Jan 29 13:17 hugetlb_shm_group
-rw-r--r--  1 root root 0 Jan 29 13:17 laptop_mode
-rw-r--r--  1 root root 0 Jan 29 13:17 legacy_va_layout
-rw-r--r--  1 root root 0 Jan 29 13:17 lower_zone_protection
-rw-r--r--  1 root root 0 Jan 29 13:17 max_map_count
-rw-r--r--  1 root root 0 Jan 29 13:17 max_queue_depth
-rw-r--r--  1 root root 0 Jan 29 13:17 min_free_kbytes
-rw-r--r--  1 root root 0 Jan 29 13:17 nr_hugepages
-r--r--r--  1 root root 0 Jan 29 13:17 nr_pdflush_threads
-rw-r--r--  1 root root 0 Jan 29 13:17 oom-kill
-rw-r--r--  1 root root 0 Jan 29 13:17 overcommit_memory
-rw-r--r--  1 root root 0 Jan 29 13:17 overcommit_ratio
-rw-r--r--  1 root root 0 Jan 29 13:17 page-cluster
-rw-r--r--  1 root root 0 Jan 29 13:17 percpu_pagelist_fraction
-rw-r--r--  1 root root 0 Jan 29 13:17 swappiness
-rw-r--r--  1 root root 0 Jan 29 13:17 vfs_cache_pressure
Could someone please help me out ?? I once rebooted these servers, the issue solved but after some time, again I found cache memory full.

Pls help !!

Thanks in advance.

Regards,
VIKAS
 
Old 01-29-2009, 07:51 AM   #2
ilikejam
Senior Member
 
Registered: Aug 2003
Location: Glasgow
Distribution: Fedora / Solaris
Posts: 3,109

Rep: Reputation: 96
The cache _should_ be full - it's a RAM cache of recently read filesystem data. If that memory is needed by an application, part of the cache will be freed and handed over.

Take a look at the 'free' output - the '-/+ buffers/cache' line is what you should be looking at.

All is well.

Dave
 
Old 01-29-2009, 06:30 PM   #3
vikas027
Senior Member
 
Registered: May 2007
Location: Sydney
Distribution: RHEL, CentOS, Debian
Posts: 1,266

Original Poster
Rep: Reputation: 99
Unhappy

Quote:
Originally Posted by ilikejam View Post
The cache _should_ be full - it's a RAM cache of recently read filesystem data. If that memory is needed by an application, part of the cache will be freed and handed over.

Take a look at the 'free' output - the '-/+ buffers/cache' line is what you should be looking at.

All is well.

Dave
I agree ilikejam, but still customer wants to know the reason for only the cache memory being utilized. There is NO such application using the cache memory.
What I need is the way to flush cache memory (without rebooting the server)

Regards,
VIKAS
 
Old 01-30-2009, 01:52 AM   #4
robertjinx
Member
 
Registered: Oct 2007
Location: Prague, CZ
Distribution: RedHat / CentOS / Ubuntu / SUSE / Debian
Posts: 561

Rep: Reputation: 58
No application is using the cache. The kernel will free the cache when there is need of memory or request for memory.
Thats how Linux kernel works. You could do stuff like memstress or something like that and you will see that the cache is being released. It doesn't matter anyways, because there is no danger and you have memory to use, just the kernel doesn't show you that
 
Old 01-30-2009, 04:01 AM   #5
ilikejam
Senior Member
 
Registered: Aug 2003
Location: Glasgow
Distribution: Fedora / Solaris
Posts: 3,109

Rep: Reputation: 96
Tell the customer that dropping the caches will cause a pretty considerable degradation in performance (which it will), and that they don't want to do this. Keep telling them this until they understand.

Dave
 
Old 01-30-2009, 05:56 AM   #6
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,232

Rep: Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024Reputation: 2024
Like ilikejam & robertjinx are telling you, Linux automatically caches as much as it can. There's no point in having empty memory. If it needs that mem, it'll adjust the caches as needed. That's what the vm part of the kernel does for you. Tell the customer to ignore it and keep telling him...

Quote:
To make the most efficient use of real memory, Linux automatically uses all free RAM for buffer cache, but also automatically makes the cache smaller when programs need more memory.
from http://www.faqs.org/docs/linux_admin/buffer-cache.html
 
Old 03-05-2009, 02:52 AM   #7
vikas027
Senior Member
 
Registered: May 2007
Location: Sydney
Distribution: RHEL, CentOS, Debian
Posts: 1,266

Original Poster
Rep: Reputation: 99
Question

Quote:
Originally Posted by robertjinx View Post
No application is using the cache. The kernel will free the cache when there is need of memory or request for memory.
Thats how Linux kernel works. You could do stuff like memstress or something like that and you will see that the cache is being released. It doesn't matter anyways, because there is no danger and you have memory to use, just the kernel doesn't show you that
Hi Robert,

Could you please explain what can be done in memstress ?
I have communicated to the customer, just for sake of my knowledge.
 
Old 03-05-2009, 03:44 AM   #8
robertjinx
Member
 
Registered: Oct 2007
Location: Prague, CZ
Distribution: RedHat / CentOS / Ubuntu / SUSE / Debian
Posts: 561

Rep: Reputation: 58
The memstress will eat up the memory, more as a bench mark to see if the kernel will handle it, but in the same time... as it eats all the memory, after stoping the script it will free up everything, as its not in use anymore... I dont know much about the logic inside it, meaning how the kernel does this, but just test it on the devel system and you will see the results.

Sorry, I didnt added the memstress script here:

[PHP]
#!/bin/bash

uprog ()
{
sleep 60

}

count=0

while [ "$count" -lt 10000000 ] ; do

uprog &
echo $count
clear
count=`expr $count + 1`

done
/[PHP]

Last edited by robertjinx; 03-05-2009 at 03:45 AM.
 
Old 03-05-2009, 03:46 AM   #9
robertjinx
Member
 
Registered: Oct 2007
Location: Prague, CZ
Distribution: RedHat / CentOS / Ubuntu / SUSE / Debian
Posts: 561

Rep: Reputation: 58
[QUOTE=robertjinx;3465535]The memstress will eat up the memory, more as a bench mark to see if the kernel will handle it, but in the same time... as it eats all the memory, after stoping the script it will free up everything, as its not in use anymore... I dont know much about the logic inside it, meaning how the kernel does this, but just test it on the devel system and you will see the results.

Sorry, I didnt added the memstress script here:

PHP Code:
#!/bin/bash

uprog ()
{
sleep 60

}

count=0

while [ "$count" -lt 10000000 ] ; do

uprog &
echo 
$count
clear
count
=`expr $count + 1`

done 
 
Old 03-05-2009, 07:43 PM   #10
vikas027
Senior Member
 
Registered: May 2007
Location: Sydney
Distribution: RHEL, CentOS, Debian
Posts: 1,266

Original Poster
Rep: Reputation: 99
Smile

[QUOTE=robertjinx;3465537]
Quote:
Originally Posted by robertjinx View Post
The memstress will eat up the memory, more as a bench mark to see if the kernel will handle it, but in the same time... as it eats all the memory, after stoping the script it will free up everything, as its not in use anymore... I dont know much about the logic inside it, meaning how the kernel does this, but just test it on the devel system and you will see the results.

Sorry, I didnt added the memstress script here:

PHP Code:
#!/bin/bash

uprog ()
{
sleep 60

}

count=0

while [ "$count" -lt 10000000 ] ; do

uprog &
echo 
$count
clear
count
=`expr $count + 1`

done 
Thanks Robert. This script, reduces that cache BUT it eats up the swap memory, more over when I tried rebooting the server it was throwing some fatal errors.

Anyways, now I can explain them a little more about the issue.

Thanks a lot.
 
Old 03-06-2009, 01:28 AM   #11
robertjinx
Member
 
Registered: Oct 2007
Location: Prague, CZ
Distribution: RedHat / CentOS / Ubuntu / SUSE / Debian
Posts: 561

Rep: Reputation: 58
It suppose to eat all the memory and when there is no memory left start with the swap... but it shouldn't kill the system, if everything is ok. I tested my own system with this and never happened a crash... also try to keep the script until it reaches like 3000 or so and Ctrl+C and you will start to see a decrees of memory usage and cache is freeing up, but its temporary until the kernel will start again caching the memory. Like all the guys said before, the Linux kernel works this way, if you client or whatever needs to know "Why?" take a tour of the Linux kernel, how it does the memory management and I think it will be explained there.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
how to set memory cache limit for nfs server libin88 Linux - Server 1 10-24-2008 08:23 AM
Consuming Memory problem aloneattack Linux - Server 2 04-08-2008 11:07 PM
Help Configuring the Memory Used by a Process in RedHat? (Cache Memory on CPU) geogecko Linux - General 3 02-23-2005 03:32 PM
How do I clear the memory cache on my linux server? robgo777 Linux - Software 2 01-09-2005 01:30 AM
Silent and low consuming server snop Slackware 4 11-10-2004 05:41 AM


All times are GMT -5. The time now is 12:14 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration