LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 05-26-2012, 06:21 PM   #1
hexalot
LQ Newbie
 
Registered: May 2012
Distribution: CentOS 6.2, BackTrack R5
Posts: 4

Rep: Reputation: Disabled
Memory Leaks in CentOS 6.2


Hello all! I was hoping to get some assistance with a memory issue i've begun to notice on a new CentOS6.2 install.

The quickie - At a fresh idle shortly after booting, my memory usage is around 400-500mb including Apache, MySQL, etc. After a few hours or so, 2.5Gbs of ram are being used, and I cannot tell by what. And after enough time, it eats up all 8Gbs of ram.


The slow -

System specs - 2 x 160gb Sata2, Athlon X2, 8Gbs of ram. CentOS 6.2 - php 5.3.10 - MySQL 5.5

Linux hosting.#######.com 2.6.32-220.17.1.el6.x86_64 #1 SMP Wed May 16 00:01:37 BST 2012 x86_64 x86_64 x86_64 GNU/Linux

Upon a clean reboot, the memory load looks good.

top - 17:14:21 up 18 min, 1 user, load average: 0.01, 0.00, 0.00
Tasks: 117 total, 1 running, 116 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.7%us, 0.6%sy, 0.0%ni, 97.6%id, 1.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 7801724k total, 328804k used, 7472920k free, 20324k buffers
Swap: 10043384k total, 0k used, 10043384k free, 115112k cached

After server has been running, the memory usage is much higher than it should be.

top - 16:55:19 up 7:20, 1 user, load average: 0.00, 0.00, 0.00
Tasks: 134 total, 1 running, 133 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 0.2%sy, 0.0%ni, 99.2%id, 0.5%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 7801724k total, 2587900k used, 5213824k free, 128964k buffers
Swap: 10043384k total, 0k used, 10043384k free, 1933744k cached


One thing I must mention is that while I have the server up and running, there is not yet any web traffic to the sites as I have not yet finished everything. The only traffic I see are the usual attack attempts, and the google bots.

Ive been reading around, searching CentOS 6.2 memory leaks and the like, and while its been happening, everyone else has been able to tag the incident to something, I cannot seem to find a process that uses that much memory. Also, its weird that it just builds up over time. Its not an instant throw, and takes hours to consume all 8gbs.


Any insight at all would be helpful

TIA
Dustin S.
 
Old 05-26-2012, 07:33 PM   #2
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942
It's just page cache (files read, filesystem internals like inodes cached, and so on), nothing to worry about.

I'm not sure why top counts it as used, though. It behaves the same on my Debian-derived Xubuntu, too. I normally look at /proc/meminfo myself, so I haven't paid a lot of attention to what memory statistics top reports. (On my workstation, top reported 3GB used out of 6GB RAM total, but actually 2.3GB of that was in page cache, inodes, and dentries, only 700MB was actually used by userspace.)

If you want to see the actual memory used by userspace applications and libraries, clear caches first:
Code:
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches ; sync'
The sync flushes unwritten data to disk. It is not necessary for safe operation; I only use it to make sure as little cache as possible is required to hold "dirty"/unwritten data. Writing 3 to /proc/sys/vm/drop_caches tells the kernel to drop clean pages from page cache, clean inodes, and clean dentries; see man 5 proc for details. The final sync is there just for good measure. "Dirty" or unwritten data will not be discarded; the commands are safe to run even when the server is heavily loaded.

I only recommend running it for diagnostic purposes since it will obviously cause a relative slowdown: immediately, because all unwritten data is written to disk at one go, and afterwards, because all new data must be read from disk, as nothing useful is cached in RAM. But, it never causes any damage, and the slowdown is just temporary.

Last edited by Nominal Animal; 05-26-2012 at 07:38 PM.
 
1 members found this post helpful.
Old 05-26-2012, 11:26 PM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,057

Rep: Reputation: 971Reputation: 971Reputation: 971Reputation: 971Reputation: 971Reputation: 971Reputation: 971Reputation: 971
Have a read of linuxatemyram for a little more background.
 
Old 05-27-2012, 08:36 AM   #4
hexalot
LQ Newbie
 
Registered: May 2012
Distribution: CentOS 6.2, BackTrack R5
Posts: 4

Original Poster
Rep: Reputation: Disabled
Nominal , syg; Thanks both for the reply Very useful information.

I was thinking about what you said Nominal, regarding the mem/disk writes, and it prompted me to check the 'messages' log. I noticed it was full of the following entry,

May 27 07:19:57 hosting kernel: [drm:atom_get_src_int] *ERROR* ATOM: fb read beyond scratch region: 1245188 vs. 16384

As far as I can tell, its a failing hdd. I will research more, but it almost seems as if the 2 are related. If anyone has ever experienced the same thing during drive death, I'd love to hear your story.

I left the server running over night, and top showed that usage was looking much better.

top - 07:16:50 up 14:20, 1 user, load average: 0.02, 0.01, 0.00
Tasks: 117 total, 1 running, 116 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 0.2%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 7801724k total, 760920k used, 7040804k free, 151460k buffers
Swap: 10043384k total, 0k used, 10043384k free, 339320k cached


Also, Clearing the caches worked great, I noticed the usage was now ~200mb at idle =D

Thanks again for the assist!

Last edited by hexalot; 05-27-2012 at 08:38 AM.
 
Old 05-27-2012, 11:26 AM   #5
robertjinx
Member
 
Registered: Oct 2007
Location: Prague, CZ
Distribution: RedHat / CentOS / Ubuntu / SUSE / Debian
Posts: 561

Rep: Reputation: 58
Quote:
May 27 07:19:57 hosting kernel: [drm:atom_get_src_int] *ERROR* ATOM: fb read beyond scratch region: 1245188 vs. 16384
Thats has nothing to do with the hdd, it's the video driver which is sending out the error. Just ignore it, it happens

Regarding the dropping of caches it's a bad idea, there is a reason why Linux is caching everything and thats for speed. If you would have memory leaks you would seems something like "OutOfMemory" error, otherwise thats just a normal behavior of the Linux kernel and should be left alone.

Caching in the memory suppose to help with speed and not doing too much I/O work on the disk. If you drop the caches, the system will have to re-read everything from the disk, which is slow, super slow compared to the RAM.

Good luck!
 
Old 05-27-2012, 04:56 PM   #6
hexalot
LQ Newbie
 
Registered: May 2012
Distribution: CentOS 6.2, BackTrack R5
Posts: 4

Original Poster
Rep: Reputation: Disabled
Thanks for your reply Robert!

The thing that worries me most about the error I posted

"May 27 07:19:57 hosting kernel: [drm:atom_get_src_int] *ERROR* ATOM: fb read beyond scratch region: 1245188 vs. 16384 "

is that it didnt start until a couple of days ago. I assumed it was hdd related mostly because I had just mounted another drive in the system, and I know it to be abit aged.

I dont mind dumping cache at the moment as im doing alot of testing on this box, but its good to know what it is exactly im dumping in that respect.

What I find even more peculiar, I have a nearly identical box next to it that does receive traffic, and it does not have this issue. It has different hardware, but the same CentOS 6.2 install with the latest basics, apache, php, mysql.

In the beginning was the worst. It seems to have died on its own? I had run top after login a couple of days ago a noticed nearly all of the 8gbs of ram were considered "used". While it seems fine now, im trying to find the point source of the issue.
 
Old 05-27-2012, 09:18 PM   #7
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,225

Rep: Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021
So you have 2 issues.

1.
Quote:
[drm:atom_get_src_int] *ERROR* ATOM: fb read beyond scratch
In this context, fb = frame buffer (ie video issue); see here for details if you really care
https://lkml.org/lkml/2011/10/23/16

2. memory: as per syg00's link to linuxatemyram, Linux will always attempt to use all your RAM, otherwise you've wasted money on unused RAM
The kernel virtual memory system is smart and will release memory (re-assign) on demand as required.
You only need to worry if you see a big performance hit (unlikely) or swap starts filling up ...
 
Old 05-29-2012, 05:02 AM   #8
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942Reputation: 942
Quote:
Originally Posted by hexalot View Post
May 27 07:19:57 hosting kernel: [drm:atom_get_src_int] *ERROR* ATOM: fb read beyond scratch region: 1245188 vs. 16384
Like robertjinx and chrism01 wrote, it is a known graphics driver bug, quite harmless.

Quote:
Originally Posted by hexalot View Post
I left the server running over night, and top showed that usage was looking much better.
Well, no. There is no problem here, but there was no problem earlier either; you're just interpreting the status wrong. There is no memory leak.

Whenever you use your system, the kernel will use free memory to cache files and filesystem details accessed. You can see this in practice if you run e.g. find /usr -type f >/dev/null , listing all files under /usr, but discarding the output. The first time you run it it takes quite a while, but if you run it again, the second run will be very fast.

This is a good thing. It means the kernel is just anticipating your future needs, without making any commitments. The memory is still completely free. The kernel just saves stuff it thinks might come handy later on in those free pages, but it is not reserving them or putting them away in any way. If an application requests memory, that RAM will be given to the application instead, always. No memory allocation will ever fail because of that caching. Caches are your friend: do not drop the caches just because you feel uneasy about them!
 
Old 05-29-2012, 12:18 PM   #9
hexalot
LQ Newbie
 
Registered: May 2012
Distribution: CentOS 6.2, BackTrack R5
Posts: 4

Original Poster
Rep: Reputation: Disabled
@chrism01 - Thanks for the link! Ive read about the message in forums before, but was never able to completely interpret. Now that I understand completely where the issue is coming from, I can take proper steps to patch or make other changes to keep the junk out of the log

And thanks to you both again for the memory input. I think its sinking in now I guess what was throwing me off was the fact that I came from limited VPS environments. In my head I was thinking "If it eats at least 1gb or more at idle, the same distro couldnt be run on a vps with 768mb of ram". It was a case of broken logic for lack of understanding linux memory usage completely.

I feel as if I have accomplished something by being better informed, and I cant thank you guys enough
 
Old 05-29-2012, 07:55 PM   #10
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5, Centos 5.10
Posts: 16,225

Rep: Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021Reputation: 2021
Good to hear.
You're certainly not the first nor will be the last to misunderstand caching; that's why they wrote this http://www.linuxatemyram.com/
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Memory leaks.. *** glibc detected *** ./SuffixTree: malloc(): memory corruption: 0x00 evansash Programming 12 03-21-2011 01:17 PM
centos memory leaks issue when playing flv tangi Linux - Newbie 2 02-14-2011 03:18 PM
Memory leaks mailsrinu28 Programming 4 08-07-2007 07:15 PM
frustrating memory leaks xushi Slackware 18 08-10-2005 06:13 AM
Memory Leaks? stampede96 Linux - Software 3 02-20-2003 12:52 PM


All times are GMT -5. The time now is 09:48 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration