LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 02-20-2012, 03:50 AM   #1
Kelly77
LQ Newbie
 
Registered: Feb 2012
Posts: 8

Rep: Reputation: Disabled
RAM gets filled up!


Hello,

I'm a new member here but I've been visiting this great site for years and have been finding solutions to my problems thanks to these wonderful forums.. but recently I had this problem I couldn't find a solution to ..

I used to have a server with 6GBs of RAM which was running perfectly until I decided to upgrade for a better deal, with a Xeon processor, 2x3TBs drives, and 16 GBs of RAM, Centos 6 & Cpanel..

the problem is, the memory gets filled up and the server crashes! especially when I copy data using rsync or wget, it fills up the RAM and the server crashes afterwards! this also goes for every process or program that fills a portion of the memory is not giving it back after it's finished.. when u open a program then close it, it should free up the memory it filled.. the server goes offline at least once a day!

to work around this issue, I've created a cron job that runs every minute to flush the memory:

* * * * * sync; echo 3 > /proc/sys/vm/drop_caches

but this is not a solution!! could someone tell me why the system is not freeing up the memory please??

Thanks..
 
Old 02-20-2012, 08:03 AM   #2
Kelly77
LQ Newbie
 
Registered: Feb 2012
Posts: 8

Original Poster
Rep: Reputation: Disabled
WOWZERS!! 97 views and no answer??

DUDE !!
 
Old 02-20-2012, 08:16 AM   #3
johnsfine
LQ Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,286

Rep: Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181
Quote:
Originally Posted by Kelly77 View Post
when I copy data using rsync or wget, it fills up the RAM
That is perfectly normal and should not need any correction.

Quote:
and the server crashes afterwards!
That is obviously not OK, but the primary cause is not ram filling as you described above. Maybe ram filling isn't a factor at all, just coincidence. Maybe it is an indirect factor effecting the direct problem.

We need more info to understand the actual problem.

Quote:
this also goes for every process or program that fills a portion of the memory is not giving it back after it's finished.
Sounds normal, since it appears you are including the cache in ram use.

Quote:
to work around this issue, I've created a cron job that runs every minute to flush the memory:
That is so unlikely to be helpful, I think you are misunderstanding something.
Quote:
could someone tell me why the system is not freeing up the memory please??
Because it isn't supposed to free that memory.
Ask instead why it crashes.
 
Old 02-20-2012, 08:26 AM   #4
Kelly77
LQ Newbie
 
Registered: Feb 2012
Posts: 8

Original Poster
Rep: Reputation: Disabled
Thanks for your reply .. but you didn't seem to understand the issue..

I used to copy over 4TBs of data on a 2GBs Ram server using rsync and wget with no issues..

here's what's goung on.. rsync is copying let's say 58GBs of data, as it starts, 'free -m' shows increase consumption of memory .. as it continues, the free memory keeps going down rapidly until eventually at some point the whole 16GBs RAM are filled and the server halts and I have to reboot it from the remote console..

I know for sure it's memory issue because when free -m shows that the server is nearly out of memory it halts after a few secs..

also I know the problem is not with rsync because this happens with wget also and other processes..

any help??

Thanks..
 
Old 02-20-2012, 08:52 AM   #5
johnsfine
LQ Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,286

Rep: Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181
Do you have a 32 bit PAE kernel or a 64 bit kernel? (You have one or the other. If you don't know which, the output of uname -a would show us the answer).
With 16GB of ram and a PAE kernel, it is easy to fill kernel virtual memory causing serious problems. While filling (32 bit PAE) kernel virtual ram is independent of filling physical ram, it might be triggered by the same workload creating an illusion of a cause and effect connection. You can't fill 64 bit kernel virtual, so with a 64 bit kernel ignore this whole side track.

Quote:
Originally Posted by Kelly77 View Post
as it continues, the free memory keeps going down rapidly until eventually at some point the whole 16GBs RAM are filled
You don't seem to understand that is perfectly normal. Cache filling physical ram under those conditions is exactly the designed and optimal behavior.

Quote:
and the server halts
What does that mean? If you copy a lot with plenty of ram available for caching, you might fill up memory with "write behind" caching. So the copy operation seems to be proceeding more than twice as fast as it really is. Then suddenly apparent progress pauses for a while before reality catches up with the previously false appearance. If you call that "halting" you are too quick on the reboot.

Quote:
Originally Posted by Kelly77 View Post
I know for sure it's memory issue because
What you don't know never does as much harm as what you do "know" that isn't so. As long as you are sure that cache using up memory is the direct cause of your problem, you will be unable to find the actual cause.

I don't normally guess hardware problem. But if you have a 64 bit Centos kernel, the info you provided hints at a hardware problem. If you have a 32 bit PAE kernel, then I would be pretty confident that you have a workload for which 16GB is too much ram for a PAE kernel, even though a PAE kernel can handle some workloads with 16GB ram. If you have a 64 bit kernel, try some offline memory diagnostics.

Last edited by johnsfine; 02-20-2012 at 09:17 AM.
 
Old 02-20-2012, 11:22 AM   #6
Kelly77
LQ Newbie
 
Registered: Feb 2012
Posts: 8

Original Poster
Rep: Reputation: Disabled
again thanks for your patience .. the system is centos 6 64 Bits..

what I mean by halts is that it stops responding to ping, ssh, websites do not open, ftp ...etc like it's turned off.. the hosting provider (hetzner.de) said that they first took the server down and did a full hardware check for 10 hours and no error were detected, then they said that they have replaced the whole server and still the same problem!!

even after reinstalling the OS several times, same problem exists..
 
Old 02-20-2012, 11:26 AM   #7
johnsfine
LQ Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,286

Rep: Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181
Is it an actual physical server? Or is it a virtual slice of some larger server?

If it is virtual, I think you are getting ripped off by some misleading service that has bugs in whatever method they use to pretend you are getting 16GB when you are actually not.

Even if it is not virtual, the fact that it is remote and under some other company's control may make it very hard for you to diagnose what they have done wrong.

So far as you have described, what you have done is not wrong. The first "symptom", cache expanding to fill almost all of free memory is also not wrong, it is normal behavior of Linux.
It should be the responsibility of the provider of the server to diagnose their problem.

Irrelevant anecdote and/or wild guess: On several Windows systems with Dell fake RAID, I found a very frustrating bug. Whenever the workload caused a very high use of write behind disk caching, some flaw in the fake RAID software deadlocked all disk access and required rebooting. I never found a correction and I know of no user controls to tune the write behind caching behavior. So the only thing I was able to do was throttle the heavy workload (by restricting which cores it could use) so file output was produced more slowly and the systems don't crash.

I don't know the tuning controls available to restrict write behind disk caching in Linux either, but I think Linux has better controls than Windows. I think you could restrict the amount of write behind caching done by adjusting some setting. Based on your description, I think there is some chance write behind caching (especially if it is interacting badly with a bug in software RAID) could be a factor. If you reduce write behind caching, physical ram should still fill up with cache, but it would almost all be read caching not write caching, so it should not place any unusual load on software RAID.

Rereading your first post, I see the work around that you think avoids the crash includes sync as well as drop_caches. Try with just sync. That should not free significant memory, but might be dealing with the actual problem, which I still think may be separate from memory being full. If it suppresses the problem, a frequent sync will cut performance, but much less so than drop_caches, so it might be an acceptable work around for longer.

Last edited by johnsfine; 02-20-2012 at 11:54 AM.
 
Old 02-20-2012, 01:37 PM   #8
Kelly77
LQ Newbie
 
Registered: Feb 2012
Posts: 8

Original Poster
Rep: Reputation: Disabled
well it is a dedicated server .. and I have several servers with them running perfectly ..

they confirmed twice that they have replaced the server but strangely it had the same IP.. they said that they have assigned the same IP to the new server..

I noticed something though.. when reinstalling the server it takes less than 5 minutes to be ready, while Iknow setting up cpanel takes from 35 - 50 minutes!

I don't know what to do!
 
Old 02-20-2012, 01:53 PM   #9
johnsfine
LQ Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,286

Rep: Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181Reputation: 1181
Quote:
Originally Posted by Kelly77 View Post
they said that they have assigned the same IP to the new server..
Perfectly plausible. I would not infer anything from the IP address not changing.

If you will be doubting a claimed hardware change in a future situation, capture more specific info about the previous hardware. MAC address can be overridden but few people would think to, so if the MAC address changes the hardware probably changed and vice versa.
I expect dmidecode might pull up a few other details likely to change in most hardware swaps.

Quote:
I don't know what to do!
A hard to diagnose failure doesn't mean impossible to diagnose.

If you can't deal with some failures along the way to diagnosing it and/or the effort, get a different server product from the same or different vendor, not a different unit of the same product.

If pushing ahead with this server product, try the sync without drop_caches that I suggested earlier and gather a lot more info about the configuration (RAID, file systems, etc.) and details of memory use etc.
 
Old 02-20-2012, 02:27 PM   #10
Kelly77
LQ Newbie
 
Registered: Feb 2012
Posts: 8

Original Poster
Rep: Reputation: Disabled
I've hired some server management company and will see what will happen..

Thanks for your great help matey : )
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Hard Drive filled up? AKviking Linux - General 2 05-30-2010 06:24 PM
how to conserve ram, 512meg 95% filled up sirius57 SUSE / openSUSE 5 03-21-2008 08:51 PM
Help, my / is 97% filled up adityavpratap Slackware 18 11-29-2005 09:13 PM
Crontab does not do what i filled in bondgirl_2170 Linux - Software 5 06-30-2005 04:29 PM
What if / is filled up? Manish Linux - Security 4 03-25-2002 02:03 PM


All times are GMT -5. The time now is 05:21 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration