LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Slackware
User Name
Password
Slackware This Forum is for the discussion of Slackware Linux.

Notices


Reply
  Search this Thread
Old 01-08-2013, 06:40 AM   #1
deckert
LQ Newbie
 
Registered: Jun 2006
Location: South Africa
Distribution: Slackware
Posts: 7

Rep: Reputation: 0
Disk cache overflow when >25GB on kernel 2.6.37.6


Hi!

We run a set of servers, each with 32GB of RAM and Slackware 13.37 with the standard 2.6.37.6 kernel (recompiled with 64GB memory model, using the 32-bit PAE extensions to access all 32GB).

Our application does not use all that much memory (max of about 2.4GB per process), so a lot of the memory gets allocated to the disk cache over time (which is fine).

However, when the size of the disk cache hits approx. 25GB (as reported by 'free'), the kernel thinks that there is no more memory left on the machine and the OOM killer starts killing off other normal processes (eg. sshd, httpd, etc).

Eg:

Code:
Dec 24 19:59:54 server kernel: [621984.510826] Out of memory: Kill process 30237 (java) score 4 or sacrifice child
Dec 24 19:59:54 server kernel: [621984.510905] Killed process 30237 (java) total-vm:2460272kB, anon-rss:153936kB, file-rss:9988kB
At the time, 'free' reported available RAM as:

Code:
             total       used       free     shared    buffers     cached
Mem:      32788272   25309772    7478500          0       2928   24677912
-/+ buffers/cache:     628932   32159340
Swap:       995992          0     995992
When we clear the disk cache using:

echo 1 > /proc/sys/vm/drop_caches

The problem goes away until we hit approx. 25GB in disk cache again. As a short-term workaround we've created a crontab entry which periodically clears the disk caches using the above command and as long-term we added mem=16G to the kernel command line to effectively limit the size of the disk caches.

As anybody come across this issue?

Our test servers do not have this amount of RAM on them, but they will shortly and we will be testing the latest 3.7.x kernel branch to see if we get the same problem.

Note that this is a 32-bit kernel (out of necessity).

--deckert
 
Old 01-08-2013, 07:00 AM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 13,360

Rep: Reputation: 1307Reputation: 1307Reputation: 1307Reputation: 1307Reputation: 1307Reputation: 1307Reputation: 1307Reputation: 1307Reputation: 1307Reputation: 1307
PAE was a kludge. Linus has said (years ago now) he has no sympathy for people running 32 bit on "big" RAM systems. And he was talking greater than 8 Gig.
I hate to say it, but you need to get onto 64-bit hardware to get the kernel devs to even listen I suspect.

You might be able to work around it using cgroups (and thus get to use all your RAM), but it's been so long since I tried on a 32-bit kernel, I can't say how successful that will be.
 
1 members found this post helpful.
Old 01-08-2013, 07:44 AM   #3
Celyr
Member
 
Registered: Mar 2012
Location: Italy
Distribution: Slackware+Debian
Posts: 316

Rep: Reputation: 80
Can you please post your
Code:
# cat /proc/meminfo
in such situation ?
 
1 members found this post helpful.
Old 01-08-2013, 08:21 AM   #4
phenixia2003
Member
 
Registered: May 2006
Location: France
Distribution: Slackware
Posts: 333

Rep: Reputation: 197Reputation: 197
Hello,

Quote:
Originally Posted by syg00 View Post
PAE was a kludge. Linus has said (years ago now) he has no sympathy for people running 32 bit on "big" RAM systems. And he was talking greater than 8 Gig.
I guess you're talking about this thread, which is really interesting to read by the way.

--
SeB
 
1 members found this post helpful.
Old 01-12-2013, 07:39 AM   #5
deckert
LQ Newbie
 
Registered: Jun 2006
Location: South Africa
Distribution: Slackware
Posts: 7

Original Poster
Rep: Reputation: 0
Smile

Hi guys,

Thanks for all the replies and info.

@syg00: yep, we really should be moving our apps and code to 64-bit. The hardware is very much 64-bit capable, but we've been using 32-bit because it is a known-stable environment for us.

@Celyr: The systems carry telco information so we have no leeway to do any experimentation. We implemented the cache-clearing and then added the mem=16G to the kernel command line to solve the issue permanently. If we ever get a chance to test it again with 32GB RAM and 32-bit kernel, I will definitely examine the contents of /proc/meminfo (and post it here).

@phenixia2003: thanks for the link to that thread, it is definitely most informative. In summary then, from Linus himself, don't use more than 16GB of RAM on 32-bit systems else you're just looking for trouble.

--deckert
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
kernel: dst cache overflow MSN Linux - Networking 1 02-08-2008 08:08 AM
dst cache overflow clutch420 Linux - Networking 0 10-24-2007 04:41 PM
Ram wiht suse, cache Disk cache??? fadelhomsi Linux - Newbie 2 02-05-2006 11:29 PM


All times are GMT -5. The time now is 08:13 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration