Disk cache overflow when >25GB on kernel 2.6.37.6

deckert · 01-08-2013, 06:40 AM

Hi!

We run a set of servers, each with 32GB of RAM and Slackware 13.37 with the standard 2.6.37.6 kernel (recompiled with 64GB memory model, using the 32-bit PAE extensions to access all 32GB).

Our application does not use all that much memory (max of about 2.4GB per process), so a lot of the memory gets allocated to the disk cache over time (which is fine).

However, when the size of the disk cache hits approx. 25GB (as reported by 'free'), the kernel thinks that there is no more memory left on the machine and the OOM killer starts killing off other normal processes (eg. sshd, httpd, etc).

Eg:

Code:

Dec 24 19:59:54 server kernel: [621984.510826] Out of memory: Kill process 30237 (java) score 4 or sacrifice child
Dec 24 19:59:54 server kernel: [621984.510905] Killed process 30237 (java) total-vm:2460272kB, anon-rss:153936kB, file-rss:9988kB

At the time, 'free' reported available RAM as:

Code:

             total       used       free     shared    buffers     cached
Mem:      32788272   25309772    7478500          0       2928   24677912
-/+ buffers/cache:     628932   32159340
Swap:       995992          0     995992

When we clear the disk cache using:

echo 1 > /proc/sys/vm/drop_caches

The problem goes away until we hit approx. 25GB in disk cache again. As a short-term workaround we've created a crontab entry which periodically clears the disk caches using the above command and as long-term we added mem=16G to the kernel command line to effectively limit the size of the disk caches.

As anybody come across this issue?

Our test servers do not have this amount of RAM on them, but they will shortly and we will be testing the latest 3.7.x kernel branch to see if we get the same problem.

Note that this is a 32-bit kernel (out of necessity).

--deckert

syg00 · 01-08-2013, 07:00 AM

PAE was a kludge. Linus has said (years ago now) he has no sympathy for people running 32 bit on "big" RAM systems. And he was talking greater than 8 Gig.
I hate to say it, but you need to get onto 64-bit hardware to get the kernel devs to even listen I suspect.

You might be able to work around it using cgroups (and thus get to use all your RAM), but it's been so long since I tried on a 32-bit kernel, I can't say how successful that will be.

Celyr · 01-08-2013, 07:44 AM

Can you please post your

Code:

# cat /proc/meminfo

in such situation ?

phenixia2003 · 01-08-2013, 08:21 AM

Hello,

Quote:

Originally Posted by syg00

PAE was a kludge. Linus has said (years ago now) he has no sympathy for people running 32 bit on "big" RAM systems. And he was talking greater than 8 Gig.

I guess you're talking about this thread, which is really interesting to read by the way.

--
SeB

deckert · 01-12-2013, 07:39 AM

Hi guys,

Thanks for all the replies and info.

@syg00: yep, we really should be moving our apps and code to 64-bit. The hardware is very much 64-bit capable, but we've been using 32-bit because it is a known-stable environment for us.

@Celyr: The systems carry telco information so we have no leeway to do any experimentation. We implemented the cache-clearing and then added the mem=16G to the kernel command line to solve the issue permanently. If we ever get a chance to test it again with 32GB RAM and 32-bit kernel, I will definitely examine the contents of /proc/meminfo (and post it here).

@phenixia2003: thanks for the link to that thread, it is definitely most informative. In summary then, from Linus himself, don't use more than 16GB of RAM on 32-bit systems else you're just looking for trouble.

--deckert