I recently had a nasty out of memory kernel freak-out which ended in oom-killer killing sshd, named, apache, mysql and a handful of other services, at which point, my server wasn't serving anything and I couldn't get to it to diagnose the problem. After it was rebooted, it was clear that it was an oom:

There's a bit more of a back-story here, but basically, the server was slowly running out of memory and, eventually, it ran out of swap as well. I've got cacti on the server which monitors the system's memory usage and I can see it decay over time, down to nothing a few hours before it crashed.

I'm trying to prevent this from recurring, but since I don't have a record of which process(es) had all the RAM, I'm trying to get some circumstantial evidence. The above snippet is the last complete oom-killer log before the server died completely. First, I'm assuming mysqld requested more memory which signaled oom-killer to kill another process and it eventually chose apache2. This server typically has fairly low resident memory usage, for example, here's what it looks like right now:

The way I understand the linux memory management is that the "free" memory will be slowly used for caching I/O (as seen in meminfo under Buffers and Cache), but since this memory is backed by actual files, it can be reclaimed when memory starts getting low.

Based on my kernel log snippet, is it clear that all the physical memory was used by resident processes? I understand that the "free" memory as shown by meminfo and top does not take reclaimable memory into account. I'm trying to get a clear picture of how much memory was used for what purpose, but there are many different numbers in the log (ie. "Mem-Info:" vs. "Node 0 DMA" vs. "Node 0 DMA32").

I have a feeling that if I were at the console at this point, I would use the top command to see why the system is so slow, and I would see 0kB of memory and swap, but the list of processes' RES memory would not add up to the 6GB of RAM+SWAP.

If it helps, I believe this issue was caused/perpetuated by a user that was hitting a memory-intensive HTTP API of mine several million times per day (this is what happens when you try to be a nice guy I guess :P )

Thanks in advance for your insight!

[EDIT]
To expand, I doubt Apache could have used all the memory since it's restricted to 10 spare servers and 25 workers max, and they typically hit about 75MB max, so this would only be 25 x 75 = 1.8GB as a worst-case scenario, and oom-killer should have been able to kill off the whole herd of workers and reclaim all kinds of memory.

Delete My account

I'm using prefork right now to spawn worker threads before they're needed, I'm just trying to make sure my system is properly reclaiming memory. What is strange to me is that this whole process took many days, so the kernel should have had time to react in order to prevent this scenario.

I've attached a graph of the free memory and swap during this period. The blank section is when the server was down. It's a little confusing because all the measurements are in kilobytes, so the top of the graph (5.0 M) is 5,000,000 kB, or 5GB. The stats are the same way, "Current: 492.65 k" really means "Current: 492.65 MB".

Attached Thumbnails