oom-killer is a kernel (operating system) program that runs to free up memory if it becomes so dangerously low as to prevent the system from being usable.
Unfortunately this gives little indication as to the cause of the problem. But there are two possibilities here; either the program legitimately needs more memory or something is running to gobble up a huge amount of memory on the server by about 2AM, possibly a disk search index update or similar.
Either way, upgrading the machine with more physical memory should fix this issue, but that's a rather expensive option.
The first thing to check is that your system has as much memory as it needs. This includes the size of your swap space on disk. If you open up a terminal and type
then you should see various memory statistics.
The first thing to check is that SwapTotal is not 0. If it is, then you have swap memory turned off; you will need to create a swap partition (or file) and then enable it.
Next, when the machine is idle after a fair amount of use, SwapFree should be a fairly high number, say at least half of SwapTotal (the actual minimum will depend on the machine. If this hits 0 then you may start to see oom-killer kicking in). If not then your swap space may be too small; enlarge your swap partition make a second swap partition (or file) and then enable it.
Generally, the sum of MemFree + Buffers + Cached should be a fairly big number, anything as low as a few thousand suggests that your system may be having memory troubles.
The other approach is that something may be gobbling up memory at 2AM in the morning. To see what is using the memory, you could try logging the top resource-hungry processes that are running at the time.
Open a console and type
Now add this line to the end of the file and save it:
Code:
* 1-2 * * * top -b -n 1 > ~/top.$(date +%Y.%m.%d.%H.%M.%S)
This will run once a minute between 1 and 3 AM and create files in your home directory of the form "top.
date". You should then be able to at least see which program was using up all the memory, by looking at the file created just before a crash.
Hope that helps:
—Robert Lee
