memoryleak? oomkiller kills processes

D.kok · 03-16-2005, 05:25 AM

Hi,

Fedora core 3 runlevel 3 used as a webserver
Since a while this system is loosing memory . When the memory is empty oom-killer kills processes f.i. httpd mysqld. So the systems becomes unusable.
I have no process withs consumes the load of memory

After a reboot the same happens again.

How can I find the "thing" that takes the memory?

Thanks in Advance
Donald

marcosdumay · 03-16-2005, 06:32 AM

There are two very useful programs to spy processes, ps and top.

Both run on a terminal, top is interative, while ps just dumps the info on the screen, so I recomend you top (if you where writing a script, ps whould be better).

To run, open a terminal and enter the command "top". You problalbly want to sort the processes by memory use, if I remember it right, you do that pressing "m", if I am wrong, try pressing "h" to see a help screen.

D.kok · 03-16-2005, 07:11 AM

Hi Marcos,
I used top and ps (and free, /proc/meminfo), but can not find strange things. Memory level is getting lower and lower though.
Greetzz
Donald

Oliv' · 03-16-2005, 08:20 AM

Hello,

from http://linux-mm.org/docs/oom-killer.php :

Quote:

The OOM killer uses the following factors to chose which process to kill in an out of memory situation:

* memory use, the more memory a process is using, the more memory we will free up and the higher the likelyhood that this program is too big for the system and couldn't have run to completion anyway
o more memory use increases the likelyhood of being killed
* CPU use, the more processor time a process has used, the more work will be lost if we kill this process
o more cpu time decreases the chance of being killed
* time since start time, the longer a process has been running, the more likely it is that the process is stable and not "guilty" of exhausting system resources
o a longer run time decreases the chance of being killed
* system administrator rights, usually only trusted programs and important system programs run as root or with capabilities enabled
o running as root decreases the chance of being killed
* direct hardware access, killing a process which has direct hardware access may lead to hardware getting confused and the machine hanging; also, programs with direct hardware access are usually important for whatever task the system is doing
o direct hardware access decreases the chance of being killed

try to run pmap on process killed. You can also have a look at your kernel messages to see if there are more info.

D.kok · 03-16-2005, 09:11 AM

oom-killer kills processes because the memory is low. It is a syptom.
I need to know why the memory on the system is running low, so oom-killer does not need to kill processes.
oom-killer helps a system, without it the system would 'die'.

marcosdumay · 03-17-2005, 09:22 AM

Are you sure you jave enogh memory + swap to run everyting you are running?
Some programs consume a lot of memory on only a few tasks, so they may be stable for a time, but suddelnly run aout of memory.

Also, if you have a memory leak, you will se the memory use of the offending program grow up all the time (it's different to grow up and keep there, several programs don't release their memory while running, this is ok). You must examine their usage several times and see if the memory usage of a program is growing.

D.kok · 03-18-2005, 02:03 AM

Hi Marcos,
The machine has 200 of the 500 Mb free after booting. It also has 2 Gb for swapping.
Basicly it runs httpd and mysql with a quite statical website.
Until last week the machine woked without problems for months without booting. The last weeks I have to to boot twice a day.
No additional software is added. I update the machine with yum daily though.
Do you know of a tool which can print the memory usage of processes, So I can put that in cron and analyse it later?

marcosdumay · 03-18-2005, 12:07 PM

Hi, Dkok.

Well, ps -A -l display all process information on stdout. You can redirect it to a log file.
The problem is that this is quite a verbhouse dump, you will problably want to make a script to walk through it. Knowing that the position of older processes on the dumps don't change will help.

But the problem you descripbe is quite a weard one. Are you sure no software has changed (even actualized)? You may want to look for a worm with those simptoms.

Sorry I can't really help you, but I never saw anything like that before.