Out of memory
 

Hi,

I've a problem with my server.
Basically it's a web server (apache 1.3) + mail server (qmail).
Sometimes his load become very high, almost 100 cpu usage, and I am forced to reboot the machine (the use of console is impossible..)

When I see the console, the error message is like:

Out of memory
VM: killing apache
VM: killing syslogd
...

I thinks it's a memory leak in some application, because sometimes the server runs fine for weeks.
How can I find the specific problem?

Thanks

You will have to check your all your logs.
Start with apache and syslog to see what is the exact error they are reporting.
You can find the logs generally in /var/log/

If its a memory leak in some application (& that is not clear from what you post), then it ought to be reasonably easy to detect.

It would probably be a good idea to monitor memory usage over some fixed period, maybe every 10 seconds, maybe longer depending on how long you think you will have to run. If it is a memory leak, that would normally gradually increase. To do this you'd probably run vmstat and you might want to store vmstat results away in a log file to try to get results close to the crash.

The other thing you would do would be to monitor individual processes to check whether any individual process has increasing memory usage. Probably top -n 1 would capture the info that you want (you may have more processes than top easily captures & I'm not sure how top determines the screen length in non-interactive sessions...) and you would want to capture a number of data dumps over a period to see how memory usage grows.

(You could do this more simply with ksysguard - given that its a server, you won't have kde installed on it, but you could use ksysguard on a desktop to monitor memory use on the server, if you place the 'sensors' remotely, but I haven't actually tried this.)

You'd be hoping (for the memory leak theory) to see some process growing over time until its memory size becomes too large; a difficulty is being sure that a process is growing 'illegitimately' (memory leak) versus one that is just growing because its worload has grown too high.

Cpu alone usually do not make a text console lag with the schedulers we have nowadays.

So, the theory of leaking memory is very consistent. Not only you have the errors, but the behavior you describe is typical of a system with a very hight swapping activity. I'd say launch an ssh session and put that window in top of your face, run top or htop on it, and pay special attention to the swap level.

If you have sysstat installed, sar has all the historical info you need to track this. Else try the "top" suggestion above, but run it in batch (-b) in addition to "-n 1". Write it out to a file - you could get creative and sort by memory and only pull (say) the top 5 if it really is a leak. May not be indicative tho' - you're likely to have tens (hundreds ???) of processes spawned by apache; or maybe a looping (cron) script. It all adds up.