Memory peak caused server crash, help with data interpretation?

aldmy · 12-16-2010, 12:17 AM

Hi,

Last week my server crashed. I'm trying to diagnose the cause.

This is the relevant error message in /var/log/messages:

Code:

Dec  9 09:47:52 ZZZZ kernel: httpd invoked oom-killer: gfp_mask=0xa01d2, order=0, oomkilladj=0
Dec  9 09:47:52 ZZZZ kernel:  [<c0464ed2>] out_of_memory+0x69/0x1a7
Dec  9 09:47:52 ZZZZ kernel:  [<c0466546>] __alloc_pages+0x21b/0x2a2
Dec  9 09:47:52 ZZZZ kernel:  [<c0467c96>] __do_page_cache_readahead+0xe2/0x1ff
Dec  9 09:47:52 ZZZZ kernel:  [<c045d399>] delayacct_end+0x70/0x77
[ ... ]

I'm assuming that I can conclude, then, that apache/httpd was the cause of the memory leak?

Next, I've been tracking my memory usage. Using top, this is an average memory load level for my server:

Code:

Mem:   2073156k total,  1982572k used,    90584k free,   421948k buffers
Swap:  2096472k total,       60k used,  2096412k free,   887700k cached

I'd like to confirm if my understanding of this data is correct, because Plesk indicates that my memory usage is only 50% or less. (Though I have read a number of reports indicating that Plesk's measurements are frequently wrong.)

top says: Of the 2,073,156K total memory, 1,982,572K (95.63%) is being used, 90,584K (4.37%) is free. Of that sum, 421,948K (20.35%) are being used as buffers. Additionally, of the 2,096,472K of Swap, 60K is used, and 887,700K (42.34%) is cached.

My questions:

Is my memory actually being 95% used?
Or is the buffered quantity (20.35%) not a use of physical/virtual memory? (i.e. is it disk usage?)
Does the amount of cached Swap influence the percentage of physical/virtual memory being used?

In other words, who is correct? Plesk says I'm using 40-50% of my memory, whereas top says 85-95%.

bpogi · 12-16-2010, 12:34 AM

Quote:

Originally Posted by aldmy

Is my memory actually being 95% used?

Yes and no. Linux likes to buffer the disk. It makes things faster, so it will buffer as much as it can, up to a certain limit (~95%). As soon as an application needs this memory, it's given up. See http://www.linuxatemyram.com/ It will also explain how to determine the memory availability for applications.

Quote:

Or is the buffered quantity (20.35%) not a use of physical/virtual memory? (i.e. is it disk usage?)

It is a usage of memory in the strictest sense, but that buffered memory is available immediately upon request by applications.

I can't comment on the error messages you provided. I don't know enough.

bpogi · 12-16-2010, 12:37 AM

Quote:

Originally Posted by aldmy

Hi,

Last week my server crashed. I'm trying to diagnose the cause.

This is the relevant error message in /var/log/messages:

Code:

Dec  9 09:47:52 ZZZZ kernel: httpd invoked oom-killer: gfp_mask=0xa01d2, order=0, oomkilladj=0
Dec  9 09:47:52 ZZZZ kernel:  [<c0464ed2>] out_of_memory+0x69/0x1a7
Dec  9 09:47:52 ZZZZ kernel:  [<c0466546>] __alloc_pages+0x21b/0x2a2
Dec  9 09:47:52 ZZZZ kernel:  [<c0467c96>] __do_page_cache_readahead+0xe2/0x1ff
Dec  9 09:47:52 ZZZZ kernel:  [<c045d399>] delayacct_end+0x70/0x77
[ ... ]

I'm assuming that I can conclude, then, that apache/httpd was the cause of the memory leak?

I can answer this last question. I don't think you can conclude that directly. apache/httpd triggered the ultimate crash, but that doesn't mean it's the source of the leak if there is one. It might be, but this doesn't rule out other processes eating up memory, then apache trying to allocate memory for a legitimate use and triggering the crash.

aldmy · 12-16-2010, 01:00 AM

Quote:

Originally Posted by bpogi

It is a usage of memory in the strictest sense, but that buffered memory is available immediately upon request by applications.

My question boils down to: is 95% of my physical memory being used?

If 20% of it is being buffered, is that buffer being stored somewhere other than physical memory?

Keeping in mind the data derived above through top, observe the contrast with what Plesk indicates:

Memory Usage
Total 1.98 GB
Used 1.93 GB
Free 50.58 MB
Shared 0 B
Buffer 137.09 MB
Cached 1.26 GB
Usage 33.8%

Plesk is fully subtracting "cached" memory from "used" memory. What form does this cached memory have? From these statistics it is clearly not in swap. Is it simply "shared" data, i.e. code data? Most importantly, is it still a use of physical memory space? Or could it be use of disk space?

If 85-95% of my physical memory is being used, base-line, then my server evidently cannot handle a spike with the hardware currently available. I need to know precisely how much physical memory is being used.

syg00 · 12-16-2010, 04:14 AM

Did you look at that linuxatemyram link ?. What didn't you understand - what didn't answer your questions ?.

There is an unfortunate overloading of terms such as memory, cache, shared, used, free, buffer, usage ...
Even the kernel devs can't agree on (actual) memory usage. Whatever that may mean.

As for the oom killing apache, the oom_killer tries to get rid of the biggest baddest (fastest consuming) task. Usually the guy you'd least want killed - like apache or your DB server.

aldmy · 12-16-2010, 09:41 AM

Quote:

Originally Posted by syg00

Did you look at that linuxatemyram link ?. What didn't you understand - what didn't answer your questions ?.

The linuxatemyram site was very helpful, thanks!

What is still unclear to me is what influence the "buffer" figure has on the figure of total usage.

The current memory situation on my server, for example, staggers me. It seems as though more memory is being used than I actually have, according to top:

According to the data, pasted, below, the cumulative sum of virtual memory used by all my apache processes is 3,439 MB, or 3.37 GB. The buffer cache is 198 MB large. Of the 2.0 GB of Swap, 0% is in use, and 653 MB are cached.

As my server only has 2.0 GB of physical memory available, how do I account for the fact that Apache is using, according to top, 3.37 GB of memory?

Code:

top - 10:16:27 up 1 day, 18:31,  1 user,  load average: 1.14, 0.89, 0.88
Tasks: 144 total,   2 running, 142 sleeping,   0 stopped,   0 zombie
Cpu(s): 34.8%us, 15.9%sy,  0.0%ni, 48.3%id,  0.8%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:   2073156k total,  2005224k used,    67932k free,   203056k buffers
Swap:  2096472k total,       84k used,  2096388k free,   669348k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2338 mysql     20   0  196m  89m 5036 S   38  4.4 268:39.71 mysqld
23209 apache    20   0  181m 134m 7044 S    0  6.6  10:10.22 httpd
23137 apache    20   0  179m 130m 6188 S    0  6.5   8:56.31 httpd
23140 apache    20   0  179m 131m 6588 S    7  6.5   8:12.75 httpd
23354 apache    20   0  175m 126m 6712 S    0  6.3   8:21.56 httpd
23249 apache    20   0  174m 126m 6016 S   11  6.2   9:30.94 httpd
23138 apache    20   0  173m 125m 6704 S    0  6.2   8:09.49 httpd
23232 apache    20   0  172m 125m 7032 S   23  6.2   9:05.62 httpd
 4389 apache    20   0  172m 125m 7044 S    2  6.2   4:45.24 httpd
26389 apache    20   0  172m 125m 7048 S   11  6.2   8:39.05 httpd
23928 apache    20   0  172m 124m 6648 S    0  6.1   6:31.91 httpd
23144 apache    20   0  172m 124m 6828 S    0  6.2   7:25.52 httpd
23142 apache    20   0  172m 124m 6996 S    0  6.2   7:38.49 httpd
23143 apache    20   0  172m 123m 6008 S    6  6.1   8:49.58 httpd
23929 apache    20   0  172m 124m 6948 S    0  6.2   7:13.57 httpd
23139 apache    20   0  172m 124m 7320 S    0  6.2   8:39.23 httpd
23141 apache    20   0  172m 125m 7780 S    0  6.2   9:54.45 httpd
 5359 apache    20   0  172m 123m 6060 R    3  6.1   3:04.63 httpd
 6336 apache    20   0  172m 123m 6108 S    0  6.1   1:34.17 httpd
 4031 apache    20   0  172m 123m 6176 S    0  6.1   6:26.62 httpd
 2652 root      20   0  159m 117m  10m S    0  5.8   1:40.13 httpd
23134 apache    20   0  142m 105m  624 S    0  5.2   0:01.02 httpd

syg00 · 12-16-2010, 04:45 PM

Linux uses shared libraries for code which can be shared. Tools like top and ps just doesn't account for it properly. Have a read of this. The comments after the article may be interesting too, or just confusing.