Hi There,

New to everything *unix and have a box that we look after which is running RHEL 3 and PLESK 8.2 which we use for web hosting.

We're finding that the box is running out of memory every few days to the point where the OOM comes along and starts killing processes. I don't really know what's causing all the memory to be chewed up. I can only guess that it's because we have a little over 1000 web sites on this box and every time a httpd request comes in, it chews up a hell of a lot of memory (see "top" ouput below sorted by memory usage).

------------------------------------------------------------------------

12:57:21 up 13:25, 2 users, load average: 0.32, 0.38, 0.67
306 processes: 305 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 1.4% 0.0% 0.2% 0.0% 0.0% 0.1% 97.9%
cpu00 0.7% 0.0% 0.0% 0.0% 0.0% 0.3% 98.8%
cpu01 0.7% 0.0% 0.7% 0.0% 0.3% 0.0% 98.0%
cpu02 3.9% 0.0% 0.3% 0.0% 0.0% 0.3% 95.2%
cpu03 0.3% 0.0% 0.0% 0.0% 0.0% 0.0% 99.6%
Mem: 4112280k av, 3863828k used, 248452k free, 0k shrd, 34888k buff
2382008k actv, 441120k in_d, 73816k in_c
Swap: 522072k av, 127076k used, 394996k free 2053264k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
18395 apache 15 0 267M 267M 8232 S 0.0 6.6 1:02 2 httpd
24405 apache 25 0 263M 222M 8508 S 0.0 5.5 1:46 0 httpd
25316 apache 25 0 265M 190M 8484 S 0.0 4.7 1:10 0 httpd
28180 apache 16 0 97248 94M 8980 S 0.0 2.3 0:07 2 httpd
28179 apache 15 0 96388 93M 8892 S 0.0 2.3 0:40 3 httpd
28555 apache 16 0 96388 93M 8648 S 0.0 2.3 0:03 3 httpd
28575 apache 15 0 96092 93M 8720 S 0.0 2.3 0:05 1 httpd
28381 apache 15 0 96044 93M 8896 S 0.0 2.3 0:07 1 httpd
29574 apache 15 0 94816 91M 8532 S 0.0 2.2 0:01 2 httpd
27752 apache 15 0 93908 91M 8864 S 0.0 2.2 0:07 3 httpd
28176 apache 15 0 93856 91M 8940 S 0.0 2.2 0:37 2 httpd

------------------------------------------------------------------------

These httpd processes are all SLEEPING so what could they be doing. When I do a lsof -p <pid>, it didn't tell me what was going on except all the open files used by the pid.

We've upgraded from 2G of RAM to 4G of RAM but this hasn't helped and gradually the RAM gets chewed up again. We are also looking at migrating some web sites off this box to see if that helps.

Any suggestions on how we can stop the memory from being chewed up or how to identify any rogue processes eating up the memory or at least identify why httpd is consuming so much memory.

Thanks.

Andy

32 bit kernel ?. Let's see the messages when you start getting oom'd - especially any that mention (memory) region.

# uname -a
Linux plesk2.netspace.net.au 2.4.21-4.ELsmp #1 SMP Fri Oct 3 17:52:56 EDT 2003 i686 i686 i386 GNU/Linux

Here's a screen shot of the DRAC console showing OOM in action when we lost access to the box eventhough it was still able to be pinged.

http://myvws.netspace.net.au/oom.jpg

That's not a lot of use - have a look at your logs.

What I'm tyring to track down is why all of the memory keeps being consumed no matter how much RAM we throw at the box up to the point where the OOM kicks in and starts killing processes.

Top shows that the memory hoggers are the HTTPD and JAVA processes, but I would think that these process would release the memory once they're down doing their thing.

-----

Right now we are sitting with 52M of memory free.

[root@plesk2 myvws.netspace.net.au]# free -m
total used free shared buffers cached
Mem: 4015 3963 52 0 44 1759
-/+ buffers/cache: 2159 1856
Swap: 509 125 384

-----

A top output sorted by memory usage.

15:34:38 up 16:02, 2 users, load average: 0.77, 1.31, 1.27
303 processes: 302 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 4.5% 0.0% 1.0% 0.0% 0.1% 17.3% 76.7%
cpu00 4.0% 0.0% 0.0% 0.0% 0.7% 3.7% 91.4%
cpu01 1.1% 0.0% 0.3% 0.0% 0.0% 30.9% 67.5%
cpu02 12.6% 0.0% 3.3% 0.0% 0.0% 3.7% 80.2%
cpu03 0.3% 0.0% 0.3% 0.3% 0.0% 31.2% 67.6%
Mem: 4112280k av, 4071208k used, 41072k free, 0k shrd, 39476k buff
2831872k actv, 237172k in_d, 88788k in_c
Swap: 522072k av, 128792k used, 393280k free 1817924k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
18395 apache 15 0 261M 261M 2072 S 0.0 6.5 1:02 2 httpd
6315 apache 25 0 260M 259M 2196 S 0.0 6.4 0:35 0 httpd
24405 apache 25 0 257M 216M 2072 S 0.0 5.3 1:46 0 httpd
25316 apache 25 0 258M 184M 2072 S 0.0 4.5 1:10 0 httpd
6032 psaadm 15 0 163M 163M 21900 S 0.0 4.0 5:37 2 httpsd
5886 psaadm 16 0 161M 161M 21024 S 0.0 4.0 4:33 2 httpsd
8641 apache 15 0 96408 93M 6084 S 0.3 2.3 0:17 0 httpd
8647 apache 15 0 95592 92M 5960 S 0.0 2.3 0:13 1 httpd
9004 apache 17 0 95456 92M 5852 S 4.1 2.3 0:19 2 httpd
6132 apache 15 0 95184 92M 5740 S 0.0 2.3 0:21 0 httpd
8769 apache 15 0 94952 92M 5700 S 0.0 2.2 0:16 1 httpd
8973 apache 15 0 94800 92M 5704 S 0.0 2.2 0:18 3 httpd
10095 apache 15 0 94796 92M 5984 S 0.0 2.2 0:12 3 httpd
8629 apache 15 0 94728 92M 5672 S 0.0 2.2 0:22 1 httpd
8770 apache 15 0 94700 92M 5828 S 0.0 2.2 0:17 3 httpd
9011 apache 15 0 94600 91M 5856 S 0.0 2.2 0:19 2 httpd
9572 apache 15 0 94576 91M 5980 S 0.0 2.2 0:10 2 httpd
8628 apache 15 0 94508 91M 5532 S 0.0 2.2 0:46 0 httpd
8640 apache 15 0 94412 91M 5672 S 0.0 2.2 0:24 2 httpd
8648 apache 15 0 94140 91M 5192 S 0.0 2.2 0:13 3 httpd
10725 apache 15 0 93812 91M 5436 S 0.0 2.2 0:05 1 httpd
10776 apache 15 0 93252 90M 5672 S 0.0 2.2 0:17 3 httpd
10288 apache 15 0 92380 89M 5324 S 0.0 2.2 0:09 3 httpd
6133 apache 15 0 92584 89M 4864 S 0.0 2.2 0:13 1 httpd
11317 apache 15 0 92260 89M 5132 S 0.2 2.2 0:04 0 httpd
11467 apache 15 0 90876 88M 4872 S 0.0 2.1 0:01 1 httpd
11320 apache 15 0 89888 87M 5092 S 0.0 2.1 0:01 1 httpd
11321 apache 15 0 89848 87M 4832 S 0.0 2.1 0:01 1 httpd
23317 apache 15 0 86656 84M 2076 S 0.0 2.1 0:41 3 httpd
32420 root 16 0 75448 73M 2300 S 0.0 1.8 0:00 0 httpd
6045 root 15 0 75332 72M 2172 S 0.0 1.8 1:03 1 httpd
5657 tomcat4 25 0 71092 69M 9524 S 0.0 1.7 0:03 3 java
5669 tomcat4 15 0 71092 69M 9524 S 0.0 1.7 0:02 0 java
5670 tomcat4 15 0 71092 69M 9524 S 0.0 1.7 0:00 0 java
5671 tomcat4 15 0 71092 69M 9524 S 0.0 1.7 0:00 3 java
5675 tomcat4 25 0 71092 69M 9524 S 0.0 1.7 0:00 2 java
5676 tomcat4 25 0 71092 69M 9524 S 0.0 1.7 0:00 2 java
5677 tomcat4 15 0 71092 69M 9524 S 0.0 1.7 0:01 0 java
5678 tomcat4 15 0 71092 69M 9524 S 0.0 1.7 0:32 1 java
5703 tomcat4 15 0 71092 69M 9524 S 0.0 1.7 0:02 3 java
5704 tomcat4 15 0 71092 69M 9524 S 0.0 1.7 0:00 0 java
5705 tomcat4 15 0 71092 69M 9524 S 0.0 1.7 0:00 3 java
5706 tomcat4 15 0 71092 69M 9524 S 0.0 1.7 0:00 2 java
5707 tomcat4 15 0 71092 69M 9524 S 0.0 1.7 0:00 2 java
5708 tomcat4 15 0 71092 69M 9524 S 0.0 1.7 0:00 2 java
5709 tomcat4 15 0 71092 69M 9524 S 0.0 1.7 0:00 2 java
5710 tomcat4 15 0 71092 69M 9524 S 0.0 1.7 0:02 1 java
5711 tomcat4 15 0 71092 69M 9524 S 0.0 1.7 0:00 2 java

Wrong - you have 1856 (efectively) free. But that is possibly irrelevant.
You were asked to peruse the logs - the top listing is pointless. If you expect people to try to help, provide the info requested.

Most processes getting "killed" (as per the screen shot) seem to be the mysql daemon.

So, it may be that mysql isn't scavenging memory. So could it be some mysql setting?

End

Thanks for the reply guys.

I will keep an eye on the free -m output. Read up a bit on it and what syg00 says makes sense now.

Since you are running RHEL and using RHEL requires a support license, why not ask RH what is going on?