We have a system which periodically (about ones a month) is generating a oom-kill. The server is running an Oracle 10.2.0.3 database and this is the process which is being killed.

The server has 4 GB of memory.

What we find very strange is that 'committed_AS' in /proc/meminfo is very large and seems to be increasing. At the moment it is:

[oracle@srv-lnx-db-02 ~]$ cat /proc/meminfo | grep Committed_AS
Committed_AS: 10364820 kB
[oracle@srv-lnx-db-02 ~]$

Is there a way to find what process is allocating so much memory (committed memory)?

Some more info is:

The top command shows:
top - 10:15:58 up 37 days, 18:01, 3 users, load average: 0.40, 0.38, 0.40
Tasks: 427 total, 1 running, 426 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.9% us, 1.9% sy, 0.0% ni, 92.5% id, 3.7% wa, 0.0% hi, 0.0% si
Mem: 4040812k total, 3971676k used, 69136k free, 29840k buffers
Swap: 2097144k total, 734040k used, 1363104k free, 2101912k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31848 oracle 17 0 6412 1276 772 R 1 0.0 0:00.10 top
17204 oracle 16 0 1578m 73m 72m S 0 1.9 0:32.55 oracle
14498 oracle 16 0 1591m 176m 163m S 0 4.5 0:08.82 oracle
28362 oracle 16 0 1591m 155m 143m S 0 3.9 0:05.39 oracle
9059 oracle 16 0 1591m 140m 127m S 0 3.5 0:03.52 oracle
15574 oracle 15 0 1581m 54m 49m S 0 1.4 0:00.30 oracle
1 root 15 0 4752 468 436 S 0 0.0 0:57.61 init[/SIZE]

/var/messages reports:
Apr 8 14:35:06 srv-lnx-db-02 kernel: oom-killer: gfp_mask=0x1d2
Apr 8 14:35:07 srv-lnx-db-02 kernel: Mem-info:
Apr 8 14:35:07 srv-lnx-db-02 kernel: Node 0 DMA per-cpu:
Apr 8 14:35:07 srv-lnx-db-02 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 8 14:35:07 srv-lnx-db-02 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 8 14:35:07 srv-lnx-db-02 kernel: cpu 1 hot: low 2, high 6, batch 1
Apr 8 14:35:07 srv-lnx-db-02 kernel: cpu 1 cold: low 0, high 2, batch 1
Apr 8 14:35:07 srv-lnx-db-02 kernel: Node 0 Normal per-cpu:
Apr 8 14:35:07 srv-lnx-db-02 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 8 14:35:07 srv-lnx-db-02 kernel: cpu 0 cold: low 0, high 32, batch 16

The free command shows:
[oracle@srv-lnx-db-02 ~]$ free
total used free shared buffers cached
Mem: 4040812 3908644 132168 0 61984 2062760
-/+ buffers/cache: 1783900 2256912
Swap: 2031608 524584 1507024
[oracle@srv-lnx-db-02 ~]$

The vmstat command shows:
oracle@srv-lnx-db-01 ~]$ vmstat 3
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 734032 48072 32012 2112932 1 1 1553 344 1 1 4 2 85 9
0 0 734032 42624 32064 2112948 0 0 212 369 2508 3870 12 2 81 5
0 0 734032 37944 32136 2112944 0 0 98 670 1321 2729 5 3 88 5
0 0 734032 33136 32192 2112956 0 0 110 281 1338 2681 4 2 90 5
0 0 734032 47384 32240 2112976 0 0 199 401 1414 2887 5 2 88 5
2 0 734032 46960 32300 2112984 0 0 120 311 1376 2590 3 1 92 4
0 0 734032 47112 32440 2112980 0 0 210 466 1488 3041 7 4 74 15
0 0 734032 46856 32496 2112992 0 0 126 523 1412 2820 6 2 85 7
0 0 734032 46568 32548 2112940 0 0 111 199 1174 2500 3 2 91 4


Thanks,

Thormk

You don't say what version of Linux you're running. Have you already tuned your kernel, per the Oracle install document(s), as they apply to Linux? There are several parameters that you have to tweak first.

Reference these docs:

http://oss.oracle.com/projects/olt/d...stCoverage.pdf
http://wiki.oracle.com/thread/128488...+Memory+Tuning

If you don't tweak the kernel parameters correctly, you'll have problems. You'll also have problems if you don't have enough swap space defined. There's a whole list of things you have to do to get Oracle running right, along with the correct kernel level. The oom_killer will hit you if things aren't done right, or if your DBA's are doing something incorrect.

Hi!

We are running Red Hat Enterprise Linux ES release 4 (Nahant Update 4).

I did not install the system so I need to check how the kernel is tuned for Oracle. I dont have access to the system right now, but will update the thread as soon as I have checked it.

Is there a way to see what processes have been committed memory and how much since /proc/meminfo shows Committed_AS: 10364820 kB?

Thanks

Not that I am aware of, other than the /proc/meminfo file.

Oracle is a bit twitchy if it's not set up right, and if the kernel parameters aren't set. Also, check with your DBA's, and see what they're up to. If they're trying to merge/sort/do-something-with a 200GB database, and you're running off 4 GB of system RAM with 512 MB of swap, chances are you'll have problems.

They have to set parameters within Oracle too, to make it play nice on the host OS.