LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 04-21-2008, 02:13 PM   #1
thormk
LQ Newbie
 
Registered: Apr 2008
Posts: 2

Rep: Reputation: 0
what process is allocating memory and causing oom-kill?


We have a system which periodically (about ones a month) is generating a oom-kill. The server is running an Oracle 10.2.0.3 database and this is the process which is being killed.

The server has 4 GB of memory.

What we find very strange is that 'committed_AS' in /proc/meminfo is very large and seems to be increasing. At the moment it is:

[oracle@srv-lnx-db-02 ~]$ cat /proc/meminfo | grep Committed_AS
Committed_AS: 10364820 kB
[oracle@srv-lnx-db-02 ~]$

Is there a way to find what process is allocating so much memory (committed memory)?

Some more info is:

The top command shows:
top - 10:15:58 up 37 days, 18:01, 3 users, load average: 0.40, 0.38, 0.40
Tasks: 427 total, 1 running, 426 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.9% us, 1.9% sy, 0.0% ni, 92.5% id, 3.7% wa, 0.0% hi, 0.0% si
Mem: 4040812k total, 3971676k used, 69136k free, 29840k buffers
Swap: 2097144k total, 734040k used, 1363104k free, 2101912k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31848 oracle 17 0 6412 1276 772 R 1 0.0 0:00.10 top
17204 oracle 16 0 1578m 73m 72m S 0 1.9 0:32.55 oracle
14498 oracle 16 0 1591m 176m 163m S 0 4.5 0:08.82 oracle
28362 oracle 16 0 1591m 155m 143m S 0 3.9 0:05.39 oracle
9059 oracle 16 0 1591m 140m 127m S 0 3.5 0:03.52 oracle
15574 oracle 15 0 1581m 54m 49m S 0 1.4 0:00.30 oracle
1 root 15 0 4752 468 436 S 0 0.0 0:57.61 init[/SIZE]

/var/messages reports:
Apr 8 14:35:06 srv-lnx-db-02 kernel: oom-killer: gfp_mask=0x1d2
Apr 8 14:35:07 srv-lnx-db-02 kernel: Mem-info:
Apr 8 14:35:07 srv-lnx-db-02 kernel: Node 0 DMA per-cpu:
Apr 8 14:35:07 srv-lnx-db-02 kernel: cpu 0 hot: low 2, high 6, batch 1
Apr 8 14:35:07 srv-lnx-db-02 kernel: cpu 0 cold: low 0, high 2, batch 1
Apr 8 14:35:07 srv-lnx-db-02 kernel: cpu 1 hot: low 2, high 6, batch 1
Apr 8 14:35:07 srv-lnx-db-02 kernel: cpu 1 cold: low 0, high 2, batch 1
Apr 8 14:35:07 srv-lnx-db-02 kernel: Node 0 Normal per-cpu:
Apr 8 14:35:07 srv-lnx-db-02 kernel: cpu 0 hot: low 32, high 96, batch 16
Apr 8 14:35:07 srv-lnx-db-02 kernel: cpu 0 cold: low 0, high 32, batch 16

The free command shows:
[oracle@srv-lnx-db-02 ~]$ free
total used free shared buffers cached
Mem: 4040812 3908644 132168 0 61984 2062760
-/+ buffers/cache: 1783900 2256912
Swap: 2031608 524584 1507024
[oracle@srv-lnx-db-02 ~]$

The vmstat command shows:
oracle@srv-lnx-db-01 ~]$ vmstat 3
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 734032 48072 32012 2112932 1 1 1553 344 1 1 4 2 85 9
0 0 734032 42624 32064 2112948 0 0 212 369 2508 3870 12 2 81 5
0 0 734032 37944 32136 2112944 0 0 98 670 1321 2729 5 3 88 5
0 0 734032 33136 32192 2112956 0 0 110 281 1338 2681 4 2 90 5
0 0 734032 47384 32240 2112976 0 0 199 401 1414 2887 5 2 88 5
2 0 734032 46960 32300 2112984 0 0 120 311 1376 2590 3 1 92 4
0 0 734032 47112 32440 2112980 0 0 210 466 1488 3041 7 4 74 15
0 0 734032 46856 32496 2112992 0 0 126 523 1412 2820 6 2 85 7
0 0 734032 46568 32548 2112940 0 0 111 199 1174 2500 3 2 91 4


Thanks,

Thormk
 
Old 04-21-2008, 02:26 PM   #2
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 22,725

Rep: Reputation: 6257Reputation: 6257Reputation: 6257Reputation: 6257Reputation: 6257Reputation: 6257Reputation: 6257Reputation: 6257Reputation: 6257Reputation: 6257Reputation: 6257
You don't say what version of Linux you're running. Have you already tuned your kernel, per the Oracle install document(s), as they apply to Linux? There are several parameters that you have to tweak first.

Reference these docs:

http://oss.oracle.com/projects/olt/d...stCoverage.pdf
http://wiki.oracle.com/thread/128488...+Memory+Tuning

If you don't tweak the kernel parameters correctly, you'll have problems. You'll also have problems if you don't have enough swap space defined. There's a whole list of things you have to do to get Oracle running right, along with the correct kernel level. The oom_killer will hit you if things aren't done right, or if your DBA's are doing something incorrect.
 
Old 04-21-2008, 02:41 PM   #3
thormk
LQ Newbie
 
Registered: Apr 2008
Posts: 2

Original Poster
Rep: Reputation: 0
Hi!

We are running Red Hat Enterprise Linux ES release 4 (Nahant Update 4).

I did not install the system so I need to check how the kernel is tuned for Oracle. I dont have access to the system right now, but will update the thread as soon as I have checked it.

Is there a way to see what processes have been committed memory and how much since /proc/meminfo shows Committed_AS: 10364820 kB?

Thanks
 
Old 04-22-2008, 08:40 AM   #4
TB0ne
LQ Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 22,725

Rep: Reputation: 6257Reputation: 6257Reputation: 6257Reputation: 6257Reputation: 6257Reputation: 6257Reputation: 6257Reputation: 6257Reputation: 6257Reputation: 6257Reputation: 6257
Not that I am aware of, other than the /proc/meminfo file.

Oracle is a bit twitchy if it's not set up right, and if the kernel parameters aren't set. Also, check with your DBA's, and see what they're up to. If they're trying to merge/sort/do-something-with a 200GB database, and you're running off 4 GB of system RAM with 512 MB of swap, chances are you'll have problems.

They have to set parameters within Oracle too, to make it play nice on the host OS.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Out of memory (oom) killer causes system crash? BusyBeeBop Linux - Software 6 06-02-2008 01:42 AM
oom kill, need suggestions bee2643 Linux - Newbie 7 08-08-2006 02:26 PM
oom-killer: How to set priorities to kill processes guarriman Linux - Security 1 01-31-2006 09:03 AM
Allocating Memory in the kernel arunachalam Linux - Software 4 10-12-2005 08:51 AM
allocating memory eshwar_ind Programming 6 02-26-2004 07:06 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 11:25 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration