LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 03-01-2008, 08:09 AM   #1
ginda
Member
 
Registered: Mar 2004
Distribution: SUSE8.2, 9.2, Knoppix
Posts: 322

Rep: Reputation: 31
Oom killing java every morning at same time?


Hi Guys

A Red Hat EL 4 web server is having problems were Java is being killed at the same time every morning around 2am, please see extract from messages log below. Can anybody tell me why this is happening, the only thing i have found is the load average went up to about 14 which is very high especially when the server should be doing nothing.

Mar 1 02:08:57 SERVER1 kernel: HighMem per-cpu:
Mar 1 02:08:57 SERVER1 kernel: cpu 0 hot: low 32, high 96, batch 16
Mar 1 02:08:57 SERVER1 kernel: cpu 0 cold: low 0, high 32, batch 16
Mar 1 02:08:57 SERVER1 kernel: cpu 1 hot: low 32, high 96, batch 16
Mar 1 02:08:57 SERVER1 kernel: cpu 1 cold: low 0, high 32, batch 16
Mar 1 02:08:57 SERVER1 kernel:
Mar 1 02:08:57 SERVER1 kernel: Free pages: 2664kB (1600kB HighMem)
Mar 1 02:08:57 SERVER1 kernel: Active:258402 inactive:1263813 dirty:0 writeback:177214 unstable:180324 free:666 slab:18560 mapped:258514 pagetables:11577
Mar 1 02:08:57 SERVER1 kernel: DMA free:16kB min:16kB low:32kB high:48kB active:0kB inactive:11928kB present:16384kB
Mar 1 02:08:57 SERVER1 kernel: protections[]: 0 0 0
Mar 1 02:08:57 SERVER1 kernel: Normal free:1048kB min:936kB low:1872kB high:2808kB active:1820kB inactive:758000kB present:901120kB
Mar 1 02:08:57 SERVER1 kernel: protections[]: 0 0 0
Mar 1 02:08:57 SERVER1 kernel: HighMem free:1600kB min:512kB low:1024kB high:1536kB active:1031788kB inactive:4285424kB present:5898240kB
Mar 1 02:08:57 SERVER1 kernel: protections[]: 0 0 0
Mar 1 02:08:57 SERVER1 kernel: DMA: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 16kB
Mar 1 02:08:57 SERVER1 kernel: Normal: 18*4kB 2*8kB 4*16kB 0*32kB 2*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1048kB
Mar 1 02:08:57 SERVER1 kernel: HighMem: 4*4kB 4*8kB 3*16kB 9*32kB 5*64kB 3*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 1600kB
Mar 1 02:08:57 SERVER1 kernel: Swap cache: add 922, delete 731, find 140/172, race 0+0
Mar 1 02:08:57 SERVER1 kernel: Out of Memory: Killed process 4000 (java).
Mar 1 02:08:57 SERVER1 kernel: oom-killer: gfp_mask=0xd0
Mar 1 02:08:57 SERVER1 kernel: DMA per-cpu:
Mar 1 02:08:57 SERVER1 kernel: cpu 0 hot: low 2, high 6, batch 1
Mar 1 02:08:57 SERVER1 kernel: cpu 0 cold: low 0, high 2, batch 1
Mar 1 02:08:57 SERVER1 kernel: cpu 1 hot: low 2, high 6, batch 1
Mar 1 02:08:57 SERVER1 kernel: cpu 1 cold: low 0, high 2, batch 1
Mar 1 02:08:57 SERVER1 kernel: Normal per-cpu:
Mar 1 02:08:57 SERVER1 kernel: cpu 0 hot: low 32, high 96, batch 16
Mar 1 02:08:57 SERVER1 kernel: cpu 0 cold: low 0, high 32, batch 16
Mar 1 02:08:58 SERVER1 kernel: cpu 1 hot: low 32, high 96, batch 16
Mar 1 02:08:58 SERVER1 kernel: cpu 1 cold: low 0, high 32, batch 16
Mar 1 02:08:58 SERVER1 kernel: HighMem per-cpu:
Mar 1 02:08:58 SERVER1 kernel: cpu 0 hot: low 32, high 96, batch 16
Mar 1 02:08:58 SERVER1 kernel: cpu 0 cold: low 0, high 32, batch 16
Mar 1 02:08:58 SERVER1 kernel: cpu 1 hot: low 32, high 96, batch 16
Mar 1 02:08:58 SERVER1 kernel: cpu 1 cold: low 0, high 32, batch 16
Mar 1 02:08:58 SERVER1 kernel:
Mar 1 02:08:58 SERVER1 kernel: Free pages: 6836kB (5824kB HighMem)
Mar 1 02:08:58 SERVER1 kernel: Active:255257 inactive:1265321 dirty:0 writeback:176528 unstable:181074 free:1709 slab:18540 mapped:254805 pagetables:11510
Mar 1 02:08:58 SERVER1 kernel: DMA free:20kB min:16kB low:32kB high:48kB active:352kB inactive:11424kB present:16384kB
Mar 1 02:08:58 SERVER1 kernel: protections[]: 0 0 0
Mar 1 02:08:58 SERVER1 kernel: Normal free:992kB min:936kB low:1872kB high:2808kB active:3732kB inactive:753624kB present:901120kB
Mar 1 02:08:58 SERVER1 kernel: protections[]: 0 0 0
Mar 1 02:08:58 SERVER1 kernel: HighMem free:5824kB min:512kB low:1024kB high:1536kB active:1016944kB inactive:4296236kB present:5898240kB
Mar 1 02:08:58 SERVER1 kernel: protections[]: 0 0 0



Thanks in advance
 
Old 03-01-2008, 10:15 AM   #2
rjlee
Senior Member
 
Registered: Jul 2004
Distribution: Ubuntu 7.04
Posts: 1,990

Rep: Reputation: 67
oom-killer is a kernel (operating system) program that runs to free up memory if it becomes so dangerously low as to prevent the system from being usable.

Unfortunately this gives little indication as to the cause of the problem. But there are two possibilities here; either the program legitimately needs more memory or something is running to gobble up a huge amount of memory on the server by about 2AM, possibly a disk search index update or similar.

Either way, upgrading the machine with more physical memory should fix this issue, but that's a rather expensive option.

The first thing to check is that your system has as much memory as it needs. This includes the size of your swap space on disk. If you open up a terminal and type
Code:
more /proc/meminfo
then you should see various memory statistics.

The first thing to check is that SwapTotal is not 0. If it is, then you have swap memory turned off; you will need to create a swap partition (or file) and then enable it.

Next, when the machine is idle after a fair amount of use, SwapFree should be a fairly high number, say at least half of SwapTotal (the actual minimum will depend on the machine. If this hits 0 then you may start to see oom-killer kicking in). If not then your swap space may be too small; enlarge your swap partition make a second swap partition (or file) and then enable it.

Generally, the sum of MemFree + Buffers + Cached should be a fairly big number, anything as low as a few thousand suggests that your system may be having memory troubles.


The other approach is that something may be gobbling up memory at 2AM in the morning. To see what is using the memory, you could try logging the top resource-hungry processes that are running at the time.

Open a console and type
Code:
crontab -e
Now add this line to the end of the file and save it:
Code:
* 1-2 * * * top -b -n 1 > ~/top.$(date +%Y.%m.%d.%H.%M.%S)
This will run once a minute between 1 and 3 AM and create files in your home directory of the form "top.date". You should then be able to at least see which program was using up all the memory, by looking at the file created just before a crash.

Hope that helps:

—Robert Lee
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Killing a process after x amount of time... virid Linux - Newbie 3 03-17-2006 06:13 AM
oom-killer is killing my programs on FC3 abarclay Fedora 1 03-08-2005 09:14 AM
Kernel OOM killing more than it should snipersock Linux - Software 1 12-01-2004 08:03 PM
help with killing all java process idandl Linux - Newbie 2 02-02-2004 05:32 AM
Java is killing me!!! sasquash Linux - Software 3 11-08-2003 06:25 PM


All times are GMT -5. The time now is 02:00 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration