server got hang . when connected through the console , in the screen it is showing error as below continuously . The error is executing as like in a loop .

cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0

lowmem_reserve[]:0 0 12877 12877
Mode 0 Normal free:476588 min127879
inactive:62354598 present :1368978
lowmem_reserve[]:0 0 0 0
Node 0 HighMem free:0kB min128kB
Node 0 DMA:3*4kB 3*8kB 4*16kB 3*32kB
Node 0 HighMem:empty
400547 pagecache pages
swap cache: add 8213770,delete 821809
Free swap =0kB
Total swap=240026632kB

The os
Linux ipl333 2.6.18-92.1.18.el5 #1 SMP Wed Nov 12 09:19:49 EST 2008 x86_64

After reboot the server came UP .

Pleae help me to trouble shoot this issue . What is the root cause for this issue .

You OOMed. Add more RAM or fix whatever was taking up so much RAM.

RAM is sufficient for the sever .. it happened onlly for 3 server . I had 15 servers with same configuration which have no issue like this

Upgrade your console firmware and see what happens.

While checking it was found that it is not due to firmware issue .
WIll it can be due to memory overflow .

If the issue is due to memory leak , can i get a solution to fix the memory leak . What sholud i do to fix the memory leak in the server . Will i get a solution for this immediately since the server is under critical stage.

i have got this error in my server . server got hang and while chekcing in /var/log/messages , this error found . Can i get it analysed .

OS :- Linux 2.6.18-92.1.18.el5 #1 SMP Wed Nov 12 09:19:49 EST 2008 x86_64
how can i identify that which application is taking much memory so that server is getting out of memory.
Call Trace:
[<ffffffff800c0b09>] out_of_memory+0x8e/0x2f5
[<ffffffff8000f263>] __alloc_pages+0x245/0x2ce
[<ffffffff80012946>] __do_page_cache_readahead+0x95/0x1d9
[<ffffffff8004a946>] try_to_del_timer_sync+0x51/0x5a
[<ffffffff8005b6aa>] del_timer_sync+0xc/0x16
[<ffffffff8006380b>] schedule_timeout+0x92/0xad
[<ffffffff8009543e>] process_timeout+0x0/0x5
[<ffffffff800132e5>] filemap_nopage+0x148/0x322
[<ffffffff80008848>] __handle_mm_fault+0x1f8/0xe23
[<ffffffff8006681f>] do_page_fault+0x4cb/0x830
[<ffffffff8000c31a>] _atomic_dec_and_lock+0x39/0x57
[<ffffffff8005dde9>] error_exit+0x0/0x84


Node 0 DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
cpu 1 hot: high 0, batch 1 used:0
cpu 1 cold: high 0, batch 1 used:0
cpu 2 hot: high 0, batch 1 used:0
cpu 2 cold: high 0, batch 1 used:0
cpu 3 hot: high 0, batch 1 used:0
cpu 3 cold: high 0, batch 1 used:0
cpu 4 hot: high 0, batch 1 used:0
cpu 4 cold: high 0, batch 1 used:0
cpu 5 hot: high 0, batch 1 used:0
cpu 5 cold: high 0, batch 1 used:0
cpu 6 hot: high 0, batch 1 used:0
cpu 6 cold: high 0, batch 1 used:0
cpu 7 hot: high 0, batch 1 used:0
cpu 7 cold: high 0, batch 1 used:0
cpu 8 hot: high 0, batch 1 used:0
cpu 8 cold: high 0, batch 1 used:0
cpu 9 hot: high 0, batch 1 used:0
cpu 9 cold: high 0, batch 1 used:0
cpu 10 hot: high 0, batch 1 used:0
cpu 10 cold: high 0, batch 1 used:0
cpu 11 hot: high 0, batch 1 used:0
cpu 11 cold: high 0, batch 1 used:0
cpu 12 hot: high 0, batch 1 used:0
cpu 12 cold: high 0, batch 1 used:0
cpu 13 hot: high 0, batch 1 used:0
cpu 13 cold: high 0, batch 1 used:0
cpu 14 hot: high 0, batch 1 used:0
cpu 14 cold: high 0, batch 1 used:0
cpu 15 hot: high 0, batch 1 used:0
cpu 15 cold: high 0, batch 1 used:0
Node 0 DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:31
cpu 0 cold: high 62, batch 15 used:53
cpu 1 hot: high 186, batch 31 used:168
cpu 1 cold: high 62, batch 15 used:50
cpu 2 hot: high 186, batch 31 used:30
cpu 2 cold: high 62, batch 15 used:57
cpu 3 hot: high 186, batch 31 used:30
cpu 3 cold: high 62, batch 15 used:54
cpu 4 hot: high 186, batch 31 used:141
cpu 4 cold: high 62, batch 15 used:57
cpu 5 hot: high 186, batch 31 used:116
cpu 5 cold: high 62, batch 15 used:48
cpu 6 hot: high 186, batch 31 used:30
cpu 6 cold: high 62, batch 15 used:35
cpu 7 hot: high 186, batch 31 used:78
cpu 7 cold: high 62, batch 15 used:52
cpu 8 hot: high 186, batch 31 used:18
cpu 8 cold: high 62, batch 15 used:60
cpu 9 hot: high 186, batch 31 used:23
cpu 9 cold: high 62, batch 15 used:52
cpu 10 hot: high 186, batch 31 used:52
cpu 10 cold: high 62, batch 15 used:26
cpu 11 hot: high 186, batch 31 used:24
cpu 11 cold: high 62, batch 15 used:33
cpu 12 hot: high 186, batch 31 used:21
cpu 12 cold: high 62, batch 15 used:60
cpu 13 hot: high 186, batch 31 used:176
cpu 13 cold: high 62, batch 15 used:54
cpu 14 hot: high 186, batch 31 used:97
cpu 14 cold: high 62, batch 15 used:55
cpu 15 hot: high 186, batch 31 used:158
cpu 15 cold: high 62, batch 15 used:60
Node 0 Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:38
cpu 0 cold: high 62, batch 15 used:47
cpu 1 hot: high 186, batch 31 used:180
cpu 1 cold: high 62, batch 15 used:51
cpu 2 hot: high 186, batch 31 used:43
cpu 2 cold: high 62, batch 15 used:61
cpu 3 hot: high 186, batch 31 used:171
cpu 3 cold: high 62, batch 15 used:56
cpu 4 hot: high 186, batch 31 used:116
cpu 4 cold: high 62, batch 15 used:55
cpu 5 hot: high 186, batch 31 used:174
cpu 5 cold: high 62, batch 15 used:61
cpu 6 hot: high 186, batch 31 used:20
cpu 6 cold: high 62, batch 15 used:55
cpu 7 hot: high 186, batch 31 used:56
cpu 7 cold: high 62, batch 15 used:53
cpu 8 hot: high 186, batch 31 used:164
cpu 8 cold: high 62, batch 15 used:50
cpu 9 hot: high 186, batch 31 used:171
cpu 9 cold: high 62, batch 15 used:58
cpu 10 hot: high 186, batch 31 used:138
cpu 10 cold: high 62, batch 15 used:58
cpu 11 hot: high 186, batch 31 used:133
cpu 11 cold: high 62, batch 15 used:55
cpu 12 hot: high 186, batch 31 used:143
cpu 12 cold: high 62, batch 15 used:53
cpu 13 hot: high 186, batch 31 used:131
cpu 13 cold: high 62, batch 15 used:60
cpu 14 hot: high 186, batch 31 used:83
cpu 14 cold: high 62, batch 15 used:50
cpu 15 hot: high 186, batch 31 used:43
cpu 15 cold: high 62, batch 15 used:57
Node 0 HighMem per-cpu: empty
Free pages: 68576kB (0kB HighMem)
Active:1953535 inactive:1944513 dirty:0 writeback:0 unstable:0 free:17144 slab:51339 mapped-file:1040 mapped-anon:3602249 pagetables:52410
Node 0 DMA free:11084kB min:8kB low:8kB high:12kB active:0kB inactive:0kB present:10724kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 3253 16130 16130
Node 0 DMA32 free:52656kB min:3276kB low:4092kB high:4912kB active:1556372kB inactive:1488548kB present:3331568kB pages_scanned:3909453756 all_unreclaimable? ye es
lowmem_reserve[]: 0 0 12877 12877
Node 0 Normal free:4836kB min:12972kB low:16212kB high:19456kB active:6257468kB inactive:6289600kB present:13186560kB pages_scanned:15668059103 all_unreclaimabl le? yes
lowmem_reserve[]: 0 0 0 0
Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 3*4kB 4*8kB 2*16kB 4*32kB 2*64kB 2*128kB 1*256kB 0*512kB 2*1024kB 0*2048kB 2*4096kB = 11084kB
Node 0 DMA32: 10*4kB 1*8kB 0*16kB 0*32kB 0*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 12*4096kB = 52656kB
Node 0 Normal: 15*4kB 5*8kB 0*16kB 0*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 4836kB
Node 0 HighMem: empty
300890 pagecache pages
Swap cache: add 15550870, delete 15549715, find 18024622/18345801, race 0+31
Free swap = 0kB
Total swap = 24002632kB
oracle invoked oom-killer: gfp_mask=0xd0, order=0, oomkilladj=0


Call Trace:
[<ffffffff800c0b09>] out_of_memory+0x8e/0x2f5
[<ffffffff8000f263>] __alloc_pages+0x245/0x2ce
[<ffffffff8003c1ad>] __get_free_pages+0xe/0x71
[<ffffffff8009b6ac>] get_pid_task+0x5/0xf
[<ffffffff800fbdc8>] proc_info_read+0x4d/0xb9
[<ffffffff8000b337>] vfs_read+0xcb/0x171
[<ffffffff80011715>] sys_read+0x45/0x6e
[<ffffffff8005d28d>] tracesys+0xd5/0xe0


Node 0 DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
cpu 1 hot: high 0, batch 1 used:0
cpu 1 cold: high 0, batch 1 used:0
cpu 2 hot: high 0, batch 1 used:0
cpu 2 cold: high 0, batch 1 used:0
cpu 3 hot: high 0, batch 1 used:0
cpu 3 cold: high 0, batch 1 used:0
cpu 4 hot: high 0, batch 1 used:0
cpu 4 cold: high 0, batch 1 used:0
cpu 5 hot: high 0, batch 1 used:0
cpu 5 cold: high 0, batch 1 used:0
cpu 6 hot: high 0, batch 1 used:0
cpu 6 cold: high 0, batch 1 used:0
cpu 7 hot: high 0, batch 1 used:0
cpu 7 cold: high 0, batch 1 used:0
cpu 8 hot: high 0, batch 1 used:0
cpu 8 cold: high 0, batch 1 used:0
cpu 9 hot: high 0, batch 1 used:0
cpu 9 cold: high 0, batch 1 used:0
cpu 10 hot: high 0, batch 1 used:0
cpu 10 cold: high 0, batch 1 used:0
cpu 11 hot: high 0, batch 1 used:0
cpu 11 cold: high 0, batch 1 used:0
cpu 12 hot: high 0, batch 1 used:0
cpu 12 cold: high 0, batch 1 used:0
cpu 13 hot: high 0, batch 1 used:0
cpu 13 cold: high 0, batch 1 used:0
cpu 14 hot: high 0, batch 1 used:0
cpu 14 cold: high 0, batch 1 used:0
cpu 15 hot: high 0, batch 1 used:0
cpu 15 cold: high 0, batch 1 used:0
Node 0 DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:31
cpu 0 cold: high 62, batch 15 used:53
cpu 1 hot: high 186, batch 31 used:168
cpu 1 cold: high 62, batch 15 used:50
cpu 2 hot: high 186, batch 31 used:30
cpu 2 cold: high 62, batch 15 used:57
cpu 3 hot: high 186, batch 31 used:30
cpu 3 cold: high 62, batch 15 used:54
cpu 4 hot: high 186, batch 31 used:141
cpu 4 cold: high 62, batch 15 used:57
cpu 5 hot: high 186, batch 31 used:116
cpu 5 cold: high 62, batch 15 used:48
cpu 6 hot: high 186, batch 31 used:30
cpu 6 cold: high 62, batch 15 used:35
cpu 7 hot: high 186, batch 31 used:78
cpu 7 cold: high 62, batch 15 used:52
cpu 8 hot: high 186, batch 31 used:18
cpu 8 cold: high 62, batch 15 used:60
cpu 9 hot: high 186, batch 31 used:23
cpu 9 cold: high 62, batch 15 used:52
cpu 10 hot: high 186, batch 31 used:52
cpu 10 cold: high 62, batch 15 used:26
cpu 11 hot: high 186, batch 31 used:24
cpu 11 cold: high 62, batch 15 used:33
cpu 12 hot: high 186, batch 31 used:21
cpu 12 cold: high 62, batch 15 used:60
cpu 13 hot: high 186, batch 31 used:176
cpu 13 cold: high 62, batch 15 used:54
cpu 14 hot: high 186, batch 31 used:97
cpu 14 cold: high 62, batch 15 used:55
cpu 15 hot: high 186, batch 31 used:158
cpu 15 cold: high 62, batch 15 used:60
Node 0 Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:38
cpu 0 cold: high 62, batch 15 used:47
cpu 1 hot: high 186, batch 31 used:180
cpu 1 cold: high 62, batch 15 used:51
cpu 2 hot: high 186, batch 31 used:43
cpu 2 cold: high 62, batch 15 used:61
cpu 3 hot: high 186, batch 31 used:171
cpu 3 cold: high 62, batch 15 used:56
cpu 4 hot: high 186, batch 31 used:116
cpu 4 cold: high 62, batch 15 used:55
cpu 5 hot: high 186, batch 31 used:174
cpu 5 cold: high 62, batch 15 used:61
cpu 6 hot: high 186, batch 31 used:20
cpu 6 cold: high 62, batch 15 used:55
cpu 7 hot: high 186, batch 31 used:56
cpu 7 cold: high 62, batch 15 used:53
cpu 8 hot: high 186, batch 31 used:164
cpu 8 cold: high 62, batch 15 used:50
cpu 9 hot: high 186, batch 31 used:171
cpu 9 cold: high 62, batch 15 used:58
cpu 10 hot: high 186, batch 31 used:138
cpu 10 cold: high 62, batch 15 used:58
cpu 11 hot: high 186, batch 31 used:133
cpu 11 cold: high 62, batch 15 used:55
cpu 12 hot: high 186, batch 31 used:143
cpu 12 cold: high 62, batch 15 used:53
cpu 13 hot: high 186, batch 31 used:131
cpu 13 cold: high 62, batch 15 used:60
cpu 14 hot: high 186, batch 31 used:83
cpu 14 cold: high 62, batch 15 used:50
cpu 15 hot: high 186, batch 31 used:43
cpu 15 cold: high 62, batch 15 used:57
Node 0 HighMem per-cpu: empty
Free pages: 68576kB (0kB HighMem)
Active:1953725 inactive:1944404 dirty:0 writeback:0 unstable:0 free:17144 slab:51339 mapped-file:1040 mapped-anon:3602249 pagetables:52410
Node 0 DMA free:11084kB min:8kB low:8kB high:12kB active:0kB inactive:0kB present:10724kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 3253 16130 16130
Node 0 DMA32 free:52656kB min:3276kB low:4092kB high:4912kB active:1556608kB inactive:1488568kB present:3331568kB pages_scanned:3909673248 all_unreclaimable? ye es
lowmem_reserve[]: 0 0 12877 12877
Node 0 Normal free:4836kB min:12972kB low:16212kB high:19456kB active:6257876kB inactive:6289408kB present:13186560kB pages_scanned:15668930596 all_unreclaimabl le? yes
lowmem_reserve[]: 0 0 0 0
Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 3*4kB 4*8kB 2*16kB 4*32kB 2*64kB 2*128kB 1*256kB 0*512kB 2*1024kB 0*2048kB 2*4096kB = 11084kB
Node 0 DMA32: 10*4kB 1*8kB 0*16kB 0*32kB 0*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 12*4096kB = 52656kB
Node 0 Normal: 15*4kB 5*8kB 0*16kB 0*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 4836kB
Node 0 HighMem: empty
300890 pagecache pages
Swap cache: add 15550870, delete 15549715, find 18024622/18345801, race 0+31
Free swap = 0kB
Total swap = 24002632kB

Looks like Oracle. But you should ready use a reporting tool like top or sar to watch the server's memory usage.

Hi,
i'm also suspecting oracle. How can i use the SAR facillity . how can i check whether the utility is installed and properly working in my server.
What are the steps ?