Hi Below is a part from a message log and all I could manage is (I hope) a frmweb process is having some issue and causing the system to run out of memory. Can some one explain whats going on and to trace the issue

I am running Redhat 5.5 64 bit, Oracle forms and report 11g2

Aug 26 12:47:43 server01v kernel: INFO: task frmweb:14379 blocked for more than 120 seconds.
Aug 26 12:47:44 server01v kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 26 12:47:44 server01v kernel: frmweb D 0000000000000014 0 14379 13596 0x00000080
Aug 26 12:47:44 server01v kernel: ffff88040fc73d28 0000000000000086 0000000000000000 0000000000000000
Aug 26 12:47:44 server01v kernel: ffff88007efde6c0 ffff88046dcb6440 ffff88007efdea98 000000018efb7197
Aug 26 12:47:44 server01v kernel: ffffffff0fc73cf8 0000000000000000 0000000000000000 ffff88006c061cc8
Aug 26 12:47:44 server01v kernel: Call Trace:
Aug 26 12:47:44 server01v kernel: [<ffffffff81058dbd>] exit_mm+0x9a/0x127
Aug 26 12:47:44 server01v kernel: [<ffffffff8105a32c>] do_exit+0x1f8/0x685
Aug 26 12:47:44 server01v kernel: [<ffffffff81067710>] ? freezing+0x13/0x15
Aug 26 12:47:44 server01v kernel: [<ffffffff8105a859>] sys_exit_group+0x0/0x1b
Aug 26 12:47:44 server01v kernel: [<ffffffff8106ab2b>] get_signal_to_deliver+0x303/0x328
Aug 26 12:47:44 server01v kernel: [<ffffffff810111f2>] do_notify_resume+0x90/0x6d7
Aug 26 12:47:48 server01v kernel: [<ffffffff8104471c>] ? pick_next_task_fair+0xa5/0xb1
Aug 26 12:47:51 server01v kernel: [<ffffffff81121cdb>] ? path_put+0x22/0x27
Aug 26 12:47:55 server01v kernel: [<ffffffff8101207e>] int_signal+0x12/0x17
Aug 26 12:47:57 server01v kernel: INFO: task frmweb:14380 blocked for more than 120 seconds.
Aug 26 12:48:00 server01v kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 26 12:48:01 server01v kernel: frmweb D 0000000000000007 0 14380 13596 0x00000080
Aug 26 12:48:03 server01v kernel: ffff88027f96fd28 0000000000000086 000000007f96fd08 0000000000000000
Aug 26 12:48:05 server01v kernel: ffff88005e5a2880 ffff88046e696840 ffff88005e5a2c58 ffffffff810425c7
Aug 26 12:48:08 server01v kernel: ffff88027f96fcf8 ffffffff810425df ffff88027f96fd08 ffff88006c061cc8
Aug 26 12:48:09 server01v kernel: Call Trace:
Aug 26 12:48:11 server01v kernel: [<ffffffff810425c7>] ? need_resched+0x23/0x2d
Aug 26 12:48:15 server01v kernel: [<ffffffff810425df>] ? should_resched+0xe/0x2f
Aug 26 12:48:17 server01v kernel: [<ffffffff81058dbd>] exit_mm+0x9a/0x127
Aug 26 12:48:18 server01v kernel: [<ffffffff8105a32c>] do_exit+0x1f8/0x685
Aug 26 12:48:20 server01v kernel: [<ffffffff81067710>] ? freezing+0x13/0x15
Aug 26 12:48:22 server01v kernel: [<ffffffff8105a859>] sys_exit_group+0x0/0x1b
Aug 26 12:48:23 server01v kernel: [<ffffffff8106ab2b>] get_signal_to_deliver+0x303/0x328
Aug 26 12:48:24 server01v kernel: [<ffffffff810111f2>] do_notify_resume+0x90/0x6d7
Aug 26 12:48:28 server01v kernel: [<ffffffff8104471c>] ? pick_next_task_fair+0xa5/0xb1
Aug 26 12:48:29 server01v kernel: [<ffffffff81121cdb>] ? path_put+0x22/0x27
Aug 26 12:48:30 server01v kernel: [<ffffffff8101207e>] int_signal+0x12/0x17
Aug 26 12:48:30 server01v kernel: INFO: task frmweb:14381 blocked for more than 120 seconds.
Aug 26 12:48:30 server01v kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 26 12:48:31 server01v kernel: frmweb D ffff88046b054800 0 14381 13596 0x00000080
Aug 26 12:48:32 server01v kernel: ffff880299e47d28 0000000000000086 0000000099e47d08 0000000000000000
Aug 26 12:48:36 server01v kernel: ffff8802b3c64300 ffff8802be428340 ffff8802b3c646d8 ffffffff810425c7
Aug 26 12:48:37 server01v kernel: ffff880299e47cf8 ffffffff810425df ffff880299e47d08 ffff88006c061cc8
Aug 26 12:48:40 server01v kernel: Call Trace:
Aug 26 12:48:41 server01v kernel: [<ffffffff810425c7>] ? need_resched+0x23/0x2d
Aug 26 12:48:42 server01v kernel: [<ffffffff810425df>] ? should_resched+0xe/0x2f
Aug 26 12:48:43 server01v kernel: [<ffffffff81058dbd>] exit_mm+0x9a/0x127
Aug 26 12:48:44 server01v kernel: [<ffffffff8105a32c>] do_exit+0x1f8/0x685
Aug 26 12:48:48 server01v kernel: [<ffffffff81067710>] ? freezing+0x13/0x15
Aug 26 12:48:49 server01v kernel: [<ffffffff8105a859>] sys_exit_group+0x0/0x1b
Aug 26 12:48:53 server01v kernel: [<ffffffff8106ab2b>] get_signal_to_deliver+0x303/0x328
Aug 26 12:48:56 server01v kernel: [<ffffffff810111f2>] do_notify_resume+0x90/0x6d7
Aug 26 12:48:59 server01v kernel: [<ffffffff8104471c>] ? pick_next_task_fair+0xa5/0xb1
Aug 26 12:48:59 server01v kernel: [<ffffffff81121cdb>] ? path_put+0x22/0x27
Aug 26 12:48:59 server01v kernel: [<ffffffff8101207e>] int_signal+0x12/0x17
Aug 26 12:48:59 server01v kernel: INFO: task frmweb:14382 blocked for more than 120 seconds.
Aug 26 12:49:02 server01v kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 26 12:49:02 server01v kernel: frmweb D 0000000000000012 0 14382 13596 0x00000080
Aug 26 12:49:03 server01v kernel: ffff88028856fd28 0000000000000086 000000008856fd08 0000000000000000
Aug 26 12:49:06 server01v kernel: ffff88035a69e680 ffff8804680121c0 ffff88035a69ea58 ffffffff810425c7
Aug 26 12:49:07 server01v kernel: ffff88028856fcf8 ffffffff810425df ffff88028856fd08 ffff88006c061cc8
Aug 26 12:49:09 server01v kernel: Call Trace:
Aug 26 12:49:10 server01v kernel: [<ffffffff810425c7>] ? need_resched+0x23/0x2d
Aug 26 12:49:10 server01v kernel: [<ffffffff810425df>] ? should_resched+0xe/0x2f
Aug 26 12:49:11 server01v kernel: [<ffffffff81058dbd>] exit_mm+0x9a/0x127
Aug 26 12:49:15 server01v kernel: [<ffffffff8105a32c>] do_exit+0x1f8/0x685
Aug 26 12:49:19 server01v kernel: [<ffffffff81067710>] ? freezing+0x13/0x15
Aug 26 12:49:20 server01v kernel: [<ffffffff8105a859>] sys_exit_group+0x0/0x1b
Aug 26 12:49:20 server01v kernel: [<ffffffff8106ab2b>] get_signal_to_deliver+0x303/0x328
Aug 26 12:49:20 server01v kernel: [<ffffffff810111f2>] do_notify_resume+0x90/0x6d7
Aug 26 12:49:25 server01v kernel: [<ffffffff8104471c>] ? pick_next_task_fair+0xa5/0xb1
Aug 26 12:49:28 server01v kernel: [<ffffffff81121cdb>] ? path_put+0x22/0x27
Aug 26 12:49:29 server01v kernel: [<ffffffff8101207e>] int_signal+0x12/0x17
Aug 26 12:49:31 server01v kernel: INFO: task frmweb:14383 blocked for more than 120 seconds.
Aug 26 12:49:31 server01v kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 26 12:49:31 server01v kernel: frmweb D 0000000000000011 0 14383 13596 0x00000080
Aug 26 12:49:31 server01v kernel: ffff8802f7415d28 0000000000000086 00000000f7415d08 0000000000000000
Aug 26 12:49:32 server01v kernel: ffff8804256d8200 ffff88046dc34140 ffff8804256d85d8 ffffffff810425c7
Aug 26 12:49:32 server01v kernel: ffff8802f7415cf8 ffffffff810425df ffff8802f7415d08 ffff88006c061cc8
Aug 26 12:49:32 server01v kernel: Call Trace:
Aug 26 12:49:32 server01v kernel: [<ffffffff810425c7>] ? need_resched+0x23/0x2d
Aug 26 12:49:32 server01v kernel: [<ffffffff810425df>] ? should_resched+0xe/0x2f
Aug 26 12:49:32 server01v kernel: [<ffffffff81058dbd>] exit_mm+0x9a/0x127
Aug 26 12:49:33 server01v kernel: [<ffffffff8105a32c>] do_exit+0x1f8/0x685
Aug 26 12:49:34 server01v kernel: [<ffffffff810128ae>] ? apic_timer_interrupt+0xe/0x20
Aug 26 12:49:35 server01v kernel: [<ffffffff8105a859>] sys_exit_group+0x0/0x1b
Aug 26 12:49:35 server01v kernel: [<ffffffff8106ab2b>] get_signal_to_deliver+0x303/0x328
Aug 26 12:49:35 server01v kernel: [<ffffffff810111f2>] do_notify_resume+0x90/0x6d7
Aug 26 12:49:37 server01v kernel: [<ffffffff8104471c>] ? pick_next_task_fair+0xa5/0xb1
Aug 26 12:49:37 server01v kernel: [<ffffffff81121cdb>] ? path_put+0x22/0x27
Aug 26 12:49:37 server01v kernel: [<ffffffff8101207e>] int_signal+0x12/0x17
Aug 26 12:49:38 server01v kernel: INFO: task frmweb:14384 blocked for more than 120 seconds.
Aug 26 12:49:39 server01v kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 26 12:49:39 server01v kernel: frmweb D ffff88021542dac0 0 14384 13596 0x00000080
Aug 26 12:49:39 server01v kernel: ffff8802b31a3d28 0000000000000086 ffff8802b31a3cf8 0000000000000000
Aug 26 12:49:40 server01v kernel: ffff880026f44080 ffff88006c35e040 ffff880026f44458 000000018efb7191
Aug 26 12:49:40 server01v kernel: 00000001b31a3cf8 0000000000000400 0000000000000000 ffff88006c061cc8
Aug 26 12:49:41 server01v kernel: Call Trace:
Aug 26 12:49:41 server01v kernel: [<ffffffff81058dbd>] exit_mm+0x9a/0x127
Aug 26 12:49:44 server01v kernel: [<ffffffff8105a32c>] do_exit+0x1f8/0x685
Aug 26 12:49:48 server01v kernel: [<ffffffff81067710>] ? freezing+0x13/0x15
Aug 26 12:49:50 server01v kernel: [<ffffffff8105a859>] sys_exit_group+0x0/0x1b
Aug 26 12:49:51 server01v kernel: [<ffffffff8106ab2b>] get_signal_to_deliver+0x303/0x328
Aug 26 12:49:51 server01v avahi-daemon[4008]: Invalid response packet from host 192.168.40.211.
Aug 26 12:49:52 server01v kernel: [<ffffffff810111f2>] do_notify_resume+0x90/0x6d7
Aug 26 12:49:56 server01v kernel: [<ffffffff81010eee>] ? __switch_to+0xc0/0x1e6
Aug 26 12:49:57 server01v kernel: [<ffffffff8104a3a4>] ? finish_task_switch+0x48/0xab
Aug 26 12:49:58 server01v kernel: [<ffffffff81121cdb>] ? path_put+0x22/0x27
Aug 26 12:50:03 server01v kernel: [<ffffffff8101207e>] int_signal+0x12/0x17
Aug 26 12:50:05 server01v kernel: INFO: task frmweb:14385 blocked for more than 120 seconds.
Aug 26 12:50:07 server01v kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 26 12:50:11 server01v kernel: frmweb D ffff88046b11eac0 0 14385 13596 0x00000080
Aug 26 12:50:19 server01v kernel: ffff880248841d28 0000000000000086 0000000048841d08 0000000000000000
Aug 26 12:50:23 server01v kernel: ffff88042a364100 ffff880392c90200 ffff88042a3644d8 ffffffff810425c7
Aug 26 12:50:26 server01v kernel: ffff880248841cf8 ffffffff810425df ffff880248841d08 ffff88006c061cc8
Aug 26 12:50:28 server01v kernel: Call Trace:
Aug 26 12:50:30 server01v kernel: [<ffffffff810425c7>] ? need_resched+0x23/0x2d
Aug 26 12:50:34 server01v kernel: [<ffffffff810425df>] ? should_resched+0xe/0x2f
Aug 26 12:50:37 server01v kernel: [<ffffffff81058dbd>] exit_mm+0x9a/0x127
Aug 26 12:50:42 server01v kernel: [<ffffffff8105a32c>] do_exit+0x1f8/0x685
Aug 26 12:50:43 server01v kernel: [<ffffffff81067710>] ? freezing+0x13/0x15
Aug 26 12:50:44 server01v kernel: [<ffffffff8105a859>] sys_exit_group+0x0/0x1b
Aug 26 12:50:48 server01v kernel: [<ffffffff8106ab2b>] get_signal_to_deliver+0x303/0x328
Aug 26 12:50:51 server01v kernel: [<ffffffff810111f2>] do_notify_resume+0x90/0x6d7
Aug 26 12:50:52 server01v kernel: [<ffffffff8104471c>] ? pick_next_task_fair+0xa5/0xb1
Aug 26 12:50:53 server01v kernel: [<ffffffff81121cdb>] ? path_put+0x22/0x27
Aug 26 12:50:55 server01v kernel: [<ffffffff8101207e>] int_signal+0x12/0x17
Aug 26 12:50:58 server01v kernel: INFO: task frmweb:14386 blocked for more than 120 seconds.
Aug 26 12:51:00 server01v kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 26 12:51:03 server01v kernel: frmweb D ffff8804679a9500 0 14386 13596 0x00000080
Aug 26 12:51:03 server01v kernel: ffff88016cbb1d28 0000000000000086 ffff88016cbb1cf8 0000000000000000
Aug 26 12:51:04 server01v kernel: ffff88006c35e040 ffff8801f89d6440 ffff88006c35e418 000000018efb7191
Aug 26 12:51:06 server01v kernel: 000000016cbb1cf8 0000000000000400 0000000000000000 ffff88006c061cc8
Aug 26 12:51:06 server01v kernel: Call Trace:
Aug 26 12:51:08 server01v kernel: [<ffffffff81058dbd>] exit_mm+0x9a/0x127
Aug 26 12:51:14 server01v kernel: [<ffffffff8105a32c>] do_exit+0x1f8/0x685
Aug 26 12:51:19 server01v kernel: [<ffffffff81067710>] ? freezing+0x13/0x15
Aug 26 12:51:23 server01v kernel: [<ffffffff8105a859>] sys_exit_group+0x0/0x1b
Aug 26 12:51:24 server01v kernel: [<ffffffff8106ab2b>] get_signal_to_deliver+0x303/0x328
Aug 26 12:51:26 server01v kernel: [<ffffffff810111f2>] do_notify_resume+0x90/0x6d7
Aug 26 12:51:33 server01v kernel: [<ffffffff8104471c>] ? pick_next_task_fair+0xa5/0xb1
Aug 26 12:51:36 server01v kernel: [<ffffffff81121cdb>] ? path_put+0x22/0x27
Aug 26 12:51:37 server01v kernel: [<ffffffff8101207e>] int_signal+0x12/0x17
Aug 26 12:51:37 server01v kernel: INFO: task frmweb:14387 blocked for more than 120 seconds.
Aug 26 12:51:37 server01v kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 26 12:51:39 server01v kernel: frmweb D 0000000000000010 0 14387 13596 0x00000080
Aug 26 12:51:39 server01v kernel: ffff8802fe6dfd28 0000000000000086 00000000fe6dfd08 0000000000000000
Aug 26 12:51:39 server01v kernel: ffff8801fe64e740 ffff88046dc1e040 ffff8801fe64eb18 ffffffff810425c7
Aug 26 12:51:42 server01v kernel: ffff8802fe6dfcf8 ffffffff810425df ffff8802fe6dfd08 ffff88006c061cc8
Aug 26 12:51:47 server01v kernel: Call Trace:
Aug 26 12:51:58 server01v kernel: [<ffffffff810425c7>] ? need_resched+0x23/0x2d
Aug 26 12:52:07 server01v kernel: [<ffffffff810425df>] ? should_resched+0xe/0x2f
Aug 26 12:52:13 server01v kernel: [<ffffffff81058dbd>] exit_mm+0x9a/0x127
Aug 26 12:52:17 server01v kernel: [<ffffffff8105a32c>] do_exit+0x1f8/0x685
Aug 26 12:52:21 server01v kernel: [<ffffffff810128ae>] ? apic_timer_interrupt+0xe/0x20
Aug 26 12:52:26 server01v kernel: [<ffffffff8105a859>] sys_exit_group+0x0/0x1b
Aug 26 12:52:29 server01v kernel: [<ffffffff8106ab2b>] get_signal_to_deliver+0x303/0x328
Aug 26 12:52:34 server01v kernel: [<ffffffff810111f2>] do_notify_resume+0x90/0x6d7
Aug 26 12:52:37 server01v kernel: [<ffffffff8104471c>] ? pick_next_task_fair+0xa5/0xb1
Aug 26 12:52:43 server01v kernel: [<ffffffff81121cdb>] ? path_put+0x22/0x27
Aug 26 12:52:43 server01v kernel: [<ffffffff8101207e>] int_signal+0x12/0x17
Aug 26 12:52:47 server01v kernel: INFO: task frmweb:14388 blocked for more than 120 seconds.
Aug 26 12:52:50 server01v kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 26 12:52:52 server01v kernel: frmweb D 000000000000000c 0 14388 13596 0x00000080
Aug 26 12:52:55 server01v kernel: ffff88014eeb5d28 0000000000000086 000000004eeb5d08 0000000000000000
Aug 26 12:53:00 server01v kernel: ffff880026ea0780 ffff88046e7844c0 ffff880026ea0b58 ffffffff810425c7
Aug 26 12:53:07 server01v kernel: ffff88014eeb5cf8 ffffffff810425df ffff88014eeb5d08 ffff88006c061cc8
Aug 26 12:53:11 server01v kernel: Call Trace:
Aug 26 12:53:13 server01v kernel: [<ffffffff810425c7>] ? need_resched+0x23/0x2d
Aug 26 12:53:16 server01v kernel: [<ffffffff810425df>] ? should_resched+0xe/0x2f
Aug 26 12:53:22 server01v kernel: [<ffffffff81058dbd>] exit_mm+0x9a/0x127
Aug 26 12:53:25 server01v kernel: [<ffffffff8105a32c>] do_exit+0x1f8/0x685
Aug 26 12:53:29 server01v kernel: [<ffffffff81067710>] ? freezing+0x13/0x15
Aug 26 12:53:37 server01v kernel: [<ffffffff8105a859>] sys_exit_group+0x0/0x1b
Aug 26 12:53:45 server01v kernel: [<ffffffff8106ab2b>] get_signal_to_deliver+0x303/0x328
Aug 26 12:53:48 server01v kernel: [<ffffffff810111f2>] do_notify_resume+0x90/0x6d7
Aug 26 12:53:52 server01v kernel: [<ffffffff8104471c>] ? pick_next_task_fair+0xa5/0xb1
Aug 26 12:53:55 server01v kernel: [<ffffffff81121cdb>] ? path_put+0x22/0x27
Aug 26 12:54:00 server01v kernel: [<ffffffff8101207e>] int_signal+0x12/0x17
Aug 26 13:02:44 server01v smbd[657]: [2015/08/26 13:02:43, 0] lib/util_sock.c:read_data(534)
Aug 26 13:02:45 server01v smbd[657]: read_data: read failure for 4 bytes to client 192.168.40.111. Error = Connection reset by peer
Aug 26 13:04:07 server01v kernel: frmweb[31820] general protection ip:7f852a78d80b sp:7fff3f18d150 error:0 in libclntsh.so.11.1[7f8528929000+2332000]
Aug 26 13:04:28 server01v smbd[1201]: [2015/08/26 13:04:28, 0] lib/util_sock.c:read_data(534)
Aug 26 13:04:33 server01v smbd[1201]: read_data: read failure for 4 bytes to client 192.168.40.111. Error = Connection reset by peer
Aug 26 13:06:23 server01v avahi-daemon[4008]: Invalid response packet from host 192.168.40.211.
Aug 26 13:13:21 server01v kernel: emdctl invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0
Aug 26 13:13:21 server01v kernel: emdctl cpuset=/ mems_allowed=0
Aug 26 13:13:21 server01v kernel: Pid: 1730, comm: emdctl Not tainted 2.6.32-100.26.2.el5 #1
Aug 26 13:13:21 server01v kernel: Call Trace:
Aug 26 13:13:21 server01v kernel: [<ffffffff8109fdaa>] ? cpuset_print_task_mems_allowed+0x92/0x9e
Aug 26 13:13:21 server01v kernel: [<ffffffff810d835a>] oom_kill_process+0x85/0x25b
Aug 26 13:13:21 server01v kernel: [<ffffffff810d8830>] ? select_bad_process+0xbc/0x102
Aug 26 13:13:21 server01v kernel: [<ffffffff810d88b3>] __out_of_memory+0x3d/0x86
Aug 26 13:13:21 server01v kernel: [<ffffffff810d8b83>] out_of_memory+0xfc/0x195
Aug 26 13:13:21 server01v kernel: [<ffffffff810dbfc5>] __alloc_pages_nodemask+0x481/0x593
Aug 26 13:13:21 server01v kernel: [<ffffffff81107093>] alloc_pages_current+0x9b/0xa4
Aug 26 13:13:21 server01v kernel: [<ffffffff810d5fcb>] __page_cache_alloc+0x5c/0x61
Aug 26 13:13:21 server01v kernel: [<ffffffff810de070>] __do_page_cache_readahead+0x81/0x196
Aug 26 13:13:39 server01v kernel: [<ffffffff81075b25>] ? wake_bit_function+0x0/0x2f
Aug 26 13:13:51 server01v kernel: [<ffffffff810de1a6>] ra_submit+0x21/0x25
Aug 26 13:13:53 server01v kernel: [<ffffffff810d758f>] filemap_fault+0x15e/0x353
Aug 26 13:14:07 server01v kernel: [<ffffffff811141c2>] ? lookup_page_cgroup+0x32/0x48
Aug 26 13:14:19 server01v kernel: [<ffffffff810ee5a9>] __do_fault+0x59/0x37c
Aug 26 13:14:31 server01v kernel: [<ffffffff81010f3e>] ? __switch_to+0x110/0x1e6
Aug 26 13:14:32 server01v kernel: [<ffffffff810f0379>] handle_mm_fault+0x14b/0x80f
Aug 26 13:14:41 server01v kernel: [<ffffffff810f4b32>] ? __vma_link_rb+0x30/0x32
Aug 26 13:14:46 server01v kernel: [<ffffffff810f4b77>] ? __vma_link+0x43/0x4f
Aug 26 13:14:49 server01v kernel: [<ffffffff810425c7>] ? need_resched+0x23/0x2d
Aug 26 13:15:01 server01v kernel: [<ffffffff810425df>] ? should_resched+0xe/0x2f
Aug 26 13:15:03 server01v kernel: [<ffffffff8143c1e6>] do_page_fault+0x210/0x299
Aug 26 13:15:06 server01v kernel: [<ffffffff8143a1b5>] page_fault+0x25/0x30
Aug 26 13:15:08 server01v kernel: Mem-Info:
Aug 26 13:15:08 server01v kernel: Node 0 DMA per-cpu:
Aug 26 13:15:12 server01v kernel: CPU 0: hi: 0, btch: 1 usd: 0
Aug 26 13:15:16 server01v kernel: CPU 1: hi: 0, btch: 1 usd: 0
Aug 26 13:15:18 server01v kernel: CPU 2: hi: 0, btch: 1 usd: 0
Aug 26 13:15:19 server01v kernel: CPU 3: hi: 0, btch: 1 usd: 0
Aug 26 13:15:20 server01v kernel: CPU 4: hi: 0, btch: 1 usd: 0
Aug 26 13:15:21 server01v kernel: CPU 5: hi: 0, btch: 1 usd: 0
Aug 26 13:15:22 server01v kernel: CPU 6: hi: 0, btch: 1 usd: 0
Aug 26 13:15:23 server01v kernel: CPU 7: hi: 0, btch: 1 usd: 0
Aug 26 13:15:25 server01v kernel: CPU 8: hi: 0, btch: 1 usd: 0
Aug 26 13:15:28 server01v kernel: CPU 9: hi: 0, btch: 1 usd: 0
Aug 26 13:15:29 server01v kernel: CPU 10: hi: 0, btch: 1 usd: 0
Aug 26 13:15:30 server01v kernel: CPU 11: hi: 0, btch: 1 usd: 0

Thanks and Regards

Please use 'code tags' on long lists of info.

Sorry about that Fatmac, was trying to figure out how to though, bit of a head scratcher

Thanks and Best Regards

I didn't notice you were using an old system (RH 5.5). Is it possible the following is causing a deadlock?

http://oracleajidba.blogspot.com/201...es-100-of.html

I also saw (along with this) reports of OOM failures due to too much data being retrieved.

You may need a newer system, and an update of the Oracle database + tools.

A "workaround" might be to increase swap space. Possibly disable oversubscription - though this would be extreme it would prevent users from deadlocking the system.

If you do disable oversubscription, I would suggest increasing swap space to help cover borderline cases.

Other things you could try are imposing memory limits (see manpage on "pam_limits"). This would help prevent the deadlocks by killing user processes when they exceed the limit.

The problem is determining what the limits should be. A rough formula (note, this is REALLY rough) is
Note, the limit computed has to account for the oracle database somewhere, so you could subtract that memory/swap usage from the p+s term to set aside that amount. If the database is on another machine (better protected isolation), this isn't necessary.

The formula can also be extended to include the number of user processes (that would be n * that value) which makes a penalty for running multiple processes (you can also limit the users process count limit too (the default is 4096 I believe). I think your defaults are:
depending on your physical memory size - check yours to get the right values.

To me, it looks like a memory deadlock. The OOM killer works fairly well - but it can be defeated by processes that grow relatively slowly (which seems to prevent any process from becoming a candidate).

If that is the case, memory limits can help - but it can take a while to identify good values, and during that time, deadlocks will still occur (which is why disabling oversubscription helps - it will kill the user immediately, which usually generates complaints, and that tells you that more swap is needed... and if not available, you can either cut down the number of users, or cut the memory limit... which will still generate complaints, but at least the system doesn't hang.

One other thought - cutting the number of users might easiest be done by reducing the number of simultaneous database connections. That way a user would get a "nicer" error message (from the database), than a potentially problem causing application abort.

Hi jpollard, I am sorry for the late reply, thanks for the reply and more to the puzzle this is an isolated incident, the system has been running with out an show stopper as this for more than 2 years now.

I have two separate machines one for the oracle database and the other for the oracle application server on weblogic (forms and reports) The issue is with oracle application server. We were able to trace a one frmweb session and once this was killed the system was back to normal. But just to be on the safe side, they have restarted the server (oracle apps) as well. So far there is no the system has been performing normal.

Thanks nad Best Regards

Good to know.

Good luck.

It would still be a good idea to update that system though - RHEL5 is up to 5.11 now, so a lot of bug fixes, inc SECURITY fixes.
https://access.redhat.com/articles/3078

Thanks you guys for the information and we are talking and how to do the patch update or to do complete upgrade to 6. But it would be nice to know what caused and if it was some kind of an bug (Oracle or linu) that caused it!

Thanks again and Best Regards

Like I said - it may be a memory deadlock due to oversubscription - and multiple processes trying to grow beyond the physical limits. I have had some multi-threaded applications do that. In my case it was a POVray render with too many objects - as the objects got created the process grew... and grew (turned out a nested loop creating a few million cubes). The OOM didn't kick in for some reason, but the system hung.

The default user limits are... unlimited (first saw that mistake on an IRIX server that suffered the same type of crash, until the PTB finally added 16GB of swap, and acknowledged that it could happen still).

Hi Jpollard, yes at the moment we are monitoring the server till we can get an upgrade agreed with the customer, strange is this is the only place that we ran in to this kind of scenario

Thanks and Best Regards

It may not happen very often, and only when a certain combination of factors happen - such as multiple processes where all make big database requests for data... If only one (or two) do it, then nothing happens.

You could try running sar to record activity and see if it detects a buildup to a possible problem. Sar might also give hints to which resource limits being overloaded. One thing it does tend to miss is the actual deadlock itself - as it might not be able to get the resources needed record the sample data

Hi Jpollard, yes we have started collecting data using sar but the systems seems to be behaving . As you say, the combination that lead to the incident may not occurs in the near future. We tried to trace the load, as in what the users, Jobs were doing at that particular period to see if we can reproduce the environment, but with not luck.

Thanks and Best Regards