Hi

I have a montavista linux OS 2.6.10 MIPs arch running on my embedded system.

The system runs with no issues for few weeks (3-4), but after that I am getting kernel oops at random locations (different locations each time) in the code. All the page fault exceptions.
Oops occur at different processes, though the examples below are showing oops in swapper context, but I have seen logs with oops in different processes as well).

Also there are few boxes with same setup/configuration that are running for 2 months or so without any issue/OOPs.

Just to add to this, there is no user activity (configuration etc) during/before the time oops occur.
But there are some backend activities constantly on-going on the system, like:
1. SNMP queries through NMS (get, bulk etc) through PPP links.
2. L2/L3 Ethernet traffic over the links on this box.

Other observations:
===================
We have memory/top command logs just before the oops and it does not show any memory leaks, also the CPU utilization is very less.

Linux kernel has:
=================
1. PREEMPTION enabled
2. SMP disabled (Its a single processor system)



Is there any similar known issues observed by anyone with the combinations mentioned above (with kernel 2.6.10 and/OR MIPs and/OR montavista etc?????
Also please do suggest what can we do to find the root cause of the issue.



Below are three such random oops (in different boxes - with 3-4 weeks) observed:

OOPS - 1:
=========
Call Trace:
handle_IRQ_event+0xc4/0x148
handle_IRQ_event+0x88/0x148
__do_IRQ+0x158/0x210
do_IRQ+0x38/0x60
do_IRQ+0x48/0x60
mipsIRQ+0x128/0x160
ret_from_exception+0x8/0x18
inet_init+0x42c/0x4b8
___do_softirq+0x48/0x130
ilat_irq_enable+0x218/0x238
ip_conntrack_in+0xb00/0xe98 [ip_conntrack]
___do_softirq+0x48/0x130
__do_softirq+0x40/0x6c
do_softirq+0x5c/0x88
irq_exit+0x44/0x5c
mipsIRQ+0x110/0x160
__call_console_drivers+0x6c/0x88
__die+0x2b8/0x474
show_registers+0xa8/0xfc
ilat_irq_enable+0x218/0x238
inet_init+0x42c/0x4b8
__die+0x2b8/0x474
release_console_sem+0x1fc/0x310
search_exception_tables+0x3c/0x4c
do_page_fault+0x230/0x370
sock_def_readable+0x44/0xa4
ilat_restore_flags+0x84/0x104
raw_rcv_skb+0x18c/0x254
kmem_cache_alloc+0x80/0x98
raw_rcv+0x64/0x7c
ilat_irq_enable+0x200/0x238
ip_local_deliver_finish+0x0/0x294
ilat_restore_flags+0x84/0x104
inet_init+0x42c/0x4b8
tlb_do_page_fault_0+0x100/0x108
nf_hook_slow+0x1bc/0x1f0
inet_init+0x42c/0x4b8
inet_init+0x42c/0x4b8
__do_softirq+0x40/0x6c

Oops in arch/mips/mm/fault.c::do_page_fault, line 171[#1]:
Cpu 0
$ 0 : 00000000 1000ff04 d2140d40 d2140d40
$ 4 : 00000000 00000039 00000000 82366e60
$ 8 : 9d207d00 ffffe000 82370000 82370000
$12 : 82370000 fffffffe 00000010 ffffffff
$16 : 82370000 8030e6b4 82360000 80333e38
$20 : 80330000 80333d10 8236ff0c 00000000
$24 : 00000001 00000003
$28 : 80332000 80333cf8 80333cf8 a01082d0
Hi : 00000240
Lo : 000001f8
epc : a01082d4 0xa01082d4 Tainted: PF
ra : a01082d0 0xa01082d0

Status: 1000ff02 KERNEL EXL Cause : 4080c008
BadVA : 00000000
PrId : 00019365
Process swapper (pid: 0, threadinfo=80332000, task=80334000)
Stack : 00000010 00000008 00000400 00000001 82370000 8030e6b4 82360000 80333e38
80330000 80333d10 8236ff0c 00000000 a01082d0 00000000 00000000 00000000
00000000 00000000 00000000 a0125e4c 00000000 00000000 82370000 8030e6b4
00000040 80333e38 80330000 80310000 a0124e9c 80333e38 a0125ecc 00000000

OOPS - 2:
=========
Call Trace:
ilat_restore_flags+0x84/0x104
run_timer_softirq+0x72c/0x918
run_timer_softirq+0x6a0/0x918
___do_softirq+0x6c/0x130
__do_softirq+0x40/0x6c
ret_from_exception+0x8/0x18
do_softirq+0x5c/0x88
irq_exit+0x44/0x5c
mipsIRQ+0x110/0x160
ilat_restore_flags+0x84/0x104
ilat_restore_flags+0xe4/0x104
release_console_sem+0x1c4/0x310
ilat_restore_flags+0x84/0x104
vprintk+0x240/0x2a4
inet_init+0x42c/0x4b8
inet_init+0x42c/0x4b8
inet_init+0x42c/0x4b8
printk+0x1c/0x28
log_panic+0x144/0x298
log_panic+0xe0/0x298
panic+0x34/0x188
do_exit+0x78/0x1108
ilat_irq_enable+0x218/0x238
sys_exit_group+0x0/0xc
get_signal_to_deliver+0x3c4/0x434
get_signal_to_deliver+0x3bc/0x434
ret_from_exception+0x8/0x18
do_signal+0x7c/0xe90
specific_send_sig_info+0xf4/0x114
force_sig_info+0xfc/0x13c
work_notifysig+0xc/0x14
inet_init+0x42c/0x4b8
unknown_bootoption+0x0/0x2a8


Break instruction in kernel code in arch/mips/kernel/traps.c::do_bp, line 1145[#1]:
Cpu 0
$ 0 : 00000000 1000fe00 9c6bfe20 8237072c
$ 4 : 8236ff0c 8237072c 410cc2bc 410cc100
$ 8 : 82370000 82370000 82370000 82370000
$12 : 82370000 fffffff2 0000000a ffffffff
$16 : 82370724 8237072c 8236ff0c 00000001
$20 : 80330000 80310000 8236ff0c e801e801
$24 : 00000000 00000009
$28 : 80332000 80333b18 4068d280 801307bc
Hi : 0003e094
Lo : dbf24000
epc : 801307ac cascade+0x40/0x7c Tainted: PF
ra : 801307bc cascade+0x50/0x7c

Status: 1000fe02 KERNEL EXL Cause : 10800024
PrId : 00019365
Process swapper (pid: 0, threadinfo=80332000, task=80334000)
Stack : 82370000 00000000 00000001 801010c4 00000000 8236ff0c 82370000 80333b58
80131130 801310a4 82370000 00000000 82370000 2246dca9 823440c0 00000003
80333b58 80333b58 00000103 00000000 8236fc90 0000000a 80310000 82370000
8033433c 8012b99c 00000003 80333bb8 82370000 2249e7eb 00000000 1000fe00
80330000 ffffffbb 1000fe01 80333d48 8012baa0 00000000 8030ae10 80102a08


OOPS - 3:
=========
Call Trace:
memcmp+0x0/0x30
ilat_restore_flags+0x84/0x104
__wake_up+0x50/0xb4
ilat_restore_flags+0x84/0x104
sock_def_readable+0x44/0xa4
skb_queue_tail+0x74/0x9c
ilat_restore_flags+0x84/0x104
raw_rcv_skb+0x18c/0x254
kmem_cache_alloc+0x80/0x98
raw_rcv+0x64/0x7c
skb_clone+0x20/0x358
ip_local_deliver_finish+0x0/0x294
raw_v4_input+0xf8/0x16c
nf_iterate+0x88/0x124
ip_ct_refresh_acct+0x108/0x1f4 [ip_conntrack]
ip_local_deliver_finish+0x134/0x294
ip_local_deliver_finish+0x0/0x294
ip_local_deliver_finish+0x0/0x294
nf_hook_slow+0xf4/0x1f0
pehci_hcd_irq+0x108/0x1d0 [pehci]
ip_local_deliver_finish+0x0/0x294
ip_local_deliver+0x2e0/0x2f8
ip_local_deliver_finish+0x0/0x294
ip_rcv_finish+0x3d4/0x4ac
ip_rcv_finish+0x0/0x4ac
ip_rcv_finish+0x0/0x4ac
ip_rcv_finish+0x0/0x4ac
nf_hook_slow+0xf4/0x1f0
ret_from_exception+0x8/0x18
isp1761_reg_read32+0x0/0x1c [hal_mips]
isp1761_pci_isr+0xec/0x108 [hal_mips]
ip_rcv_finish+0x0/0x4ac
inet_init+0x42c/0x4b8
ip_rcv+0x740/0x81c
ilat_restore_flags+0x84/0x104


Oops in arch/mips/mm/fault.c::do_page_fault, line 171[#1]:
Cpu 0
$ 0 : 00000000 1000ff00 00000000 00100100
$ 4 : 8030f69c 000004dd 00000000 82366e60
$ 8 : 00000000 00000800 00000000 00000000
$12 : 00004000 00000000 0000000a ffffffff
$16 : 00000000 00000001 00100100 93863e98
$20 : 00000000 00000000 00000002 9c9b1000
$24 : 00000000 8012bb94
$28 : 80332000 80333840 80333840 8011f570
Hi : 00000e7e
Lo : 00078bd8
epc : 8011f54c __wake_up_common+0x44/0xb8 Tainted: PF
ra : 8011f570 __wake_up_common+0x68/0xb8

Status: 1000ff02 KERNEL EXL Cause : 80800008
BadVA : 00100100
PrId : 00019365
Process swapper (pid: 0, threadinfo=80332000, task=80334000)
Stack : 8021340c 801010c4 9d1e38e0 9d1e38e0 1000ff01 00000044 00000000 00000010
00000000 82380000 00000000 80333878 8011f610 00000044 00000000 00000010
00000000 801010c4 00000000 ffffffff 82112400 00000002 8026f034 80270354