Hi guys.

I´ve just joined this community in the hope that anybody could help us out with a problem that is killing us.
We have an Advantech system running OpenSuse 13.1 and Kernel 3.4.6 (although the problem also happened in the Kernel 3.11.10)

The loop of Kernel Oops happens intermittently and is not reproducible, but I suspect it might be related to load, CPU or IRQs I guess.
I am not expecting any miraculous answer but any hint, guess or directions for the problem we are facing will be very, very much appreciated !!

Thanks

[134031.334446] BUG: unable to handle kernel NULL pointer dereference at 0000000a
[134031.342277] IP: [<c0204d74>] print_context_stack+0x54/0xa0
[134031.347974] *pdpt = 00000000336c7001 *pde = 0000000000000000
[134031.354115] Oops: 0000 [#1] SMP
[134031.357709] Modules linked in: binfmt_misc ppp_generic slhc wctdm24xxp(O) wctdm(O) wcfxo(O) wctc4xxp(O) dahdi_transcode(O) wcb4xxp(O) wcte12xp(O) dahdi_voicebus(O) ip6table_mangle ip6table_filter ip6_tables iptable_nat xt_DSCP iptable_mangle ipt_REJECT xt_tcpudp xt_LOG xt_limit xt_multiport xt_addrtype xt_conntrack iptable_filter ip_tables ipt_MASQUERADE nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 x_tables nf_conntrack wct4xxp(O) dahdi(O) crc_ccitt loop acpi_cpufreq mperf coretemp video serio_raw shpchp pcspkr iTCO_wdt pci_hotplug i2c_i801 iTCO_vendor_support crc32c_intel aesni_intel cryptd aes_i586 microcode button 8021q garp stp xfrm_user ipcomp xfrm_ipcomp esp4 ah4 af_key sg autofs4 af_packet ata_generic ata_piix ehci_hcd usbcore e1000e usb_common fan thermal processor thermal_sys scsi_dh_rdac scsi_dh_hp_sw scsi_dh_alua scsi_dh_emc scsi_dh mptspi mptscsih mptbase scsi_transport_spi
[134031.437900]
[134031.439505] Pid: 3762, comm: kamailio Tainted: G O 3.4.6-2.10.1-pae #1 Advantech SYS-2USM03-6M01E/SYS-2USM03-6M01E
[134031.451013] EIP: 0060:[<c0204d74>] EFLAGS: 00210293 CPU: 1
[134031.456641] EIP is at print_context_stack+0x54/0xa0
[134031.461632] EAX: ffffe000 EBX: 0000000a ECX: 00000000 EDX: 0000000a
[134031.468090] ESI: 00000000 EDI: 00000000 EBP: f410deb8 ESP: f410de80
[134031.474514] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[134031.480061] CR0: 80050033 CR2: 0000000a CR3: 1f4e3000 CR4: 000407f0
[134031.486465] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[134031.492880] DR6: ffff0ff0 DR7: 00000400
[134031.496844] Process kamailio (pid: 3762, ti=f410c000 task=f244b0d0 task.ti=f2d64000)
[134031.504712] Stack:
[134031.506822] ffffe000 00000000 00000000 00001ffc c069f1a8 00000000 c07f56fc f410deb8
[134031.514822] c020401b c069f1a8 c07f56fc 00000000 f410deb8 0000000a 00000000 00000000
[134031.522879] c07f56fc 00000000 00000000 f410df10 c020513f 00000000 c069f1a8 c07f56fc
[134031.530938] Call Trace:
[134031.533508] [<c020401b>] dump_trace+0x9b/0xf0
[134031.538088] [<c020513f>] show_trace_log_lvl+0x3f/0x50
[134031.543398] [<c068b77d>] dump_stack+0x75/0x7a
[134031.547992] [<c0230c0c>] warn_slowpath_common+0x6c/0xa0
[134031.553461] [<c0230c73>] warn_slowpath_fmt+0x33/0x40
[134031.558656] [<c05d70d2>] dev_watchdog+0x1d2/0x1e0
[134031.563580] [<c023db91>] run_timer_softirq+0xe1/0x290
[134031.568850] [<c023725e>] __do_softirq+0x8e/0x170
[134031.573705] [<c0203eb9>] do_softirq+0x59/0xa0
[134031.578244] [<f410e108>] 0xf410e107
[134031.581950] DWARF2 unwinder stuck at 0xf410e108
[134031.586599]
[134031.588209] Leftover inexact backtrace:
[134031.588210]
[134031.593746] [<c0694ea1>] ? __schedule+0x341/0x730
[134031.598702] [<c0694ea1>] ? __schedule+0x341/0x730
[134031.603633] [<c0350065>] ? bioset_free+0xb5/0xd0
[134031.608501] [<c0350063>] ? bioset_free+0xb3/0xd0
[134031.613348] <IRQ>
[134031.615394] BUG: unable to handle kernel NULL pointer dereference at 0000000a
[134031.622936] IP: [<c0204d74>] print_context_stack+0x54/0xa0
[134031.628589] *pdpt = 00000000336c7001 *pde = 0000000000000000
[134031.634509] Oops: 0000 [#2] SMP
[134031.637884] Modules linked in: binfmt_misc ppp_generic slhc wctdm24xxp(O) wctdm(O) wcfxo(O) wctc4xxp(O) dahdi_transcode(O) wcb4xxp(O) wcte12xp(O) dahdi_voicebus(O) ip6table_mangle ip6table_filter ip6_tables iptable_nat xt_DSCP iptable_mangle ipt_REJECT xt_tcpudp xt_LOG xt_limit xt_multiport xt_addrtype xt_conntrack iptable_filter ip_tables ipt_MASQUERADE nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 x_tables nf_conntrack wct4xxp(O) dahdi(O) crc_ccitt loop acpi_cpufreq mperf coretemp video serio_raw shpchp pcspkr iTCO_wdt pci_hotplug i2c_i801 iTCO_vendor_support crc32c_intel aesni_intel cryptd aes_i586 microcode button 8021q garp stp xfrm_user ipcomp xfrm_ipcomp esp4 ah4 af_key sg autofs4 af_packet ata_generic ata_piix ehci_hcd usbcore e1000e usb_common fan thermal processor thermal_sys scsi_dh_rdac scsi_dh_hp_sw scsi_dh_alua scsi_dh_emc scsi_dh mptspi mptscsih mptbase scsi_transport_spi
[134031.717927]
[134031.719513] Pid: 3762, comm: kamailio Tainted: G O 3.4.6-2.10.1-pae #1 Advantech SYS-2USM03-6M01E/SYS-2USM03-6M01E
[134031.730981] EIP: 0060:[<c0204d74>] EFLAGS: 00210093 CPU: 1
[134031.736662] EIP is at print_context_stack+0x54/0xa0
[134031.741687] EAX: ffffe000 EBX: 0000000a ECX: 00000000 EDX: 0000000a
[134031.748086] ESI: 00000000 EDI: 00000000 EBP: f410dcac ESP: f410dc74
[134031.754498] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[134031.760055] CR0: 80050033 CR2: 0000000a CR3: 1f4e3000 CR4: 000407f0
[134031.766496] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[134031.772892] DR6: ffff0ff0 DR7: 00000400
[134031.776846] Process kamailio (pid: 3762, ti=f410c000 task=f244b0d0 task.ti=f2d64000)
[134031.784740] Stack:
[134031.786852] ffffe000 00000000 00000000 00001ffc c069f1a8 00000000 c07e5c64 f410dcac
[134031.794859] c020401b c069f1a8 c07e5c64 00000000 f410dcac 0000000a 00000000 f410de44
[134031.802932] c07e5c64 00000000 f410de44 f410de80 c020513f 00000000 c069f1a8 c07e5c64
[134031.810939] Call Trace:
[134031.813515] [<c020401b>] dump_trace+0x9b/0xf0
[134031.818092] [<c020513f>] show_trace_log_lvl+0x3f/0x50
[134031.823417] [<c02040c0>] show_stack_log_lvl+0x50/0xd0
[134031.828677] [<c0204212>] show_registers+0xd2/0x1e0
[134031.833671] [<c069759f>] __die+0x8f/0xf0
[134031.837794] [<c068c507>] no_context+0x17f/0x1ac
[134031.842544] [<c068c66e>] __bad_area_nosemaphore+0x13a/0x142
[134031.848351] [<c068c685>] bad_area_nosemaphore+0xf/0x11
[134031.853709] [<c0699717>] do_page_fault+0x3f7/0x450
[134031.858721] [<c0696d6a>] error_code+0x5a/0x60
[134031.863326] [<c0204d74>] print_context_stack+0x54/0xa0
[134031.868699] [<c020401b>] dump_trace+0x9b/0xf0
[134031.873253] [<c020513f>] show_trace_log_lvl+0x3f/0x50
[134031.878498] [<c068b77d>] dump_stack+0x75/0x7a
[134031.883077] [<c0230c0c>] warn_slowpath_common+0x6c/0xa0
[134031.888537] [<c0230c73>] warn_slowpath_fmt+0x33/0x40
[134031.893732] [<c05d70d2>] dev_watchdog+0x1d2/0x1e0
[134031.898640] [<c023db91>] run_timer_softirq+0xe1/0x290
[134031.903911] [<c023725e>] __do_softirq+0x8e/0x170
[134031.908757] [<c0203eb9>] do_softirq+0x59/0xa0
[134031.913354] [<f410e108>] 0xf410e107
[134031.917097] DWARF2 unwinder stuck at 0xf410e108
[134031.921739]
[134031.923332] Leftover inexact backtrace:
[134031.923332]
[134031.928858] [<c0694ea1>] ? __schedule+0x341/0x730
[134031.933805] [<c0694ea1>] ? __schedule+0x341/0x730
[134031.938764] [<c0350065>] ? bioset_free+0xb5/0xd0
[134031.943616] [<c0350063>] ? bioset_free+0xb3/0xd0
[134031.948431] <IRQ>
/usr/sbin/kamail[134031.950494] BUG: unable to handle kernel NULL pointer dereference at 0000000a
[134031.959424] IP:io[3886]: ERROR: [<c0204d74>] print_context_stack+0x54/0xa0
[134031.961181] [dahdi] wct4xxp: Need to increase latency. Estimated latency should be 3
[134031.961246] [dahdi] wct4xxp: Increased latency to 3
<core> [tcp_mai[134031.979596] *pdpt = 00000000336c7001 *pde = 0000000000000000 n.c:4553]: conne
[134031.988266] Oops: 0000 [#3] ct 10.100.182.15SMP
[134031.991148] [dahdi] wct4xxp: Need to increase latency. Estimated latency should be 4
[134031.991188] [dahdi] wct4xxp: Increased latency to 4
[134032.006182]
6:5061 failed (t[134032.007765] Modules linked in: binfmt_misc ppp_generic slhc wctdm24xxp(O) wctdm(O) wcfxo(O) wctc4xxp(O) dahdi_transcode(O) wcb4xxp(O) wcte12xp(O) dahdi_voicebus(O) ip6table_mangle ip6table_filter ip6_tables iptable_nat xt_DSCP iptable_mangle ipt_REJECT xt_tcpudp xt_LOG xt_limit xt_multiport xt_addrtype xt_conntrack iptable_filter ip_tables ipt_MASQUERADE nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 x_tables nf_conntrack wct4xxp(O) dahdi(O) crc_ccitt loop acpi_cpufreq mperf coretemp video serio_raw shpchp pcspkr iTCO_wdt pci_hotplug i2c_i801 iTCO_vendor_support crc32c_intel aesni_intel cryptd aes_i586 microcode button 8021q garp stp xfrm_user ipcomp xfrm_ipcomp esp4 ah4 af_key sg autofs4 af_packet ata_generic ata_piix ehci_hcd usbcore e1000e usb_common fan thermal processor thermal_sys scsi_dh_rdac scsi_dh_hp_sw scsi_dh_alua scsi_dh_emc scsi_dh mptspi mptscsih mptbase scsi_transport_spi
[134032.089453]
[134032.091065] Pid: 3762, comm: kamailio Tainted: G O 3.4.6-2.10.1-pae #1 Advantech SYS-2USM03-6M01E/SYS-2USM03-6M01E
[134032.102551] EIP: 0060:[<c0204d74>] EFLAGS: 00210093 CPU: 1
[134032.108161] EIP is at print_context_stack+0x54/0xa0
[134032.113186] EAX: ffffe000 EBX: 0000000a ECX: 00000000 EDX: 0000000a
imeout)
[134032.119618] ESI: 00000000 EDI: 00000000 EBP: f410daa0 ESP: f410da68
[134032.119662] [dahdi] wct4xxp: Need to increase latency. Estimated latency should be 7
[134032.119701] [dahdi] wct4xxp: Increased latency to 7
[134032.139916] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[134032.145437] CR0: 80050033 CR2: 0000000a CR3: 1f4e3000 CR4: 000407f0
[134032.151825] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[134032.158194] DR6: ffff0ff0 DR7: 00000400
[134032.162150] Process kamailio (pid: 3762, ti=f410c000 task=f244b0d0 task.ti=f2d64000)
[134032.170026] Stack:
[134032.172137] ffffe000 00000000 00000000 00001ffc c069f1a8 00000000 c07e5c64 f410daa0
[134032.180135] c020401b c069f1a8 c07e5c64 00000000 f410daa0 0000000a 00000000 f410dc38
[134032.188193] c07e5c64 00000000 f410dc38 f410dc74 c020513f 00000000 c069f1a8 c07e5c64
[134032.196227] Call Trace:
[134032.198802] [<c020401b>] dump_trace+0x9b/0xf0
[134032.203366] [<c020513f>] show_trace_log_lvl+0x3f/0x50
[134032.208679] [<c02040c0>] show_stack_log_lvl+0x50/0xd0
[134032.213940] [<c0204212>] show_registers+0xd2/0x1e0
[134032.218977] [<c069759f>] __die+0x8f/0xf0
[134032.223107] [<c068c507>] no_context+0x17f/0x1ac
[134032.227874] [<c068c66e>] __bad_area_nosemaphore+0x13a/0x142
[134032.233689] [<c068c685>] bad_area_nosemaphore+0xf/0x11
[134032.239013] [<c0699717>] do_page_fault+0x3f7/0x450
[134032.244058] [<c0696d6a>] error_code+0x5a/0x60
[134032.248647] [<c0204d74>] print_context_stack+0x54/0xa0
[134032.254012] [<c020401b>] dump_trace+0x9b/0xf0
[134032.258589] [<c020513f>] show_trace_log_lvl+0x3f/0x50
[134032.263879] [<c02040c0>] show_stack_log_lvl+0x50/0xd0
[134032.269176] [<c0204212>] show_registers+0xd2/0x1e0
[134032.274203] [<c069759f>] __die+0x8f/0xf0
[134032.278367] [<c068c507>] no_context+0x17f/0x1ac
[134032.283102] [<c068c66e>] __bad_area_nosemaphore+0x13a/0x142
[134032.288936] [<c068c685>] bad_area_nosemaphore+0xf/0x11
[134032.294319] [<c0699717>] do_page_fault+0x3f7/0x450
[134032.299340] [<c0696d6a>] error_code+0x5a/0x60
[134032.303936] [<c0204d74>] print_context_stack+0x54/0xa0
[134032.309328] [<c020401b>] dump_trace+0x9b/0xf0
[134032.313891] [<c020513f>] show_trace_log_lvl+0x3f/0x50
[134032.313923] [dahdi] wct4xxp: Need to increase latency. Estimated latency should be 71
[134032.314011] [dahdi] wct4xxp: Increased latency to 71
[134032.332475] [<c068b77d>] dump_stack+0x75/0x7a
[134032.337062] [<c0230c0c>] warn_slowpath_common+0x6c/0xa0
[134032.342566] [<c0230c73>] warn_slowpath_fmt+0x33/0x40
[134032.347760] [<c05d70d2>] dev_watchdog+0x1d2/0x1e0
[134032.352702] [<c023db91>] run_timer_softirq+0xe1/0x290
[134032.357972] [<c023725e>] __do_softirq+0x8e/0x170
[134032.362820] [<c0203eb9>] do_softirq+0x59/0xa0
[134032.367419] [<f410e108>] 0xf410e107
[134032.371141] DWARF2 unwinder stuck at 0xf410e108
[134032.375801]
[134032.377383] Leftover inexact backtrace:
[134032.377384]
[134032.382972] [<c0694ea1>] ? __schedule+0x341/0x730
[134032.387954] [<c0694ea1>] ? __schedule+0x341/0x730
[134032.392868] [<c0350065>] ? bioset_free+0xb5/0xd0
[134032.397684] [<c0350063>] ? bioset_free+0xb3/0xd0
[134032.402514] <IRQ>
[134032.404551] BUG: unable to handle kernel NULL pointer dereference at 0000000a
[134032.412083] IP: [<c0204d74>] print_context_stack+0x54/0xa0
[134032.417710] *pdpt = 00000000336c7001 *pde = 0000000000000000
[134032.423623] Oops: 0000 [#4] SMP
[134032.427007] Modules linked in: binfmt_misc ppp_generic slhc wctdm24xxp(O) wctdm(O) wcfxo(O) wctc4xxp(O) dahdi_transcode(O) wcb4xxp(O) wcte12xp(O) dahdi_voicebus(O) ip6table_mangle ip6table_filter ip6_tables iptable_nat xt_DSCP iptable_mangle ipt_REJECT xt_tcpudp xt_LOG xt_limit xt_multiport xt_addrtype xt_conntrack iptable_filter ip_tables ipt_MASQUERADE nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 x_tables nf_conntrack wct4xxp(O) dahdi(O) crc_ccitt loop acpi_cpufreq mperf coretemp video serio_raw shpchp pcspkr iTCO_wdt pci_hotplug i2c_i801 iTCO_vendor_support crc32c_intel aesni_intel cryptd aes_i586 microcode button 8021q garp stp xfrm_user ipcomp xfrm_ipcomp esp4 ah4 af_key sg autofs4 af_packet ata_generic ata_piix ehci_hcd usbcore e1000e usb_common fan thermal processor thermal_sys scsi_dh_rdac scsi_dh_hp_sw scsi_dh_alua scsi_dh_emc scsi_dh mptspi mptscsih mptbase scsi_transport_spi
[134032.507302]
[134032.508904] Pid: 3762, comm: kamailio Tainted: G O 3.4.6-2.10.1-pae #1 Advantech SYS-2USM03-6M01E/SYS-2USM03-6M01E
[134032.520337] EIP: 0060:[<c0204d74>] EFLAGS: 00210093 CPU: 1
[134032.526000] EIP is at print_context_stack+0x54/0xa0
[134032.531011] EAX: ffffe000 EBX: 0000000a ECX: 00000000 EDX: 0000000a
[134032.537439] ESI: 00000000 EDI: 00000000 EBP: f410d894 ESP: f410d85c
[134032.543837] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[134032.549411] CR0: 80050033 CR2: 0000000a CR3: 1f4e3000 CR4: 000407f0
[134032.555807] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[134032.562211] DR6: ffff0ff0 DR7: 00000400
[134032.566168] Process kamailio (pid: 3762, ti=f410c000 task=f244b0d0 task.ti=f2d64000)
[134032.574078] Stack:
[134032.576181] ffffe000 00000000 00000000 00001ffc c069f1a8 00000000 c07e5c64 f410d894
[134032.584214] c020401b c069f1a8 c07e5c64 00000000 f410d894 0000000a 00000000 f410da2c
[134032.592238] c07e5c64 00000000 f410da2c f410da68 c020513f 00000000 c069f1a8 c07e5c64
[134032.600245] Call Trace:
[134032.602837] [<c020401b>] dump_trace+0x9b/0xf0
[134032.607411] [<c020513f>] show_trace_log_lvl+0x3f/0x50
[134032.612684] [<c02040c0>] show_stack_log_lvl+0x50/0xd0
[134032.617955] [<c0204212>] show_registers+0xd2/0x1e0
[134032.622951] [<c069759f>] __die+0x8f/0xf0
wdt[858]: W: Kee[134032.627077] [<c068c507>] no_context+0x17f/0x1ac
p Alive
[134032.633192] [<c068c66e>] __bad_area_nosemaphore+0x13a/0x142
[134032.639719] [<c068c685>] bad_area_nosemaphore+0xf/0x11
[134032.645076] [<c0699717>] do_page_fault+0x3f7/0x450
[134032.650088] [<c0696d6a>] error_code+0x5a/0x60
[134032.654656] [<c0204d74>] print_context_stack+0x54/0xa0
[134032.660049] [<c020401b>] dump_trace+0x9b/0xf0
[134032.664645] [<c020513f>] show_trace_log_lvl+0x3f/0x50
[134032.669906] [<c02040c0>] show_stack_log_lvl+0x50/0xd0
[134032.675180] [<c0204212>] show_registers+0xd2/0x1e0
[134032.680192] [<c069759f>] __die+0x8f/0xf0
[134032.684321] [<c068c507>] no_context+0x17f/0x1ac
[134032.689103] [<c068c66e>] __bad_area_nosemaphore+0x13a/0x142
[134032.694896] [<c068c685>] bad_area_nosemaphore+0xf/0x11
[134032.700263] [<c0699717>] do_page_fault+0x3f7/0x450
[134032.705266] [<c0696d6a>] error_code+0x5a/0x60
[134032.709835] [<c0204d74>] print_context_stack+0x54/0xa0
[134032.715189] [<c020401b>] dump_trace+0x9b/0xf0
[134032.719752] [<c020513f>] show_trace_log_lvl+0x3f/0x50
[134032.725044] [<c02040c0>] show_stack_log_lvl+0x50/0xd0
[134032.730312] [<c0204212>] show_registers+0xd2/0x1e0
[134032.735342] [<c069759f>] __die+0x8f/0xf0
[134032.739462] [<c068c507>] no_context+0x17f/0x1ac
[134032.744205] [<c068c66e>] __bad_area_nosemaphore+0x13a/0x142
[134032.750029] [<c068c685>] bad_area_nosemaphore+0xf/0x11
[134032.755422] [<c0699717>] do_page_fault+0x3f7/0x450
[134032.760436] [<c0696d6a>] error_code+0x5a/0x60
[134032.764996] [<c0204d74>] print_context_stack+0x54/0xa0
[134032.770354] [<c020401b>] dump_trace+0x9b/0xf0
[134032.774940] [<c020513f>] show_trace_log_lvl+0x3f/0x50
[134032.780221] [<c068b77d>] dump_stack+0x75/0x7a
[134032.784790] [<c0230c0c>] warn_slowpath_common+0x6c/0xa0
[134032.790241] [<c0230c73>] warn_slowpath_fmt+0x33/0x40
[134032.795412] [<c05d70d2>] dev_watchdog+0x1d2/0x1e0
[134032.800310] [<c023db91>] run_timer_softirq+0xe1/0x290
[134032.805626] [<c023725e>] __do_softirq+0x8e/0x170
[134032.810473] [<c0203eb9>] do_softirq+0x59/0xa0
[134032.815052] [<f410e108>] 0xf410e107
[134032.818733] DWARF2 unwinder stuck at 0xf410e108
[134032.823388]
[134032.824989] Leftover inexact backtrace:
[134032.824990]
[134032.830557] [<c0694ea1>] ? __schedule+0x341/0x730
[134032.835478] [<c0694ea1>] ? __schedule+0x341/0x730
[134032.840423] [<c0350065>] ? bioset_free+0xb5/0xd0
[134032.845220] [<c0350063>] ? bioset_free+0xb3/0xd0
[134032.850097] <IRQ>
[134032.852149] BUG: unable to handle kernel NULL pointer dereference at 0000000a
[134032.859688] IP: [<c0204d74>] print_context_stack+0x54/0xa0
[134032.865341] *pdpt = 00000000336c7001 *pde = 0000000000000000
[134032.871263] Oops: 0000 [#5] SMP
[134032.874611] Modules linked in: binfmt_misc ppp_generic slhc wctdm24xxp(O) wctdm(O) wcfxo(O) wctc4xxp(O) dahdi_transcode(O) wcb4xxp(O) wcte12xp(O) dahdi_voicebus(O) ip6table_mangle ip6table_filter ip6_tables iptable_nat xt_DSCP iptable_mangle ipt_REJECT xt_tcpudp xt_LOG xt_limit xt_multiport xt_addrtype xt_conntrack iptable_filter ip_tables ipt_MASQUERADE nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 x_tables nf_conntrack wct4xxp(O) dahdi(O) crc_ccitt loop acpi_cpufreq mperf coretemp video serio_raw shpchp pcspkr iTCO_wdt pci_hotplug i2c_i801 iTCO_vendor_support crc32c_intel
<4>[134032.925890] [<c0204d74>] print_context_stack+0x54/0xa0
<4>[134032.925893] [<c020401b>] dump_trace+0x9b/0xf0
<4>[134032.925896] [<c020513f>] show_trace_log__registers+0xd2/0x1e0
<4>[134032.940712] [<c069759f>] __die+0x8f/0xf0
<4>[134032.940715] [<c068c507>] no_context+0x17f/0x1ac
<4>[134032.940718] [<c068c66e>] __bad_area_nosb>] dump_trace+0x9Waiting for device /dev/root to appear: ok

The first place that I would begin is by asking yourself the usual questions: "What changed, recently?" Is this a custom(ized) kernel? Are you running an unusual device? Is this "the latest" kernel version available from the vendor, and are all updates to it installed? And so on.

A problem with device configuration certainly could be a problem ... the wrong driver is being used, it wasn't compiled with the right options, and so on. To better answer such questions, we need to know much more about your system, and about how it came to be set-up in just the way that it now is.

1 members found this post helpful.

Thanks!

Basically what we know is that the problem did not happen in the old kernel version 2.6.34.7, Suse 11.3.
The kernel has very few modules touched (ppp_generic.c and xt_hashlimit.c for netfilter), nothing else has been changed.
Basically it is a dedicated server running with a PCI card from Digium with its own dahdi driver.
We looked at updated with this kernel but did not found anything that could ring a bell.
Here is the some system output (lsmod, lspci -vvv) if helps.

Thanks again.
500idq02:~ # uname --all
Linux 500idq02 3.4.6-2.10.1-pae #1 SMP Thu Jul 26 09:36:26 UTC 2012 (641c197) i686 i686 i386 GNU/Linux
500idq02:~ # lsmod
Module Size Used by
binfmt_misc 13135 1
ppp_generic 32096 0
slhc 13234 1 ppp_generic
wctdm24xxp 112175 0
wctdm 40596 0
wcfxo 17547 0
wctc4xxp 44061 0
dahdi_transcode 13926 1 wctc4xxp
wcb4xxp 57123 0
wcte12xp 40495 0
dahdi_voicebus 61514 2 wctdm24xxp,wcte12xp
ip6table_mangle 12620 0
ip6table_filter 12711 0
ip6_tables 17969 2 ip6table_mangle,ip6table_filter
iptable_nat 13138 0
xt_DSCP 12549 2
iptable_mangle 12615 1
ipt_REJECT 12485 2
xt_tcpudp 12756 45
xt_LOG 17239 13
xt_limit 12541 13
xt_multiport 12694 4
xt_addrtype 12563 5
xt_conntrack 12664 20
iptable_filter 12706 1
ip_tables 17987 3 iptable_nat,iptable_mangle,iptable_filter
ipt_MASQUERADE 12760 0
nf_nat 25110 2 iptable_nat,ipt_MASQUERADE
nf_conntrack_ipv4 14501 23 iptable_nat,nf_nat
nf_defrag_ipv4 12649 1 nf_conntrack_ipv4
x_tables 26087 16 ip6table_mangle,ip6table_filter,ip6_tables,iptable_nat,xt_DSCP,iptable_mangle,ipt_REJECT,xt_tcpudp,x t_LOG,xt_limit,xt_multiport,xt_addrtype,xt_conntrack,iptable_filter,ip_tables,ipt_MASQUERADE
nf_conntrack 84834 5 iptable_nat,xt_conntrack,ipt_MASQUERADE,nf_nat,nf_conntrack_ipv4
wct4xxp 236261 124
dahdi 225919 500 wctdm24xxp,wctdm,wcfxo,dahdi_transcode,wcb4xxp,wcte12xp,dahdi_voicebus,wct4xxp
crc_ccitt 12595 2 wctdm24xxp,dahdi
loop 22990 4
acpi_cpufreq 14281 1
mperf 12603 1 acpi_cpufreq
coretemp 13204 0
crc32c_intel 12781 0
aesni_intel 18134 0
cryptd 15578 1 aesni_intel
aes_i586 16956 1 aesni_intel
microcode 22824 0
serio_raw 13155 0
pcspkr 12630 0
i2c_i801 17499 0
iTCO_wdt 17652 1
iTCO_vendor_support 13502 1 iTCO_wdt
shpchp 32189 0
pci_hotplug 33907 1 shpchp
video 18798 0
button 13626 0
8021q 27546 0
garp 14020 1 8021q
stp 12811 1 garp
xfrm_user 34640 0
ipcomp 12581 0
xfrm_ipcomp 13163 1 ipcomp
esp4 12835 0
ah4 12834 0
af_key 39480 0
sg 35608 0
autofs4 33231 2
af_packet 34793 0
ata_generic 12779 0
ata_piix 26359 3
ehci_hcd 59412 0
usbcore 190002 2 ehci_hcd
e1000e 192254 0
usb_common 12525 1 usbcore
thermal 18230 0
fan 12823 0
processor 43505 1 acpi_cpufreq
thermal_sys 19935 4 video,thermal,fan,processor
scsi_dh_rdac 13075 0
scsi_dh_hp_sw 12745 0
scsi_dh_alua 12844 0
scsi_dh_emc 12965 0
scsi_dh 14178 4 scsi_dh_rdac,scsi_dh_hp_sw,scsi_dh_alua,scsi_dh_emc
mptspi 18062 0
mptscsih 30752 1 mptspi
mptbase 75141 2 mptspi,mptscsih
scsi_transport_spi 29662 1 mptspi
500idq02:~ # lspci -vvv
00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09)
Subsystem: Intel Corporation 2nd Generation Core Processor Family DRAM Controller
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
Latency: 0
Capabilities: [e0] Vendor Specific Information: Len=0c <?>
Kernel driver in use: agpgart-intel
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=01, subordinate=02, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: f7d00000-f7dfffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [88] Subsystem: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port
Capabilities: [80] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
Address: fee0f00c Data: 4151
Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s L1, Latency L0 <256ns, L1 <4us
ClockPM- Surprise- LLActRep- BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #1, PowerLimit 75.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet+ LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [140 v1] Root Complex Link
Desc: PortNumber=02 ComponentID=01 EltType=Config
Link0: Desc: TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+
Addr: 00000000fed19000
Kernel driver in use: pcieport
Kernel modules: shpchp
00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09) (prog-if 00 [VGA controller])
Subsystem: Intel Corporation Device 2010
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 11
Region 0: Memory at f7800000 (64-bit, non-prefetchable) [size=4M]
Region 2: Memory at e0000000 (64-bit, prefetchable) [size=256M]
Region 4: I/O ports at f000 [size=64]
Expansion ROM at <unassigned> [disabled]
Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit-
Address: 00000000 Data: 0000
Capabilities: [d0] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [a4] PCI Advanced Features
AFCap: TP+ FLR+
AFCtrl: FLR-
AFStatus: TP-
00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
Subsystem: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Latency: 0
Interrupt: pin A routed to IRQ 11
Region 0: Memory at f7e2b000 (64-bit, non-prefetchable) [size=16]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [8c] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Kernel modules: mei

(Continue in the next post...)

(Continuation from the previous output..)

00:16.3 Serial controller: Intel Corporation 6 Series/C200 Series Chipset Family KT Controller (rev 04) (prog-if 02 [16550])
Subsystem: Intel Corporation 6 Series/C200 Series Chipset Family KT Controller
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin B routed to IRQ 19
Region 0: I/O ports at f140 [size=8]
Region 1: Memory at f7e29000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [c8] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Kernel driver in use: serial
00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 05)
Subsystem: Intel Corporation Device 0000
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 41
Region 0: Memory at f7e00000 (32-bit, non-prefetchable) [size=128K]
Region 1: Memory at f7e28000 (32-bit, non-prefetchable) [size=4K]
Region 2: I/O ports at f060 [size=32]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee0300c Data: 4181
Capabilities: [e0] PCI Advanced Features
AFCap: TP+ FLR+
AFCtrl: FLR-
AFStatus: TP-
Kernel driver in use: e1000e
Kernel modules: e1000e
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05) (prog-if 20 [EHCI])
Subsystem: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin D routed to IRQ 19
Region 0: Memory at f7e27000 (32-bit, non-prefetchable) [size=1K]
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [58] Debug port: BAR=1 offset=00a0
Capabilities: [98] PCI Advanced Features
AFCap: TP+ FLR+
AFCtrl: FLR-
AFStatus: TP-
Kernel driver in use: ehci_hcd
Kernel modules: ehci_hcd
00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 05)
Subsystem: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 5
Region 0: Memory at f7e20000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [70] Express (v1) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- RBE- FLReset+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed unknown, Width x0, ASPM unknown, Latency L0 <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01
Status: NegoPending- InProgress-
VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=1 ArbSelect=Fixed TC/VC=22
Status: NegoPending- InProgress-
Capabilities: [130 v1] Root Complex Link
Desc: PortNumber=0f ComponentID=00 EltType=Config
Link0: Desc: TargetPort=00 TargetComponent=00 AssocRCRB- LinkType=MemMapped LinkValid+
Addr: 00000000fed1c000
Kernel modules: snd_hda_intel

00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <1us, L1 <16us
ClockPM- Surprise- LLActRep+ BwNot-
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #0, PowerLimit 10.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
Changed: MRL- PresDet- LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Address: 00000000 Data: 0000
Capabilities: [90] Subsystem: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1
Capabilities: [a0] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Kernel driver in use: pcieport
Kernel modules: shpchp

00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
I/O behind bridge: 0000e000-0000efff
Memory behind bridge: f7c00000-f7cfffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #8, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <16us
ClockPM- Surprise- LLActRep+ BwNot-
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #7, PowerLimit 10.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet- LinkState+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Address: 00000000 Data: 0000
Capabilities: [90] Subsystem: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8
Capabilities: [a0] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Kernel driver in use: pcieport
Kernel modules: shpchp

00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05) (prog-if 20 [EHCI])
Subsystem: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 23
Region 0: Memory at f7e26000 (32-bit, non-prefetchable) [size=1K]
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [58] Debug port: BAR=1 offset=00a0
Capabilities: [98] PCI Advanced Features
AFCap: TP+ FLR+
AFCtrl: FLR-
AFStatus: TP-
Kernel driver in use: ehci_hcd
Kernel modules: ehci_hcd

00:1f.0 ISA bridge: Intel Corporation QM67 Express Chipset Family LPC Controller (rev 05)
Subsystem: Intel Corporation QM67 Express Chipset Family LPC Controller
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Capabilities: [e0] Vendor Specific Information: Len=0c <?>
Kernel modules: iTCO_wdt

00:1f.2 IDE interface: Intel Corporation 6 Series/C200 Series Chipset Family 4 port SATA IDE Controller (rev 05) (prog-if 8f [Master SecP SecO PriP PriO])
Subsystem: Intel Corporation 6 Series/C200 Series Chipset Family 4 port SATA IDE Controller
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin B routed to IRQ 19
Region 0: I/O ports at f130 [size=8]
Region 1: I/O ports at f120 [size=4]
Region 2: I/O ports at f110 [size=8]
Region 3: I/O ports at f100 [size=4]
Region 4: I/O ports at f0f0 [size=16]
Region 5: I/O ports at f0e0 [size=16]
Capabilities: [70] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [b0] PCI Advanced Features
AFCap: TP+ FLR+
AFCtrl: FLR-
AFStatus: TP-
Kernel driver in use: ata_piix
Kernel modules: ata_piix, pata_acpi, ata_generic

00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)
Subsystem: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin C routed to IRQ 18
Region 0: Memory at f7e25000 (64-bit, non-prefetchable) [size=256]
Region 4: I/O ports at f040 [size=32]
Kernel modules: i2c_i801

00:1f.5 IDE interface: Intel Corporation 6 Series/C200 Series Chipset Family 2 port SATA IDE Controller (rev 05) (prog-if 85 [Master SecO PriO])
Subsystem: Intel Corporation 6 Series/C200 Series Chipset Family 2 port SATA IDE Controller
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin B routed to IRQ 19
Region 0: I/O ports at f0d0 [size=8]
Region 1: I/O ports at f0c0 [size=4]
Region 2: I/O ports at f0b0 [size=8]
Region 3: I/O ports at f0a0 [size=4]
Region 4: I/O ports at f090 [size=16]
Region 5: I/O ports at f080 [size=16]
Capabilities: [70] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [b0] PCI Advanced Features
AFCap: TP+ FLR+
AFCtrl: FLR-
AFStatus: TP-
Kernel driver in use: ata_piix
Kernel modules: ata_piix, pata_acpi, ata_generic

01:00.0 PCI bridge: Texas Instruments XIO2000(A)/XIO2200A PCI Express-to-PCI Bridge (rev 03) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=01, secondary=02, subordinate=02, sec-latency=32
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: f7d00000-f7dfffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Bridge: PM- B3+
Capabilities: [60] MSI: Enable- Count=1/16 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [80] Subsystem: Device 0000:0000
Capabilities: [90] Express (v1) PCI/PCI-X Bridge, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ BrConfRtry-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <16us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn-
Kernel modules: shpchp

02:08.0 Communication controller: Digium, Inc. Wildcard TE420 quad-span T1/E1/J1 card 3.3V (PCI-Express) (5th gen) (rev 02)
Subsystem: Device 0005:0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 32 (8000ns min, 2000ns max)
Interrupt: pin A routed to IRQ 16
Region 0: Memory at f7d00000 (32-bit, non-prefetchable) [size=128]
Kernel driver in use: wct4xxp
Kernel modules: wct4xxp

04:00.0 Ethernet controller: Intel Corporation 82583V Gigabit Network Connection
Subsystem: Intel Corporation Device 0000
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 42
Region 0: Memory at f7c00000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at e000 [size=32]
Region 3: Memory at f7c20000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee0300c Data: 4191
Capabilities: [e0] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <128ns, L1 <64us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [a0] MSI-X: Enable- Count=1 Masked-
Vector table: BAR=3 offset=00000000
PBA: BAR=3 offset=00002000
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr+ BadTLP+ BadDLLP+ Rollover- Timeout+ NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [140 v1] Device Serial Number 00-0b-ab-ff-ff-3c-f0-3c
Kernel driver in use: e1000e
Kernel modules: e1000e

500idq02:~ #

Hi.

We have updated the Kernel, but we still have the same problem.
The Oops goes over and over until we get a Kernel panic:

239000.666672] BUG: unable to handle kernel NULL pointer dereference at 0000000a
[239000.674200] IP: [<c0204c94>] print_context_stack+0x54/0xa0
[239000.679836] *pdpt = 000000002fac4001 *pde = 0000000000000000
[239000.685792] Oops: 0000 [#1] SMP
[239000.689208] Modules linked in: nfnetlink_log nfnetlink binfmt_misc ppp_generic slhc wctdm24xxp(O) wcaxx(O) wctdm(O) wcfxo(O) wctc4xxp(O) dahdi_transcode(O) wcb4xxp(O) wcte13xp(O) wcte12xp(O) dahdi_voicebus(O) ip6table_mangle ip6table_filter ip6_tables iptable_nat nf_nat_ipv4 xt_DSCP iptable_mangle ipt_REJECT xt_tcpudp xt_LOG xt_limit nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_addrtype xt_conntrack iptable_filter ip_tables ipt_MASQUERADE nf_nat x_tables nf_conntrack wct4xxp(O) wcte43x(O) oct612x(O) dahdi(O) crc_ccitt loop x86_pkg_temp_thermal intel_powerclamp intel_rapl coretemp crc32_pclmul iTCO_wdt iTCO_vendor_support aesni_intel ablk_helper cryptd lrw aes_i586 xts gf128mul lpc_ich pcspkr mfd_core serio_raw i2c_i801 mei_me mei shpchp battery tpm_infineon video tpm_tis tpm tpm_bios button 8021q mrp garp stp llc xfrm_user ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo sg autofs4 af_packet ata_generic ata_piix crc32c_intel ehci_pci ehci_hcd e1000e ptp pps_core usbcore usb_common fan thermal processor thermal_sys scsi_dh_rdac scsi_dh_alua scsi_dh_emc scsi_dh_hp_sw scsi_dh mptspi mptscsih mptbase scsi_transport_spi
[239000.791579] CPU: 1 PID: 3917 Comm: ssm Tainted: G O 3.12.53-40.1-pae #1
[239000.799394] Hardware name: Advantech SYS-2USM03-6M01E/SYS-2USM03-6M01E, BIOS 4.6.4 11/22/2011
[239000.808153] task: efbeef50 ti: f48f6000 task.ti: ea5c6000
[239000.813737] EIP: 0060:[<c0204c94>] EFLAGS: 00210293 CPU: 1
[239000.819365] EIP is at print_context_stack+0x54/0xa0
[239000.824448] EAX: ffffe000 EBX: 0000000a ECX: 00000000 EDX: 0000000a
[239000.830950] ESI: 00000000 EDI: 00000000 EBP: f48f7e6c ESP: f48f7e34
[239000.837433] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[239000.842971] CR0: 80050033 CR2: 0000000a CR3: 2a5b0000 CR4: 000407f0
[239000.849386] Stack:
[239000.851507] ffffe000 00000000 00000000 00001ffc c071b1e0 00000000 c08a17f6 f48f7e6c
[239000.859634] c02040db c071b1e0 c08a17f6 00000000 f48f7e6c 0000000a 00000000 ffffffff
[239000.867759] c08a17f6 00000000 00000000 f48f7ed0 c020504f 00000000 c071b1e0 c08a17f6
[239000.875829] Call Trace:
[239000.878485] [<c02040db>] dump_trace+0x9b/0xf0
[239000.883092] [<c020504f>] show_trace_log_lvl+0x3f/0x50
[239000.888382] [<c0204180>] show_stack_log_lvl+0x50/0xd0
[239000.893730] [<c020509f>] show_stack+0x1f/0x40
[239000.898380] [<c070b5df>] dump_stack+0x3e/0x4e
[239000.902941] [<c02461a8>] warn_slowpath_common+0x88/0xc0
[239000.908402] [<c0246213>] warn_slowpath_fmt+0x33/0x40
[239000.913621] [<c065857e>] dev_watchdog+0x1de/0x1f0
[239000.918584] [<c0250c24>] call_timer_fn+0x24/0xe0
[239000.923454] [<c0251958>] run_timer_softirq+0x178/0x210
[239000.928810] [<c024adb6>] __do_softirq+0xb6/0x1d0
[239000.933719] [<c0203f79>] do_softirq+0x59/0xa0
[239000.938364] [<f48f8140>] 0xf48f813f
[239000.942124] DWARF2 unwinder stuck at 0xf48f8140
[239000.946932]
[239000.948543] Leftover inexact backtrace:
[239000.948543]
[239000.954132] [<c03074d0>] ? mempool_free_slab+0x10/0x10
[239000.959563] [<c03074e0>] ? mempool_alloc_pages+0x10/0x10
[239000.965132] [<c02ab4f0>] ? try_to_force_load+0x50/0x50
[239000.970571] [<f7b8a000>] ? 0xf7b89fff
[239000.974481] [<c02ad509>] ? module_alloc_update_bounds+0x9/0x50
[239000.980603] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239000.986547] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239000.992457] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239000.998385] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.004278] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.010208] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.016127] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.022135] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.028035] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.033977] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.039897] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.045954] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.051857] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.057832] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.063784] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.069695] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.075640] <IRQ>
[239001.077680] BUG: unable to handle kernel NULL pointer dereference at 0000000a
[239001.085270] IP: [<c0204c94>] print_context_stack+0x54/0xa0
[239001.090957] *pdpt = 000000002fac4001 *pde = 0000000000000000
[239001.096970] Oops: 0000 [#2] SMP
[239001.100368] Modules linked in: nfnetlink_log nfnetlink binfmt_misc ppp_generic slhc wctdm24xxp(O) wcaxx(O) wctdm(O) wcfxo(O) wctc4xxp(O) dahdi_transcode(O) wcb4xxp(O) wcte13xp(O) wcte12xp(O) dahdi_voicebus(O) ip6table_mangle ip6table_filter ip6_tables iptable_nat nf_nat_ipv4 xt_DSCP iptable_mangle ipt_REJECT xt_tcpudp xt_LOG xt_limit nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_addrtype xt_conntrack iptable_filter ip_tables ipt_MASQUERADE nf_nat x_tables nf_conntrack wct4xxp(O) wcte43x(O) oct612x(O) dahdi(O) crc_ccitt loop x86_pkg_temp_thermal intel_powerclamp intel_rapl coretemp crc32_pclmul iTCO_wdt iTCO_vendor_support aesni_intel ablk_helper cryptd lrw aes_i586 xts gf128mul lpc_ich pcspkr mfd_core serio_raw i2c_i801 mei_me mei shpchp battery tpm_infineon video tpm_tis tpm tpm_bios button 8021q mrp garp stp llc xfrm_user ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo sg autofs4 af_packet ata_generic ata_piix crc32c_intel ehci_pci ehci_hcd e1000e ptp pps_core usbcore usb_common fan thermal processor thermal_sys scsi_dh_rdac scsi_dh_alua scsi_dh_emc scsi_dh_hp_sw scsi_dh mptspi mptscsih mptbase scsi_transport_spi
[239001.202926] CPU: 1 PID: 3917 Comm: ssm Tainted: G O 3.12.53-40.1-pae #1
[239001.210897] Hardware name: Advantech SYS-2USM03-6M01E/SYS-2USM03-6M01E, BIOS 4.6.4 11/22/2011
[239001.219609] task: efbeef50 ti: f48f6000 task.ti: ea5c6000
[239001.225187] EIP: 0060:[<c0204c94>] EFLAGS: 00210093 CPU: 1
[239001.230878] EIP is at print_context_stack+0x54/0xa0
[239001.235944] EAX: ffffe000 EBX: 0000000a ECX: 00000000 EDX: 0000000a
[239001.242462] ESI: 00000000 EDI: 00000000 EBP: f48f7c6c ESP: f48f7c34
[239001.249024] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[239001.254650] CR0: 80050033 CR2: 0000000a CR3: 2a5b0000 CR4: 000407f0
[239001.261175] Stack:
[239001.263314] ffffe000 00000000 00000000 00001ffc c071b1e0 00000000 c0886102 f48f7c6c
[239001.271459] c02040db c071b1e0 c0886102 00000000 f48f7c6c 0000000a 00000000 ffffffff
[239001.279568] c0886102 00000000 f48f7df8 f48f7e34 c020504f 00000000 c071b1e0 c0886102
[239001.287608] Call Trace:
[239001.290181] [<c02040db>] dump_trace+0x9b/0xf0
[239001.294787] [<c020504f>] show_trace_log_lvl+0x3f/0x50
[239001.300120] [<c0204180>] show_stack_log_lvl+0x50/0xd0
[239001.305478] [<c0204291>] show_regs+0x91/0x1a0
[239001.310057] [<c0712524>] __die+0x94/0x100
[239001.314341] [<c0706f90>] no_context+0x1c7/0x242
[239001.319117] [<c070712e>] __bad_area_nosemaphore+0x123/0x12b
[239001.324953] [<c0707145>] bad_area_nosemaphore+0xf/0x11
[239001.330325] [<c0714640>] __do_page_fault+0xe0/0x4d0
[239001.335503] [<c0711cb3>] error_code+0x67/0x6c
[239001.340057] [<c0204c94>] print_context_stack+0x54/0xa0
[239001.345449] [<c02040db>] dump_trace+0x9b/0xf0
[239001.349990] [<c020504f>] show_trace_log_lvl+0x3f/0x50
[239001.355314] [<c0204180>] show_stack_log_lvl+0x50/0xd0
[239001.360708] [<c020509f>] show_stack+0x1f/0x40
[239001.365290] [<c070b5df>] dump_stack+0x3e/0x4e
[239001.369928] [<c02461a8>] warn_slowpath_common+0x88/0xc0
[239001.375440] [<c0246213>] warn_slowpath_fmt+0x33/0x40
[239001.380642] [<c065857e>] dev_watchdog+0x1de/0x1f0
[239001.385654] [<c0250c24>] call_timer_fn+0x24/0xe0
[239001.390532] [<c0251958>] run_timer_softirq+0x178/0x210
[239001.395970] [<c024adb6>] __do_softirq+0xb6/0x1d0
[239001.400876] [<c0203f79>] do_softirq+0x59/0xa0
[239001.405539] [<f48f8140>] 0xf48f813f
[239001.409220] DWARF2 unwinder stuck at 0xf48f8140
[239001.413899]
[239001.415501] Leftover inexact backtrace:
[239001.415501]
[239001.421110] [<c03074d0>] ? mempool_free_slab+0x10/0x10
[239001.426472] [<c03074e0>] ? mempool_alloc_pages+0x10/0x10
[239001.432117] [<c02ab4f0>] ? try_to_force_load+0x50/0x50
[239001.437559] [<f7b8a000>] ? 0xf7b89fff
[239001.441436] [<c02ad509>] ? module_alloc_update_bounds+0x9/0x50
[239001.447567] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.453450] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.459380] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.465299] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.471275] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.477194] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.483191] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.489095] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.494988] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.500929] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.506865] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.512817] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.518729] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.524607] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.530561] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.536515] [<c054a170>] ? tty_prepare_flip_string+0x40/0x40
[239001.542407] <IRQ>
[239001.544464] BUG: unable to handle kernel NULL pointer dereference at 0000000a
[239001.552032] IP: [<c0204c94>] print_context_stack+0x54/0xa0
[239001.557761] *pdpt = 000000002fac4001 *pde = 0000000000000000
[239001.563733] Oops: 0000 [#3] SMP

etc...until:

[239013.910284] BUG: unable to handle kernel NULL pointer dereference at 0000000a
[239013.917875] IP: [<c0204c94>] print_context_stack+0x54/0xa0
[239013.923569] *pdpt = 000000002fac4001 *pde = 0000000000000000
[239013.929526] Oops: 0000 [#17] SMP
[239013.932979] Modules linked in: nfnetlink_log nfnetlink binfmt_misc ppp_generic slhc wctdm24xxp(O) wcaxx(O) wctdm(O) wcfxo(O) wctc4xxp(O) dahdi_transcode(O) wcb4xxp(O) wcte13xp(O) wcte12xp(O) dahdi_voicebus(O) ip6table_mangle ip6table_filter ip6_tables iptable_nat nf_nat_ipv4 xt_DSCP iptable_mangle ipt_REJECT xt_tcpudp xt_LOG xt_limit nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_addrtype xt_conntrack iptable_filter ip_tables ipt_MASQUERADE nf_nat x_tables nf_conntrack wct4xxp(O) wcte43x(O) oct612x(O) dahdi(O) crc_ccitt loop x86_pkg_temp_thermal intel_powerclamp intel_rapl coretemp crc32_pclmul iTCO_wdt iTCO_vendor_support aesni_intel ablk_helper cryptd lrw aes_i586 xts gf128mul lpc_ich pcspkr mfd_core serio_raw i2c_i801 mei_me mei shpchp battery tpm_infineon video tpm_tis tpm tpm_bios button 8021q mrp garp stp llc xfrm_user ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo sg autofs4 af_packet ata_generic ata_piix crc32c_intel ehci_pci ehci_hcd e1000e ptp pps_core usbcore usb_common fan thermal processor thermal_sys scsi_dh_rdac scsi_dh_alua scsi_dh_emc scsi_dh_hp_sw scsi_dh mptspi mptscsih mptbase scsi_transport_spi
[239014.035382] CPU: 1 PID: 3917 Comm: ssm Tainted: G O 3.12.53-40.1-pae #1
[239014.043172] Hardware name: Advantech SYS-2USM03-6M01E/SYS-2USM03-6M01E, BIOS 4.6.4 11/22/2011
[239014.051922] task: efbeef50 ti: f48f4000 task.ti: ea5c6000
[239014.057459] EIP: 0060:[<c0204c94>] EFLAGS: 00210093 CPU: 1
[239014.063124] EIP is at print_context_stack+0x54/0xa0
[239014.068175] EAX: ffffe000 EBX: 0000000a ECX: 00000000 EDX: 0000000a
[239014.074692] ESI: 00000000 EDI: 00000000 EBP: f48f5e6c ESP: f48f5e34
[239014.081173] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[239014.086785] CR0: 80050033 CR2: 0000000a CR3: 2a5b0000 CR4: 000407f0
[239014.093222] Stack:
[239014.095396] ffffe000 00000000 00000000 00001ffc c071b1e0 00000000 c0886102 f48f5e6c
[239014.103524] c02040db c071b1e0 c0886102 00000000 f48f5e6c 0000000a 00000000 ffffffff
[239014.111608] c0886102 00000000 f48f5ff8 f48f6034 c020504f 00000000 c071b1e0 c0886102
[239014.119717] Call Trace:
[239014.122278] [<c02040db>] dump_trace+0x9b/0xf0
[239014.126891] [<c020504f>] show_trace_log_lvl+0x3f/0x50
[239014.132259] [<c0204180>] show_stack_log_lvl+0x50/0xd0
[239014.137520] [<c0204291>] show_regs+0x91/0x1a0
[239014.142176] [<c0712524>] __die+0x94/0x100
[239014.146476] [<c0706f90>] no_context+0x1c7/0x242
[239014.151265] [<c070712e>] __bad_area_nosemaphore+0x123/0x12b
[239014.157166] [<c0707145>] bad_area_nosemaphore+0xf/0x11
[239014.162586] [<c0714640>] __do_page_fault+0xe0/0x4d0
[239014.167801] [<c0711cb3>] error_code+0x67/0x6c
[239014.172458] DWARF2 unwinder stuck at error_code+0x67/0x6c
[239014.178037]
[239014.179648] Leftover inexact backtrace:
[239014.179648]
[239014.185216] Code: 8d b4 26 00 00 00 00 85 f6 74 14 39 f3 73 05 3b 1c 24 73 17 83 c4 10 89 f8 5b 5e 5f 5d c3 90 3b 5c 24 08 76 ef 3b 5c 24 0c 73 e9 <8b> 2b 89 e8 e8 c3 bb 05 00 85 c0 74 16 8d 47 04 39 c3 74 18 89
[239014.205458] EIP: [<c0204c94>] print_context_stack+0x54/0xa0 SS:ESP 0068:f48f5e34
[239014.213110] CR2: 000000000000000a
[239014.216921] ---[ end trace b05d221280869788 ]---
[239014.221705] Kernel panic - not syncing: Fatal exception in interrupt
[239014.228263] Rebooting in 5 seconds..
[239019.260555] ACPI MEMORY or I/O RESET_REG.

Hi Guys, just an update.


Debugging a little bit more, we found that the Oops (Null Pointer) has happen inside a warning function in the dev_watchdog.
We do not know what this warning was supposed to show to use, because we did not have the right debug level.
We are changing it now to see what it is.


But, is there a reason for a warning to produce a Kernel Oops? Is this correct?

if (some_queue_timedout) {
WARN_ONCE(1, KERN_INFO "NETDEV WATCHDOG: %s (%s): transmit queue %u timed out\n",
dev->name, netdev_drivername(dev), i);
dev->netdev_ops->ndo_tx_timeout(dev);