Hi,
I am newbie in Linux kernel, I am running 3.14.25-rt22 kernel on Xilinx zc702 board.
I have created simple tcp client-server application to check network.
In this application client is sending data(128Kbyte) to server, when server has received data it will reply with same data to client, this will run in infinite loop and Application does not store any data on machine.
When I run this application on target (ZC702 board as server)-host (dextop ubuntu machine as client), after few hours my board is hanged and it does not responds to serial console or ping.
There is no any panic information come on serial console of board.
Please suggest how to debug this kind of errors.
Further findings:
I have run same application without applying RT patch to kernel 3.14.25, Application has run for more then 50 Hrs. so this seems to be issue of rt-patch. I guess issue might be some race around condition or memory leak in kernel, But I don't know how to find this.
Further findings 2:
I have enabled kmemleak in kernel 3.14.25-rt22 (with rt patch), this time I have run Iperf (simple TCP client-server testing) for network testing, then my board is hanged after few hours and got following message on console:
Quote:
root@/ # ------------[ cut here ]------------
Kernel BUG at c03b4394 [verbose debug info unavailable]
Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARM
Modules linked in:
CPU: 0 PID: 618 Comm: irq/54-eth0 Not tainted 3.14.25-rt22-xilinx-00008-g92b3222-dirty #2
task: eea0be80 ti: edaca000 task.ti: edaca000
PC is at rt_spin_lock_slowlock+0x274/0x2dc
LR is at task_blocks_on_rt_mutex+0x220/0x234
pc : [<c03b4394>] lr : [<c0056f48>] psr: a0000013
sp : edacbd18 ip : edaca020 fp : ec2f6300
r10: 00000001 r9 : 00000000 r8 : edacbd18
r7 : eea0c314 r6 : 00000002 r5 : eea0be80 r4 : ee844180
r3 : 0000a417 r2 : a417a416 r1 : 00000000 r0 : ffffffdd
Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 18c5387d Table: 2c51c04a DAC: 00000015
Process irq/54-eth0 (pid: 618, stack limit = 0xedaca248)
Stack: (0xedacbd18 to 0xedacc000)
bd00: 00000001 00000000
bd20: 00000000 edacbd24 00000000 00000000 eea0be80 ee844180 edacbd01 00000031
bd40: ed29576c ee844180 ee844180 edacbd7c 00000000 00000000 c05aaa88 c002c2f0
bd60: c05a2200 ed29576c 0000000f 00236577 c0357848 c002c99c ed29576c 00000000
bd80: 00236577 ed295400 00002ee0 ed295860 00000000 ec78bf4e c033af28 c0309de4
bda0: ed295400 c03593ac ed295400 ec6de180 ec6de180 ed295400 ed295458 c035b7d0
bdc0: ec78bf4e 138934b6 239710ac c0344538 000034b6 ec6de180 ec78bf4e ec78bf00
bde0: 00000000 ec6de180 c03e9c5c ec78bf00 00000000 c05acbc8 00000000 c033af28
be00: ec2f6300 c033acc8 c033abcc ec6de180 ec78bf4e c033b730 ec6de180 ee973d40
be20: c05abd44 ee973d40 00000008 c031860c 00000600 00000600 ef587f20 ec6de180
be40: 00000600 c05abd58 ef587160 ec6de180 c05abd34 f08e44e0 2c7f9540 00000001
be60: ee97438c c0318934 ee97438c c0019008 c001cc64 00000f40 ec6de180 ee973d40
be80: c02b8d8c ee9742c0 ec6de180 c02b8dc8 00000000 00000000 ee973d40 000005ea
bea0: ee974314 000005ea 00000040 00000040 edacbeb8 ee97438c eefd8800 c02b8bc0
bec0: 0000012c 00000040 00000000 eefd8808 00236569 c0318b78 c0318ad8 edaca000
bee0: 00000003 00000000 00208040 c059c3b8 00000000 c05a2140 00000001 c0026404
bf00: 00000001 edacbf30 00000048 edaca000 ee80e900 00000001 edaca000 00000000
bf20: 00000000 00000000 00000000 c00265c8 eda86500 c005d348 ee80e900 eda86500
bf40: c005d300 c005ce5c 00000000 c005d220 00000000 00000002 eda86500 c005cd7c
bf60: 00000000 c003cbfc edacbf64 00000000 eda7b000 eda86500 00000000 00000000
bf80: edacbf80 edacbf80 00000000 00000000 edacbf90 edacbf90 edacbfac eda86800
bfa0: c003cb3c 00000000 00000000 c000e5d8 00000000 00000000 00000000 00000000
bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 4980e123 19619042
[<c03b4394>] (rt_spin_lock_slowlock) from [<c002c2f0>] (lock_timer_base+0x2c/0x50)
[<c002c2f0>] (lock_timer_base) from [<c002c99c>] (mod_timer+0xb4/0x1d4)
[<c002c99c>] (mod_timer) from [<c0309de4>] (sk_reset_timer+0x14/0x3c)
[<c0309de4>] (sk_reset_timer) from [<c03593ac>] (tcp_prequeue+0x254/0x284)
[<c03593ac>] (tcp_prequeue) from [<c035b7d0>] (tcp_v4_rcv+0x320/0x708)
[<c035b7d0>] (tcp_v4_rcv) from [<c033acc8>] (ip_local_deliver+0xfc/0x35c)
[<c033acc8>] (ip_local_deliver) from [<c033b730>] (ip_rcv+0x808/0x95c)
[<c033b730>] (ip_rcv) from [<c031860c>] (__netif_receive_skb_core+0x490/0x4f0)
[<c031860c>] (__netif_receive_skb_core) from [<c0318934>] (netif_receive_skb_internal+0xa0/0xb4)
[<c0318934>] (netif_receive_skb_internal) from [<c02b8dc8>] (xemacps_rx_poll+0x208/0x38c)
[<c02b8dc8>] (xemacps_rx_poll) from [<c0318b78>] (net_rx_action+0xa0/0x1a0)
[<c0318b78>] (net_rx_action) from [<c0026404>] (do_current_softirqs+0x114/0x234)
[<c0026404>] (do_current_softirqs) from [<c00265c8>] (__local_bh_enable+0x54/0x7c)
[<c00265c8>] (__local_bh_enable) from [<c005d348>] (irq_forced_thread_fn+0x48/0x50)
[<c005d348>] (irq_forced_thread_fn) from [<c005ce5c>] (irq_thread+0xe0/0x144)
[<c005ce5c>] (irq_thread) from [<c003cbfc>] (kthread+0xc0/0xd0)
[<c003cbfc>] (kthread) from [<c000e5d8>] (ret_from_fork+0x14/0x3c)
Code: 0a000002 e59d2000 e1520008 0a000000 (e7f001f2)
---[ end trace 0000000000000002 ]---
note: irq/54-eth0[618] exited with preempt_count 1
Unable to handle kernel paging request at virtual address ffffffec
pgd = c0004000
[ffffffec] *pgd=2f7fd821, *pte=00000000, *ppte=00000000
Internal error: Oops: 17 [#2] PREEMPT SMP ARM
Modules linked in:
CPU: 0 PID: 618 Comm: irq/54-eth0 Tainted: G D 3.14.25-rt22-xilinx-00008-g92b3222-dirty #2
task: eea0be80 ti: edaca000 task.ti: edaca000
PC is at kthread_data+0x4/0xc
LR is at irq_thread_dtor+0x50/0xac
pc : [<c003ceec>] lr : [<c005d270>] psr: 20000093
sp : edacbb78 ip : 00000000 fp : 60000093
r10: eea0be80 r9 : 00000001 r8 : edaca018
r7 : 1c251c25 r6 : eea0be80 r5 : eea0be80 r4 : eea0be80
r3 : 00000000 r2 : edacbb78 r1 : edaca038 r0 : eea0be80
Flags: nzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user
Control: 18c5387d Table: 2c51c04a DAC: 00000015
Process irq/54-eth0 (pid: 618, stack limit = 0xedaca248)
Stack: (0xedacbb78 to 0xedacc000)
bb60: c005d220 eea0be80
bb80: 00000000 c003a238 0000006b eea0be80 edacbb9c c00249d8 0000000b c05dcef8
bba0: 00000001 00000000 0000000b c03b4396 edacbc02 00000000 c03b4394 00000000
bbc0: 00000008 c0011ab0 edaca248 0000000b 30ffffff 30303061 20323030 64393565
bbe0: 30303032 35316520 30303032 61302038 30303030 28203030 30663765 32663130
bc00: ee002029 0000002b 00000000 e7f001f2 edacbcd0 c03b4394 00000000 e7100000
bc20: c00122cc edaca000 ec2f6300 c000837c 00000006 00000000 00000004 00000000
bc40: 00030001 c03b4394 edacbccc ec468c80 00000000 c005cc74 c05a107c ee80fe00
bc60: 0000002b 00000000 ffffffff edacbccc ec468c80 c000ee40 0000002b f8f00100
bc80: edacbc98 c0008504 c0056e9c c03b4fb0 a0000013 c0012200 ec469114 a0000013
bca0: 0000428b a0000013 edacbd24 c00569a4 80000013 ee844180 ee844180 c03b4398
bcc0: 00000000 c0012558 00000000 c00122cc ffffffdd 00000000 a417a416 0000a417
bce0: ee844180 eea0be80 00000002 eea0c314 edacbd18 00000000 00000001 ec2f6300
bd00: edaca020 edacbd18 c0056f48 c03b4394 a0000013 ffffffff 00000001 00000000
bd20: 00000000 edacbd24 00000000 00000000 eea0be80 ee844180 edacbd01 00000031
bd40: ed29576c ee844180 ee844180 edacbd7c 00000000 00000000 c05aaa88 c002c2f0
bd60: c05a2200 ed29576c 0000000f 00236577 c0357848 c002c99c ed29576c 00000000
bd80: 00236577 ed295400 00002ee0 ed295860 00000000 ec78bf4e c033af28 c0309de4
bda0: ed295400 c03593ac ed295400 ec6de180 ec6de180 ed295400 ed295458 c035b7d0
bdc0: ec78bf4e 138934b6 239710ac c0344538 000034b6 ec6de180 ec78bf4e ec78bf00
bde0: 00000000 ec6de180 c03e9c5c ec78bf00 00000000 c05acbc8 00000000 c033af28
be00: ec2f6300 c033acc8 c033abcc ec6de180 ec78bf4e c033b730 ec6de180 ee973d40
be20: c05abd44 ee973d40 00000008 c031860c 00000600 00000600 ef587f20 ec6de180
be40: 00000600 c05abd58 ef587160 ec6de180 c05abd34 f08e44e0 2c7f9540 00000001
be60: ee97438c c0318934 ee97438c c0019008 c001cc64 00000f40 ec6de180 ee973d40
be80: c02b8d8c ee9742c0 ec6de180 c02b8dc8 00000000 00000000 ee973d40 000005ea
bea0: ee974314 000005ea 00000040 00000040 edacbeb8 ee97438c eefd8800 c02b8bc0
bec0: 0000012c 00000040 00000000 eefd8808 00236569 c0318b78 c0318ad8 edaca000
bee0: 00000003 00000000 00208040 c059c3b8 00000000 c05a2140 00000001 c0026404
bf00: 00000001 edacbf30 00000048 edaca000 ee80e900 00000001 edaca000 00000000
bf20: 00000000 00000000 00000000 c00265c8 eda86500 c005d348 ee80e900 eda86500
bf40: c005d300 c005ce5c 00000000 c005d220 00000000 00000002 eda86500 c005cd7c
bf60: 00000000 c003cbfc edacbf64 00000000 eda7b000 eda86500 00000000 00000000
bf80: edacbf80 edacbf80 00000001 00010001 edacbf90 edacbf90 edacbfac eda86800
bfa0: c003cb3c 00000000 00000000 c000e5d8 00000000 00000000 00000000 00000000
bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 4980e123 19619042
[<c003ceec>] (kthread_data) from [<c005d270>] (irq_thread_dtor+0x50/0xac)
[<c005d270>] (irq_thread_dtor) from [<c003a238>] (task_work_run+0xb8/0xcc)
[<c003a238>] (task_work_run) from [<c00249d8>] (do_exit+0x3ac/0x86c)
[<c00249d8>] (do_exit) from [<c0011ab0>] (die+0x35c/0x3ac)
[<c0011ab0>] (die) from [<c000837c>] (do_undefinstr+0x138/0x154)
[<c000837c>] (do_undefinstr) from [<c00122cc>] (__und_svc_finish+0x0/0x14)
Exception stack(0xedacbcd0 to 0xedacbd18)
bcc0: ffffffdd 00000000 a417a416 0000a417
bce0: ee844180 eea0be80 00000002 eea0c314 edacbd18 00000000 00000001 ec2f6300
bd00: edaca020 edacbd18 c0056f48 c03b4394 a0000013 ffffffff
[<c00122cc>] (__und_svc_finish) from [<c03b4394>] (rt_spin_lock_slowlock+0x274/0x2dc)
[<c03b4394>] (rt_spin_lock_slowlock) from [<c002c2f0>] (lock_timer_base+0x2c/0x50)
[<c002c2f0>] (lock_timer_base) from [<c002c99c>] (mod_timer+0xb4/0x1d4)
[<c002c99c>] (mod_timer) from [<c0309de4>] (sk_reset_timer+0x14/0x3c)
[<c0309de4>] (sk_reset_timer) from [<c03593ac>] (tcp_prequeue+0x254/0x284)
[<c03593ac>] (tcp_prequeue) from [<c035b7d0>] (tcp_v4_rcv+0x320/0x708)
[<c035b7d0>] (tcp_v4_rcv) from [<c033acc8>] (ip_local_deliver+0xfc/0x35c)
[<c033acc8>] (ip_local_deliver) from [<c033b730>] (ip_rcv+0x808/0x95c)
[<c033b730>] (ip_rcv) from [<c031860c>] (__netif_receive_skb_core+0x490/0x4f0)
[<c031860c>] (__netif_receive_skb_core) from [<c0318934>] (netif_receive_skb_internal+0xa0/0xb4)
[<c0318934>] (netif_receive_skb_internal) from [<c02b8dc8>] (xemacps_rx_poll+0x208/0x38c)
[<c02b8dc8>] (xemacps_rx_poll) from [<c0318b78>] (net_rx_action+0xa0/0x1a0)
[<c0318b78>] (net_rx_action) from [<c0026404>] (do_current_softirqs+0x114/0x234)
[<c0026404>] (do_current_softirqs) from [<c00265c8>] (__local_bh_enable+0x54/0x7c)
[<c00265c8>] (__local_bh_enable) from [<c005d348>] (irq_forced_thread_fn+0x48/0x50)
[<c005d348>] (irq_forced_thread_fn) from [<c005ce5c>] (irq_thread+0xe0/0x144)
[<c005ce5c>] (irq_thread) from [<c003cbfc>] (kthread+0xc0/0xd0)
[<c003cbfc>] (kthread) from [<c000e5d8>] (ret_from_fork+0x14/0x3c)
Code: e513001c e7e00150 e12fff1e e590326c (e5130014)
|
kernel debugging methods;3.14.25-rt22 Kernel hangs during TCP server client communica
