LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 04-12-2014, 02:39 PM   #1
seighalani
Member
 
Registered: Aug 2007
Posts: 122

Rep: Reputation: 15
BUG: soft lockup - CPU#2x stuck for 10s!


hi everyone

i have centos 5.3 server and recently i recieve "BUG: soft lockup - CPU#24 stuck for 10s!". in my server i have oracle10g.what should i do?thanks in advance

server : hp dl580 g5
kernel: 2.6.18-128

Apr 12 13:11:10 db1 kernel: BUG: soft lockup - CPU#63 stuck for 10s! [oracle:2245]
Apr 12 13:11:10 db1 kernel: CPU 63:
Apr 12 13:11:10 db1 kernel: Modules linked in: nfsd exportfs lockd nfs_acl auth_rpcgss oracleasm(U) autofs4 hidp rfcomm l2cap bluetooth sunrpc netxen_nic bonding dm_mirror dm_multipath scsi_dh video hwmon backlight sbs i2c_ec i2c_core button battery asus_acpi acpi_memhotplug ac xfrm_nalgo crypto_api parport_pc lp parport joydev sr_mod cdrom st sg serio_raw pcspkr bnx2 dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache qla2400(U) qla2300(U) usb_storage ata_piix libata shpchp cciss ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(FU) sd_mod scsi_mod qla2xxx_conf(FU) intermodule(U)
Apr 12 13:11:10 db1 kernel: Pid: 2245, comm: oracle Tainted: GF 2.6.18-128.el5 #1
Apr 12 13:11:10 db1 kernel: RIP: 0010:[<ffffffff80064cb4>] [<ffffffff80064cb4>] .text.lock.spinlock+0x2/0x30
Apr 12 13:11:10 db1 kernel: RSP: 0018:ffff8104bba399e0 EFLAGS: 00000286
Apr 12 13:11:10 db1 kernel: RAX: 0000000000000259 RBX: ffff81104e695e98 RCX: 0000000000000034
Apr 12 13:11:10 db1 kernel: RDX: ffff8104bba39b38 RSI: 0000000000000000 RDI: ffff81103e1f78a8
Apr 12 13:11:10 db1 kernel: RBP: 0000000000000202 R08: ffff811040001600 R09: 0000000000000020
Apr 12 13:11:10 db1 kernel: R10: 0000000000000020 R11: ffff81081e0247a0 R12: 0000000000000010
Apr 12 13:11:10 db1 kernel: R13: ffffffff800cbca1 R14: ffffffffffffff10 R15: 0000000b30516000
Apr 12 13:11:10 db1 kernel: FS: 00002abade256c30(0000) GS:ffff81183ffc5ac0(0000) knlGS:0000000000000000
Apr 12 13:11:12 db1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr 12 13:11:42 db1 snmpd[19161]: Connection from UDP: [10.1.60.102]:1048
Apr 12 13:11:55 db1 kernel: CR2: 000000107a381922 CR3: 000000012b3a5000 CR4: 00000000000006e0
Apr 12 13:11:55 db1 snmpd[19161]: Received SNMP packet(s) from UDP: [10.1.60.102]:1048
Apr 12 13:11:55 db1 kernel:
Apr 12 13:11:55 db1 kernel: Call Trace:
Apr 12 13:11:55 db1 kernel: [<ffffffff80032f92>] page_referenced_file+0x42/0xc3
Apr 12 13:11:55 db1 kernel: [<ffffffff8003bafe>] page_referenced+0xcb/0xe4
Apr 12 13:11:55 db1 snmpd[19161]: Connection from UDP: [10.1.60.102]:1048
Apr 12 13:11:55 db1 kernel: [<ffffffff800c6d80>] shrink_active_list+0x192/0x426
Apr 12 13:11:55 db1 snmpd[19161]: Connection from UDP: [10.1.60.102]:1048
Apr 12 13:11:55 db1 kernel: [<ffffffff80012ce4>] shrink_zone+0xd8/0x11c
Apr 12 13:11:55 db1 snmpd[19161]: Connection from UDP: [10.1.60.102]:1048
Apr 12 13:11:55 db1 kernel: [<ffffffff800c801b>] try_to_free_pages+0x197/0x2b9
Apr 12 13:12:05 db1 kernel: [<ffffffff8000f271>] __alloc_pages+0x1cb/0x2ce
Apr 12 13:12:05 db1 kernel: [<ffffffff8002de8b>] __alloc_skb+0x77/0x123
Apr 12 13:12:05 db1 kernel: [<ffffffff80025be7>] tcp_sendmsg+0x564/0xb2f
Apr 12 13:12:05 db1 kernel: [<ffffffff80037696>] do_sock_write+0xc4/0xce
Apr 12 13:12:05 db1 kernel: [<ffffffff80047191>] sock_aio_write+0x4f/0x5e
Apr 12 13:12:05 db1 snmpd[19161]: Connection from UDP: [10.1.60.102]:1048
Apr 12 13:12:05 db1 kernel: [<ffffffff80017d2d>] do_sync_write+0xc7/0x104
Apr 12 13:12:05 db1 snmpd[19161]: Connection from UDP: [10.1.60.102]:1048
Apr 12 13:12:05 db1 kernel: [<ffffffff8009db21>] autoremove_wake_function+0x0/0x2e
Apr 12 13:12:05 db1 snmpd[19161]: Connection from UDP: [10.1.60.102]:1048
Apr 12 13:12:05 db1 kernel: [<ffffffff800165b1>] vfs_write+0xe1/0x174
Apr 12 13:12:05 db1 snmpd[19161]: Connection from UDP: [10.1.60.102]:1048
Apr 12 13:12:05 db1 kernel: [<ffffffff80016e6b>] sys_write+0x45/0x6e
Apr 12 13:12:05 db1 snmpd[19161]: Connection from UDP: [10.1.60.102]:1048
Apr 12 13:12:05 db1 kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0
Apr 12 13:12:05 db1 kernel:
Apr 12 13:12:05 db1 kernel: BUG: soft lockup - CPU#23 stuck for 10s! [oracle:665]
Apr 12 13:12:05 db1 kernel: CPU 23:
Apr 12 13:12:05 db1 kernel: Modules linked in: nfsd exportfs lockd nfs_acl auth_rpcgss oracleasm(U) autofs4 hidp rfcomm l2cap bluetooth sunrpc netxen_nic bonding dm_mirror dm_multipath scsi_dh video hwmon backlight sbs i2c_ec i2c_core button battery asus_acpi acpi_memhotplug ac xfrm_nalgo crypto_api parport_pc lp parport joydev sr_mod cdrom st sg serio_raw pcspkr bnx2 dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache qla2400(U) qla2300(U) usb_storage ata_piix libata shpchp cciss ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(FU) sd_mod scsi_mod qla2xxx_conf(FU) intermodule(U)
Apr 12 13:12:05 db1 kernel: Pid: 665, comm: oracle Tainted: GF 2.6.18-128.el5 #1
Apr 12 13:12:05 db1 kernel: RIP: 0010:[<ffffffff80064cb4>] [<ffffffff80064cb4>] .text.lock.spinlock+0x2/0x30
Apr 12 13:12:05 db1 kernel: RSP: 0018:ffff8106235afac0 EFLAGS: 00000286
Apr 12 13:12:05 db1 kernel: RAX: 0000000000000259 RBX: ffff81104dd80298 RCX: 0000000000000034
Apr 12 13:12:05 db1 kernel: RDX: ffff8106235afc18 RSI: 0000000000000000 RDI: ffff81103e1f78a8
Apr 12 13:12:05 db1 kernel: RBP: 0000000000000000 R08: ffff811040001600 R09: 0000000000000020
Apr 12 13:12:05 db1 kernel: R10: 0000000000000020 R11: 00000000ffffffff R12: 0000000000000000
Apr 12 13:12:05 db1 kernel: R13: 0000000000008000 R14: ffffffff8002ae00 R15: ffff8106235afb04
Apr 12 13:12:05 db1 kernel: FS: 00002b8d58a38c30(0000) GS:ffff81105c3f9840(0000) knlGS:0000000000000000
Apr 12 13:12:05 db1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr 12 13:12:05 db1 kernel: CR2: 000000147021a200 CR3: 000000059d13e000 CR4: 00000000000006e0
Apr 12 13:12:05 db1 kernel:
Apr 12 13:12:05 db1 kernel: Call Trace:
Apr 12 13:12:05 db1 kernel: [<ffffffff80032f92>] page_referenced_file+0x42/0xc3
Apr 12 13:12:05 db1 kernel: [<ffffffff8003bafe>] page_referenced+0xcb/0xe4
Apr 12 13:12:05 db1 kernel: [<ffffffff800c6d80>] shrink_active_list+0x192/0x426
Apr 12 13:12:05 db1 kernel: [<ffffffff8009db21>] autoremove_wake_function+0x0/0x2e
Apr 12 13:12:05 db1 kernel: [<ffffffff80012ce4>] shrink_zone+0xd8/0x11c
Apr 12 13:12:05 db1 kernel: [<ffffffff800c801b>] try_to_free_pages+0x197/0x2b9
Apr 12 13:12:05 db1 kernel: [<ffffffff8000f271>] __alloc_pages+0x1cb/0x2ce
Apr 12 13:12:05 db1 kernel: [<ffffffff8002b56e>] get_zeroed_page+0x21/0x82
Apr 12 13:12:05 db1 kernel: [<ffffffff800161fc>] __pte_alloc+0x1a/0x138
Apr 12 13:12:05 db1 kernel: [<ffffffff80008798>] __handle_mm_fault+0x12d/0xe5c
Apr 12 13:12:05 db1 kernel: [<ffffffff80066b9a>] do_page_fault+0x4cb/0x830
Apr 12 13:12:05 db1 kernel: [<ffffffff8005dde9>] error_exit+0x0/0x84
Apr 12 13:12:05 db1 kernel:
Apr 12 13:12:05 db1 kernel: BUG: soft lockup - CPU#26 stuck for 10s! [oracle:9942]
Apr 12 13:12:05 db1 kernel: CPU 26:
Apr 12 13:12:05 db1 kernel: Modules linked in: nfsd exportfs lockd nfs_acl auth_rpcgss oracleasm(U) autofs4 hidp rfcomm l2cap bluetooth sunrpc netxen_nic bonding dm_mirror dm_multipath scsi_dh video hwmon backlight sbs i2c_ec i2c_core button battery asus_acpi acpi_memhotplug ac xfrm_nalgo crypto_api parport_pc lp parport joydev sr_mod cdrom st sg serio_raw pcspkr bnx2 dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache qla2400(U) qla2300(U) usb_storage ata_piix libata shpchp cciss ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(FU) sd_mod scsi_mod qla2xxx_conf(FU) intermodule(U)
Apr 12 13:12:05 db1 kernel: Pid: 9942, comm: oracle Tainted: GF 2.6.18-128.el5 #1
Apr 12 13:12:05 db1 kernel: RIP: 0010:[<ffffffff80064cb4>] [<ffffffff80064cb4>] .text.lock.spinlock+0x2/0x30
Apr 12 13:12:05 db1 kernel: RSP: 0018:ffff811b97e55ac0 EFLAGS: 00000286
Apr 12 13:12:05 db1 kernel: RAX: 0000000000000259 RBX: ffff8101066d7348 RCX: 0000000000000034
Apr 12 13:12:05 db1 kernel: RDX: ffff811b97e55c18 RSI: 0000000000000000 RDI: ffff81103e1f78a8
Apr 12 13:12:05 db1 kernel: RBP: 0000000000000000 R08: ffff81000008b600 R09: 000000000000000c
Apr 12 13:12:05 db1 kernel: R10: 0000000000000000 R11: ffff81183fca3ee8 R12: 0000000000000000
Apr 12 13:12:05 db1 kernel: R13: 0000000000008000 R14: ffffffff8002ae00 R15: ffff811b97e55b04
Apr 12 13:12:05 db1 kernel: FS: 00002acf3a929c30(0000) GS:ffff81183fc86640(0000) knlGS:0000000000000000
Apr 12 13:12:05 db1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr 12 13:12:05 db1 kernel: CR2: 0000000386ffbf98 CR3: 000000082c51b000 CR4: 00000000000006e0
Apr 12 13:12:05 db1 kernel:
Apr 12 13:12:05 db1 kernel: Call Trace:
Apr 12 13:12:05 db1 kernel: [<ffffffff80032f92>] page_referenced_file+0x42/0xc3
Apr 12 13:12:05 db1 kernel: [<ffffffff8003bafe>] page_referenced+0xcb/0xe4
Apr 12 13:12:05 db1 kernel: [<ffffffff800c6d80>] shrink_active_list+0x192/0x426
Apr 12 13:12:05 db1 kernel: [<ffffffff8009db21>] autoremove_wake_function+0x0/0x2e
Apr 12 13:12:05 db1 kernel: [<ffffffff80012ce4>] shrink_zone+0xd8/0x11c
Apr 12 13:12:05 db1 kernel: [<ffffffff800c801b>] try_to_free_pages+0x197/0x2b9
Apr 12 13:12:05 db1 kernel: [<ffffffff8000f271>] __alloc_pages+0x1cb/0x2ce
Apr 12 13:12:05 db1 kernel: [<ffffffff8002b56e>] get_zeroed_page+0x21/0x82
Apr 12 13:12:05 db1 kernel: [<ffffffff800161fc>] __pte_alloc+0x1a/0x138
Apr 12 13:12:05 db1 kernel: [<ffffffff80008798>] __handle_mm_fault+0x12d/0xe5c
Apr 12 13:12:05 db1 kernel: [<ffffffff80066b9a>] do_page_fault+0x4cb/0x830
Apr 12 13:12:05 db1 kernel: [<ffffffff8005dde9>] error_exit+0x0/0x84
Apr 12 13:12:05 db1 kernel:
Apr 12 13:12:05 db1 kernel: BUG: soft lockup - CPU#66 stuck for 10s! [oracle:9897]
Apr 12 13:12:05 db1 kernel: CPU 66:
Apr 12 13:12:05 db1 kernel: Modules linked in: nfsd exportfs lockd nfs_acl auth_rpcgss oracleasm(U) autofs4 hidp rfcomm l2cap bluetooth sunrpc netxen_nic bonding dm_mirror dm_multipath scsi_dh video hwmon backlight sbs i2c_ec i2c_core button battery asus_acpi acpi_memhotplug ac xfrm_nalgo crypto_api parport_pc lp parport joydev sr_mod cdrom st sg serio_raw pcspkr bnx2 dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache qla2400(U) qla2300(U) usb_storage ata_piix libata shpchp cciss ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(FU) sd_mod scsi_mod qla2xxx_conf(FU) intermodule(U)
Apr 12 13:12:05 db1 kernel: Pid: 9897, comm: oracle Tainted: GF 2.6.18-128.el5 #1
Apr 12 13:12:05 db1 kernel: RIP: 0010:[<ffffffff80064cb7>] [<ffffffff80064cb7>] .text.lock.spinlock+0x5/0x30
Apr 12 13:12:05 db1 kernel: RSP: 0018:ffff8101410839e0 EFLAGS: 00000286
Apr 12 13:12:05 db1 kernel: RAX: 0000000000000259 RBX: ffff811856a16988 RCX: 0000000000000034
Apr 12 13:12:05 db1 kernel: RDX: ffff810141083b38 RSI: 0000000000000000 RDI: ffff81103e1f78a8
Apr 12 13:12:05 db1 kernel: RBP: 0000000000000206 R08: ffff811840001600 R09: 000000000000000b
Apr 12 13:12:05 db1 kernel: R10: 0000000000000000 R11: 00000000ffffffff R12: 0000000000000010
Apr 12 13:12:05 db1 kernel: R13: ffffffff800cbca1 R14: ffffffffffffff10 R15: 00000009df148000
Apr 12 13:12:05 db1 kernel: FS: 00002af2371fdc30(0000) GS:ffff81183f867840(0000) knlGS:0000000000000000
Apr 12 13:12:05 db1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr 12 13:12:05 db1 kernel: CR2: 0000001460a86240 CR3: 0000000335422000 CR4: 00000000000006e0
Apr 12 13:12:05 db1 kernel:
Apr 12 13:12:05 db1 kernel: Call Trace:
Apr 12 13:12:05 db1 kernel: [<ffffffff80032f92>] page_referenced_file+0x42/0xc3
Apr 12 13:12:05 db1 kernel: [<ffffffff8003bafe>] page_referenced+0xcb/0xe4
Apr 12 13:12:05 db1 kernel: [<ffffffff800c6d80>] shrink_active_list+0x192/0x426
Apr 12 13:12:05 db1 kernel: [<ffffffff80012ce4>] shrink_zone+0xd8/0x11c
Apr 12 13:12:05 db1 kernel: [<ffffffff800c801b>] try_to_free_pages+0x197/0x2b9
Apr 12 13:12:05 db1 kernel: [<ffffffff8000f271>] __alloc_pages+0x1cb/0x2ce
Apr 12 13:12:05 db1 kernel: [<ffffffff8002de8b>] __alloc_skb+0x77/0x123
Apr 12 13:12:05 db1 kernel: [<ffffffff80025be7>] tcp_sendmsg+0x564/0xb2f
Apr 12 13:12:05 db1 kernel: [<ffffffff80037696>] do_sock_write+0xc4/0xce
Apr 12 13:12:05 db1 kernel: [<ffffffff80047191>] sock_aio_write+0x4f/0x5e
Apr 12 13:12:05 db1 kernel: [<ffffffff80017d2d>] do_sync_write+0xc7/0x104
Apr 12 13:12:05 db1 kernel: [<ffffffff8009db21>] autoremove_wake_function+0x0/0x2e
Apr 12 13:12:05 db1 kernel: [<ffffffff800165b1>] vfs_write+0xe1/0x174
Apr 12 13:12:05 db1 kernel: [<ffffffff80016e6b>] sys_write+0x45/0x6e
Apr 12 13:12:05 db1 kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0
Apr 12 13:12:05 db1 kernel:
Apr 12 13:12:05 db1 kernel: BUG: soft lockup - CPU#64 stuck for 10s! [oracle:9775]
Apr 12 13:12:05 db1 kernel: CPU 64:
Apr 12 13:12:05 db1 kernel: Modules linked in: nfsd exportfs lockd nfs_acl auth_rpcgss oracleasm(U) autofs4 hidp rfcomm l2cap bluetooth sunrpc netxen_nic bonding dm_mirror dm_multipath scsi_dh video hwmon backlight sbs i2c_ec i2c_core button battery asus_acpi acpi_memhotplug ac xfrm_nalgo crypto_api parport_pc lp parport joydev sr_mod cdrom st sg serio_raw pcspkr bnx2 dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache qla2400(U) qla2300(U) usb_storage ata_piix libata shpchp cciss ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(FU) sd_mod scsi_mod qla2xxx_conf(FU) intermodule(U)
Apr 12 13:12:05 db1 kernel: Pid: 9775, comm: oracle Tainted: GF 2.6.18-128.el5 #1
Apr 12 13:12:05 db1 kernel: RIP: 0010:[<ffffffff80064cb4>] [<ffffffff80064cb4>] .text.lock.spinlock+0x2/0x30
Apr 12 13:12:05 db1 kernel: RSP: 0018:ffff8111a70b5ac0 EFLAGS: 00000286
Apr 12 13:12:05 db1 kernel: RAX: 0000000000000259 RBX: ffff811856a26cf8 RCX: 0000000000000034
Apr 12 13:12:05 db1 kernel: RDX: ffff8111a70b5c18 RSI: 0000000000000000 RDI: ffff81103e1f78a8
Apr 12 13:12:05 db1 kernel: RBP: 0000000000000202 R08: ffff811840001600 R09: 000000000000000b
Apr 12 13:12:05 db1 kernel: R10: 0000000000000000 R11: ffff81183f5407a0 R12: 0000000000000010
Apr 12 13:12:05 db1 kernel: R13: ffffffff800cbca1 R14: ffffffffffffff10 R15: 00000010c338a000
Apr 12 13:12:05 db1 kernel: FS: 00002b78a3bcfc30(0000) GS:ffff81183f80f9c0(0000) knlGS:0000000000000000
Apr 12 13:12:05 db1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr 12 13:12:05 db1 kernel: CR2: 00000002b6fd9868 CR3: 00000015467d9000 CR4: 00000000000006e0
Apr 12 13:12:05 db1 kernel:
Apr 12 13:12:05 db1 kernel: Call Trace:
Apr 12 13:12:05 db1 kernel: [<ffffffff80032f92>] page_referenced_file+0x42/0xc3
Apr 12 13:12:05 db1 kernel: [<ffffffff8003bafe>] page_referenced+0xcb/0xe4
Apr 12 13:12:05 db1 kernel: [<ffffffff800c6d80>] shrink_active_list+0x192/0x426
Apr 12 13:12:05 db1 kernel: [<ffffffff80012ce4>] shrink_zone+0xd8/0x11c
Apr 12 13:12:05 db1 kernel: [<ffffffff800c801b>] try_to_free_pages+0x197/0x2b9
Apr 12 13:12:05 db1 kernel: [<ffffffff8000f271>] __alloc_pages+0x1cb/0x2ce
Apr 12 13:12:05 db1 kernel: [<ffffffff8002b56e>] get_zeroed_page+0x21/0x82
Apr 12 13:12:05 db1 kernel: [<ffffffff800161fc>] __pte_alloc+0x1a/0x138
Apr 12 13:12:05 db1 kernel: [<ffffffff80008798>] __handle_mm_fault+0x12d/0xe5c
Apr 12 13:12:05 db1 kernel: [<ffffffff80066b9a>] do_page_fault+0x4cb/0x830
Apr 12 13:12:05 db1 kernel: [<ffffffff8005dde9>] error_exit+0x0/0x84
Apr 12 13:12:05 db1 kernel:
Apr 12 13:12:05 db1 kernel: BUG: soft lockup - CPU#24 stuck for 10s! [oracle:4682]
Apr 12 13:12:05 db1 kernel: CPU 24:
Apr 12 13:12:05 db1 kernel: Modules linked in: nfsd exportfs lockd nfs_acl auth_rpcgss oracleasm(U) autofs4 hidp rfcomm l2cap bluetooth sunrpc netxen_nic bonding dm_mirror dm_multipath scsi_dh video hwmon backlight sbs i2c_ec i2c_core button battery asus_acpi acpi_memhotplug ac xfrm_nalgo crypto_api parport_pc lp parport joydev sr_mod cdrom st sg serio_raw pcspkr bnx2 dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache qla2400(U) qla2300(U) usb_storage ata_piix libata shpchp cciss ext3 jbd uhci_hcd ohci_hcd ehci_hcd qla2xxx(FU) sd_mod scsi_mod qla2xxx_conf(FU) intermodule(U)
Apr 12 13:12:05 db1 kernel: Pid: 4682, comm: oracle Tainted: GF 2.6.18-128.el5 #1
Apr 12 13:12:05 db1 kernel: RIP: 0010:[<ffffffff80064cb4>] [<ffffffff80064cb4>] .text.lock.spinlock+0x2/0x30
Apr 12 13:12:05 db1 kernel: RSP: 0018:ffff81018d6a19c0 EFLAGS: 00000286
Apr 12 13:12:05 db1 kernel: RAX: 0000000000000219 RBX: ffff811046d67e88 RCX: 0000000114443e67
Apr 12 13:12:06 db1 kernel: RDX: 0000000000000062 RSI: 0000000000000001 RDI: ffff81103e1f78a8
Apr 12 13:12:06 db1 kernel: RBP: 000000122f766047 R08: 0000000000000041 R09: 000000000000003f
Apr 12 13:12:06 db1 kernel: R10: ffff81105c143cf0 R11: 0000000000000000 R12: ffffffff00000018
Apr 12 13:12:06 db1 kernel: R13: ffff81122f92c268 R14: 000000000000002e R15: ffff810f96b31b88
Apr 12 13:12:06 db1 kernel: FS: 00002b91bc78fc30(0000) GS:ffff81183fc40340(0000) knlGS:0000000000000000
Apr 12 13:12:06 db1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr 12 13:12:06 db1 kernel: CR2: 000000089761f15d CR3: 0000000688bb8000 CR4: 00000000000006e0
Apr 12 13:12:06 db1 kernel:
Apr 12 13:12:06 db1 kernel: Call Trace:
Apr 12 13:12:06 db1 kernel: [<ffffffff80032f92>] page_referenced_file+0x42/0xc3
Apr 12 13:12:06 db1 kernel: [<ffffffff800c2d5f>] __remove_from_page_cache+0x1b/0x3a
Apr 12 13:12:06 db1 kernel: [<ffffffff800cdf94>] swap_duplicate+0x5a/0xf1
Apr 12 13:12:06 db1 kernel: [<ffffffff8003badf>] page_referenced+0xac/0xe4
Apr 12 13:12:06 db1 kernel: [<ffffffff800c72b6>] shrink_inactive_list+0x191/0x7f9
Apr 12 13:12:06 db1 kernel: [<ffffffff8003bb08>] page_referenced+0xd5/0xe4
Apr 12 13:12:06 db1 kernel: [<ffffffff80047ab0>] __pagevec_release+0x19/0x22
Apr 12 13:12:06 db1 kernel: [<ffffffff800c7004>] shrink_active_list+0x416/0x426
Apr 12 13:12:06 db1 kernel: [<ffffffff8009db21>] autoremove_wake_function+0x0/0x2e
Apr 12 13:12:06 db1 kernel: [<ffffffff80012d02>] shrink_zone+0xf6/0x11c
Apr 12 13:12:06 db1 kernel: [<ffffffff800c801b>] try_to_free_pages+0x197/0x2b9
Apr 12 13:12:06 db1 kernel: [<ffffffff8000f271>] __alloc_pages+0x1cb/0x2ce
Apr 12 13:12:06 db1 kernel: [<ffffffff800076ad>] find_get_page+0x21/0x50
Apr 12 13:12:06 db1 kernel: [<ffffffff8002b56e>] get_zeroed_page+0x21/0x82
Apr 12 13:12:06 db1 kernel: [<ffffffff800161fc>] __pte_alloc+0x1a/0x138
Apr 12 13:12:07 db1 kernel: [<ffffffff80008798>] __handle_mm_fault+0x12d/0xe5c
Apr 12 13:12:07 db1 kernel: [<ffffffff80066b9a>] do_page_fault+0x4cb/0x830
Apr 12 13:12:07 db1 kernel: [<ffffffff8005dde9>] error_exit+0x0/0x84
Apr 12 13:12:07 db1 kernel:
 
Old 04-14-2014, 12:01 PM   #2
metaschima
Senior Member
 
Registered: Dec 2013
Distribution: Slackware
Posts: 1,982

Rep: Reputation: 491Reputation: 491Reputation: 491Reputation: 491Reputation: 491
I only found this:
http://bugs.centos.org/view.php?id=3095
Looks like the bug is still open.

Could be some obscure kernel bug.
 
Old 04-14-2014, 09:17 PM   #3
btmiller
Senior Member
 
Registered: May 2004
Location: In the DC 'burbs
Distribution: Arch, Scientific Linux, Debian, Ubuntu
Posts: 4,290

Rep: Reputation: 378Reputation: 378Reputation: 378Reputation: 378
If possible, I'd recommend updating to a newer version of the CentOS 5.x series. it could also be bad RAM or a disk drive starting to go bad; testing both (with memtest86 and smartmontools) might be a good idea.
 
Old 04-14-2014, 11:58 PM   #4
seighalani
Member
 
Registered: Aug 2007
Posts: 122

Original Poster
Rep: Reputation: 15
i consult with DB department in our company. they double check and said that it is related to oracle's process parameters . these parameters set to for example 1200 and they changed it to 4000 .now with this change oracle db can support 4000processes Simultaneously. i thinks database generated this error after high load . after this change we dont see that bug error. but recently i will Under consideration it.
 
Old 04-22-2014, 04:18 AM   #5
seighalani
Member
 
Registered: Aug 2007
Posts: 122

Original Poster
Rep: Reputation: 15
hi again

when doublechek i found that swap use is around 15GB! i think we need performance monitoring for swap space, shmax parameter, swappiness and so on.

Last edited by seighalani; 04-22-2014 at 04:20 AM.
 
Old 04-22-2014, 11:24 AM   #6
metaschima
Senior Member
 
Registered: Dec 2013
Distribution: Slackware
Posts: 1,982

Rep: Reputation: 491Reputation: 491Reputation: 491Reputation: 491Reputation: 491
Also check for memory leaks from programs.
 
Old 04-26-2014, 03:20 PM   #7
robertjinx
Member
 
Registered: Oct 2007
Location: Prague, CZ
Distribution: RedHat / CentOS / Ubuntu / SUSE / Debian
Posts: 749

Rep: Reputation: 73
Userspace shouldn't affect the kernel, basically it means that userspace programs that do this or end-up in kernel crash are usually kernel issue. This is not a oracle issue, but oracle helps to generate it.
 
Old 04-29-2014, 12:47 AM   #8
seighalani
Member
 
Registered: Aug 2007
Posts: 122

Original Poster
Rep: Reputation: 15
thanks everyone for good notes. i will explain our tuning or correction as soon as possilbe.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] BUG: soft lockup - CPU#1 stuck for 10s! [swapper:0] john lee Linux - Newbie 4 01-05-2015 11:30 PM
BUG: soft lockup - CPU# stuck for 4278190091s! knspradeep Linux - Server 1 12-31-2013 11:41 AM
BUG: soft lockup - CPU#1 stuck for 61s saurin Linux - Kernel 1 10-22-2010 02:14 PM
Crashing "BUG: soft lockup - CPU#1 stuck for 11s" DavidDiggs Linux - Server 2 06-05-2009 12:43 AM
BUG: soft lockup - CPU#3 stuck for 10s! chakkerz Linux - Server 2 06-16-2008 05:34 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 05:19 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration