LinuxQuestions.org
Visit Jeremy's Blog.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel
User Name
Password
Linux - Kernel This forum is for all discussion relating to the Linux kernel.

Notices


Reply
  Search this Thread
Old 10-31-2008, 07:36 AM   #1
qaiser
LQ Newbie
 
Registered: Oct 2008
Posts: 1

Rep: Reputation: 0
Help Troubleshooting Kernel Panic


Hi,

I am running Ubuntu x64 server edition. This has worked flawlessly for serveral years, however I have recently started to experience kernel crashes. Crashes seem to occur when downloading using hellanzb from news servers and only with large files (4GB+, though this varies. I updated from kernel 2.6.24-21 to 2.6.27-7 but problem stil exists.

It smells hardware related but only occurs wuth heavy downloads. I captured the output of the call trace and hardware info is attached. Any ideas on how to troubleshoot this issue?

I am relatively new at doing this.

Thank you!

Call Trace

? enqueue_hrtimer+0x8a/0x140
error_entry+0x50/0x62
? search_extable+0x4c/0x80
? search_extable+0x9/0x80
search_exception_tables+0x21/0x50
fixup_exception+0x0/0x40
do_page_fault+0x19e/0x750
? set_cyc2ms_scale+0x26/0xb0
? getnstimeofday+0x57/0xd0
? set_normalized_timespec+0x32/0x90
? sys_nanoslep+0x20/0x80
? sched_clock_cpu+0x74/0x160
? enqueue_hrtimer+0x40/0x140
error_exit+0x0/0x70

Hardware Info

id:
datastore
description: Desktop Computer
product: System Product Name
vendor: System manufacturer
version: System Version
serial: System Serial Number
width: 32 bits
capabilities: smbios-2.4 dmi-2.4 smp-1.4 smp
configuration:
boot = normal
chassis = desktop
cpus = 0
uuid = C991B198-F7A7-2DB2-11BA-37C89DD8B5E5
id:
core
description: Motherboard
product: StrikerExtreme
vendor: ASUSTeK Computer INC.
physical id:
0
version: 1.XX
serial: 123456789000
id:
firmware
description: BIOS
vendor: Phoenix Technologies, LTD
physical id:
0
version: ASUS StrikerExtreme ACPI BIOS Revision 1305 (08/22/2007)
size: 128KiB
capacity: 960KiB
capabilities: pci pnp apm upgrade shadowing cdboot bootselect socketedrom edd int13floppy360 int13floppy1200 int13floppy720 int13floppy2880 int5printscreen int9keyboard int14serial int17printer int10video acpi usb ls120boot zipboot biosbootspecification
id:
cpu
description: CPU
product: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
vendor: Intel Corp.
physical id:
5
bus info:
cpu@0
version: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
slot: Socket 775
size: 2400MHz
capacity: 3800MHz
width: 64 bits
clock: 266MHz
capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx x86-64 constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr lahf_lm
id:
cache:0
description: L1 cache
physical id:
b
slot: L1 Cache
size: 32KiB
capacity: 32KiB
capabilities: synchronous internal write-back data
id:
cache:1
description: L2 cache
physical id:
c
slot: L2 Cache
size: 4MiB
capacity: 4MiB
capabilities: synchronous internal write-back unified
id:
cache:2
description: L3 cache
physical id:
d
slot: L3 Cache
capabilities: synchronous internal varies
id:
memory
description: System Memory
physical id:
41
slot: System board or motherboard
size: 4GiB
id:
bank:0
description: DIMM 800 MHz (1.2 ns)
product: None
vendor: None
physical id:
0
serial: None
slot: DIMM_A1
size: 1GiB
clock: 800MHz (1.2ns)
id:
bank:1
description: DIMM 800 MHz (1.2 ns)
product: None
vendor: None
physical id:
1
serial: None
slot: DIMM_A2
size: 1GiB
clock: 800MHz (1.2ns)
id:
bank:2
description: DIMM 800 MHz (1.2 ns)
product: None
vendor: None
physical id:
2
serial: None
slot: DIMM_B1
size: 1GiB
clock: 800MHz (1.2ns)
id:
bank:3
description: DIMM 800 MHz (1.2 ns)
product: None
vendor: None
physical id:
3
serial: None
slot: DIMM_B2
size: 1GiB
clock: 800MHz (1.2ns)
id:
pci
description: Host bridge
product: C55 Host Bridge
vendor: nVidia Corporation
physical id:
100
bus info:
pci@0000:00:00.0
version: a2
width: 32 bits
clock: 66MHz
id:
memory:0
description: RAM memory
product: C55 Memory Controller
vendor: nVidia Corporation
physical id:
0.1
bus info:
pci@0000:00:00.1
version: a1
width: 32 bits
clock: 66MHz (15.2ns)
configuration:
latency = 0
id:
memory:1
description: RAM memory
product: C55 Memory Controller
vendor: nVidia Corporation
physical id:
0.2
bus info:
pci@0000:00:00.2
version: a1
width: 32 bits
clock: 66MHz (15.2ns)
configuration:
latency = 0
id:
memory:2
description: RAM memory
product: C55 Memory Controller
vendor: nVidia Corporation
physical id:
0.3
bus info:
pci@0000:00:00.3
version: a1
width: 32 bits
clock: 66MHz (15.2ns)
capabilities: bus_master
configuration:
latency = 0
id:
memory:3
description: RAM memory
product: C55 Memory Controller
vendor: nVidia Corporation
physical id:
0.4
bus info:
pci@0000:00:00.4
version: a1
width: 32 bits
clock: 66MHz (15.2ns)
capabilities: bus_master
configuration:
latency = 0
id:
memory:4
description: RAM memory
product: C55 Memory Controller
vendor: nVidia Corporation
physical id:
0.5
bus info:
pci@0000:00:00.5
version: a2
width: 32 bits
clock: 66MHz (15.2ns)
capabilities: bus_master
configuration:
latency = 0
id:
memory:5
description: RAM memory
product: C55 Memory Controller
vendor: nVidia Corporation
physical id:
0.6
bus info:
pci@0000:00:00.6
version: a1
width: 32 bits
clock: 66MHz (15.2ns)
configuration:
latency = 0
id:
memory:6
description: RAM memory
product: C55 Memory Controller
vendor: nVidia Corporation
physical id:
0.7
bus info:
pci@0000:00:00.7
version: a1
width: 32 bits
clock: 66MHz (15.2ns)
configuration:
latency = 0
id:
memory:7
description: RAM memory
product: C55 Memory Controller
vendor: nVidia Corporation
physical id:
1
bus info:
pci@0000:00:01.0
version: a1
width: 32 bits
clock: 66MHz (15.2ns)
configuration:
latency = 0
id:
memory:8
description: RAM memory
product: C55 Memory Controller
vendor: nVidia Corporation
physical id:
1.1
bus info:
pci@0000:00:01.1
version: a1
width: 32 bits
clock: 66MHz (15.2ns)
configuration:
latency = 0
id:
memory:9
description: RAM memory
product: C55 Memory Controller
vendor: nVidia Corporation
physical id:
1.2
bus info:
pci@0000:00:01.2
version: a1
width: 32 bits
clock: 66MHz (15.2ns)
configuration:
latency = 0
id:
memory:10
description: RAM memory
product: C55 Memory Controller
vendor: nVidia Corporation
physical id:
1.3
bus info:
pci@0000:00:01.3
version: a1
width: 32 bits
clock: 66MHz (15.2ns)
configuration:
latency = 0
id:
memory:11
description: RAM memory
product: C55 Memory Controller
vendor: nVidia Corporation
physical id:
1.4
bus info:
pci@0000:00:01.4
version: a1
width: 32 bits
clock: 66MHz (15.2ns)
configuration:
latency = 0
id:
memory:12
description: RAM memory
product: C55 Memory Controller
vendor: nVidia Corporation
physical id:
1.5
bus info:
pci@0000:00:01.5
version: a1
width: 32 bits
clock: 66MHz (15.2ns)
configuration:
latency = 0
id:
memory:13
description: RAM memory
product: C55 Memory Controller
vendor: nVidia Corporation
physical id:
1.6
bus info:
pci@0000:00:01.6
version: a1
width: 32 bits
clock: 66MHz (15.2ns)
configuration:
latency = 0
id:
memory:14
description: RAM memory
product: C55 Memory Controller
vendor: nVidia Corporation
physical id:
2
bus info:
pci@0000:00:02.0
version: a1
width: 32 bits
clock: 66MHz (15.2ns)
configuration:
latency = 0
id:
memory:15
description: RAM memory
product: C55 Memory Controller
vendor: nVidia Corporation
physical id:
2.1
bus info:
pci@0000:00:02.1
version: a1
width: 32 bits
clock: 66MHz (15.2ns)
capabilities: bus_master
configuration:
latency = 0
id:
memory:16
description: RAM memory
product: C55 Memory Controller
vendor: nVidia Corporation
physical id:
2.2
bus info:
pci@0000:00:02.2
version: a1
width: 32 bits
clock: 66MHz (15.2ns)
configuration:
latency = 0
id:
pci:0
description: PCI bridge
product: C55 PCI Express bridge
vendor: nVidia Corporation
physical id:
3
bus info:
pci@0000:00:03.0
version: a1
width: 32 bits
clock: 33MHz
capabilities: pci pm msi ht pciexpress normal_decode bus_master cap_list
configuration:
driver = pcieport-driver
id:
pci:0
description: PCI bridge
product: 80333 Segment-A PCI Express-to-PCI Express Bridge
vendor: Intel Corporation
physical id:
0
bus info:
pci@0000:01:00.0
version: 00
width: 32 bits
clock: 33MHz
capabilities: pci pciexpress msi pm pcix normal_decode bus_master cap_list
id:
storage
description: RAID bus controller
product: PowerEdge Expandable RAID controller 5
vendor: Dell
physical id:
e
bus info:
pci@0000:02:0e.0
logical name:
scsi0
version: 00
width: 32 bits
clock: 66MHz
capabilities: storage pm msi pcix bus_master cap_list
configuration:
driver = megaraid_sas
latency = 32
mingnt = 128
module = megaraid_sas
id:
disk
description: SCSI Disk
product: PERC 5/i
vendor: DELL
physical id:
2.0.0
bus info:
scsi@0:2.0.0
logical name:
/dev/sda
version: 1.12
serial: 0xsdQD-YpP0-Vmqi-y0jC-rdml-3nMH-vb9xp2
size: 3256GiB
capacity: 3256GiB
capabilities: lvm2
configuration:
ansiversion = 5
id:
pci:1
description: PCI bridge
product: 80333 Segment-B PCI Express-to-PCI Express Bridge
vendor: Intel Corporation
physical id:
0.2
bus info:
pci@0000:01:00.2
version: 00
width: 32 bits
clock: 33MHz
capabilities: pci pciexpress msi pm pcix normal_decode bus_master cap_list
id:
pci:1
description: PCI bridge
product: C55 PCI Express bridge
vendor: nVidia Corporation
physical id:
6
bus info:
pci@0000:00:06.0
version: a1
width: 32 bits
clock: 33MHz
capabilities: pci pm msi ht pciexpress normal_decode bus_master cap_list
configuration:
driver = pcieport-driver
id:
pci:2
description: PCI bridge
product: C55 PCI Express bridge
vendor: nVidia Corporation
physical id:
7
bus info:
pci@0000:00:07.0
version: a1
width: 32 bits
clock: 33MHz
capabilities: pci pm msi ht pciexpress normal_decode bus_master cap_list
configuration:
driver = pcieport-driver
id:
memory:17
description: RAM memory
product: MCP55 Memory Controller
vendor: nVidia Corporation
physical id:
9
bus info:
pci@0000:00:09.0
version: a1
width: 32 bits
clock: 66MHz (15.2ns)
capabilities: ht bus_master cap_list
configuration:
latency = 0
id:
isa
description: ISA bridge
product: MCP55 LPC Bridge
vendor: nVidia Corporation
physical id:
a
bus info:
pci@0000:00:0a.0
version: a2
width: 32 bits
clock: 66MHz
capabilities: isa bus_master
configuration:
latency = 0
id:
serial
description: SMBus
product: MCP55 SMBus
vendor: nVidia Corporation
physical id:
a.1
bus info:
pci@0000:00:0a.1
version: a2
width: 32 bits
clock: 66MHz
capabilities: pm cap_list
configuration:
driver = nForce2_smbus
latency = 0
module = i2c_nforce2
id:
usb:0
description: USB Controller
product: MCP55 USB Controller
vendor: nVidia Corporation
physical id:
b
bus info:
pci@0000:00:0b.0
version: a1
width: 32 bits
clock: 66MHz
capabilities: pm ohci bus_master cap_list
configuration:
driver = ohci_hcd
latency = 0
maxlatency = 1
mingnt = 3
module = ohci_hcd
id:
usb:1
description: USB Controller
product: MCP55 USB Controller
vendor: nVidia Corporation
physical id:
b.1
bus info:
pci@0000:00:0b.1
version: a2
width: 32 bits
clock: 66MHz
capabilities: debug pm ehci bus_master cap_list
configuration:
driver = ehci_hcd
latency = 0
maxlatency = 1
mingnt = 3
module = ehci_hcd
id:
ide
description: IDE interface
product: MCP55 IDE
vendor: nVidia Corporation
physical id:
d
bus info:
pci@0000:00:0d.0
logical name:
scsi1
version: a1
width: 32 bits
clock: 66MHz
capabilities: ide pm bus_master cap_list emulated
configuration:
driver = pata_amd
latency = 0
maxlatency = 1
mingnt = 3
module = pata_amd
id:
disk:0
description: ATA Disk
product: ST3400832A
vendor: Seagate
physical id:
0.0.0
bus info:
scsi@1:0.0.0
logical name:
/dev/sdb
version: 3.02
serial: 5NF018CZ
size: 372GiB (400GB)
capabilities: partitioned partitioned:dos
configuration:
ansiversion = 5
signature = c1d1010e
id:
volume:0
description: EXT3 volume
vendor: Linux
physical id:
1
bus info:
scsi@1:0.0.0,1
logical name:
/dev/sdb1
logical name:
/boot
version: 1.0
serial: 7edf785d-d67e-4039-8a95-e5b3f4407d3c
size: 243MiB
capacity: 243MiB
capabilities: primary bootable journaled extended_attributes large_files huge_files recover ext3 ext2 initialized
configuration:
created = 2007-10-26 17:06:03
filesystem = ext3
modified = 2008-08-11 20:30:48
mount.fstype = ext3
mount.options = rw,relatime,data=ordered
mounted = 2008-08-10 13:07:25
state = mounted
id:
volume:1
description: Extended partition
physical id:
2
bus info:
scsi@1:0.0.0,2
logical name:
/dev/sdb2
size: 372GiB
capacity: 372GiB
capabilities: primary extended partitioned partitioned:extended
id:
logicalvolume
description: Linux LVM Physical Volume partition
physical id:
5
logical name:
/dev/sdb5
serial: vUPmjz-Bdfg-smt4-1tUY-FsAc-qQD1-ojY6qd
size: 372GiB
capacity: 372GiB
capabilities: multi lvm2
id:
disk:1
description: ATA Disk
product: ST3400832A
vendor: Seagate
physical id:
0.1.0
bus info:
scsi@1:0.1.0
logical name:
/dev/sdc
version: 3.02
serial: 5NF01872
size: 372GiB (400GB)
capabilities: partitioned partitioned:dos
configuration:
ansiversion = 5
signature = 000e073d
id:
volume
description: Linux LVM Physical Volume partition
physical id:
1
bus info:
scsi@1:0.1.0,1
logical name:
/dev/sdc1
serial: Sfv2nc-BKmc-yV0p-LCeX-ta2D-ue5m-iAM2Rt
size: 372GiB
capacity: 372GiB
capabilities: primary multi lvm2
id:
pci:3
description: PCI bridge
product: MCP55 PCI bridge
vendor: nVidia Corporation
physical id:
f
bus info:
pci@0000:00:0f.0
version: a2
width: 32 bits
clock: 66MHz
capabilities: pci ht subtractive_decode bus_master cap_list
id:
display
description: VGA compatible controller
product: MGA 2164W [Millennium II]
vendor: Matrox Graphics, Inc.
physical id:
6
bus info:
pci@0000:06:06.0
version: 00
width: 32 bits
clock: 33MHz
capabilities: vga_controller bus_master
configuration:
latency = 32
id:
network:0
description: Ethernet interface
product: 82546GB Gigabit Ethernet Controller
vendor: Intel Corporation
physical id:
7
bus info:
pci@0000:06:07.0
logical name:
eth0
version: 03
serial: 00:04:23:cb:5c:ea
size: 1GB/s
capacity: 1GB/s
width: 64 bits
clock: 66MHz
capabilities: pm pcix bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
configuration:
autonegotiation = on
broadcast = yes
driver = e1000
driverversion = 7.3.20-k2-NAPI
duplex = full
firmware = N/A
latency = 32
link = yes
mingnt = 255
module = e1000
multicast = yes
port = twisted pair
slave = yes
speed = 1GB/s
id:
network:1
description: Ethernet interface
product: 82546GB Gigabit Ethernet Controller
vendor: Intel Corporation
physical id:
7.1
bus info:
pci@0000:06:07.1
logical name:
eth1
version: 03
serial: 00:04:23:cb:5c:ea
size: 1GB/s
capacity: 1GB/s
width: 64 bits
clock: 66MHz
capabilities: pm pcix bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
configuration:
autonegotiation = on
broadcast = yes
driver = e1000
driverversion = 7.3.20-k2-NAPI
duplex = full
firmware = N/A
latency = 32
link = yes
mingnt = 255
module = e1000
multicast = yes
port = twisted pair
slave = yes
speed = 1GB/s
id:
firewire
description: FireWire (IEEE 1394)
product: IEEE 1394 Host Controller
vendor: VIA Technologies, Inc.
physical id:
b
bus info:
pci@0000:06:0b.0
version: c0
width: 32 bits
clock: 33MHz
capabilities: pm ohci bus_master cap_list
configuration:
driver = ohci1394
latency = 32
maxlatency = 32
module = ohci1394
id:
bridge:0
description: Ethernet interface
product: MCP55 Ethernet
vendor: nVidia Corporation
physical id:
11
bus info:
pci@0000:00:11.0
logical name:
eth2
version: a2
serial: 00:04:23:cb:5c:ea
size: 1000000000
capacity: 1000000000
width: 32 bits
clock: 66MHz
capabilities: bridge pm msix msi ht bus_master cap_list ethernet physical mii 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
configuration:
autonegotiation = on
broadcast = yes
driver = forcedeth
driverversion = 0.61
duplex = full
latency = 0
link = yes
maxlatency = 20
mingnt = 1
module = forcedeth
multicast = yes
port = MII
slave = yes
speed = 1GB/s
id:
bridge:1
description: Ethernet interface
product: MCP55 Ethernet
vendor: nVidia Corporation
physical id:
12
bus info:
pci@0000:00:12.0
logical name:
eth3
version: a2
serial: 00:04:23:cb:5c:ea
size: 1000000000
capacity: 1000000000
width: 32 bits
clock: 66MHz
capabilities: bridge pm msix msi ht bus_master cap_list ethernet physical mii 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
configuration:
autonegotiation = on
broadcast = yes
driver = forcedeth
driverversion = 0.61
duplex = full
latency = 0
link = yes
maxlatency = 20
mingnt = 1
module = forcedeth
multicast = yes
port = MII
slave = yes
speed = 1GB/s
id:
pci:4
description: PCI bridge
product: MCP55 PCI Express bridge
vendor: nVidia Corporation
physical id:
13
bus info:
pci@0000:00:13.0
version: a2
width: 32 bits
clock: 33MHz
capabilities: pci pm msi ht pciexpress normal_decode bus_master cap_list
configuration:
driver = pcieport-driver
id:
pci:5
description: PCI bridge
product: MCP55 PCI Express bridge
vendor: nVidia Corporation
physical id:
14
bus info:
pci@0000:00:14.0
version: a2
width: 32 bits
clock: 33MHz
capabilities: pci pm msi ht pciexpress normal_decode bus_master cap_list
configuration:
driver = pcieport-driver
id:
pci:6
description: PCI bridge
product: MCP55 PCI Express bridge
vendor: nVidia Corporation
physical id:
15
bus info:
pci@0000:00:15.0
version: a2
width: 32 bits
clock: 33MHz
capabilities: pci pm msi ht pciexpress normal_decode bus_master cap_list
configuration:
driver = pcieport-driver
id:
pci:7
description: PCI bridge
product: MCP55 PCI Express bridge
vendor: nVidia Corporation
physical id:
16
bus info:
pci@0000:00:16.0
version: a2
width: 32 bits
clock: 33MHz
capabilities: pci pm msi ht pciexpress normal_decode bus_master cap_list
configuration:
driver = pcieport-driver
id:
pci:8
description: PCI bridge
product: MCP55 PCI Express bridge
vendor: nVidia Corporation
physical id:
17
bus info:
pci@0000:00:17.0
version: a2
width: 32 bits
clock: 33MHz
capabilities: pci pm msi ht pciexpress normal_decode bus_master cap_list
configuration:
driver = pcieport-driver
id:
pci:9
description: PCI bridge
product: MCP55 PCI Express bridge
vendor: nVidia Corporation
physical id:
18
bus info:
pci@0000:00:18.0
version: a2
width: 32 bits
clock: 33MHz
capabilities: pci pm msi ht pciexpress normal_decode bus_master cap_list
configuration:
driver = pcieport-driver
id:
network
description: Ethernet interface
physical id:
1
logical name:
bond0
serial: 00:04:23:cb:5c:ea
capabilities: ethernet physical
configuration:
broadcast = yes
driver = bonding
driverversion = 3.2.3
firmware = 2
ip = 192.168.0.8
master = yes
multicast = yes
 
Old 11-02-2008, 02:19 PM   #2
Mara
Moderator
 
Registered: Feb 2002
Location: Grenoble
Distribution: Debian
Posts: 9,696

Rep: Reputation: 232Reputation: 232Reputation: 232
It seems it has crashed in enqueue_hrtimer, which doesn't do any hardware stuff - it uses kernel timer structures. It may mean (you need to convert the addresses to assembly positions to find out where exactly it was, do you know how?) that the timer structures got corrupted. That option means not a very easy debug. As a first shot I would write a script that will copy cat /proc/timer_list to file and try to generate the crash while having it running. The last content of timer_list should be the most interesting one...
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Troubleshooting lockups and kernel panics. oskar Linux - Hardware 5 01-21-2008 04:35 AM
Troubleshooting Newly Compiled Kernel binarybob0001 Linux - General 2 11-20-2005 04:06 AM
Troubleshooting a kernel build Fritz_Monroe Linux - Newbie 8 06-13-2005 03:23 PM
kernel panic (narius panic) narius Linux - Newbie 3 06-20-2002 03:56 PM
Kernel panic making me panic! Linux_N0oby Linux - Software 19 05-29-2002 12:31 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel

All times are GMT -5. The time now is 07:08 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration