Kernel Panic after Kernel Updates

getlin · 12-09-2017, 06:49 PM

Hi @all,

after Kernel updates today I restarted my rootserver.
These are the last lines I can see from the serial console:

Code:

[    1.574358] Code: c0 0f 85 46 05 00 00 48 8b 74 24 08 48 c7 c2 00 dd a6 81 bf ff ff ff ff e8 91 78 21 00 48 98 49 8b 56 10 48 8b 04 c5 a0 1e 8e 81 <48> 8b 14 10 b8 01 00 00 00 49 89 54 24 10 f0 0f c1 02 85 c0 75 
[    1.617795] RIP  [<ffffffff8109be3d>] build_sched_domains+0x72d/0xcf0
[    1.630821]  RSP <ffff88081c5cbdf8>
[    1.637874] ---[ end trace 042f68a87a238ae5 ]---
[    1.647192] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    1.647192] 
[    1.665602] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    1.665602]

What can I do now?

rtmistler · 12-11-2017, 06:35 AM

Please see if this other thread https://www.linuxquestions.org/quest...it-4175614101/ has any information which can help you with your problem.

getlin · 12-11-2017, 07:00 AM

Thanks, but I already knew what "Kernel panic" means.

Hardware is ok.
I've made a downgrade of the kernel and the server is up again.
There were no logs written about the Kernel panic.

rtmistler · 12-11-2017, 07:13 AM

Yes, I agree that the hardware is OK because it runs the older kernel.

I think what it may mean is that the new kernel has a driver which has a problem with your hardware.

If you have the source, you can look where the reference, build_sched_domains, is located and see whether it relates to any particular part of the system.

sundialsvcs · 12-11-2017, 07:24 AM

Most commonly, this problem occurs because the kernel cannot find the init (-equivalent) process, e.g. due to some problem with filesystem drivers or other similar issues. Messages which precede this one should provide clues.

getlin · 12-11-2017, 08:11 AM

OS is Debian 8 (Jessie), kernel package is linux-image-3.16.0-4-amd64
working kernel version: 3.16.43-2+deb8u5
panic kernel version: 3.16.51-2

Debian maintainers changed a lot, the changelogs are very big.

Here is the output from the last boot with the kernel panic:

There is a warning in line 204, is this the problem? What can I do? Any ideas?
https://pastebin.com/h2FYxeek

rtmistler · 12-11-2017, 08:26 AM

Clearly the GPF is a major problem, it may have to do with SMP.

I know there are a ton of kernel changes, but sometimes I compare between the good boot log and the bad boot log.

I've found that ACPI is a major reason for the kernel not booting on a system, because it has a problem with mapping the interrupts or I/O ports, however that would be a new system where it hasn't yet worked.

Still, comparing the good kernel log to the bad one, while they may be vastly different in sections due to the software changes, I feel you may see a case where the working kernel does detection and initialization of a part of the system fine, but the non-working kernel says it didn't detect something. And it may not be noticeable, the non-working one may just say something like, "Detecting ... blah-blah .... Found: None" and then move on, not also noting that it may be a future critical error. If there's any way to turn on more verbose logging, suggest you do that.

getlin · 12-11-2017, 09:19 AM

Thanks for your input.

Well, there are some differences. But I don't understand if they are the root of the problem ...

Are there important differences?

not working, line 26-27:

Code:

[    0.000000] SMBIOS 2.6 present.
[    0.000000] AGP: No AGP bridge found

working:

Code:

[    0.000000] SMBIOS 2.6 present.
[    0.000000] DMI: Supermicro H8SGL/H8SGL, BIOS 3.5        11/25/2013
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] AGP: No AGP bridge found

not working, line 28-29:

Code:

[    0.000000] e820: last_pfn = 0x101f000 max_arch_pfn = 0x400000000
[    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106

working:

Code:

[    0.000000] e820: last_pfn = 0x101f000 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: uncachable
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-EFFFF uncachable
[    0.000000]   F0000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 000000000000 mask FFFF80000000 write-back
[    0.000000]   1 base 000080000000 mask FFFFC0000000 write-back
[    0.000000]   2 base 0000C0000000 mask FFFFE0000000 write-back
[    0.000000]   3 disabled
[    0.000000]   4 disabled
[    0.000000]   5 disabled
[    0.000000]   6 disabled
[    0.000000]   7 disabled
[    0.000000] TOM2: 000000101f000000 aka 66032M
[    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
[    0.000000] e820: update [mem 0xe0000000-0xffffffff] usable ==> reserved
[    0.000000] Base memory trampoline at [ffff880000095000] 95000 size 24576

not working, line 32-39:

Code:

[    0.000000] Using GB pages for direct mapping
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000] init_memory_mapping: [mem 0x101ee00000-0x101effffff]
[    0.000000] init_memory_mapping: [mem 0x101c000000-0x101edfffff]
[    0.000000] init_memory_mapping: [mem 0x1000000000-0x101bffffff]
[    0.000000] init_memory_mapping: [mem 0x00100000-0xdfe8ffff]
[    0.000000] init_memory_mapping: [mem 0x100000000-0xfffffffff]
[    0.000000] RAMDISK: [mem 0x37062000-0x37feffff]

working:

Code:

[    0.000000] Using GB pages for direct mapping
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000]  [mem 0x00000000-0x000fffff] page 4k
[    0.000000] BRK [0x01af6000, 0x01af6fff] PGTABLE
[    0.000000] BRK [0x01af7000, 0x01af7fff] PGTABLE
[    0.000000] BRK [0x01af8000, 0x01af8fff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x101ee00000-0x101effffff]
[    0.000000]  [mem 0x101ee00000-0x101effffff] page 2M
[    0.000000] BRK [0x01af9000, 0x01af9fff] PGTABLE
[    0.000000] init_memory_mapping: [mem 0x101c000000-0x101edfffff]
[    0.000000]  [mem 0x101c000000-0x101edfffff] page 2M
[    0.000000] init_memory_mapping: [mem 0x1000000000-0x101bffffff]
[    0.000000]  [mem 0x1000000000-0x101bffffff] page 2M
[    0.000000] init_memory_mapping: [mem 0x00100000-0xdfe8ffff]
[    0.000000]  [mem 0x00100000-0x001fffff] page 4k
[    0.000000]  [mem 0x00200000-0x3fffffff] page 2M
[    0.000000]  [mem 0x40000000-0xbfffffff] page 1G
[    0.000000]  [mem 0xc0000000-0xdfdfffff] page 2M
[    0.000000]  [mem 0xdfe00000-0xdfe8ffff] page 4k
[    0.000000] init_memory_mapping: [mem 0x100000000-0xfffffffff]
[    0.000000]  [mem 0x100000000-0xfffffffff] page 1G
[    0.000000] RAMDISK: [mem 0x37065000-0x37feffff]

not working, line 81-82

Code:

[    0.000000] [ffffea001c700000-ffffea001c7fffff] potential offnode page_structs
[    0.000000] Zone ranges:

working:

Code:

[    0.000000] [ffffea001c700000-ffffea001c7fffff] potential offnode page_structs
[    0.000000]  [ffffea0000000000-ffffea001c7fffff] PMD -> [ffff8807ffe00000-ffff88081bffffff] on node 0
[    0.000000]  [ffffea001c800000-ffffea00387fffff] PMD -> [ffff880ffe600000-ffff88101a5fffff] on node 1
[    0.000000] Zone ranges:

not working, line 133-134

Code:

[    0.000000] PERCPU: Embedded 27 pages/cpu @ffff88081fc00000 s81664 r8192 d20736 u262144
[    0.000000] Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 16543314

working:

Code:

[    0.000000] PERCPU: Embedded 27 pages/cpu @ffff88081fc00000 s80960 r8192 d21440 u262144
[    0.000000] pcpu-alloc: s80960 r8192 d21440 u262144 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 00 01 02 03 04 05 12 14 [1] 06 07 08 09 10 11 13 15 
[    0.000000] Built 2 zonelists in Zone order, mobility grouping on.  Total pages: 16543314

not working, line 158:

Code:

[    0.000000] tsc: Detected 2299.832 MHz processor

working:

Code:

[    0.000000] tsc: Detected 2300.189 MHz processor

not working, line 202

Code:

[    0.910133] smpboot: Total of 12 processors activated (55195.96 BogoMIPS)
[    0.946236] ------------[ cut here ]------------
[    0.955552] WARNING: CPU: 0 PID: 1 at /build/linux-i4LtiY/linux-3.16.51/kernel/sched/core.c:5811 build_sched_domains+0xc6d/0xcf0()
[    0.979123] Modules linked in:
[...]
and here comes the kernel panic ...

working:

Code:

[    0.910223] smpboot: Total of 12 processors activated (55204.53 BogoMIPS)
[    0.946657] devtmpfs: initialized
[    0.953515] memory block size : 128MB
[    0.969996] PM: Registering ACPI NVS region [mem 0xdfeb2000-0xdfedffff] (188416 bytes)
[    0.986116] futex hash table entries: 4096 (order: 6, 262144 bytes)
[    0.999986] pinctrl core: initialized pinctrl subsystem
[...]
normal boot process

miquels · 12-11-2017, 09:34 AM

Debian bug report:
https://bugs.debian.org/cgi-bin/bugr...cgi?bug=883938

getlin · 12-11-2017, 09:44 AM

Quote:

Originally Posted by miquels

Debian bug report:
https://bugs.debian.org/cgi-bin/bugr...cgi?bug=883938

Thank you very much!
I updated many servers and only one had this kernel panic.
So I thought I made something wrong ... or the server gone wild ;-)