[SOLVED] Problems getting crash dumps created on VirtualBox guests

pacolino · 04-23-2021, 02:15 PM

Hello, I am trying to get some kernel crashes recorded by using the kexec utility, but my kernels are crashing and freezing, rather than booting into the crash kernel and thus getting to the stage where the vmcore can be copied to /var/crash or similar.

I am using various kernels 5.10 / 5.11 on the guests and VirtualBox 6.1.18 on Windows or Linux as hosts. Problem repros in both hosts, but does not happen if I use a real "physical" Linux host.

I prepare the crash kernel like this:

Code:

# kexec -p /boot/vmlinuz-$(uname -r) --reuse-cmdline

I then trigger the crash with

Code:

 
echo c > /proc/sysrq-trigger

and what I get in my serial console is a "not syncing" kernel panic and the machine DOES NOT boot into the crash kernel, but rather freezes forever.

Code:

[    0.462686] Code: 48 c7 c7 60 3e 92 bd 41 5c 41 5d 41 5e 41 5f e9 34 05 b9 00 0f 0b cc cc 39 3d 6a f9 7a 01 74 24 31 ff 65 48 8b 05 1e 15 54 44 <f6> 80 88 00 00 00 03 40 0f 95 c7 e8 1e 15 ff ff bf 01 00 00 00 e9
[    0.462687] RSP: 0000:ffffb12fc0003ea0 EFLAGS: 00010046
[    0.462689] RAX: 0000000000000000 RBX: ffffffffbd1ff300 RCX: 0000000000000000
[    0.462691] RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000000
[    0.462693] RBP: 7fffffffffffffff R08: 0000000000000000 R09: 0000000000000001
[    0.462695] R10: ffff9b87db800270 R11: 0000000000000000 R12: 0000000000000000
[    0.462696] R13: ffffb12fc0003f0c R14: 0000000000000000 R15: ffff9b87db451800
[    0.462698] FS:  0000000000000000(0000) GS:ffff9b87fac00000(0000) knlGS:0000000000000000
[    0.462700] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.462702] CR2: 0000000000000088 CR3: 000000007a00c001 CR4: 00000000000606b0
[    0.462704] Kernel panic - not syncing: Fatal exception in interrupt
[    0.692058] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

Note: Sometimes it's something other than Fatal exception in interrupt. The constant is the "not syncing".

Your input is highly appreciated.

pacolino · 04-23-2021, 10:57 PM

Ok, I found the solution for now. It seems that the normal kexec -p command line does not work in new kernels. It's more than just reserving crashkernel memory. I had one systemd based OpenSuse tumbleweed and I installed in it the package kdump, which brought in several things, but the most important that upon starting kdump systemd unit, then the correct arguments are fed into kexec -p which then allows the kernel to both boot into the crash mode kernel as well as record the core dump under /var/crash.

So, to recap, once the kdump package was installed I had to do:

Code:

systemctl start kdump

and then to peek into the kexec command line it created, I did journalctl and looked for the entries created.

Now, I will copy those parameters such that I can use them in my non-systemd based installs.