We are testing a 2-node cluster, with both nodes running Debian Squeeze (6.0.2.1) + Xen 4.0.2. The vms are stored on SAN using an ocfs2 filesystem.

We are stress-testing the system by creating 2 vms, and migrating these vms back and forth from one node to the other. Meanwhile, the 2 vms are passing back and forth between them a large file (4.6 Gb) via nfs filesharing.

Under this network load, the vms run for several hours, then crash. Sometimes one or both nodes also crash and then reboot.

We want to capture the logs recording the failure.

But all relevant logs (messages, kern.log, boot, etc.) seem to be erased on reboot. All we have are messages beginning thus:

imklog 4.2.0, log source = /proc/kmsg started.


What is the best way to capture a log of the last few seconds of system operation without the subsequent crash or reboot erasing or overwriting these logs?

if you're sure (or not...) the logs are erased on reboot rather just not being written, perhaps immediately after crash, bootup from other partition or live, then read the logs?

Thanks!--this is a good idea.