KVM Guest hangs and require restart of host
 

Following is environment on KVM host server

Server Model : ProLiant DL380 Gen9
OS : Cent 6.5 ( for host and all guest VM)
Guest VM : Cent OS 6.5 using raw disk through virtio
qemu-kvm-tools-0.12.1.2-2.415.el6.x86_64
qemu-kvm-0.12.1.2-2.415.el6.x86_64
---------------------------------
libvirt-0.10.2-29.el6.x86_64
libvirt-java-devel-0.4.9-1.el6.noarch
libvirt-cim-0.6.1-9.el6.x86_64
libvirt-python-0.10.2-29.el6.x86_64
libvirt-java-0.4.9-1.el6.noarch
libvirt-snmp-0.0.2-4.el6.x86_64
libvirt-devel-0.10.2-29.el6.x86_64
libvirt-client-0.10.2-29.el6.x86_64
fence-virtd-libvirt-0.2.3-15.el6.x86_64
---------------------------------

Problem Details :

All guest VM are in state of running but not accessible through ssh & even not pingable from mother host
There is no significant increase in load/memory or any other system resources
I am not able to destroy them using virsh command and following in logs

2018-02-16 04:05:05.760+0000: 3776: warning : qemuProcessKill:4174 : Timed out waiting after SIGTERM to process 4047, sending SIGKILL
2018-02-16 04:05:10.762+0000: 3776: warning : qemuProcessKill:4206 : Timed out waiting after SIGKILL to process 4047
2018-02-16 04:05:10.762+0000: 3776: error : qemuDomainDestroyFlags:2098 : operation failed: failed to kill qemu process with SIGTERM
2018-02-16 04:05:44.001+0000: 3780: warning : qemuProcessKill:4174 : Timed out waiting after SIGTERM to process 4047, sending SIGKILL
2018-02-16 04:05:49.003+0000: 3780: warning : qemuProcessKill:4206 : Timed out waiting after SIGKILL to process 4047
2018-02-16 04:05:49.003+0000: 3780: error : qemuDomainDestroyFlags:2098 : operation failed: failed to kill qemu process with SIGTERM

I have to performed physical reboot this mother host and restore KVM guest OS.

What could be possible reason...
Any logs file that can assist to trace this issue

I'm sorry for your virtualization troubles. I don't have solid solutions for you, but some questions that might help us get your VMs working correctly.

What kind of networking are the VMs using? That is, are they bridged, or NATted?

Have you thought about modifying the VM definitions to include a console on a virtual serial port you can connect to? https://rwmj.wordpress.com/2011/07/0...u-and-libvirt/ shows something similar to what I have done in the past. It doesn't look identical to how I did it, but I'm away from my virtual environment right now.

Perhaps you could turn qemu logging on: https://unix.stackexchange.com/quest...rtual-machines. I'm assuming you would have to modify the init script for the qemu service or maybe a /etc/sysconfig/qemu file. I've never tried changing an option like this.

Might also look in any /var logs in host.

Seems that some part of qemu is failing would be my first assumption. It could be some issue with how it interacts with VM technologies on host I guess.