[SOLVED] slackware64-current libvirt 2.2.0 - virsh / virt-manager hangs.

af7567 · 10-20-2016, 05:48 AM

Hi.
I have been using libvirt with virt-manager daily for a while and never had any problems until today. I started virt-manager and it just got stuck showing "connecting..." So I tried virsh and that just hangs if I do "virsh list" or "virsh connect".

I turned on debug logging in libvirtd.conf

Code:

log_level = 1
log_filters=""
log_outputs="1:file:/tmp/libvirtd.log"

but nothing is logged at all when I run virsh or virt-manager as if they aren't even connecting to libvirtd. Only the libvirtd startup messages are logged but I don't see any errors there.

I haven't installed anything new, or any updates for a few days. Haven't changed any settings either. I have checked dmesg, syslog, messages and the other log files in /var/log/libvirt but there are no errors in any of them either. Does anyone have any ideas where else I could look for clues?

af7567 · 10-20-2016, 07:53 AM

It looks like I have one of those problems that can't be possible. I have moved all my libvirt stuff out the way, removed the package, and deleted every trace of it from the hard drive. I recompiled libvirt and dependencies (yajl and urlgrabber) from the SlackBuilds. I also built and installed qemu again. So I now have a completely "empty" libvirt installation using the default config that was installed with the package - it still doesn't work. virsh and virt-admin just hang whenever trying to connect to libvirtd.

I have run fsck, memtest, prime95 without problems, and everything else seems to run OK - so I don't think it's a hardware error.
I checked that hardware virtualization didn't somehow get disabled in the BIOS.
I tested with my 4.7.4 kernel as well as the slackware 4.4.22 kernel.

I don't know how it is possible that it doesn't work even after completely reinstalling everything to do with libvirt. I am now reinstalling every slackware64 package using slackpkg to see if that makes a difference.

willysr · 10-20-2016, 08:04 AM

is this on 14.2 or in current?

af7567 · 10-20-2016, 09:23 AM

This is on slackware64 current.

I ran virsh with strace and it seems to freeze while waiting for data from the socket.

Code:

access("/etc/libvirt/libvirt.conf", F_OK) = 0
open("/etc/libvirt/libvirt.conf", O_RDONLY) = 6
read(6, "#\n# This can be used to setup UR"..., 8192) = 547
read(6, "", 7645)                       = 0
close(6)                                = 0
getuid()                                = 0
geteuid()                               = 0
access("/proc/vz", F_OK)                = -1 ENOENT (No such file or directory)
geteuid()                               = 0
getuid()                                = 0
geteuid()                               = 0
socket(PF_LOCAL, SOCK_STREAM, 0)        = 6
connect(6, {sa_family=AF_LOCAL, sun_path="/var/run/libvirt/libvirt-sock"}, 110) = 0
getsockname(6, {sa_family=AF_LOCAL, NULL}, [2]) = 0
futex(0x7fa4cf959188, FUTEX_WAKE_PRIVATE, 2147483647) = 0
fcntl(6, F_GETFD)                       = 0
fcntl(6, F_SETFD, FD_CLOEXEC)           = 0
fcntl(6, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
futex(0x7fa4cf959088, FUTEX_WAKE_PRIVATE, 2147483647) = 0
pipe2([7, 8], O_CLOEXEC)                = 0
write(5, "\0", 1)                       = 1
futex(0x7fa4cf959150, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7fa4cf9590a0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(5, "\0", 1)                       = 1
futex(0x7fa4cf958560, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigprocmask(SIG_BLOCK, [PIPE CHLD WINCH], [], 8) = 0
poll([{fd=6, events=POLLOUT}, {fd=7, events=POLLIN}], 2, -1) = 1 ([{fd=6, revents=POLLOUT}])
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
write(6, "\0\0\0\34 \0\200\206\0\0\0\1\0\0\0B\0\0\0\0\0\0\0\0\0\0\0\0", 28) = 28
rt_sigprocmask(SIG_BLOCK, [PIPE CHLD WINCH], [], 8) = 0
poll([{fd=6, events=POLLIN}, {fd=7, events=POLLIN}], 2, -1

It will just hang at that -1, I left virsh running for about 20 mins in case something timed out but nothing happened.

Nothing has been updated or changed on the system since yesterday, just shut down at night then turned back on again in the morning.

edit:
I just decided to try running virtlockd, virtlogg and libvirtd as a normal user instead of as root and it seems to work then. I didn't try creating and running a virtual machine but virsh list works properly.

af7567 · 10-20-2016, 11:59 AM

I finally got libvirt / virt-manager working again and can use my virtual machines. I decided to move all the files from /usr/lib64/libvirt/connection-driver/ out the way and then I could see libvirtd actually responded to connections on its socket. So I restored the files back one at a time and found that libvirt_driver_nwfilter.so was causing libvirtd to not respond to data sent to its socket.

I have now restored my libvirt configuration and images from this morning and removed that libvirt_driver_nwfilter.so and it all seems to be working OK and I have internet access inside my virtual machines.

What does libvirt_driver_nwfilter do? I wonder if I even need it. I did try building libvirt with debugging symbols and attempted running it in gdb, but I have no idea how to use gdb at all so that was a waste of time

especially since it doesn't actually crash anywhere.

edit:
After a system restart virsh and virt-manager are still freezing when trying to communicate with the libvirtd socket. I had to also move libvirt_driver_network.so out of the connection-driver directory and restart libvirt for it to respond to the socket again. Then shut it down and move libvirt_driver_network.so back (since libvirt is unusable without it), then for some reason it works after restarting libvirt again.

I don't know what's going on, maybe libvirt doesn't like starting at system boot anymore.

af7567 · 10-25-2016, 04:07 PM

After 5 days I finally found the problem. When libvirtd is starting it executes ebtables as

Code:

/usr/sbin/ebtables --concurrent -L

and this creates a lock file in /var/lib/ebtables/lock

The lockfile was still there on my PC from the morning of the 20th and it wasn't removed after restarts. This means that libvirt couldn't start properly because it was always stuck waiting for the ebtables lock to go away. I only noticed tonight that libvirtd was running ebtables and when running the ebtables command manually I saw the warning message that it was waiting for the lock.

(The reason for the lockfile being left behind was because the nvidia driver crashed at system startup that morning so I had to force the power off)