libvirt: libvirtd often takes ~45 seconds to start
 

Hi,

libvirt version: 3.0.0.

libvirtd often takes ~45 seconds to start, where normally it should be around 2 seconds (on my machine). It happens very often, but sometimes it starts quickly as it should.

I actually debugged /etc/rc.d/rc.libvirt and know that it hangs here:
I tried disabling network auto-start with no change.
Also, the log does not say much:
Running /etc/rc.d/rc.libvirt restart once the system is booted, is fast.
Also, /usr/sbin/libvirtd when run by regular user (i.e. in qemu:///session) starts quickly.

Do you have any ideas?
Thanks in advance!

--
Best regards,
Andrzej Telszewski

Hi,

I enabled logging with debug level set and something can be observed, but I'm far from knowing what it means ;-)
The logs are ~20M in size, so I post only the part where the problem occurs. Please note the bold font.

When libvirtd hangs:
You can see there is a gap of ~50 seconds between the logged messages.
And that is more less what I measured with stopwatch.

Now for quickly starting libvirtd:
Here, the gap is much smaller, ~3 seconds only.

--
Best regards,
Andrzej Telszewski

Configuration
Automatic startup

If you want to have the libvirt daemon started automatically, add the following section to /etc/rc.d/rc.local:

# start libvirt
if [ -x /etc/rc.d/rc.libvirt ]; then
/etc/rc.d/rc.libvirt start
fi

Make sure /etc/rc.d/rc.libvirt is executable.

Ref: http://docs.slackware.com/howtos:gen...in:kvm_libvirt

Hi,

I'd say you misunderstood my question (alternatively my wording wasn't precise enough ;-)).
It's about different matter.

--
Best regards,
Andrzej Telszewski

I had the same, and put this in my rc.local:
Although now I realize it's perhaps possible to use the hardware random generator instead...

Hi,

Would than mean there is not enough entropy?
If so, then I will first try to solve that, maybe I have RNG that I don't know of.

I would prefer not to go with the solution you did.
Although, If my problem is the same as yours, this is really important piece of information.

I'll try it, thanks!

--
Best regards,
Andrzej Telszewski

Yes, it was a lack of entropy problem. Using strace I found it was waiting on reading /dev/random.
It's worth a try to test if you also have the lack of entropy problem.

I would also prefer not to use this solution, but I haven't found a better one.

Some cpu's have a hardware random generator, however there are some issue with it

If entropy is a problem go to kernel 4.8.

It may be compiled with GCC plagins ported from Grsecurity.

https://lwn.net/Articles/691102/

In

For my desktop was about 900 bits now with kernel 4.8 is about 4000 bits of entropy.

Look into 'haveged' if you need additional entropy.

Tutorial:
https://www.digitalocean.com/communi...-using-haveged

Homepage:
http://www.issihosts.com/haveged/

Hi,

@PhilipH Funny, because when I display the available entropy just before libvirtd, it's around ~40 for both situations, i.e. when libvirtd starts quickly or slowly. Still, I think that entropy is the problem and increasing that is the first thing I want to do, not just because of libvirtd's slow startup.

Are GCC plugins and Grsecurity of any relevance here or just 4.8+ kernel?
I'm not entirely sure, but GCC plugins require greater GCC version than I have (gcc-5.3.0) and I don't intend to upgrade GCC now.
I'm now building kernel 4.9, since 4.8 is EOL.

That's an option too.

Anyways, I had an initial look at TPM and rng-tools, it looks like it might work on my system.

--
Best regards,
Andrzej Telszewski

Well, it's easy enough to test if entropy is the problem.
Put the lines I posted earlier in rc.local and reboot.
If it's fast, it's an lack of entropy problem, if still slow it's something else.
After testing just remove the lines from rc.local and reboot, /dev is a sort of tmpfs and will be rebuild at boot so /dev/urandom will be restored.

If it's not an entropy problem, perhaps it will help to change this line in /etc/rc.d/rc.libvirt:
/usr/sbin/libvirtd -d -l $LIBVIRTD_OPTS
into
strace -f -o /tmp/libvirttrace.log /usr/sbin/libvirtd -d -l $LIBVIRTD_OPTS

CONFIG_GCC_PLUGINS option in kernel config. Gcc-5.3.0 is enough.
4.8 has better entropy seen in

but in 4.9 is better during boot - Latent entropy patch.

I think all is done by ported gcc plugins from Grsecurity team.

Hi,

In the end, I managed to increase the entropy availability with 4.4.x kernel, rng-tools and TPM.
I had to disable IOMMU, otherwise TPM is not accessible due to some errors.
That's not a big deal, since I don't have any device to pass to virtual machines.
In this setup my entropy is at around 3k bits.

I decided not to go with haveged and RDRAND/RDSEED due to some concerns mentioned elsewhere on the Internet.
I haven't dug deep enough, so I'm not really sure.

One thing I can say about RDRAND/RDSEED is that it looks to be really fast.
It is surely faster than TPM (but TPM seems to be fast enough).

--
Best regards,
Andrzej Telszewski

Hi,

It looks like low entropy availability was the problem of libvirtd slow startup.
Now it starts faster than it ever used to ;-)

@PhilipH Thank you for pointing out this problem.
And, I encourage you to find better entropy source too ;-)

Thanks.

--
Best regards,
Andrzej Telszewski

Good to hear you have it working !
My work computer seems to have a tpm module also, now I just have to persuade the people who know the bios password to enable it ;-)
Maybe I will just start libvirt in the background, by the time I have logged in and started the virtual machine manager it probably has enough entropy.

btw, today some more fun with libvirt after upgrading it to 3.0.0.
It won't start the vm if the configured disk is a link, for example /dev/vg01/lvname when using lvm or /dev/disk/by-uuid/<uuid>
It just says: error: An error occurred, but the cause is unknown
More info is in the Arch Linux bugreport in case somebody runs into this...