I started my virtualisation experiments with VirtualBox because of the quick and easy graphical frontend, but grew tired of having to compile the associated kernel modules with every kernel upgrade.
I much prefer qemu+kvm now. Once set up, it has proved very reliable and easy to maintain. I do not have sufficient need to build virt-manager, but if I was starting from scratch it would things much easier.
Xen has a primary goal of virtualising Linux systems, which is not my focus, so I have not explored it. It has needed custom kernel compilation in the past.
VMware I have used on Windows. It is very good, the granddaddy to other virtualisation attempts, but has a commercial focus.