Slackware-specific guide to KVM-Qemu VGA passthrough
I had a stroke of luck in January and as a result I have finally been able to build a powerful new desktop.
:twocents: I bought two fairly good video cards, and would like to assign one of them exclusively to a Windows guest under KVM-Qemu. Thought I'd try here first for Slackware-specific instructions, before heading over to the virtualization forum. There are Debian and Arch guides floating around on the internet, but, as usual, there's always something in these to trip you up. So slacker suggestions, instructions, tips and whatnot welcome! |
Hi gezley,
I'm very interested in this as well! With regards to the 2 video cards, are they identical or is one just for gaming and the other one just for the display/multimedia? |
Just some general hints:
1) Your mainboard's BIOS needs to support this, otherwise PCI-passthrough doesn't work. 2) After booting you have to unbind (from what?) the video card and assign it to the pci-stub driver. I did this with a simple script: Code:
#!/bin/bash 3) Start qemu as root and pass it the PCI- and USB-ids wou want your virtual machine to handle. It took me a while and many reboots to figure out the correct commands. After that everything was running fine as long as I didn't touch anything, i.e. restarting the software within the VM would result in a complete host lock up. Some filesystems don't like this, so better have a backup ready. The situation might be improved with more modern hardware and newer QEMUs. |
I have just been through this process. This is what you have to do on Slackware.
Be sure that your motherboard and the graphics card that you want to pass through both support UEFI, and that your CPU supports IOMMU. Most modern ones do. The disk image of your guest OS must also support UEFI. This rules out anything before, and some versions of, Windows 7. Windows 8+ should be OK. Be sure that VT-d (for Intel CPUs) or AMD-Vi (for AMD CPUs) is enabled in BIOS. Also make sure that the graphics card that you want to pass through is not set as the primary one. You will need to recompile the kernel, because the default "huge" one, at least in current, doesn't include CONFIG_VFIO_PCI_VGA, so enable it. You can do the few steps below before rebooting into the new kernel, although it's good to reboot first to make debugging easier in case you fucked something up. Remember to keep the old kernel! In lilo.conf, for an Intel CPU, add "intel_iommu=on" to the kernel parameters (that's the "append" line). I'm not sure what it should be for AMD CPUs. It might be "amd_iommu=on", but the kernel documentation isn't clear on how to enable it for AMD. You might also add "iommu=pt" (see here for why you might not want to). The idea is that you want the vfio-pci driver to take control of any PCI device that you want to pass through. Some guides will mention pci-stub. That's older tech. and you don't need it for Slackware. Sure, you can use it, but it adds an extra unnecessary step. Or at least, it did for me. You can see what driver is assigned to a device with "lspci -nnk". Look for "Kernel driver in use". The next steps are necessary to assign vfio-pci to the device(s). If you intend to pass through an Nvidia card, blacklist the nouveau driver in /etc/modprobe.d/nouveau.conf or some such. You don't want it claiming the card before you want to use it. Of course, also blacklist the official Nvidia binary driver, if you have it installed. With an Nvidia card, you might notice that lspci shows it has an audio device as well as a VGA controller. I wanted this audio device to also pass through (in fact, I might have read that it makes the whole exercise easier, if not possible), but I noticed later that the snd_hda_intel driver was always claiming it instead of vfio-pci. To prevent that, I blacklisted snd_hda_intel. However, that meant that even my Intel motherboard's audio didn't work, so I simply loaded the driver at a stage after the vfio-pci is loaded - in /etc/rc.d/rc.local ("modprobe snd_hda_intel"). You'll need to tell the system to load the vfio-pci driver and assign it to the PCI devices (Nvidia VGA and, for me, it's audio counterpart). To do that, you'll need the addresses of the PCI devices. You can either use the addresses of the devices themselves, or, as I did, use a string which accounts for all Nvidia devices. You can see the addresses with the lspci command above. They look like "10de:13c2". Create an entry in /etc/modprobe.d/vfio.conf, separating each address with a comma, thusly: Quote:
Quote:
You'll need to install QEMU and its dependencies. I used the SBo 14.1 repository for this. Here're the contents of a queue file that you can load into sbopkg to make life easier: Quote:
I had to change libvirt and virt-manager (which I didn't end up using, but it still might be necessary) to the latest versions. Here are the diffs for libvirt: Quote:
Quote:
I found I had to change stdio_handler to "file", in qemu.conf, and uncomment cgroup_device_acl and add /dev/vfio/1: Code:
cgroup_device_acl = [ Quote:
Create this file as kvm-install.sh: Code:
#!/bin/sh Run kvm-install.sh as root, and you should see a QEMU guest window appear and Windows start to install. When it asks you to set up the disk, click "Load Driver" and point it to the drive where virtio-win is. You want to select the viostor driver. After Windows has installed, shut it down, don't reboot. Create another file, kvm-start.sh: Code:
#!/bin/sh I had a go at passing through a USB host, so my peripherals would work directly in the guest as well, but this was hit and miss. Sometimes Windows would show, for example, my mouse as a generic device, and sometimes it would show as the Logitech mouse that it is. Anyway, that should be enough to get you started with PCI VGA passthrough on Slackware. I have to give credit to other guys who already did most of the actual work, I just put things together for Slackware. Their links below. As far as performance, with the Unigine Valley benchmark, I saw 61.7 average FPS with the basic settings on an Nvidia GTX 970 in a Windows 10 guest, whereas in Windows 7 running natively, I had 89.1 average FPS. Bear in mind that I didn't do much tweaking either with QEMU or in the guest. I note also that I couldn't get the guest to show all four of my CPU cores, they would only ever show as one.
|
Don't use libvirt. It's bloated FOSS crap that serves no purpose other than to waste your precious disk space. The syntax is plain awful imho.
As previously mentioned you need to recompile your kernel with: - CONFIG_VFIO_PCI_VGA=y I also suggest - CONFIG_JUMP_LABEL=y (optimizes likely / unlikely branches, maybe gives a small perf. boost) - CONFIG_HUGETLBFS=y - CONFIG_HUGETLB_PAGE=y - CONFIG_KSM=y - CONFIG_MCORE2=y (depends on your arch, this is for Intel C2D and higher) - CONFIG_HZ_1000=y - CONFIG_HZ=1000 - CONFIG_PREEMPT_VOLUNTARY=y - CONFIG_CC_STACKPROTECTOR_STRONG=y (this should be a default setting.) Here are my scripts: /etc/rc.d/rc.vfio Quote:
Quote:
/usr/local/sbin/winnet Quote:
Quote:
Note: I have installed Windows directly on my secondary SSD. This gives me option to run Windows natively just in case something doesn't work as expected with QEMU. Additionally you need to get edk2 from here: kraxel.org/repos/jenkins/edk2/ Simply download edk2.git-ovmf-x64-(...).rpm from there and run rpm2tgz to create a Slackware package. /etc/modprobe.d/vfio.conf Quote:
/etc/mkinitrd.conf Quote:
We also need to change the /etc/fstab file. Add this: Quote:
Quote:
GRUB_CMDLINE_LINUX_DEFAULT="hugepagesz=1GB default_hugepagesz=1GB hugepages=8 intel_iommu=on iommu=pt" Other useful kernel options are: - pcie_acs_override=downstream (requires the acs kernel patch. Only add this if passthrough doesn't work.) - hugepages=8 -> 1x8 GB hugepages. Make sure you have enough ram. |
Opinions differ on libvirtd.
|
Indeed. :)
I did some benchmarks recently and here are some results. Might be of interest for some. Benchmark on native Windows 10 x64: http://www.3dmark.com/fs/7942691 The same benchmark on QEMU: http://www.3dmark.com/fs/7931626 There wasn't really much difference in terms of performance between QEMU and the native run. |
Quote:
|
Just one more quick tip. In case you're using an integrated Intel iGPU as primary GPU and plan to use a secondary card for VT-d passthrough you might encounter a strange error where grub is unable to boot the system. Some error about "file '/grub2/locale/en.mo.gz' not found" or something along the lines.
Just uncomment "GRUB_TERMINAL=console" in /etc/default/grub and create a new config with grub-mkconfig. In extreme cases you might have to fix your ACPI tables and recompile the dsdt.hex into your kernel in order to be able to boot with the onboard GPU as primary device. If someone encounters this problem just PM me or ask here. I might know how to fix it but it requires further testing. Cheers |
Sorry for the late reply. One of the video cards I bought is a GeForce GTX 750 Ti; the other a Radeon R7 200.
The GeForce does not work properly under Linux. The fans power up like jet turbines every 20 seconds or so, making the computer unusable. I also had trouble installing Linux on the motherboard, an Asrock Extreme9 990FX. Admittedly this was probably down to my complete ignorance regarding UEFI. At the end of the day, I have neither time nor inclination to fight these battles over and over again. In 2016 I expect not to have to fight the battles I was fighting with Linux in 2001. Gnome, Red Hat, the systemd cabal, KDE, Debian, Google, Ubuntu - all of them creating a never-ending stream of new bugs for future generations to tackle, but none of them remotely interested in solving today's bugs. But that's what you get when Wall Street venture capital dictates the terms on which Linux should proceed. It's sad: what was an international project that had so much promise 15 years ago has now been hijacked and monopolised by the big bullies on the block for their own ends. Truth to tell, I am increasingly sick of Linux, sick of the immaturity that drives so-called progress in Linux, sick of the constant breakage in Linux and last but not least sick and tired of the fanboys making false claims about Linux. At the moment I am back with Windows 8.1, though we all know where Microsoft are going, and undoubtedly they've had a hand in nudging Linux down the cul-de-sac it's in anyway, so Microsoft is not a long-term option either. NetBSD is a beautifully engineered project - sane, conservative, and predictable. The trouble is, you still need to slap a desktop on it for day-to-day use, so what do you go with: Xfce, which is struggling with Gnome's constant breakage and its bully-boy indifference to other projects? Gnome, which has been an insulting Fisher-Price PoS since version 3 was imposed by the powers-that-be in corporate America? or KDE, which will eventually culminate in a stable version of 5 only to decide they want to abandon it and devote themselves exclusively to 6 instead? Great choice there. Of course we all know they're pushing us to use the oh-so-great cloud anyway and the desktop is oh-so-nineties (when We were still in nappies), why would you even care! Well anyway, they're my thoughts. Slackware and Crux are great. Hard to see them holding out against the tide for the next decade though, and who can blame them if they eventually do succumb? Sorry. Probably not the place to take out my anger on Linux, but I see the options narrowing, and that is not supposed to be what Linux was about. I am so, so angry with those responsible, and their stupid, brain-dead, immature pet projects designed to keep breakage to the fore in Linux. |
Quote:
|
Quote:
I will probably go picking a few brains here when I get the second video card replaced. Thank you for your detailed reply. |
I'm happy to help. Don't worry it really looks harder than it is.
Though I don't necessarily disagree with your statement. It's a complex system but once it works it works. I haven't encountered any critical bugs nor crashes during the two years I have used KVM-pt on my system. The developers have really put a lot of effort into this technology and it shows. I'm still impressed how smoothly it performs on my system. When it comes to passthrough Radeon cards are probably the better choice. Nvidia has some nasty anti-features to prevent hyper-v enlightenments in their drivers whereas AMD has no such mechanisms. These extensions are IMHO necessary for flawless performance. Without them my system felt too unresponsive and sluggish under load. |
While Radeon is preferred, is there anything major the prevent Nvidia cards from working?
Been on the fence about giving this a try for quite some time, and this thread (with Slack-specific instructions) may be what pushes me over the edge. |
Well apparently the problem with Nvidia GPUs has been solved in Qemu.
Just quoting from the Arch wiki here: Quote:
|
All times are GMT -5. The time now is 07:28 AM. |