Slackware-specific guide to KVM-Qemu VGA passthrough
 

I had a stroke of luck in January and as a result I have finally been able to build a powerful new desktop.

:twocents:

I bought two fairly good video cards, and would like to assign one of them exclusively to a Windows guest under KVM-Qemu.

Thought I'd try here first for Slackware-specific instructions, before heading over to the virtualization forum. There are Debian and Arch guides floating around on the internet, but, as usual, there's always something in these to trip you up. So slacker suggestions, instructions, tips and whatnot welcome!

Hi gezley,

I'm very interested in this as well!

With regards to the 2 video cards, are they identical or is one just for gaming and the other one just for the display/multimedia?

Just some general hints:

1) Your mainboard's BIOS needs to support this, otherwise PCI-passthrough doesn't work.

2) After booting you have to unbind (from what?) the video card and assign it to the pci-stub driver. I did this with a simple script:

Obviously this only works in a very specific setup, but it should help to explain the concept.

3) Start qemu as root and pass it the PCI- and USB-ids wou want your virtual machine to handle.

It took me a while and many reboots to figure out the correct commands. After that everything was running fine as long as I didn't touch anything, i.e. restarting the software within the VM would result in a complete host lock up. Some filesystems don't like this, so better have a backup ready. The situation might be improved with more modern hardware and newer QEMUs.

I have just been through this process. This is what you have to do on Slackware.

Be sure that your motherboard and the graphics card that you want to pass through both support UEFI, and that your CPU supports IOMMU. Most modern ones do. The disk image of your guest OS must also support UEFI. This rules out anything before, and some versions of, Windows 7. Windows 8+ should be OK.

Be sure that VT-d (for Intel CPUs) or AMD-Vi (for AMD CPUs) is enabled in BIOS. Also make sure that the graphics card that you want to pass through is not set as the primary one.

You will need to recompile the kernel, because the default "huge" one, at least in current, doesn't include CONFIG_VFIO_PCI_VGA, so enable it. You can do the few steps below before rebooting into the new kernel, although it's good to reboot first to make debugging easier in case you fucked something up. Remember to keep the old kernel!

In lilo.conf, for an Intel CPU, add "intel_iommu=on" to the kernel parameters (that's the "append" line). I'm not sure what it should be for AMD CPUs. It might be "amd_iommu=on", but the kernel documentation isn't clear on how to enable it for AMD. You might also add "iommu=pt" (see here for why you might not want to).

The idea is that you want the vfio-pci driver to take control of any PCI device that you want to pass through. Some guides will mention pci-stub. That's older tech. and you don't need it for Slackware. Sure, you can use it, but it adds an extra unnecessary step. Or at least, it did for me. You can see what driver is assigned to a device with "lspci -nnk". Look for "Kernel driver in use". The next steps are necessary to assign vfio-pci to the device(s).

If you intend to pass through an Nvidia card, blacklist the nouveau driver in /etc/modprobe.d/nouveau.conf or some such. You don't want it claiming the card before you want to use it. Of course, also blacklist the official Nvidia binary driver, if you have it installed.

With an Nvidia card, you might notice that lspci shows it has an audio device as well as a VGA controller. I wanted this audio device to also pass through (in fact, I might have read that it makes the whole exercise easier, if not possible), but I noticed later that the snd_hda_intel driver was always claiming it instead of vfio-pci. To prevent that, I blacklisted snd_hda_intel. However, that meant that even my Intel motherboard's audio didn't work, so I simply loaded the driver at a stage after the vfio-pci is loaded - in /etc/rc.d/rc.local ("modprobe snd_hda_intel").

You'll need to tell the system to load the vfio-pci driver and assign it to the PCI devices (Nvidia VGA and, for me, it's audio counterpart). To do that, you'll need the addresses of the PCI devices. You can either use the addresses of the devices themselves, or, as I did, use a string which accounts for all Nvidia devices. You can see the addresses with the lspci command above. They look like "10de:13c2". Create an entry in /etc/modprobe.d/vfio.conf, separating each address with a comma, thusly:

Of course, those are the addresses of my own device. Yours may differ. Alternatively, use the Nvidia catch all string:

You can reboot now. Check that the vfio-pci driver has been assiged to the devices. Check also that the new kernel is using the correct setting ("zgrep CONFIG_VFIO_PCI_VGA /proc/config.gz"), and that the IOMMU kernel parameters worked ("find /sys/kernel/iommu_groups/ -type l" - if you see lots of entries, it worked; if you see none, it didn't).

You'll need to install QEMU and its dependencies. I used the SBo 14.1 repository for this. Here're the contents of a queue file that you can load into sbopkg to make life easier:

Maybe some of them could be found at a repository in slackpkg...

I had to change libvirt and virt-manager (which I didn't end up using, but it still might be necessary) to the latest versions. Here are the diffs for libvirt:

The diffs for virt-manager:

You'll need a copy of the latest OVMF UEFI BIOS. Get edk2.git-ovmf-x64 and extract OVMF-pure-efi.fd somewhere. I followed the steps here, but I spent too long trying to figure out how to get "UEFI" in the BIOS setting. I guess the RPM spec file would have set it up all nicely on an RPM based platform, but all I managed was seeing the full path to the OVMF file in virt-manager, which didn't work. I did that by changing the nvram variable in /etc/libvirt/qemu.conf. Instead, I gave up on virt-manager and used qemu-system-x86_64 directly, as below.

I found I had to change stdio_handler to "file", in qemu.conf, and uncomment cgroup_device_acl and add /dev/vfio/1:

Restart libvirt: When you install a guest, it'll most likely need an external driver in order to recognise the VirtIO drive that you'll use. I found that the stable virtio-win ISO worked in Microsoft's Windows 7, and the latest virtio-win ISO worked in Microsoft's Windows 10.

Create this file as kvm-install.sh:

INSTALLFILE is the name of the image of the guest that will be created. Substitute DEVICE1 and DEVICE2 for your device addresses ("lspci"). Notice that I commented out the QEMU_* audio driver line. The QEMU that I used didn't support Pulseaudio, but I still heard sound from line out, if somewhat crackly. Edit the smp argument according to your system. Notice also that -bios points to the .fd file that was downloaded earlier. Just go through the file and check it looks good for you.

Run kvm-install.sh as root, and you should see a QEMU guest window appear and Windows start to install. When it asks you to set up the disk, click "Load Driver" and point it to the drive where virtio-win is. You want to select the viostor driver. After Windows has installed, shut it down, don't reboot.

Create another file, kvm-start.sh:

I changed some arguments here because I was tweaking it for my system. YMMV. Again, check it over for yourself, before running it as root. This time, the guest window should display a message saying that the display has not yet been initialised. That's good. Check the output of the video card that you're passing through. It should show Windows!

I had a go at passing through a USB host, so my peripherals would work directly in the guest as well, but this was hit and miss. Sometimes Windows would show, for example, my mouse as a generic device, and sometimes it would show as the Logitech mouse that it is. Anyway, that should be enough to get you started with PCI VGA passthrough on Slackware. I have to give credit to other guys who already did most of the actual work, I just put things together for Slackware. Their links below.

As far as performance, with the Unigine Valley benchmark, I saw 61.7 average FPS with the basic settings on an Nvidia GTX 970 in a Windows 10 guest, whereas in Windows 7 running natively, I had 89.1 average FPS. Bear in mind that I didn't do much tweaking either with QEMU or in the guest. I note also that I couldn't get the guest to show all four of my CPU cores, they would only ever show as one.

• VFIO tips and tricks seems to be the first place to go for all this
• Video tutorial is how I finally got mine working. Check the links in the description
• PCI passthrough via OVMF on the Arch wiki
• The old Arch forum thread that now refers to the first link above

Don't use libvirt. It's bloated FOSS crap that serves no purpose other than to waste your precious disk space. The syntax is plain awful imho.


As previously mentioned you need to recompile your kernel with:
- CONFIG_VFIO_PCI_VGA=y

I also suggest
- CONFIG_JUMP_LABEL=y (optimizes likely / unlikely branches, maybe gives a small perf. boost)
- CONFIG_HUGETLBFS=y
- CONFIG_HUGETLB_PAGE=y
- CONFIG_KSM=y
- CONFIG_MCORE2=y (depends on your arch, this is for Intel C2D and higher)
- CONFIG_HZ_1000=y
- CONFIG_HZ=1000
- CONFIG_PREEMPT_VOLUNTARY=y
- CONFIG_CC_STACKPROTECTOR_STRONG=y (this should be a default setting.)

Here are my scripts:

/etc/rc.d/rc.vfio

/etc/vfio_devices.conf
-> lspci -nk gets you a nice tree with device IDs.

/usr/local/sbin/winnet
/usr/local/sbin/winvm
If you intend to use QMP you need to get these files from the QEMU source tarball. They're not included by default for some odd reason.

Note: I have installed Windows directly on my secondary SSD. This gives me option to run Windows natively just in case something doesn't work as expected with QEMU.

Additionally you need to get edk2 from here: kraxel.org/repos/jenkins/edk2/

Simply download edk2.git-ovmf-x64-(...).rpm from there and run rpm2tgz to create a Slackware package.

/etc/modprobe.d/vfio.conf
After compiling make sure to get your initramfs right.
/etc/mkinitrd.conf
Then use mkinitrd -F to create a new initramfs.

We also need to change the /etc/fstab file. Add this:
Last but not least I have changed the /etc/rc.d/rc.udev script in order to automatically mount hugetlbfs on boot and initialize the rc.vfio script.

And in case you're using grub:
GRUB_CMDLINE_LINUX_DEFAULT="hugepagesz=1GB default_hugepagesz=1GB hugepages=8 intel_iommu=on iommu=pt"

Other useful kernel options are:
- pcie_acs_override=downstream (requires the acs kernel patch. Only add this if passthrough doesn't work.)
- hugepages=8 -> 1x8 GB hugepages. Make sure you have enough ram.

Opinions differ on libvirtd.

Indeed. :)

I did some benchmarks recently and here are some results. Might be of interest for some.

Benchmark on native Windows 10 x64: http://www.3dmark.com/fs/7942691
The same benchmark on QEMU: http://www.3dmark.com/fs/7931626

There wasn't really much difference in terms of performance between QEMU and the native run.

That's some useful information. Thanks for posting it.

Just one more quick tip. In case you're using an integrated Intel iGPU as primary GPU and plan to use a secondary card for VT-d passthrough you might encounter a strange error where grub is unable to boot the system. Some error about "file '/grub2/locale/en.mo.gz' not found" or something along the lines.

Just uncomment "GRUB_TERMINAL=console" in /etc/default/grub and create a new config with grub-mkconfig.

In extreme cases you might have to fix your ACPI tables and recompile the dsdt.hex into your kernel in order to be able to boot with the onboard GPU as primary device. If someone encounters this problem just PM me or ask here. I might know how to fix it but it requires further testing.

Cheers

Sorry for the late reply. One of the video cards I bought is a GeForce GTX 750 Ti; the other a Radeon R7 200.

The GeForce does not work properly under Linux. The fans power up like jet turbines every 20 seconds or so, making the computer unusable. I also had trouble installing Linux on the motherboard, an Asrock Extreme9 990FX. Admittedly this was probably down to my complete ignorance regarding UEFI.

At the end of the day, I have neither time nor inclination to fight these battles over and over again. In 2016 I expect not to have to fight the battles I was fighting with Linux in 2001. Gnome, Red Hat, the systemd cabal, KDE, Debian, Google, Ubuntu - all of them creating a never-ending stream of new bugs for future generations to tackle, but none of them remotely interested in solving today's bugs.

But that's what you get when Wall Street venture capital dictates the terms on which Linux should proceed. It's sad: what was an international project that had so much promise 15 years ago has now been hijacked and monopolised by the big bullies on the block for their own ends. Truth to tell, I am increasingly sick of Linux, sick of the immaturity that drives so-called progress in Linux, sick of the constant breakage in Linux and last but not least sick and tired of the fanboys making false claims about Linux.

At the moment I am back with Windows 8.1, though we all know where Microsoft are going, and undoubtedly they've had a hand in nudging Linux down the cul-de-sac it's in anyway, so Microsoft is not a long-term option either. NetBSD is a beautifully engineered project - sane, conservative, and predictable. The trouble is, you still need to slap a desktop on it for day-to-day use, so what do you go with: Xfce, which is struggling with Gnome's constant breakage and its bully-boy indifference to other projects? Gnome, which has been an insulting Fisher-Price PoS since version 3 was imposed by the powers-that-be in corporate America? or KDE, which will eventually culminate in a stable version of 5 only to decide they want to abandon it and devote themselves exclusively to 6 instead? Great choice there. Of course we all know they're pushing us to use the oh-so-great cloud anyway and the desktop is oh-so-nineties (when We were still in nappies), why would you even care!

Well anyway, they're my thoughts. Slackware and Crux are great. Hard to see them holding out against the tide for the next decade though, and who can blame them if they eventually do succumb?

Sorry. Probably not the place to take out my anger on Linux, but I see the options narrowing, and that is not supposed to be what Linux was about. I am so, so angry with those responsible, and their stupid, brain-dead, immature pet projects designed to keep breakage to the fore in Linux.

Thank you for such a detailed write-up! I've had problems with the Nvidia card I bought so I've postponed this for the time being. When I get a replacement video card I will use these instructions.

Cripes it's a lot more complex than I was expecting! Video card pass-through with NetBSD Xen is far less complex but unfortunately the NetBSD team haven't yet updated Xen 4 to support it, and Xen 3, which does support it, is quite old now.

I will probably go picking a few brains here when I get the second video card replaced.

Thank you for your detailed reply.

I'm happy to help. Don't worry it really looks harder than it is.

Though I don't necessarily disagree with your statement. It's a complex system but once it works it works. I haven't encountered any critical bugs nor crashes during the two years I have used KVM-pt on my system. The developers have really put a lot of effort into this technology and it shows. I'm still impressed how smoothly it performs on my system.

When it comes to passthrough Radeon cards are probably the better choice. Nvidia has some nasty anti-features to prevent hyper-v enlightenments in their drivers whereas AMD has no such mechanisms. These extensions are IMHO necessary for flawless performance. Without them my system felt too unresponsive and sluggish under load.

While Radeon is preferred, is there anything major the prevent Nvidia cards from working?

Been on the fence about giving this a try for quite some time, and this thread (with Slack-specific instructions) may be what pushes me over the edge.

Well apparently the problem with Nvidia GPUs has been solved in Qemu.

Just quoting from the Arch wiki here:

So who of you guys is going to add this crucial information as a new article on the Slackware Documentation Project Wiki?

Do I need to do anything fancy re:mouse and keyboard? FWIW, I have two monitors, and my plan was to either split them, or maybe plug one into both cards, and just switch it's input when gaming under Windows. Can I move the mouse between the two, or do I need to set something up for that?

Not necessarily. A USB switch for keyboard and mouse and an additional USB card for passthrough is highly recommended though. I also use an external USB sound card for my VM as sound emulation is sort of terrible.

If you don't want to spend money you can always use Synergy or QEMU USB passthrough for your USB peripheral devices.

Synergy works well. Although I am currently using Arch, I had VGA passthrough working on Slackware 14.1. Agree with the notes previously given. Qemu script and libvirtd each have advantages other than the GUI, but to each his/her own. My notes on setup are on my LQ blog pages, though they're a bit dated now. I'd recommend the vfio-kvm mailing list for help, and make sure you check out the guide on vfio.blogspot.com.

Greetings!
I've recently dusted my old GTX550Ti in order to try and prepare a virtual machine (x64 Windows 10) with GPU pass-through but I've hit a few issues on the way.
I've followed the guides in this topic (as well as the linked ones) and I managed to go as far as binding the GTX550Ti to vfio-pci but I cannot manage to have a working VM with qemu.

I'm using an up-to-date Slackware64-current installation: I had to recompile the kernel (I used the default kernel's config file and enabled vfio), blacklisted both snd_hda_intel and the nvidia drivers (later to be loaded with modprobe using /etc/rc.d/rc.local) and bound the graphic card to vfio, both the GPU and the audio portion.
lspci -nnk outputs the following (relevant entries only):
iommu groups are ok, as seen here:
The other graphic card (a GTX 950) works as usual with the nVidia proprietary drivers.
I'm using alien's qemu and vde packages if this can be relevant. I also checked if the GTX550Ti works properly and I verified both a Windows installation (dual boot) and my Slackware system (prior to binding the GTX550Ti to vfio) could use the card without issues and output to the monitor I connected.

Once I reached this point, I started to run into several issues.

- Problem #1: no output on the GTX550Ti
I cannot output anything on the GTX550Ti using qemu (the monitor I hooked up keeps giving the NO SIGNAL warning).
I'm using this script to launch qemu:
- Problem #2: BSOD on the VM if using -vga std
If I start the vm with the -vga std option I can install Windows 10; doing so allowed me to verify that the VM does actually see the GTX550Ti with no issues. Windows proceeded to download the latest updates and the drivers for the nVidia card but at the next reboot it started going into a BSOD loop giving one of the two following errors:
or This happens both if I let Windows download the updates and the drivers by itself or if I install the latest drivers myself.

- Problem #3: freezes and garbled audio.
Prior to installing the drivers, the VM works reasonably well, although it's a bit slow at times and I had better success by using
instead of
in the qemu script I use.
It does, however, freeze every time it needs to output sound: this happens both with alsa and pulseaudio. The terminal shows this warning:
On top of that, the sound is also heavily garbled. The sound coming from the host is not affected.

Does anyone have any idea about how to solve any of those issues, or where to look for a possible solution?

If I'll manage to have a working VM with proper pass-through and virtualisation I'll also write a step by step guide taken to solve the aforementioned issues.

Thanks in advance!
Cheers,
~kthxbye

A quick update: I managed to get output from the GTX550Ti by NOT using the OVMF BIOS, which solves issue #1.
Issue #2 changed as now I don't have to use -vga std to work on the VM; however, after installing the nVidia drivers the GTX550Ti does not output anything (monitor remains blank for a while before giving the no signal warning).
Issue #3 is unchanged.
I have a new issue as well: if I reboot the guest OS and/or restart the VM, I have no output on the GTX550Ti and I have the following warning on the terminal:
It only works the first time the VM is started after a reboot of the host OS. I tried to extract the rom to use with the romfile= option but I've been unsuccessful as of yet, as even on a fresh reboot of the host OS it says the rom is invalid when I try to cat it to a new file.
For reference, I've tried this method:

Hi - I'm the OP, and I'm still interested in getting this done, but I still haven't replaced the nvidia card that was giving me trouble so I'm not much use to you I'm afraid.

Some question about vga passtrough
I have two video cards
one is nvidia,on host
another is ati,i want to pass it to qemu/libvirt.
I have one tv/monitor with multiple inputs
my card is connected to vga(nvidia) the ati
is on dv.
I have installed with libvirt(i know is foss,
but has a nice virt-manager to add and remove
stuff..) windows 8.1
And on display i can see the windows
But my dream is to see something like this(pic is fake!
i hope it get reality someday..)

https://s12.postimg.org/t4kgq4yzh/ricevuta_ram5.jpg

the gpu card(real) on a screen emulated(vnc,spice...)
Is possible to do something like this with qemu/vga passtrough?

On some youtube video seems possible,but they don't put a simply howto
only the video..:(

https://www.youtube.com/watch?v=Qi1LdFkRzIs

I see the qemu config on this video of the same user,
he get a very nice result,can play 3d game in little window
using spice client!

https://www.youtube.com/watch?v=_6K9Sxbb_lU

Lucky user,i don't understand how he can get the 3d of real card
on spicy client.
As i understand this configuration works only with intel host+amd guest gpu.
I have tried a similar configuration running this script
but spicy client connect only to qxl card...whitout 3d

I use vfio instead of stub because i cannot find the "unbind" voice

Return nothing

I'd be happy to, but since I haven't done passthrough since I wrote that post, I'm afraid it would only be a copy-paste job.