[SOLVED] Not clue how to put pci device in passthrough in current

the_zone · 01-01-2023, 01:27 PM

I have 2 the same network cards.

One of them will be the WAN side of pfsense which runs in qemu.
I'd like to isolate this card, instead of bridging it, and give it to pfsense solely.

Code:

01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
	Subsystem: Hewlett-Packard Company RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [103c:82a2]
	Kernel driver in use: r8169
	Kernel modules: r8169
02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 06)
	Subsystem: TP-LINK Technologies Co., Ltd. TG-3468 Gigabit PCI Express Network Adapter [7470:3468]
	Kernel driver in use: r8169
	Kernel modules: r8169

I have tried to add a vfio.conf in /lib/modprobe.d which contains:

Code:

options vfio-pci ids=02:00.0|10ec:8168

First of all I am not sure whether the syntax is right. Cause by adding the device "10ec:8168" I will passthrough bot NICS. So i added the bus.
It did not work.

Secondly I tried by overriding the device by cat-ting "vfio-pci" in

/sys/bus/pci/devices/0000:02:00.0/driver_override

Also this did not work because after boot it defaulted.

Third, I tried by using vfio-bind.
This command does not seems to be available on current.

Also the following does not work:

Code:

virsh nodedev-dettach pci_0000_02_00_0
error: Failed to detach device pci_0000_02_00_0
error: argument unsupported: VFIO device assignment is currently not supported on this system

To be sure the kernel module vfio-pci was loaded I did a

Code:

modeprobe vfio-pci

Code:

lsmod

returned with :

Code:

Module                  Size  Used by
vfio_pci               16384  0
vfio_pci_core          69632  1 vfio_pci
vfio_virqfd            16384  1 vfio_pci_core
vfio_iommu_type1       45056  0
vfio                   32768  2 vfio_pci_core,vfio_iommu_type1

So that seems to be OK.

I did a grep on vfio in dmesg and I got

Code:

[ 3018.707503] vfio_pci: add [0002:0000[ffffffff:ffffffff]] class 0x000000/00000000

That looks OK, isn't it ?

However qemu still complains that the host is not able to put pci devices in passthrough.
And, yes, I have enabled the correct BIOS options (VT-X etc...).

Anyone suggestions ?

br,

ST

linuxdaddy · 01-01-2023, 02:41 PM

Hi the_zone,

I'm not familiar with qemu. Would the config file let you use eth0/eth1 names instead of the
device ID? Or the subsystem number which is different.

henca · 01-01-2023, 02:45 PM

I haven't tried it myself, I have only used bridging and tun/tap with qemu for purposes like that, but maybe this page will help:

https://www.theseus-os.com/Theseus/b...ssthrough.html

That page adds "0000:" before the pci id, what should be added can probably be found in /sys/bus/pci/devices/

regards Henrik

the_zone · 01-03-2023, 12:46 PM

@linuxdaddy
I am using qemu via virt-manager. It lets you use the NICs via NAT or bridged mode.

@henca
That link gave away part of the trick indeed

I made it work.

Initially I changed my elilo.conf file, adding intel_iommu=on as kernel parameter, and ran eliloconfig, which overwrote my changes. I am used to LILO

Now that intel_iommu=on has been corretly passed I can set my NIC in passthrough with the help of the following script which I found.
Here is the link to the source of the script
https://docs.opennebula.io/6.4/open_...ssthrough.html

Code:

#!/bin/sh
modprobe vfio-pci
for dev in "$@"; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/\$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done

paste it in to a file called vfio-bind.

Calling it with the correct pci bus/device parameter

Code:

./vfio-bind 0000:00:02.0

puts it in passthrough.

yvesjv · 01-03-2023, 01:30 PM

Nice, I could use that solution.
If you were Gryffindor... 10 points to you! ( ͡~ ͜ʖ ͡°)

the_zone · 01-04-2023, 01:06 AM

@yvesjv
Glad this thread is of help to someone else to.
If you have further questions, please pm me.