Secondary GPU: No video signal

TcShadowWalker · 12-21-2013, 02:44 PM

I'm currently setting up my new desktop computer.
There are two graphics cards installed on the Mainboard (MSI 970A-G43):
- Nvidia Geforce GTX 650 Ti Boost (Vendor: Giga-byte)
- Nvidia Geforce 210 (Vendor: EVGA)

My intention was to setup a KVM Virtual Machine with Windows, use VGA Passthrough with the high-performance card (GTX 650), while running my linux desktop on the Geforce 210. But I don't get that far.

Both of the cards are plugged into the mainboard's PCI Express slots.
However, the Geforce 210 card is currently not working as expected:
During boot, only the GTX650 (connected to the first PCI-E slot in the mainboard) will output a video signal.
Even if I only connect the 210, the screen remains black.

If I remove the 650 from the mainboard, the 210 card works.
I also tried to boot up Windows (on the host computer) and both cards are working there.

It's not that linux isn't recognizing the secondary GPU (it does; see hwinfo below), but I can't use it to display anything. KDE Monitor settings does not allow me to configure the second gpu either.

hwinfo reports:

Code:

root@JPS-Desktop:~ # hwinfo --gfxcard
34: PCI 100.0: 0300 VGA compatible controller (VGA)             
  [Created at pci.319]
  Unique ID: VCu0.73qzEg7BLY1
  Parent ID: _Znp.FMDgKnDywJ4
  SysFS ID: /devices/pci0000:00/0000:00:02.0/0000:01:00.0
  SysFS BusID: 0000:01:00.0
  Hardware Class: graphics card
  Model: "nVidia VGA compatible controller"
  Vendor: pci 0x10de "nVidia Corporation"
  Device: pci 0x11c2 
  SubVendor: pci 0x1458 "Giga-byte Technology"
  SubDevice: pci 0x356d 
  Revision: 0xa1
  Driver: "nvidia"
  Driver Modules: "nvidia"
  Memory Range: 0xf6000000-0xf6ffffff (rw,non-prefetchable)
  Memory Range: 0xc8000000-0xcfffffff (ro,non-prefetchable)
  Memory Range: 0xd0000000-0xd1ffffff (ro,non-prefetchable)
  I/O Ports: 0xe000-0xefff (rw)
  Memory Range: 0xf7000000-0xf707ffff (ro,non-prefetchable,disabled)
  IRQ: 88 (19922 events)
  Module Alias: "pci:v000010DEd000011C2sv00001458sd0000356Dbc03sc00i00"
  Driver Info #0:
    Driver Status: nvidiafb is not active
    Driver Activation Cmd: "modprobe nvidiafb"
  Driver Info #1:
    Driver Status: nouveau is not active
    Driver Activation Cmd: "modprobe nouveau"
  Driver Info #2:
    Driver Status: nvidia is active
    Driver Activation Cmd: "modprobe nvidia"
  Config Status: cfg=no, avail=yes, need=no, active=unknown
  Attached to: #10 (PCI bridge)

36: PCI 200.0: 0300 VGA compatible controller (VGA)
  [Created at pci.319]
  Unique ID: B35A.krMl9Oq+G5C
  Parent ID: 8otl.HB6AIsoa_f3
  SysFS ID: /devices/pci0000:00/0000:00:04.0/0000:02:00.0
  SysFS BusID: 0000:02:00.0
  Hardware Class: graphics card
  Model: "nVidia GT200 [GeForce 210]"
  Vendor: pci 0x10de "nVidia Corporation"
  Device: pci 0x0a65 "GT200 [GeForce 210]"
  SubVendor: pci 0x3842 "eVga.com. Corp."
  SubDevice: pci 0x1313 
  Revision: 0xa2
  Driver: "nvidia"
  Driver Modules: "nvidia"
  Memory Range: 0xf4000000-0xf4ffffff (rw,non-prefetchable)
  Memory Range: 0xb0000000-0xbfffffff (ro,non-prefetchable)
  Memory Range: 0xc0000000-0xc1ffffff (ro,non-prefetchable)
  I/O Ports: 0xd000-0xdfff (rw)
  Memory Range: 0xf5000000-0xf507ffff (ro,non-prefetchable,disabled)
  IRQ: 89 (4706 events)
  Module Alias: "pci:v000010DEd00000A65sv00003842sd00001313bc03sc00i00"
  Driver Info #0:
    Driver Status: nvidiafb is not active
    Driver Activation Cmd: "modprobe nvidiafb"
  Driver Info #1:
    Driver Status: nouveau is not active
    Driver Activation Cmd: "modprobe nouveau"
  Driver Info #2:
    Driver Status: nvidia is active
    Driver Activation Cmd: "modprobe nvidia"
  Config Status: cfg=no, avail=yes, need=no, active=unknown
  Attached to: #11 (PCI bridge)

Primary display adapter: #34

I'd like to switch the GPUs in their PCI-E slots, to see if that helps. But unfortunately, the 650 is too large to be installed into the lower slot.

Any ideas?

albinard · 12-21-2013, 04:19 PM

What distro are you running? Reason I ask is that on another forum there are three very recent issues with Nvidia drivers and updates. In fact, I'm running a GeForce 210 and had one of them myself. If you installed your driver from Additional Drivers, try

Code:

jockey-text --list

and see what is running and what is disabled.

TcShadowWalker · 12-22-2013, 10:14 AM

Thanks for the reply.
I'm currently running OpenSuse 12.3.
(CPU is AMD FX-6300)

In the meantime, I already partially solved the problem.
Running the NVIDIA configuration tool, I can enable the second GPU, so at least I can run my linux desktop.
It still doesn't work during startup, but I don't really need that anyway.

Now, after many hours of reading up on VGA Passthrough and trying to pass the primary GPU to KVM,
I still couldn't really figure out how to make it work.

The secondary GPU (running my host desktop) is using the nouveau driver, 'nvidia' kernel module is not loaded.

Both the 01:00.0 and 01:00.1 PCI devices (the primary vga card and it's HDMI audio device) are assigned to the VM.
I'm running the host kernel with all the required options:
amd_iommu=on pci-stub.ids=10de:11c2,10de:0e0b vfio_iommu_type1.allow_unsafe_interrupts=1
IOMMU is enabled in BIOS.

If I start the VM, however, all I get is:

Code:

Error starting domain: internal error Unable to reset PCI device 0000:01:00.1: internal error Active 0000:01:00.0 devices on bus with 0000:01:00.1, not doing bus reset

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 96, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 117, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1098, in startup
    self._backend.create()
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 681, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: internal error Unable to reset PCI device 0000:01:00.1: internal error Active 0000:01:00.0 devices on bus with 0000:01:00.1, not doing bus reset

Explicitly detaching the devices using virsh doesn't work either.

Code:

root@JPS-Desktop:/$  virsh nodedev-dettach pci_0000_01_00_1
Device pci_0000_01_00_1 detached
root@JPS-Desktop:/$  virsh nodedev-dettach pci_0000_01_00_0
Device pci_0000_01_00_0 detached

mostlyharmless · 12-23-2013, 07:24 PM

Quote:

Now, after many hours of reading up on VGA Passthrough and trying to pass the primary GPU to KVM,
I still couldn't really figure out how to make it work.

If you read about it, you would know that VGA OpenSuSE's 12.3 has kernel 3.7.10 and that kernel 3.9+ is recommended. In fact you should follow the tutorial on
https://bbs.archlinux.org/viewtopic.php?id=162768

Written for Arch but easily adaptable to any distro...

Looks like the latest "factory" version of OpenSUSE has kernel 3.12.2, so you'll just need to use the latest qemu and seabios if you update your version. If you're not wedded to OpenSUSE you can try the Arch tutorial with Arch. There's also a Xen version that uses Linux Mint. http://forums.linuxmint.com/viewtopic.php?t=112013&f=42

TcShadowWalker · 12-24-2013, 08:28 AM

Quote:

Originally Posted by mostlyharmless

If you read about it, you would know that VGA OpenSuSE's 12.3 has kernel 3.7.10 and that kernel 3.9+ is recommended.
In fact you should follow the tutorial on
https://bbs.archlinux.org/viewtopic.php?id=162768

Written for Arch but easily adaptable to any distro...

That is exactly the tutorial I was following.
However, I didn't pay enough attention to the required kernel version.
I guess I just assumed that current OpenSuse kernel was adequate.
Thanks a lot for the tip.

mostlyharmless · 12-24-2013, 12:16 PM

Ok, well then good luck. I noticed in your original post that you used virt-manager, not the qemu command line. If you use the test commandline from the Arch thread above:

Code:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

with of course the appropriate substitutions for the devices and paths, does that produce the expected output?

Did you also use the qemu-git with the patches? How did you build that and incorporate it into OpenSUSE?