You are not logged in.

#1 2016-06-24 01:33:46

maiz
Member
Registered: 2014-09-30
Posts: 11

Failing to passthrough a graphics device to a kvm-qemu guest

Hi,
This is my hardware:

lspci | grep -i -e vga -e graphics
01:00.0 VGA compatible controller: NVIDIA Corporation GK110 [GeForce GTX 780 Rev. 2] (rev a1)
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series]
00:02.0 Display controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
lscpu | grep -e virt -e name -e flags
Model name:            Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
Virtualization:        VT-x
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt dtherm ida arat pln pts

I multiseat the 2 graphics cards with the help of logind and lightdm (https://wiki.archlinux.org/index.php/Xo … s_hardware).

I'm trying to passthrough the nvidia card (although I tried both) to a kvm-qemu guest running windows (although I also tried another archlinux).

I've tried using the vfio-linux kernel which comes with the acs-override patch builtin.

cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux-vfio root=UUID=14791614-f54e-4299-bc62-c3328a777f56 rw quiet splash pcie_acs_override=downstream intel_iommu=on video=efifb:off pci-stub.ids=10de:1007,10de:0e1a,1787:2291,1787:aa68

Everything does work as expected: each graphics device is in an isolated IOMMU group

IOMMU group 14
    01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK110 [GeForce GTX 780 Rev. 2] [10de:1007] (rev a1)
    01:00.1 Audio device [0403]: NVIDIA Corporation GK110 HDMI Audio [10de:0e1a] (rev a1)
IOMMU group 15
    02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] [1002:68f9]
    02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] [1002:aa68]

The pci-stub module is effectively the kernel driver in use for the graphics card I wanna passthrough.

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK110 [GeForce GTX 780 Rev. 2] [10de:1007] (rev a1)
    Subsystem: ASUSTeK Computer Inc. Device [1043:84ca]
    Kernel driver in use: pci-stub
    Kernel modules: nouveau
--
02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] [1002:68f9]
    Subsystem: Hightech Information System Ltd. Device [1787:2291]
    Kernel driver in use: pci-stub
    Kernel modules: radeon

After the vfio-binding:

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK110 [GeForce GTX 780 Rev. 2] [10de:1007] (rev a1)
    Subsystem: ASUSTeK Computer Inc. Device [1043:84ca]
    Kernel driver in use: vfio-pci
    Kernel modules: nouveau
--
02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] [1002:68f9]
    Subsystem: Hightech Information System Ltd. Device [1787:2291]
    Kernel driver in use: vfio-pci
    Kernel modules: radeon

Everything so far seems to be right, yet this qemu cmdline doesn't work:

qemu-system-x86_64 -enable-kvm -m 4096 -M q35 -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 -device vfio-pci,host=01:00.0 -device vfio-pci,host=01:00.1 -vga none

Surely, it's supposed to display the seabios bootscreen on the monitor attached with the devices passthrough (in this case pci devices 01:00.0 and 01:00.1), but it doesn't. The screen just stays black; there is a signal (the monitor can tell).

Now here is the bizarre part: if I try the same qemu cmdline with an actual archinstall image (this -> https://www.archlinux.org/download/), it still blanks, but after a while, it boots into the image.

Of course, I've tried many different configurations, I've tried booting an actual real disk with a passed through card, I've tried the same processes with ovmf instead of seabios, I've tried from a uefi-based arch install (instead of legacy). Desperate, I've even tried to flash my VBIOS.

Indeed, if I attempt to read the VBIOS of my nvidia graphics card (for example) from /sys/path/to/the/card/rom with rom-parser (https://github.com/awilliam/rom-parser), I get this:

Valid ROM signature found @0h, PCIR offset 190h
    PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1007, class: 030000
    PCIR: revision 0, vendor revision: 1
Error, ran off the end

However, if I use nvflash to backup the VBIOS:

 nvflash --save save.rom 

The 2 roms (the one from sysfs and the one from nvflash) are different; they're not even of the same size:

-rw-r--r-- 1 iz   iz    59K May 31 18:54 sysfs.rom
-rw-r--r-- 1 root root 205K May 31 18:52 save.rom

After a few comparisons, I remarked they are the same, except that the sysfs.rom is truncated at 59K.
Without much hope, I did flash the bios with another rom (the right one hopefully, since the card still works), but the symptoms persist, nothing has changed.

In the end, I can't seem to passthrough any device, and I have no idea why.

Offline

#2 2016-06-24 14:35:19

extreme9
Member
Registered: 2015-05-17
Posts: 4

Re: Failing to passthrough a graphics device to a kvm-qemu guest

maiz wrote:

Everything so far seems to be right, yet this qemu cmdline doesn't work:

qemu-system-x86_64 -enable-kvm -m 4096 -M q35 -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 -device vfio-pci,host=01:00.0 -device vfio-pci,host=01:00.1 -vga none

Surely, it's supposed to display the seabios bootscreen on the monitor attached with the devices passthrough (in this case pci devices 01:00.0 and 01:00.1), but it doesn't. The screen just stays black; there is a signal (the monitor can tell).

Now here is the bizarre part: if I try the same qemu cmdline with an actual archinstall image (this -> https://www.archlinux.org/download/), it still blanks, but after a while, it boots into the image.

For seabios you should probably use the x-vga parameter: ... -device vfio-pci,host=01:00.0,x-vga=on ...

Offline

Board footer

Powered by FluxBB