Arch Linux

gtmcclinton

Member

Registered: 2016-07-31

Posts: 3

Attempting GPU passthrough using Wiki Guide, and getting Code 43

Hi, quite new to Arch Linux, but use RHEL extensively at work though, so not a complete n00b.

I am attempting the wiki guide PCI_passthrough_via_OVMF, and getting stuck with the card showing up with Code 43.

I have a MSI GTX 750 Ti and an i3-6300 in a MSI H170M PRO VDH Mobo (which only has one x16 slot, so I can't try another!), running Arch linux 2016.07.01

The first IOMMU part looks a bit different from what's in the guide:

# dmesg |grep -e DMAR -e IOMMU
[    0.000000] ACPI: DMAR 0x000000008F5A2D08 0000DC (v01 INTEL  SKL      00000001 INTL 00000001)
[    0.000000] DMAR: IOMMU enabled
[    0.039934] DMAR: Host address width 39
[    0.039935] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[    0.039940] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.039941] DMAR: RMRR base: 0x0000008eb3e000 end: 0x0000008eb5dfff
[    0.039942] DMAR: ANDD device: 1 name: \_SB.PCI0.I2C0
[    0.039942] DMAR: ANDD device: 2 name: \_SB.PCI0.I2C1
[    0.039943] DMAR: ANDD device: 9 name: \_SB.PCI0.UA00
[    0.039944] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed90000 IOMMU 0
[    0.039944] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
[    0.039945] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.039946] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.041240] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.614603] DMAR: ACPI device "device:72" under DMAR at fed90000 as 00:15.0
[    0.614605] DMAR: ACPI device "device:73" under DMAR at fed90000 as 00:15.1
[    0.614606] DMAR: ACPI device "device:74" under DMAR at fed90000 as 00:1e.0
[    0.614612] DMAR: No ATSR found
[    0.614701] DMAR: dmar0: Using Queued invalidation
[    0.614709] DMAR: Setting RMRR:
[    0.614719] DMAR: Setting identity map for device 0000:00:14.0 [0x8eb3e000 - 0x8eb5dfff]
[    0.614724] DMAR: Prepare 0-16MiB unity mapping for LPC
[    0.614728] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.614731] DMAR: Intel(R) Virtualization Technology for Directed I/O

Is this anything to worry about?

IOMMU groups: I have my GTX card and associated Audio hardware in a group with the PCI controller, which the guide says should be ok.

# for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo "IOMMU group $(basename "$iommu_group")"; for device in $(ls -1 "$iommu_group"/devices/); do echo -n $'\t'; lspci -nns "$device"; done; done
IOMMU group 0
	00:00.0 Host bridge [0600]: Intel Corporation Skylake Host Bridge/DRAM Registers [8086:190f] (rev 07)
IOMMU group 1
	00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 07)
	01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] [10de:1380] (rev a2)
	01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbc] (rev a1)
IOMMU group 2
	00:08.0 System peripheral [0880]: Intel Corporation Skylake Gaussian Mixture Model [8086:1911]
IOMMU group 3
	00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller [8086:a12f] (rev 31)
	00:14.2 Signal processing controller [1180]: Intel Corporation Sunrise Point-H Thermal subsystem [8086:a131] (rev 31)
IOMMU group 4
	00:15.0 Signal processing controller [1180]: Intel Corporation Sunrise Point-H Serial IO I2C Controller #0 [8086:a160] (rev 31)
	00:15.1 Signal processing controller [1180]: Intel Corporation Sunrise Point-H Serial IO I2C Controller #1 [8086:a161] (rev 31)
IOMMU group 5
	00:16.0 Communication controller [0780]: Intel Corporation Sunrise Point-H CSME HECI #1 [8086:a13a] (rev 31)
IOMMU group 6
	00:17.0 SATA controller [0106]: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] [8086:a102] (rev 31)
IOMMU group 7
	00:1c.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #5 [8086:a114] (rev f1)
	02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU group 8
	00:1e.0 Signal processing controller [1180]: Intel Corporation Sunrise Point-H Serial IO UART #0 [8086:a127] (rev 31)
IOMMU group 9
	00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point-H LPC Controller [8086:a144] (rev 31)
	00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-H PMC [8086:a121] (rev 31)
	00:1f.3 Audio device [0403]: Intel Corporation Sunrise Point-H HD Audio [8086:a170] (rev 31)
	00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-H SMBus [8086:a123] (rev 31)

I don't like the look of these vfio errors:  Are these going to kill me?

dmesg|grep -i vfio
[    0.727048] VFIO - User Level meta-driver version: 0.3
[    0.739568] vfio_pci: add [10de:1380[ffff:ffff]] class 0x000000/00000000
[    0.752875] vfio_pci: add [10de:0fbc[ffff:ffff]] class 0x000000/00000000
[  404.565236] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e@0x258
[  404.565248] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[  404.566424] vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xd0000000-0xd1ffffff 64bit pref]
[  406.269726] vfio-pci 0000:01:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff
[  406.274465] vfio-pci 0000:01:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff

I couldn't get ovmf-git to build, so I did what a suggestion in one of the comments on that page did, and got the binaries from fedora

nvram = [ 
        "/usr/share/ovmf/x64/OVMF_CODE-pure-efi.fd:/usr/share/ovmf/x64/OVMF_VARS-pure-efi.fd"
]

There were another 2 binaries included in the package, OVMF_CODE-need-smm.fd and OVMF_CODE-with-csm.fd. I don't know what's different about these, and what they are supposed to achieve. I haven't tried them yet. Would it be worth trying these?

I got Windows 10 installed, and attached the 2 Nvidia devices to this VM, while also removing the virtual display hardware,

I can now RDP to this VM, and can see the Code 43 error in device manager, but I see nothing in the virt-manager monitor (obviously due to no virtual display hardware), and crucially, nothing on the physical monitor.

The physical monitor shows the POST screen when the host is booting, and the display freezes not very far into the boot process.

But when the VM is started, the display is lost completely, and goes to a blue screen.

I have tried numerous Nvidia drivers, from the latest, back to the first one which officially supports Win 10, with the same results.

I also have tried the hv_vendor_id=blah

libvirt/qemu log shown here:

2016-07-31 10:24:47.893+0000: starting up libvirt version: 2.0.0, qemu version: 2.6.0, hostname: argon.hopto.org
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/sbin/qemu-system-x86_64 -name guest=gallium,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-gallium/master-key.aes -machine pc-i440fx-2.6,accel=kvm,usb=off,vmport=off -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=blah,kvm=off -drive file=/usr/share/ovmf/x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/gallium_VARS.fd,if=pflash,format=raw,unit=1 -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid c94ad2bd-0690-4f21-b750-a8f0b6503e56 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-gallium/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x5 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive file=/images/windies/2008.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -drive if=none,id=drive-ide0-0-1,readonly=on -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -drive if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=26,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:50:52:81,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -device usb-host,hostbus=1,hostaddr=5,id=hostdev0 -device usb-host,hostbus=1,hostaddr=4,id=hostdev1 -device vfio-pci,host=01:00.0,id=hostdev2,bus=pci.0,addr=0x2,rombar=1 -device vfio-pci,host=01:00.1,id=hostdev3,bus=pci.0,addr=0x9,rombar=1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
Domain id=1 is tainted: host-cpu
char device redirected to /dev/pts/2 (label charserial0)
2016-07-31T10:24:48.345032Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev2,bus=pci.0,addr=0x2,rombar=1: Failed to mmap 0000:01:00.0 BAR 3. Performance may be slow
ehci warning: guest updated active QH
2016-07-31T10:26:58.489181Z qemu-system-x86_64: terminating on signal 15 from pid 389
2016-07-31 10:26:59.883+0000: shutting down

Any ideas where to go from here?

Thanks in advance

headkase

Member

Registered: 2011-12-06

Posts: 1,976

Re: Attempting GPU passthrough using Wiki Guide, and getting Code 43

Hide the hypervisor signature: Linky.

gtmcclinton

Member

Registered: 2016-07-31

Posts: 3

Re: Attempting GPU passthrough using Wiki Guide, and getting Code 43

If you look at the qemu log I posted, I have KVM=off in the CPU options already, no worky

gtmcclinton

Member

Registered: 2016-07-31

Posts: 3

Re: Attempting GPU passthrough using Wiki Guide, and getting Code 43

I figured it out, it was a host BIOS setting

I had to set my on board video to be the primary.

Thanks for your consideration though