Arch Linux

PunisherHD

Member

Registered: 2014-04-21

Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,
I'm trying to passthrough my Nvidia GTX 760 card to Windows 7 KVM guest.
I got through all steps in the first post and now I have Windows 7 booting and displaying on my monitor, but nvidia driver says "error code 43".
Everything else works fine (USB, Sata drives ...), only the Nvidia driver has problem.

I'm using ArchLinux, kernel 3.14 from OP, Qemu 2.0.0 from "testing" repository.

HW setup:
ASUS H81I-Plus
Core i5 4670
Geforce GTX 760

Qemu command line:

qemu-system-x86_64 \
  -enable-kvm \
  -M q35 \
  -m 4096 \
  -cpu host \
  -smp 2,sockets=1,cores=2,threads=1 \
  -bios /usr/share/qemu/bios.bin \
  -vga none \
  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
  -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
  -device ahci,bus=pcie.0,id=ahci \
  -drive file=/dev/sda,id=disk1,format=raw -device ide-hd,bus=ahci.0,drive=disk1 \
  -drive file=/dev/sdc,id=disk2,format=raw -device ide-hd,bus=ahci.1,drive=disk2 \
  -device vfio-pci,host=00:14.0,bus=pcie.0 \
  -device vfio-pci,host=00:1a.0,bus=pcie.0

I tried to rebuild Qemu 2.0.0 adding this patch: http://lists.nongnu.org/archive/html/qe … 00597.html but no luck, still code 43.

Any hint to solve this mysterious code 43 ?

Krobar

Member

Registered: 2014-04-12

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,
I'm trying to passthrough my Nvidia GTX 760 card to Windows 7 KVM guest.
I got through all steps in the first post and now I have Windows 7 booting and displaying on my monitor, but nvidia driver says "error code 43".
Everything else works fine (USB, Sata drives ...), only the Nvidia driver has problem.

I'm using ArchLinux, kernel 3.14 from OP, Qemu 2.0.0 from "testing" repository.

HW setup:
ASUS H81I-Plus
Core i5 4670
Geforce GTX 760

Qemu command line:

qemu-system-x86_64 \
  -enable-kvm \
  -M q35 \
  -m 4096 \
  -cpu host \
  -smp 2,sockets=1,cores=2,threads=1 \
  -bios /usr/share/qemu/bios.bin \
  -vga none \
  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
  -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
  -device ahci,bus=pcie.0,id=ahci \
  -drive file=/dev/sda,id=disk1,format=raw -device ide-hd,bus=ahci.0,drive=disk1 \
  -drive file=/dev/sdc,id=disk2,format=raw -device ide-hd,bus=ahci.1,drive=disk2 \
  -device vfio-pci,host=00:14.0,bus=pcie.0 \
  -device vfio-pci,host=00:1a.0,bus=pcie.0

I tried to rebuild Qemu 2.0.0 adding this patch: http://lists.nongnu.org/archive/html/qe … 00597.html but no luck, still code 43.

Any hint to solve this mysterious code 43 ?

Have you tried these tweaks?

EDIT 2: Well, thanks to this post from MacCoaster, who had a similar configuration (Intel Haswell + Asus MB + R7770), I added the following to the kernel command line:

vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0 nohz=off

I don't know which of these got it working with the Radeon HD 7770, but it seems quite stable, I can even run the latest 3D Mark smoothly (BSOD with the NVIDIA, but I should try with the new kernel command line).

spam

Member

Registered: 2013-06-15

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have been abble to make some tries, with an Ubuntu host I had hda-codec errors but gfx was working both with nouveau and nvidia default configuration. I tried to disable i915 module and any sound module yet I still got code 43 under windows. Do you believe I'll had better luck with an AMD card ?  Is there a way to do pci-assign with kvm and a GTX 660 ?

Btw, i did not test nohz=off.

My last attempt was with

$ cat /etc/modprobe.d/*
blacklist nouveau
blacklist nvidia
blacklist i915
blacklist snd
blacklist snd_hda_intel
blacklist snd_hda_codec_realtek
blacklist snd_hda_codec_generic
blacklist snd_hda_intel
blacklist snd_hda_codec
blacklist snd_hwdep
blacklist snd_pcm
blacklist snd_timer
blacklist snd
blacklist soundcore
#options kvm ignore_msrs=1
options kvm_intel emulate_invalid_guest_state=0
options pci-stub ids=10de:11c0,10de:0e0b
options vfio_iommu_type1 allow_unsafe_interrupts=1

$ cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=0f8e3d34-5e38-4276-969f-c64aaf7f3245 rw quiet intel_iommu=on pci-stub.ids=10de:11c0,10de:0e0b

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Anyone know if VFIO can be used with SR-IOV/VNICs? Have an Intel 1000ET dual card and it looks like Intel Virtual Function NICs might be easier to setup in Arch than a bridge.

Yes

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

doubledr

Member

Registered: 2013-12-26

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, is there any kernel option to active ACS patch v2? Right now I applied the patch but it is not activated. My PCH ports are still stucked in a big group...

shweng

Member

Registered: 2014-03-13

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello nbhs and all,
I have tried serval days to pass-through my Nvidia GT640 and AMD HD7750 GPU to Windows7 KVM guest but with no succeed.
I followed all steps in the first post and now I could pass-through GPU device to Windows 7 VM but still has the follow issues.
1.VM with AMD HD7750 GPU, win7 start up BSOD with 0x00000116 on atikmpag.sys
    same as ilya80 post at https://bbs.archlinux.org/viewtopic.php?pid=1351243,but the solution of change "clear_emulator_capabilities = 0" on qemu.conf take no effect.
2.VM with Nvidia GT640 GPU, both vm and host hangup when vm OS booting to the twinkling mircorsoft windows logo.
3.there is no display on Qemu's Window,need to connect pass-throughed GPU's DVI port to display VM desktop.
for issue 1 and 2,enter VM on "Safe Mode" will be OK, but without graphics driver thus no hardware acceleration.
could you please help me to check my issue.
Hardware:
CPU:Intel Core i5-3470
MB:Gigabyte Z77P-D3, BIOS F7 08/24/2012 (VT-d enabled in BIOS)
RAM:4GB
Host:Intel HD IGP
Guest nVidia GeForce GT 640 or AMD RADEON HD7750
 
On the software side, I'm running Fedora 20 with kernel 3.14.1 which apply the patchset of linux-mainline.tar.gz except kernel-38-gcc48-2.patch
fix_reading_of_DR6.patch and debug_registers.patch
The kernel was compiled with the following config: 
CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_VGA=y

IOMMU is enabled through kernel switch.
disabled all the modules which would interfere with guest graphics cards (radeon, nouveau, nvidia,fglrx)
latest qemu:qemu-2.0.0-1(also tried NoSnoop patch from http://lists.nongnu.org/archive/html/qe … 00597.html but no effect)
latest seabios:seabios-1.7.4-5

GPU
[root@localhost home]# lspci |grep VGA
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750]
06:00.0 VGA compatible controller: NVIDIA Corporation GK107 [GeForce GT 640] (rev a1)

kernel param
[root@localhost home]# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.14.1 root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/swap vconsole.font=latarcyrheb-sun16 rd.lvm.lv=fedora/root rhgb quiet intel_iommu=on pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 LANG=en_US.UTF-8

update initramfs
/etc/modprobe.d/
[root@localhost modprobe.d]# cat blacklist.conf
blacklist radeon
blacklist redeonhd
blacklist nvidia
blacklist nouveau
blacklist fglrx
[root@localhost modprobe.d]# cat vfio_iommu_type1.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

[root@localhost modprobe.d]#dracut -f -v /boot/initramfs-$(uname -r).img $(uname -r)

qemu command:
qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 1,sockets=1,cores=1,threads=1 \
-bios /usr/share/qemu/bios.bin \
-vga none \
-drive file=/home/images/nv.qcow2,id=disk,format=qcow2 -device ide-hd,bus=ide.0,drive=disk \
-drive file=/ISO/win7.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd -boot order=cd,menu=on \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1                                      \
-device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=06:00.1,bus=root.1,addr=00.1 \
-usbdevice tablet
(host=01:00.0 for AMD gpu).
(host=06:00.0 for Nvidia gpu).

dmesg:
[root@localhost modprobe.d]# dmesg |grep IOMMU
[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.000000] Intel-IOMMU: enabled
[    0.018151] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.018155] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.018226] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.783685] IOMMU 0 0xfed90000: using Queued invalidation
[    0.783686] IOMMU 1 0xfed91000: using Queued invalidation
[    0.783688] IOMMU: Setting RMRR:
[    0.783696] IOMMU: Setting identity map for device 0000:00:02.0 [0x9f800000 - 0xaf9fffff]
[    0.784850] IOMMU: Setting identity map for device 0000:00:1d.0 [0x9d5c9000 - 0x9d5f3fff]
[    0.784865] IOMMU: Setting identity map for device 0000:00:1a.0 [0x9d5c9000 - 0x9d5f3fff]
[    0.784878] IOMMU: Setting identity map for device 0000:00:14.0 [0x9d5c9000 - 0x9d5f3fff]
[    0.784887] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.784892] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

[root@localhost modprobe.d]# dmesg |grep DMAR
[    0.000000] ACPI: DMAR 000000009d775558 0000B8 (v01 INTEL      SNB  00000001 INTL 00000001)
[    0.783661] DMAR: No ATSR found
[   24.261510] [drm] DMAR active, disabling use of stolen memory

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@shweng
Don't know about the NVIDIA problem, you might want to try the qemu/seabios from git posted on page 1

However for the AMD problem

.VM with AMD HD7750 GPU, win7 start up BSOD with 0x00000116 on atikmpag.sys
    same as ilya80 post at https://bbs.archlinux.org/viewtopic.php?pid=1351243,but the solution of change "clear_emulator_capabilities = 0" on qemu.conf take no effect.

I would recommend (1) don't pass the audio part 1:00.1, (2) in safe mode uninstall the AMD drivers; then after that presumably you can boot windows with the generic VGA driver (and you won't have the atimpag.sys error because there won't be an ati driver).  (3) then reinstall only the graphics driver, not the CCC. When you install the drivers, there is an option to do that. You will still have accelerated graphics, just not the tuning available with the CCC.  Anyway, that worked for me with my HD7970.

ohetfi

Member

Registered: 2014-04-07

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, nbhs and all!

I would like to share my configuration of my sort of wannabe "All-in-One" KVM VFIO on Arch Linux. Using all nbhs' ingredients (kernel 3.14), I have successfully passed through my GeForce GTX 770 and IBM M1015 to my KVM guests. Using this configuration I can also natively boot to both Linux Mint Debian Edition (for whole disk backup of Arch Linux disk) and to Windows 7 when I need the true bare metal performance (Metro: Last Light +/-4 FPS increase on Windows 7 native).

3DMark Results
Bare Metal: http://www.3dmark.com/fs/1945959
KVM Guest: http://www.3dmark.com/fs/2040597
Any idea on how to make my CPU recognized by 3DMark when on KVM Guest?

Hardware Specification
CPU: Intel Core i7-4770
Memory: 4 x 8 GB Patriot Signature Line Kit
Motherboard: ASRock Z87 Extreme6
GPU:
    - Intel HD Graphics 4600 (host)
    - Gigabyte GeForce GTX 770 Windforce (guest1)
    - Cirrus (guest2)
Disks:
    - SanDisk FDD 32 GB -> Arch Linux (host)
    - Hitachi HDD 500 GB -> Windows 7 (guest1)
    - Seagate HDD 160 GB -> Linux Mint Debian Edition (guest2)
    - 4 x Hitachi HDD 2 TB -> ZFS (guest2)
HBA: IBM M1015 RAID Card flashed with IT-mode (guest2)

Boot Parameters (/proc/cmdline)

BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=d6205fde-e945-4ded-b4e1-fe3bd98b1f84 rw quiet intel_iommu=on pcie_acs_override=downstream

I need to add the pcie_acs_override parameter to force PCIe devices to be on separate IOMMU groups, and of cource intel_iommu parameter to make all this magic come true.

Kernel Module Parameters (/etc/modprobe.d/*.conf)

# Blacklist Gigabyte GeForce GTX 770 Windforce
blacklist nouveau
blacklist nvidia
# Bonding Modes: balance-rr, active-backup, balance-xor, broadcast, 802.3ad, balance-tlb, balance-alb
options bonding mode=balance-alb miimon=100 updelay=200
# Needed to avoid crashing KVM guest when running Passmark Performance Test and SiSoftware Sandra
options kvm ignore_msrs=1
# 01:00.0 10de:1184 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1)
# 01:00.1 10de:0e0a Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
# 02:00.0 1000:0072 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
options pci-stub ids=10de:1184,10de:0e0a,1000:0072
# Enable interrupt remapping
options vfio_iommu_type1 allow_unsafe_interrupts=1

I bond 2 of my NICs and then setup a bridge device for all my guests.

VFIO Bind Configuration (/etc/vfio.cfg)

# 01:00.0 10de:1184 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1)
# 01:00.1 10de:0e0a Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
# 02:00.0 1000:0072 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
DEVICES="0000:01:00.0 0000:01:00.1 0000:02:00.0"

Hugepages Parameter (/etc/sysctl.d/40-vm.conf)

# Allocate 26 GB of RAM for Hugepages
vm.nr_hugepages = 13312

The host memory is 32 GB, I allocated around 26 GB for all my KVM guests. 6 GB for the host.

Windows 7 Guest Parameters (kvm-windows7.sh)

#!/bin/bash

# Use PulseAudio as the audio driver
QEMU_PA_SAMPLES=128
export QEMU_AUDIO_DRV=pa

# Startup Windows 7 guest
sudo qemu-system-x86_64 \
	-name Windows-7 \
	-cpu host \
	-smp 8,sockets=1,cores=4,threads=2 \
	-enable-kvm \
	-m 8G \
	-mem-path /dev/hugepages \
	-mem-prealloc \
	-balloon none \
	-rtc clock=host \
	-vga none \
	-serial null \
	-parallel null \
	-monitor none \
	-display none \
	-daemonize \
	-soundhw hda \
	-k en-us \
	-machine type=q35,accel=kvm \
	-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
	-boot order=c \
	-drive if=none,id=drive0,cache=none,aio=native,format=raw,file=/dev/disk/by-id/ata-Hitachi_HDS721050CLA660_JP1570FR1ZWP7K \
	-device virtio-blk,drive=drive0,scsi=off,config-wce=off,x-data-plane=on \
	-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
	-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
	-net nic,model=virtio,macaddr=52:54:00:6b:03:05 -net bridge,br=br0 \
	-usb \
	-device ich9-usb-uhci3,id=uhci \
	-device usb-ehci,id=ehci \
	-usbdevice host:093a:2510 \
	-usbdevice host:04d9:1603 \
	-usbdevice host:046d:c21e \
	-usbdevice host:0bc2:2300

I use host clock (UTC), since I sometimes accidentally boot the Windows 7 natively and wanted to avoid my hardware clock confusion. So far no need to eject or no issues on rebooting the guest, thank God.

Linux Mint Debian Edition Guest Parameters (kvm-lmde.sh)

#!/bin/bash

# Use PulseAudio as the audio driver
export QEMU_PA_SAMPLES=128
export QEMU_AUDIO_DRV=pa

# Starts up Linux Mint Debian Edition guest
sudo qemu-system-x86_64 \
	-name LMDE \
	-cpu host \
	-smp 8,sockets=1,cores=4,threads=2 \
	-enable-kvm \
	-m 8G \
	-mem-path /dev/hugepages \
	-mem-prealloc \
	-balloon none \
	-rtc clock=host \
	-vga std \
	-vnc localhost:1 \
	-daemonize \
	-soundhw hda \
	-k en-us \
	-machine type=q35,accel=kvm \
	-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
	-boot order=c \
	-drive if=none,id=drive0,cache=none,aio=native,format=raw,file=/dev/disk/by-id/ata-ST3160815AS_5RA7CPYR \
	-device virtio-blk,drive=drive0,scsi=off,config-wce=off,x-data-plane=on \
	-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,romfile=/opt/sas2008/x64sas2.rom \
	-net nic,model=virtio,macaddr=52:54:00:3b:cf:09 -net bridge,br=br0 \
	-usb \
	-device ich9-usb-uhci3,id=uhci \
	-device usb-ehci,id=ehci

My ZFS storage server.

THANKS A LOT, Guys!

anickname

Member

Registered: 2014-01-07

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, nbhs and all!

I would like to share my configuration of my sort of wannabe "All-in-One" KVM VFIO on Arch Linux. Using all nbhs' ingredients (kernel 3.14), I have successfully passed through my GeForce GTX 770 and IBM M1015 to my KVM guests. Using this configuration I can also natively boot to both Linux Mint Debian Edition (for whole disk backup of Arch Linux disk) and to Windows 7 when I need the true bare metal performance (Metro: Last Light +/-4 FPS increase on Windows 7 native).

THANKS A LOT, Guys!

You will also get some network speed improvement using vhost=on.

Eg.:

-netdev tap,vhost=on,ifname=$VM,script=$KVM_ROOT/scripts/vm_ifup_brlan,id=brlan -device virtio-net-pci,mac=52:54:00:00:00:21,netdev=brlan \

where vm_ifup_brlan is :

#!/bin/sh

BRIDGE=brlan

echo "Executing $0"

echo "Bringing up $1 for bridged mode..."
ip link set $1 up promisc on

echo "Disabling STP for bridge $BRIDGE"
brctl stp $BRIDGE off

echo "Adding $1 to $BRIDGE ..."
brctl addif $BRIDGE $1

sleep 2

ohetfi

Member

Registered: 2014-04-07

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, nbhs and all!

I would like to share my configuration of my sort of wannabe "All-in-One" KVM VFIO on Arch Linux. Using all nbhs' ingredients (kernel 3.14), I have successfully passed through my GeForce GTX 770 and IBM M1015 to my KVM guests. Using this configuration I can also natively boot to both Linux Mint Debian Edition (for whole disk backup of Arch Linux disk) and to Windows 7 when I need the true bare metal performance (Metro: Last Light +/-4 FPS increase on Windows 7 native).

THANKS A LOT, Guys!

You will also get some network speed improvement using vhost=on.

Eg.:

-netdev tap,vhost=on,ifname=$VM,script=$KVM_ROOT/scripts/vm_ifup_brlan,id=brlan -device virtio-net-pci,mac=52:54:00:00:00:21,netdev=brlan \

where vm_ifup_brlan is :

#!/bin/sh

BRIDGE=brlan

echo "Executing $0"

echo "Bringing up $1 for bridged mode..."
ip link set $1 up promisc on

echo "Disabling STP for bridge $BRIDGE"
brctl stp $BRIDGE off

echo "Adding $1 to $BRIDGE ..."
brctl addif $BRIDGE $1

sleep 2

Hi, anickname!

Thanks for pointing that, I was having difficulties on applying the new QEMU network parameters thus ended up using the ones that I have mentioned in my post. Will definitely try that. In my case, my bridge device is called br0, so I assume your brlan is my br0?

I also found this slide http://www.linux-kvm.org/wiki/images/c/ … 008_11.pdf, probably will try to enable/disable the kvm_intel module parameters and see if it would increase my benchmarks: vpid, flexpriority, and ept.

Thanks!

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear All,

I am trying to follow guide http://fds-team.de/cms/articles/2013-12 … yer-u.html , but faced with issue:

A disk read error occurred
Press Ctrl+Alt+Del to restart

I think is due to different types of booting: I use UEFI (and GPT) during real booting, but VM uses seabios. Could you please point me to right direction: what should I do to boot windows in VM?
BTW, I have installed Windows on one partition (recovery is also placed on the C drive, I have no 200MB hide partition)

belliash

Member

Registered: 2014-04-25

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone successfully passed through a discrete GPU on notebook with nvidia optimus technology? I have already spent a whole day on this, but all the time Windows reports code 10 (Device cannot start) in Device Manager.

I use vanilla kernel 3.14.1, qemu 2.0 and seabios 1.7.3.2. Do you have any ideas?

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

A disk read error occurred

The message has been fixed, but I faced another:

missing operating system

Even if I run Windows installation from VM - installation process can not perform first reboot

Original drive:

Disk /dev/sdb: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: E0A01D64-268D-4491-88AB-B4498B9F4600

Device           Start          End   Size Type
/dev/sdb1         8192   1952571391 931.1G Microsoft basic data
/dev/sdb2   1952571392   3905134591 931.1G Microsoft basic data
/dev/sdb3   3905134592   5857697791 931.1G Microsoft basic data
/dev/sdb4   5857697792   5860532223   1.4G Microsoft basic data

sdb2 - windows partition

Creation procedures:

dd if=/dev/zero of=boot.mbr count=0 seek=1952571392
sudo losetup /dev/loop0 boot.mgr
sudo mdadm --build /dev/md0 --level=linear --raid-devices=2 /dev/loop0 /dev/sdb2

fdisk -l /dev/md0

Disk /dev/md0: 1.8 TiB, 1999428911104 bytes, 3905134592 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0xf057e88f

Device     Boot      Start        End    Blocks  Id System
/dev/md0p1 *    1952571392 3905134591 976281600   7 HPFS/NTFS/exFAT

Could you please help me to fix this issue.
Thanks in advance!

Last edited by myweb (2014-04-26 00:03:01)

belliash

Member

Registered: 2014-04-25

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone successfully passed through a discrete GPU on notebook with nvidia optimus technology? I have already spent a whole day on this, but all the time Windows reports code 10 (Device cannot start) in Device Manager.

I use vanilla kernel 3.14.1, qemu 2.0 and seabios 1.7.3.2. Do you have any ideas?

Now i got Error 43, like others using nvidia cards. Is there any way to fix this?

zzz3000

Member

Registered: 2014-02-16

Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I was able to get it work on Fedora rawhide(with qemu 2.0).
It works without patches but with corrupted host graphics.

There are first time it works I think because of new fedora rawhide(qemu 2.0 may be)

The guest windows show black screen with while lines and host crashed before(qemu 1.6).

My config
MB: MSI h67ma-e45
CPU Intel i5-2400
host GPU HD2000
guest GPU HD7750 (sapphire 1gb ddr5)

Last edited by zzz3000 (2014-04-27 13:00:05)

apex8

Member

Registered: 2014-03-29

Posts: 60

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@zzz3000
Can you tell, what you mean with corrupted graphics?

zzz3000

Member

Registered: 2014-02-16

Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@zzz3000
Can you tell, what you mean with corrupted graphics?

Host image become spoiled(I was able to solve this host problem with i915 patch before but guest did not work)
Now with qemu 2.0 quest works fine I think  kernel patching will solve host image problem

apex8

Member

Registered: 2014-03-29

Posts: 60

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@zzz3000
Can you tell, what you mean with corrupted graphics?

Host image become spoiled(I was able to solve this host problem with i915 patch before but guest did not work)
Now with qemu 2.0 quest works fine I think  kernel patching will solve host image problem

Ok Thanks. So that means, I still need the i915 patch, despite using intel exclusively on the host and nvidia exclusively in the VM.

zzz3000

Member

Registered: 2014-02-16

Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok Thanks. So that means, I still need the i915 patch, despite using intel exclusively on the host and nvidia exclusively in the VM.

It is true in my case with fedora rawhide (kernel 3.15 qemu 2.0)

ohetfi

Member

Registered: 2014-04-07

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

...

You will also get some network speed improvement using vhost=on.

Eg.:

-netdev tap,vhost=on,ifname=$VM,script=$KVM_ROOT/scripts/vm_ifup_brlan,id=brlan -device virtio-net-pci,mac=52:54:00:00:00:21,netdev=brlan \

...

Hi, anickname! Your suggestion worked. When I ps -ef | grep vhost, I can see the vhost processes on my host Arch Linux. Don't know if it's only my feeling or not, but I can now download Steam games at my full Internet bandwidth speed and very much stable just like when I passthrough one of my Intel NIC to a VM. Before using vhost, I experienced the bandwidth speed were very fluctuate and almost never full speed. Thanks!

...
The message has been fixed, but I faced another:
...
Could you please help me to fix this issue.
Thanks in advance!

Don't know if my experience will help you in any way, but I have different disk for each VM. My Arch Linux host resides in a 32 GB flash disk, which is very sufficient because I want my Arch Linux lean and simple, in addition I will be periodically do full offline disk backup, so 32 GB is enough for me. The flash disk is GPT, and I use EFI since my motherboard supports it and Arch Linux Beginner's Guide recommend it to use EFI/GPT for new installation if possible.

My Windows VM resides in a separate dedicated 500 GB hard disk, which I supply by giving a direct access to /dev/sdb (for instance). So the Windows VM have full access to the whole disk, but still via Arch Linux host drivers. I then install the Windows VM from Seabios and follow the installation as usual. After installing drivers etc inside the VM, I tried rebooting to the 500 GB disk (native reboot not VM reboot). And to my surprise, the Windows installation still runs, although needs to reinstall drivers for most part of the hardware. I didn't setup the grub to probe for the Windows partition, I switch manually to different boot drive via EFI/BIOS. So I think if you have spare disk, it would be easier to setup. Hope it helps.

rabcor

Banned

Registered: 2013-02-09

Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm trying this once again. I'm using the nouveau drivers with a GTX 550-Ti this time around and then I plan to passthrough a GTX 670.

So far, I installed qemu-git from AUR. I added pci-stub to the module section of mkinitcpio.conf and then ran the mkinitcpio command, now here's where I get stuck.

Now add this to your grub cfg file:

pci-stub.ids=1002:6719,1002:aa80

those instructions aren't specific enough for me; which grub cfg file? the one in /etc/default/ or the one in /boot/grub? and more importantly, where in that file?

Then there's this step that's confusing me as well

If your board doesn't enable interrupt remapping, you need to add this to your grub cfg:

vfio_iommu_type1.allow_unsafe_interrupts=1

Or if vfio-pci was built as a module ( default on arch )

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf

I've got a board that supports interrupt remapping I think, but i'm also using the default arch kernel so vfio-pci is built as a module, the part that's confusing me here is that in the former example we have:

vfio_iommu_type1.allow_unsafe_interrupts=1

but in the latter we have:

options vfio_iommu_type1 allow_unsafe_interrupts=1

The part that's confusing me is that one has a period between "type1" and "allow" and the other has a space, shouldn't both have a period? I.e. shouldn't it be:

options vfio_iommu_type1.allow_unsafe_interrupts=1

Sorry for being a noob

Last edited by rabcor (2014-04-28 10:30:01)

---

Making Logitech G510 Work on Linux!

dwe11er

Member

Registered: 2014-03-18

Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm trying this once again. I'm using the nouveau drivers with a GTX 550-Ti this time around and then I plan to passthrough a GTX 670.

So far, I installed qemu-git from AUR. I added pci-stub to the module section of mkinitcpio.conf and then ran the mkinitcpio command, now here's where I get stuck.

Now add this to your grub cfg file:

pci-stub.ids=1002:6719,1002:aa80

those instructions aren't specific enough for me; which grub cfg file? the one in /etc/default/ or the one in /boot/grub? and more importantly, where in that file?

Then there's this step that's confusing me as well

If your board doesn't enable interrupt remapping, you need to add this to your grub cfg:

vfio_iommu_type1.allow_unsafe_interrupts=1

Or if vfio-pci was built as a module ( default on arch )

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf

I've got a board that supports interrupt remapping I think, but i'm also using the default arch kernel so vfio-pci is built as a module, the part that's confusing me here is that in the former example we have:

vfio_iommu_type1.allow_unsafe_interrupts=1

but in the latter we have:

options vfio_iommu_type1 allow_unsafe_interrupts=1

The part that's confusing me is that one has a period between "type1" and "allow" and the other has a space, shouldn't both have a period? I.e. shouldn't it be:

options vfio_iommu_type1.allow_unsafe_interrupts=1

Sorry for being a noob

The one with period is kernel parameter in grub.cfg (or /etc/default/grub), the other one put inside the file (with .conf extension) in /etc/modprobe.d/. Then add modconf hook to HOOKS in mkinicpio.conf. For best result choose latter method

tswindell

Member

Registered: 2014-04-10

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, when passing through the HD Audio and Video card of my nVidia 680GTX, under windows Audio works great, 3D is a bit buggy. But when using Ubuntu as the guest, with both audio and video passed through, the 3D accel works great, better than under Windows, but Audio doesn't work at all Get a weird feedback loop of some kind.

Has anyone managed to get an nVidia card's audio parts to work using VFIO passthrough to a Linux guest?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, when passing through the HD Audio and Video card of my nVidia 680GTX, under windows Audio works great, 3D is a bit buggy. But when using Ubuntu as the guest, with both audio and video passed through, the 3D accel works great, better than under Windows, but Audio doesn't work at all Get a weird feedback loop of some kind.

Has anyone managed to get an nVidia card's audio parts to work using VFIO passthrough to a Linux guest?

You might try enabling MSI on the snd_hda_intel driver.  You can do this on the kernel command line with snd_hda_intel.enable_msi=1 or via modprobe.d with "options snd_hda_intel enable_msi=1".  Nvidia apparently has a poor track record with MSI on the audio card as there's code specifically to disable it on Nvidia devices, but they also have a poor track record of making device that properly re-assert INTx if we mask it using the PCI 2.3 spec defined disable bit.  MSI is more efficient in a virt environment anyway, so it it works, it's better to use it.  I haven't had any trouble with audio on a GTX660, but on a Quadro K4000 (which uses the same device ID for the audio function), it's a complete fail.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

rabcor

Banned

Registered: 2013-02-09

Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks @dwe11er!

dmesg | grep pci-stub
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=298f5fd5-65b3-4f84-9de7-046660a66597 rw quiet pci-stub.ids=10de:1189,10de:0e0a
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=298f5fd5-65b3-4f84-9de7-046660a66597 rw quiet pci-stub.ids=10de:1189,10de:0e0a
[    0.625505] pci-stub: add 10DE:1189 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.625513] pci-stub 0000:02:00.0: claimed by stub
[    0.625518] pci-stub: add 10DE:0E0A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.625522] pci-stub 0000:02:00.1: claimed by stub

Now I've reached the qemu testing stage.

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host \
-smp 6,sockets=1,cores=4,threads=2 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

Results in

WARNING: failed to find q35-acpi-dsdt.aml
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

What did I do wrong? I googled around and found this. That suggests I should copy the file to /usr/share/qemu/ but copy it from where?

Edit: I found the file here.

but I still get

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

how do I fix that?

Digging deeper:

$journalctl -b | grep vfio
systemd[1]: Starting Binds devices to vfio-pci...
apr 28 21:38:20 Necembel kernel: vfio-pci: probe of 0000:02:00.0 failed with error -22
apr 28 21:38:20 Necembel kernel: vfio-pci: probe of 0000:02:00.0 failed with error -22
apr 28 21:38:20 Necembel kernel: vfio-pci: probe of 0000:02:00.1 failed with error -22

"ls /sys/bus/pci/devices/0000\:02\:00.0/" also lists no "iommu_group" link.

Last edited by rabcor (2014-04-28 21:51:33)

---

Making Logitech G510 Work on Linux!