Arch Linux

thearcherblog

Member

Registered: 2014-10-30

Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How exactly are you guys using your keyboard and mouse in the guest? I saw a youtube video of someone using synergy so that seems like a good option, but other than plugging in a separate keyboard and passing it through as well I'm at a loss as to how I'm supposed to share the keyboard and mouse without them potentially getting stuck in the guest.

Don't know if will help but in my case, if I know that I will not use Linux when I'm running the Guest (Windows), I pass the USB for my Logitech keyboard and mouse and works like a charm. As soon as I turn off the Windows VM then I recover my keyboard and mouse on Linux.

In case that I need to go to my Linux host, then I have ssh and in the worst case, Teamviewer and both works fine from the Windows setup.

Sometimes I'm on Linux and my wife is connecting to the Windows machine from the iPad too watch movies / play simple games and then I'm running same script but without passing the USB for keyboard and mouse so I have full control on Linux and she is connecting with Teamviewer to the Windows computer.

Working with Photoshop I connect from the Linux computer using rdesktop that works pretty cool.

By the way, I know some people uses synergy but I tried myself and didn't get as good results as passing the USB directly (0 delay )

Hope it helps for you.

Best regards,

TheArcher

Last edited by thearcherblog (2015-01-17 17:56:17)

tully

Member

Registered: 2015-01-16

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Update: We have video! Using the 970 as the passthrough and OVMF I was able to get output to the screen. However, I'm unable to get the VM to boot from the windows iso. When trying it I see "Boot failed. EFI DVD/CDROM 1", it tries to pxe boot, then it kicks over to the UEFI Interactive shell v2.1.

Use this to assign the ISO to the VM :

-device virtio-scsi-pci,id=scsi \
-drive file=/VMs/win8.iso,id=iso_install,if=none \
-device scsi-cd,drive=iso_install \

Good luck !

I managed to boot an arch cd using that. Thanks. Just need to figure out why my windows 8 iso is refusing to boot.

Thanks Myranti and archer for your usb help. For now I'm using

"-vga none -device secondary-vga"

This just gives me a black box that will capture input until I ctrl+alt. So it's a good temp solution for now. Will explore both options to find a suitable method once I've got everything stable.

DelusionalLogic

Member

Registered: 2013-05-07

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Okay, so i got everything working (i forgot kvm=off in the cpu parameter last).

The only problem now seems to be that i can't reboot the VM. vfio complains that the bios is corrupted. I'm guessing that's because the windows VM shuts the GFX card off. What can i do about this?

To reiterate: It's a GTX 460

How exactly are you guys using your keyboard and mouse in the guest? I saw a youtube video of someone using synergy so that seems like a good option, but other than plugging in a separate keyboard and passing it through as well I'm at a loss as to how I'm supposed to share the keyboard and mouse without them potentially getting stuck in the guest.

Don't know if will help but in my case, if I know that I will not use Linux when I'm running the Guest (Windows), I pass the USB for my Logitech keyboard and mouse and works like a charm. As soon as I turn off the Windows VM then I recover my keyboard and mouse on Linux.

In case that I need to go to my Linux host, then I have ssh and in the worst case, Teamviewer and both works fine from the Windows setup.

Sometimes I'm on Linux and my wife is connecting to the Windows machine from the iPad too watch movies / play simple games and then I'm running same script but without passing the USB for keyboard and mouse so I have full control on Linux and she is connecting with Teamviewer to the Windows computer.

Working with Photoshop I connect from the Linux computer using rdesktop that works pretty cool.

By the way, I know some people uses synergy but I tried myself and didn't get as good results as passing the USB directly (0 delay )

Hope it helps for you.

Best regards,

TheArcher

I'm using synergy, and it's working well. I'm using a bridged connection with netctl, and connecting with the hostnames. No noticeable delay anywhere in the system. It's great because it allows me to have one monitor windows and the other linux (I disable one monitor on linux with xrandr in my VM startup script).

Last edited by DelusionalLogic (2015-01-17 23:54:40)

tully

Member

Registered: 2015-01-16

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Good news everyone! I'm up and running. I wasn't able to get the windows 8 iso to load. I managed to get the windows 10 preview to load and that works just fine. I tried a few windows 8 isos so no idea what that was about. Anyway, I used a gtx 970 FTW with a i7 4970k. Nvidia drivers worked without a hitch, gpu shows up just fine in dev manager, and I have no problem rebooting the machine. I ended up using OVMF as the older method wasn't cutting it for me. I also used the kernel OP provided although I'm not exactly sure how much of the patches were needed.

Here's some code

/etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="pcie_acs_override=downstream i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:13c2,10de:0fbb"

and my script to start the vm

#!/bin/bash

qemu-system-x86_64 -enable-kvm -m 8192 -cpu host,kvm=off -smp 4,cores=4,threads=1 \
-pflash /usr/share/ovmf/x64/ovmf_x64.bin \
-vga none \
-device vfio-pci,host=06:00.0,multifunction=on \
-device vfio-pci,host=06:00.1 \
-device virtio-scsi-pci,id=scsi \
-drive file=/home/tully/VMs/win10.iso,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd \
-drive file=/dev/sdb,id=disk,format=raw,if=none -device scsi-hd,drive=disk \
-drive file=/home/tully/VMs/virt.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \
-boot menu=on

I'm writing directly to a 300GB velociraptor that I had lying around. I'm not exactly sure the % hit I took from using kvm but it's negligible. Insurgency was at 180fps maxed out and league was locked at 60fps and never dipped down once. I'll try out some more demanding games in a few days when I can but for now it looks to be performing wonderfully. Synergy doesn't exactly do well with certain games, but it does well in others. I'll just need to find out the quirks and work around them. Thanks everyone for helping me get this up and running.

DelusionalLogic

Member

Registered: 2013-05-07

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Good news everyone! I'm up and running. I wasn't able to get the windows 8 iso to load. I managed to get the windows 10 preview to load and that works just fine. I tried a few windows 8 isos so no idea what that was about. Anyway, I used a gtx 970 FTW with a i7 4970k. Nvidia drivers worked without a hitch, gpu shows up just fine in dev manager, and I have no problem rebooting the machine. I ended up using OVMF as the older method wasn't cutting it for me. I also used the kernel OP provided although I'm not exactly sure how much of the patches were needed.

Here's some code

/etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="pcie_acs_override=downstream i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:13c2,10de:0fbb"

and my script to start the vm

#!/bin/bash

qemu-system-x86_64 -enable-kvm -m 8192 -cpu host,kvm=off -smp 4,cores=4,threads=1 \
-pflash /usr/share/ovmf/x64/ovmf_x64.bin \
-vga none \
-device vfio-pci,host=06:00.0,multifunction=on \
-device vfio-pci,host=06:00.1 \
-device virtio-scsi-pci,id=scsi \
-drive file=/home/tully/VMs/win10.iso,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd \
-drive file=/dev/sdb,id=disk,format=raw,if=none -device scsi-hd,drive=disk \
-drive file=/home/tully/VMs/virt.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \
-boot menu=on

I'm writing directly to a 300GB velociraptor that I had lying around. I'm not exactly sure the % hit I took from using kvm but it's negligible. Insurgency was at 180fps maxed out and league was locked at 60fps and never dipped down once. I'll try out some more demanding games in a few days when I can but for now it looks to be performing wonderfully. Synergy doesn't exactly do well with certain games, but it does well in others. I'll just need to find out the quirks and work around them. Thanks everyone for helping me get this up and running.

Firstly, How did you get past the interactive shell? Even with a parameter-list that looks functionally identical to yours i can't get it past the PXE boot.

Secondly. What i was doing (I'm trying OVMF now) was passing the usb keyboard/mouse through, but then running a synergy server ON THE VM, with a client on the host. so the vm sees them as native input devices, and the linux host sees them as synergy virtual stuff.

EDIT: Figured it out. If you want to fix it you need to get a windows8 iso that support UEFI (most iso's seem to have that stripped).

Last edited by DelusionalLogic (2015-01-18 23:06:09)

kainet

Member

Registered: 2014-12-22

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just switched from gtx 670 to gtx 770 and everything works fine. Even more stable now. With gtx 670 sound sometimes become very laggy and only way to get back to normal sound is restart. I suppose this is because i do not set vga bios explicitly for gtx 670.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just switched from gtx 670 to gtx 770 and everything works fine. Even more stable now. With gtx 670 sound sometimes become very laggy and only way to get back to normal sound is restart. I suppose this is because i do not set vga bios explicitly for gtx 670.

Technically 770 is 680 with overclocked memory, so there shouldn't be major differences between those cards.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just switched from gtx 670 to gtx 770 and everything works fine. Even more stable now. With gtx 670 sound sometimes become very laggy and only way to get back to normal sound is restart. I suppose this is because i do not set vga bios explicitly for gtx 670.

I have had same problems on 660ti, 680 and 770... (560ti works as i remember), after some time (random 1-60mins) sound would go crazy, like it's lagging (slowed down) or something... I couldn't get rid of it (changed mbo and cpu=same reslut), so i bought usb sound card.

kainet

Member

Registered: 2014-12-22

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just switched from gtx 670 to gtx 770 and everything works fine. Even more stable now. With gtx 670 sound sometimes become very laggy and only way to get back to normal sound is restart. I suppose this is because i do not set vga bios explicitly for gtx 670.

I have had same problems on 660ti, 680 and 770... (560ti works as i remember), after some time (random 1-60mins) sound would go crazy, like it's lagging (slowed down) or something... I couldn't get rid of it (changed mbo and cpu=same reslut), so i bought usb sound card.

With 670 i solved this by such a way. For me sound goes crazy before 10 min after start. If that did not happened then i have normal sound till i shut down VM. Otherwise i shutdown (restart in VM not helps) VM and start it again, usually it takes 2-5 restarts to achieve stable sound With 770 sound seems is normal but i start vm only few times.

DelusionalLogic

Member

Registered: 2013-05-07

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I wish i could suspend the graphics card when it's not in use without keeping the vm on at all times.

I've been unbinding the vfio-pci driver when i stop the vm, which does make sysfs report the device as suspended. The problem is that the graphics card still has the fan on at full power, because it's recieving juice. Can i somehow properly suspend the device?

EDIT: I can't seem to get libvirt to boot the VM correctly either. Is there any catch to running the VM with libvirt compared to qemu?

Last edited by DelusionalLogic (2015-01-19 19:05:18)

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I got updated to the lates kernel and Qemu and my Linux guest could not boot wuth vga=none:

journalctl -f:

Jan 19 21:35:37 localhost sudo[20006]:  : TTY=pts/2 ; PWD=/home/ ; USER=root ; COMMAND=./vfio-bind 0000:01:00.0 0000:01:00.1
Jan 19 21:35:37 localhost sudo[20022]:  : TTY=pts/2 ; PWD=/home/ ; USER=root ; COMMAND=/usr/bin/losetup /dev/loop1 bootL.mgr
Jan 19 21:35:37 localhost sudo[20025]:  : TTY=pts/2 ; PWD=/home/ ; USER=root ; COMMAND=/usr/bin/mdadm --build /dev/md1 --level=linear --force --raid-devices=2 /dev/loop1 /dev/sda2
Jan 19 21:35:37 localhost sudo[20028]:  : TTY=pts/2 ; PWD=/home/ ; USER=root ; COMMAND=/usr/bin/qemu-system-x86_64 -M q35 -m 6148 -enable-kvm -cpu host -boot c -vga none -display none -monitor telnet:10.10.10.92:4441,server,nowait -smp 4,sockets=1,cores=2,threads=2 -device ahci,bus=pcie.0,id=ahci -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1 -device vfio-pci,host=01:00.1,bus=pcie.0 -drive file=/dev/md1,id=dataDiskL,format=raw,if=none -device ide-hd,drive=dataDiskL -net nic,macaddr=52:54:00:12:34:56 -net bridge,br=br0
Jan 19 21:35:37 localhost kernel: device tap0 entered promiscuous mode
Jan 19 21:35:37 localhost kernel: br0: port 2(tap0) entered forwarding state
Jan 19 21:35:37 localhost kernel: br0: port 2(tap0) entered forwarding state
Jan 19 21:35:37 localhost kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
Jan 19 21:35:38 localhost ntpd[1076]: bind(31) AF_INET6 fe80::fc43:e2ff:fefd:e905%92%92#123 flags 0x11 failed: Cannot assign requested address
Jan 19 21:35:38 localhost ntpd[1076]: unable to create socket on tap0 (88) for fe80::fc43:e2ff:fefd:e905%92#123
Jan 19 21:35:38 localhost ntpd[1076]: failed to init interface for address fe80::fc43:e2ff:fefd:e905%92
Jan 19 21:35:40 localhost ntpd[1076]: new interface(s) found: waking up resolver
Jan 19 21:35:52 localhost kernel: br0: port 2(tap0) entered forwarding state

dmesg:

[2087126.096421] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[2087140.619697] br0: port 2(tap0) entered forwarding state
[2087205.484345] br0: port 2(tap0) entered disabled state
[2087205.484415] device tap0 left promiscuous mode
[2087205.484432] br0: port 2(tap0) entered disabled state
[2087307.712013] device tap0 entered promiscuous mode
[2087307.712105] br0: port 2(tap0) entered forwarding state
[2087307.712113] br0: port 2(tap0) entered forwarding state
[2087308.213505] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[2087322.757144] br0: port 2(tap0) entered forwarding state

Execution:

sudo qemu-system-x86_64 -M q35  -m 6148  -enable-kvm -cpu host -boot c -vga none -display none  -monitor telnet:10.10.10.92:4441,server,nowait \
-smp 4,sockets=1,cores=2,threads=2 \
-device ahci,bus=pcie.0,id=ahci \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1 \
-device vfio-pci,host=01:00.1,bus=pcie.0 \
-drive file=/dev/md1,id=dataDiskL,format=raw,if=none -device ide-hd,drive=dataDiskL \
-net nic,macaddr=52:54:00:12:34:56 -net bridge,br=br0

running without vga=none:

dmesg:

[2087884.573891] kvm: zapping shadow pages for mmio generation wraparound
[2087893.724076] kvm [24007]: vcpu0 unhandled rdmsr: 0x1c9
[2087893.724565] kvm [24007]: vcpu0 unhandled rdmsr: 0x1a6
[2087893.725027] kvm [24007]: vcpu0 unhandled rdmsr: 0x1a7
[2087893.725463] kvm [24007]: vcpu0 unhandled rdmsr: 0x3f6
[2087894.981414] kvm [24007]: vcpu0 unhandled rdmsr: 0x606
[2087895.836766] br0: port 2(tap0) entered forwarding state
[2087896.384502] kvm [24007]: vcpu0 unhandled rdmsr: 0x611
[2087896.384932] kvm [24007]: vcpu0 unhandled rdmsr: 0x639
[2087896.385350] kvm [24007]: vcpu0 unhandled rdmsr: 0x641
[2087896.385764] kvm [24007]: vcpu0 unhandled rdmsr: 0x619
[2087896.397773] vfio-pci 0000:01:00.1: irq 30 for MSI/MSI-X
[2087896.405814] kvm [24007]: vcpu0 unhandled rdmsr: 0x611
[2087904.772580] vfio-pci 0000:01:00.0: irq 33 for MSI/MSI-X

Last edited by myweb (2015-01-19 18:47:54)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I got updated to the lates kernel and Qemu and my Linux guest could not boot wuth vga=none:

This is a useless bug report, nothing in the provided logs indicates an error, you don't provide version of either component or details how the guest fails to boot.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I got updated to the lates kernel and Qemu and my Linux guest could not boot wuth vga=none:

This is a useless bug report, nothing in the provided logs indicates an error, you don't provide version of either component or details how the guest fails to boot.

Yes, I do not see the reason in logs, monitor is without signal and Guest is not available via network.
Could you please specify which log I should take a look?

Without vga=none - guest booted.

qemu-system-x86_64 --version
QEMU emulator version 2.2.0, Copyright (c) 2003-2008 Fabrice Bellard

uname -a

Linux localhost 3.18.0-1-mainline #1 SMP PREEMPT Wed Dec 10 19:09:20 FET 2014 x86_64 GNU/Linux

qemu 2.2.0-1
seabios 1.7.5-2

Last edited by myweb (2015-01-19 18:52:56)

Flyser

Member

Registered: 2013-12-19

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I fixed the sound problem by switching the hdmi audio of my gtx 660 to use MSI.

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I got updated to the lates kernel and Qemu and my Linux guest could not boot wuth vga=none:

This is a useless bug report, nothing in the provided logs indicates an error, you don't provide version of either component or details how the guest fails to boot.

Yes, I do not see the reason in logs, monitor is without signal and Guest is not available via network.
Could you please specify which log I should take a look?

Without vga=none - guest booted.

qemu-system-x86_64 --version
QEMU emulator version 2.2.0, Copyright (c) 2003-2008 Fabrice Bellard

uname -a

Linux localhost 3.18.0-1-mainline #1 SMP PREEMPT Wed Dec 10 19:09:20 FET 2014 x86_64 GNU/Linux

qemu 2.2.0-1
seabios 1.7.5-2

Without vga=none I am able to boot and use my Nvidia GT64 card with legacy 403 driver. But with new 346.45 I got:

nvidia-smi -q

Unable to determine the device handle for GPU 0000:01:00.0: Unknown Error

dmesg

[   11.797417] nvidia 0000:01:00.0: irq 28 for MSI/MSI-X
[   11.799246] NVRM: RmInitAdapter failed! (0x23:0x56:606)
[   11.799251] NVRM: rm_init_adapter failed for device bearing minor number 0
[   11.799381] NVRM: nvidia_frontend_open: minor 0, module->open() failed, error -5

Xorg

[    11.283] (**) NVIDIA(0): ConnectedMonitor string: "DFP-1"
[    11.285] (EE) NVIDIA(GPU-0): The NVIDIA GPU at PCI:1:0:0 is not supported by the 346.35
[    11.285] (EE) NVIDIA(GPU-0):     NVIDIA driver.
[    11.285] (EE) NVIDIA(GPU-0): Failed to initialize the NVIDIA graphics device!
[    11.285] (EE) NVIDIA(0): Failing initialization of X screen 0

kainet

Member

Registered: 2014-12-22

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I fixed the sound problem by switching the hdmi audio of my gtx 660 to use MSI.

Could you please explain this. Not fully understand you.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I got updated to the lates kernel and Qemu and my Linux guest could not boot wuth vga=none:

This is a useless bug report, nothing in the provided logs indicates an error, you don't provide version of either component or details how the guest fails to boot.

Yes, I do not see the reason in logs, monitor is without signal and Guest is not available via network.
Could you please specify which log I should take a look?

Without vga=none - guest booted.

qemu-system-x86_64 --version
QEMU emulator version 2.2.0, Copyright (c) 2003-2008 Fabrice Bellard

uname -a

Linux localhost 3.18.0-1-mainline #1 SMP PREEMPT Wed Dec 10 19:09:20 FET 2014 x86_64 GNU/Linux

qemu 2.2.0-1
seabios 1.7.5-2

Sounds like a VGA arbitration problem... if only we knew your hardware config and had some idea what the last version was that worked, someone might be able to help.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

DelusionalLogic

Member

Registered: 2013-05-07

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm trying to get libvirt to boot my vm. I have it working in pure Qemu from the commandline. I'm using OVMF with a 660ti.

For some reason when i boot through libvirt the nvidia driver fails (in the same way that it did before i started using OVMF, exception code 0x40000015). It doesn't crash, the GPU just doesn't seem to be passed through properly. I have no idea what's causing it, although i suspect it's not booting OVMF/EFI.

Here's my current config:

<domain type='kvm'>
  <name>OVMF-Gaming</name>
  <uuid>523502ec-9ff4-46de-9935-6582c468dc25</uuid>
  <memory unit='KiB'>8392704</memory>
  <currentMemory unit='KiB'>8392704</currentMemory>
  <vcpu placement='static'>6</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
    <loader type='pflash'>/var/lib/libvirt/images/ovmf_x64.bin</loader>
    <boot dev='hd'/>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-passthrough'>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sdb'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:f1:a0:4b'/>
      <source bridge='br0'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <sound model='ac97'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </sound>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
</domain>

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>

    <timer name='hypervclock' present='yes'/>

The above lines need to be removed for nvidia

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sounds like a VGA arbitration problem... if only we knew your hardware config and had some idea what the last version was that worked, someone might be able to help.

The last working kernel for me was 3.15
qemu-git-2.0.r32368.ge2da502-1-x86_64.pkg.tar.xz

My configuration:
i5-3470
AsRock Z77 pro4
Nvidia GT640 (01:00.0) - Arch Linux - vga=none does not work
ATI HD 7750 (02:00.0) - Windows 8 guest - vga=none works

Curent kernel command line:

Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=b1f615cd-86ea-4c6a-ba12-b6e0e4b2f3a4 rw intel_iommu=on pcie_acs_override=downstream pci-stub.ids=10de:0fc1,10de:0e1b,1002:683f,1002:aab0 i915.enable_hd_vgaarb=1 quiet systemd.unit=multi-user.target

Last edited by myweb (2015-01-19 20:30:41)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sounds like a VGA arbitration problem... if only we knew your hardware config and had some idea what the last version was that worked, someone might be able to help.

The last working kernel for me was 3.15
qemu-git-2.0.r32368.ge2da502-1-x86_64.pkg.tar.xz

My configuration:
i5-3470
AsRock Z77 pro4
Nvidia GT640 (01:00.0) - Arch Linux - vga=none does not work
ATI HD 7750 (02:00.0) - Windows 8 guest - vga=none works

Are you using the i5 Intel graphics for the host?  Did you apply the i915 patch to your kernel?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Are you using the i5 Intel graphics for the host?  Did you apply the i915 patch to your kernel?

Yes, I am using i5 Intel graphics for the host.
i915_317.patch patch was used in linux-mainline-3.18.0.tar.gz (3.18.0 + acs override patch + i915 vga arbiter fixes)

I added the following kernel option:i915.enable_hd_vgaarb=1 and now I see in logs:

Jan 19 23:40:56 localhost kernel: device tap1 entered promiscuous mode
Jan 19 23:40:56 localhost kernel: br0: port 3(tap1) entered forwarding state
Jan 19 23:40:56 localhost kernel: br0: port 3(tap1) entered forwarding state
Jan 19 23:40:57 localhost ntpd[1641]: bind(32) AF_INET6 fe80::fc1e:56ff:fe2b:fdb2%13%13#123 flags 0x11 failed: Cannot assign requested address
Jan 19 23:40:57 localhost ntpd[1641]: unable to create socket on tap1 (17) for fe80::fc1e:56ff:fe2b:fdb2%13#123
Jan 19 23:40:57 localhost ntpd[1641]: failed to init interface for address fe80::fc1e:56ff:fe2b:fdb2%13
Jan 19 23:40:57 localhost kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
Jan 19 23:40:59 localhost ntpd[1641]: Listen normally on 18 tap1 [fe80::fc1e:56ff:fe2b:fdb2%13]:123
Jan 19 23:40:59 localhost ntpd[1641]: new interface(s) found: waking up resolver
Jan 19 23:41:01 localhost kernel: kvm: zapping shadow pages for mmio generation wraparound
Jan 19 23:41:11 localhost kernel: br0: port 3(tap1) entered forwarding state
Jan 19 23:41:13 localhost kernel: kvm_get_msr_common: 3 callbacks suppressed
Jan 19 23:41:13 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x1c9
Jan 19 23:41:13 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x1a6
Jan 19 23:41:13 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x1a7
Jan 19 23:41:13 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x3f6
Jan 19 23:41:14 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x606
Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x611
Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x639
Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x641
Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x619
Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x611
Jan 19 23:41:16 localhost kernel: vfio-pci 0000:01:00.1: irq 33 for MSI/MSI-X

But display is still black and guest does not boot

Last edited by myweb (2015-01-19 20:45:32)

DelusionalLogic

Member

Registered: 2013-05-07

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>

    <timer name='hypervclock' present='yes'/>

The above lines need to be removed for nvidia

Fantastic! it's working perfectly now. Thank you.

Screw you asus, linux wasn't the problem, your shoddy IOMMU table was. If anyone still cares (after so long), the Asus P6T deluxe V2 board has a broken IOMMU bios. The fix is as easy as dumping the table, decompiling it then changing a single hex char and recompiling. Inserting it into the initramfs was a breeze since the relevant compile flag is already enabled on arch-mainline.

The functionality that enables this (boot time overriding of ACPI tables) is https://www.kernel.org/doc/Documentatio … erride.txt
The only problem with it is that i have to run my injection script every time i run mkinitcpio, but that's a small price to pay.

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Are you using the i5 Intel graphics for the host?  Did you apply the i915 patch to your kernel?

Yes, I am using i5 Intel graphics for the host.
i915_317.patch patch was used in linux-mainline-3.18.0.tar.gz (3.18.0 + acs override patch + i915 vga arbiter fixes)

I added the following kernel option:i915.enable_hd_vgaarb=1 and now I see in logs:

Jan 19 23:40:56 localhost kernel: device tap1 entered promiscuous mode
Jan 19 23:40:56 localhost kernel: br0: port 3(tap1) entered forwarding state
Jan 19 23:40:56 localhost kernel: br0: port 3(tap1) entered forwarding state
Jan 19 23:40:57 localhost ntpd[1641]: bind(32) AF_INET6 fe80::fc1e:56ff:fe2b:fdb2%13%13#123 flags 0x11 failed: Cannot assign requested address
Jan 19 23:40:57 localhost ntpd[1641]: unable to create socket on tap1 (17) for fe80::fc1e:56ff:fe2b:fdb2%13#123
Jan 19 23:40:57 localhost ntpd[1641]: failed to init interface for address fe80::fc1e:56ff:fe2b:fdb2%13
Jan 19 23:40:57 localhost kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
Jan 19 23:40:59 localhost ntpd[1641]: Listen normally on 18 tap1 [fe80::fc1e:56ff:fe2b:fdb2%13]:123
Jan 19 23:40:59 localhost ntpd[1641]: new interface(s) found: waking up resolver
Jan 19 23:41:01 localhost kernel: kvm: zapping shadow pages for mmio generation wraparound
Jan 19 23:41:11 localhost kernel: br0: port 3(tap1) entered forwarding state
Jan 19 23:41:13 localhost kernel: kvm_get_msr_common: 3 callbacks suppressed
Jan 19 23:41:13 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x1c9
Jan 19 23:41:13 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x1a6
Jan 19 23:41:13 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x1a7
Jan 19 23:41:13 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x3f6
Jan 19 23:41:14 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x606
Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x611
Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x639
Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x641
Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x619
Jan 19 23:41:16 localhost kernel: kvm [13684]: vcpu0 unhandled rdmsr: 0x611
Jan 19 23:41:16 localhost kernel: vfio-pci 0000:01:00.1: irq 33 for MSI/MSI-X

But display is still black and guest does not boot

I added i915 module to black list and now I see the following information in dmesg:

[   41.665779] br0: port 2(tap0) entered forwarding state
[   41.665790] br0: port 2(tap0) entered forwarding state
[   42.167464] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[   42.167597] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[   42.171227] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)
[   45.893325] kvm: zapping shadow pages for mmio generation wraparound
[   56.689435] br0: port 2(tap0) entered forwarding state
[   57.805957] kvm [1248]: vcpu0 unhandled rdmsr: 0x1c9
[   57.807847] kvm [1248]: vcpu0 unhandled rdmsr: 0x1a6
[   57.809960] kvm [1248]: vcpu0 unhandled rdmsr: 0x1a7
[   57.812242] kvm [1248]: vcpu0 unhandled rdmsr: 0x3f6
[   58.991381] kvm [1248]: vcpu0 unhandled rdmsr: 0x606
[   60.451997] kvm [1248]: vcpu0 unhandled rdmsr: 0x611
[   60.453933] kvm [1248]: vcpu0 unhandled rdmsr: 0x639
[   60.455845] kvm [1248]: vcpu0 unhandled rdmsr: 0x641
[   60.457879] kvm [1248]: vcpu0 unhandled rdmsr: 0x619
[   60.740637] vfio-pci 0000:01:00.1: irq 31 for MSI/MSI-X
[   74.566563] saa7146: interrupt_hw: warning: interrupt enabled, but not handled properly.(0xe7fcfbff)
[   74.569760] saa7146: interrupt_hw: disabling interrupt source(s)!
[  100.167145] saa7146: saa7146 (0): unexpected i2c irq: isr e7fffbff psr 00040420 ssr 00810000
[  100.167149] saa7146: interrupt_hw: warning: interrupt enabled, but not handled properly.(0xe7fcfbff)
[  100.171143] saa7146: interrupt_hw: disabling interrupt source(s)!
[  118.367566] saa7146: saa7146 (0): unexpected i2c irq: isr e7fffbff psr 00040420 ssr 00810000
[  118.367570] saa7146: interrupt_hw: warning: interrupt enabled, but not handled properly.(0xe7fcfbff)
[  118.371014] saa7146: interrupt_hw: disabling interrupt source(s)!

old nvidia driver without vga=none works fine:
nvidia-smi -q

==============NVSMI LOG==============

Timestamp                       : Tue Jan 20 00:15:22 2015
Driver Version                  : 304.125

Attached GPUs                   : 1
GPU 0000:01:00.0
    Product Name                : GeForce GT 640
    Display Mode                : N/A
    Persistence Mode            : Disabled
    Driver Model
        Current                 : N/A
        Pending                 : N/A
    Serial Number               : N/A
    GPU UUID                    : GPU-ae36e4e0-8e34-e7d2-e571-159b9bfe3020
    VBIOS Version               : 80.07.26.00.5B
    Inforom Version
        Image Version           : N/A
        OEM Object              : N/A
        ECC Object              : N/A
        Power Management Object : N/A
    GPU Operation Mode
        Current                 : N/A
        Pending                 : N/A
    PCI
        Bus                     : 0x01
        Device                  : 0x00
        Domain                  : 0x0000
        Device Id               : 0x0FC110DE
        Bus Id                  : 0000:01:00.0
        Sub System Id           : 0x840E1043
        GPU Link Info
            PCIe Generation
                Max             : N/A
                Current         : N/A
            Link Width
                Max             : N/A
                Current         : N/A
    Fan Speed                   : 37 %
    Performance State           : N/A
    Clocks Throttle Reasons     : N/A
    Memory Usage
        Total                   : 1023 MB
        Used                    : 77 MB
        Free                    : 946 MB
    Compute Mode                : Default
    Utilization
        Gpu                     : N/A
        Memory                  : N/A
    Ecc Mode
        Current                 : N/A
        Pending                 : N/A
    ECC Errors
        Volatile
            Single Bit
                Device Memory   : N/A
                Register File   : N/A
                L1 Cache        : N/A
                L2 Cache        : N/A
                Texture Memory  : N/A
                Total           : N/A
            Double Bit
                Device Memory   : N/A
                Register File   : N/A
                L1 Cache        : N/A
                L2 Cache        : N/A
                Texture Memory  : N/A
                Total           : N/A
        Aggregate
            Single Bit
                Device Memory   : N/A
                Register File   : N/A
                L1 Cache        : N/A
                L2 Cache        : N/A
                Texture Memory  : N/A
                Total           : N/A
            Double Bit
                Device Memory   : N/A
                Register File   : N/A
                L1 Cache        : N/A
                L2 Cache        : N/A
                Texture Memory  : N/A
                Total           : N/A
    Temperature
        Gpu                     : 44 C
    Power Readings
        Power Management        : N/A
        Power Draw              : N/A
        Power Limit             : N/A
        Default Power Limit     : N/A
        Min Power Limit         : N/A
        Max Power Limit         : N/A
    Clocks
        Graphics                : N/A
        SM                      : N/A
        Memory                  : N/A
    Applications Clocks
        Graphics                : N/A
        Memory                  : N/A
    Max Clocks
        Graphics                : N/A
        SM                      : N/A
        Memory                  : N/A
    Compute Processes           : N/A

Last edited by myweb (2015-01-19 21:20:00)

DelusionalLogic

Member

Registered: 2013-05-07

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Am i right in assuming i can't enable Spice at all on nividia passthrough? I'd love to hotplug usb devices to the VM.