Arch Linux

ironicbadger

Member

Registered: 2013-11-12

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for this great thread. With all the information here I was able to get VGA passthrough working.

My setup is:
Debian testing host
Kernel 3.12.8 with i915 and acs override patches
QEMU-git with nosnoop patch
Windows 7 guest

Motherboard: Asus Z87-K
CPU: Intel i7-4770
GPU host: Intel HD4600
GPU guest: GeForce GTX560

I've only used this for gaming and so far every game I've tried has worked. Only problem I have with this is if I shutdown the VM I have to reboot the host to get VGA passthrough working again.

Does ejecting your GPU before reboot from the guest resolve this issue? Also I had problems with Win7 and found Win8 much more 'friendly' in this regard. I used Xen up until recently (and havn't tried this under KVM yet but I automated the process using devmanview and a tiny script I wrote. I wrote about it in my blog post http://blog.ktz.me/?p=219. (Sorry if linking isn't allowed I'll remove it, just PM me).

I have just purchased a Supermicro X10SAT board and using the information found in the thread thought I'd give KVM a try, having used Xen for months it would be nice to use something without having to install a package. Anyway, my issue is this. I have vt-d / passthrough of my RAID controller working great, except for one thing. It stops HDMI audio working as soon as I add intel_iommu=on to my grub_default bit. Here is the relevant part of my grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet ipv6.disable=1 intel_iommu=on vfio_iommu_type"

Here is my mkinitcpio.conf

MODULES="i915 kvm kvm_intel pci-stub"

If iommu isn't present in the default line audio via HDMI works great, as soon as I switch it on and redo grub and reboot (regardless of whether the host is actually running a guest of not) it doesn't play audio. So weird.

Is this likely to be a motherboard issue? I'm thinking so as I have an identical software setup on another machine (in my sig) and that works flawlessly.

---

linuxserver.io

mbirkis

Member

Registered: 2013-12-12

Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

[…]
Finally, how do i apply the patch for using the nvidia proprietary drivers on the host? i can see a patch, but i ahve no idea what file to patch or how to get my hands on it (i thought they were closed source these drivers...(

Since mbirkis also had a problem with this, here is my modified PKGBUILD for current nvidia-dkms.

Thanks alot for the PKGBUILD, i was able to build the driver using that one. One slight problem is it does not work using the linux-mainline kernel (works fine with stock arch).
Errors could have something to do with this:

% uname -r
3.13.0-2-mainline

% dmesg | grep nvidia
[    2.642180] nvidia: module license 'NVIDIA' taints kernel.
[    2.652591] nvidia: Unknown symbol acpi_os_wait_events_complete (err 0)
[    2.653150] nvidia: Unknown symbol acpi_os_wait_events_complete (err 0)
[    3.475852] nvidia: Unknown symbol acpi_os_wait_events_complete (err 0)
[    3.586826] nvidia: Unknown symbol acpi_os_wait_events_complete (err 0)
[    3.683180] nvidia: Unknown symbol acpi_os_wait_events_complete (err 0)
[    3.706485] nvidia: Unknown symbol acpi_os_wait_events_complete (err 0)
[    3.815154] nvidia: Unknown symbol acpi_os_wait_events_complete (err 0)
[    3.922719] nvidia: Unknown symbol acpi_os_wait_events_complete (err 0)
[    4.037577] nvidia: Unknown symbol acpi_os_wait_events_complete (err 0)

% systemctl status gdm.service 
gdm.service - GNOME Display Manager
   Loaded: loaded (/usr/lib/systemd/system/gdm.service; enabled)
   Active: active (running) since Mon 2014-01-20 09:33:58 CET; 1h 14min ago
 Main PID: 325 (gdm)
   CGroup: /system.slice/gdm.service
           325 /usr/bin/gdm

Jan 20 09:33:59 archxen gdm[325]: GdmDisplay: display lasted 0,107528 seconds
Jan 20 09:33:59 archxen gdm[325]: Child process 678 was already dead.
Jan 20 09:33:59 archxen gdm[325]: GLib-GObject: g_object_ref: assertion 'object->ref_count > 0' failed
Jan 20 09:33:59 archxen gdm[325]: GLib-GObject: g_object_unref: assertion 'object->ref_count > 0' failed
Jan 20 09:33:59 archxen gdm[325]: GdmDisplay: display lasted 0,106267 seconds
Jan 20 09:33:59 archxen gdm[325]: Child process 690 was already dead.
Jan 20 09:33:59 archxen gdm[325]: GLib-GObject: g_object_ref: assertion 'object->ref_count > 0' failed
Jan 20 09:33:59 archxen gdm[325]: GLib-GObject: g_object_unref: assertion 'object->ref_count > 0' failed
Jan 20 09:33:59 archxen gdm[325]: GdmDisplay: display lasted 0,114289 seconds
Jan 20 09:33:59 archxen gdm[325]: GdmLocalDisplayFactory: maximum number of X display failures reached: ch...rrors
Hint: Some lines were ellipsized, use -l to show in full.

Edit: Found similar issue here: https://devtalk.nvidia.com/default/topi … c1-kernel/ testing that patch now.

Edit2: The patch worked, i now have working nvidia driver on the host.

Last edited by mbirkis (2014-01-20 10:16:51)

andy123

Member

Registered: 2011-11-04

Posts: 169

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Oops, yeah. That makes sense. I forgot to tell you that I use this with 3.12.x not 3.13, sorry for the confusion.

---

i'm sorry for my poor english wirting skills…

alexis_evo

Member

Registered: 2013-07-04

Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How would one get 2x of the same GPU working with this? I have to pass 1002:6738 into pci-stub, but that matches *both* of my video cards.

Edit: Here's my graphics adapters in lspci -nn.

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870] [1002:6738]
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series] [1002:aa88]
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870] [1002:6738]
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series] [1002:aa88]

I am using the open source radeon driver on the host, so the issue mentioned in OP shouldn't effect me?

Edit: nevermind, looks like it isn't an issue at all. I just put the above IDs into pci-stub in bootloader, started it, ran the steps for vfio-pci and it's working fine. However, now I'm getting stuck in the windows 7 installation. the first step, "copying windows files", goes fine, but the next step (expanding windows files) has been sitting at 0% for 20 minutes now. pretty sure it's stalled and not actually working, as monitoring disk IO on the host shows there is practically no IO coming from qemu. Looking at CPU usage, qemu is using 10% of one core.

I know I should expect it to be kinda slow, because I'm using a file as raw disk image, but this seems wrong.

-device ahci,bus=pcie.0,id=ahci -drive file=/mnt/oldarch/windows.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk -drive file=/mnt/2tb/win7/Windows7Ultimate.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd

Last edited by alexis_evo (2014-01-20 16:04:46)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

However, now I'm getting stuck in the windows 7 installation. the first step, "copying windows files", goes fine, but the next step (expanding windows files) has been sitting at 0% for 20 minutes now. pretty sure it's stalled and not actually working, as monitoring disk IO on the host shows there is practically no IO coming from qemu. Looking at CPU usage, qemu is using 10% of one core.

I know I should expect it to be kinda slow, because I'm using a file as raw disk image, but this seems wrong.

QEMU AHCI is barely functional, wait it out and switch to virtio once installed (or maybe you can do it during install with a virtual driver disk)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For your testing pleasure (Intel root port ACS):

https://lkml.org/lkml/2014/1/20/453

Also, why so many reports of needing to reboot the host system to make the guest work a 2nd time?  This should be solved by the secondary bus reset support.  Could someone experiencing this problem report exactly what kernel and qemu you're using and provide lspci -vvv of the bus with the assigned video card?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Lauri

Member

Registered: 2014-01-18

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does ejecting your GPU before reboot from the guest resolve this issue? Also I had problems with Win7 and found Win8 much more 'friendly' in this regard. I used Xen up until recently (and havn't tried this under KVM yet but I automated the process using devmanview and a tiny script I wrote. I wrote about it in my blog post http://blog.ktz.me/?p=219. (Sorry if linking isn't allowed I'll remove it, just PM me).

Ejecting the GPU from the guest doesn't help. If I eject the GPU from the guest and shut down and try start it again the host freezes.

Also, why so many reports of needing to reboot the host system to make the guest work a 2nd time?  This should be solved by the secondary bus reset support.  Could someone experiencing this problem report exactly what kernel and qemu you're using and provide lspci -vvv of the bus with the assigned video card?

I'm using kernel 3.12.8 with i915 and acs override patches, latest development version of QEMU with nosnoop patch. I also tried it with kernel 3.13.0 and QEMU without nosnoop patch but trying to start the guest for a 2nd time make the host completely freeze. Can't even ssh to it or use sysrq key combinations to reboot.

01:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: ASUSTeK Computer Inc. Device 83b5
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 16
	Region 0: Memory at f4000000 (32-bit, non-prefetchable) [size=32M]
	Region 1: Memory at e0000000 (64-bit, prefetchable) [size=128M]
	Region 3: Memory at e8000000 (64-bit, prefetchable) [size=64M]
	Region 5: I/O ports at e000 [size=128]
	Expansion ROM at f6000000 [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [78] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 128 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [128 v1] Power Budgeting <?>
	Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Kernel driver in use: vfio-pci

Last edited by Lauri (2014-01-21 05:43:19)

alexis_evo

Member

Registered: 2013-07-04

Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After letting it sit for like 3 hours it finally installed. Got virtio up and working. Games run, but I get random stalling. Lowering graphics settings seems to help but I can manage 60 FPS with highest settings in some games so it shouldn't be necessary.

Trying to run Mirror's Edge, get a solid framerate, highest settings on everything but 2xAA. Runs fine for about 5 minutes, will then stall for 5-10 seconds. Sometimes it recovers and lets me play a bit further. Sometimes it corrupts the display and I have to forcibly close qemu. In one situation after a stall, I got some weird screen tearing that persisted even after exiting the game. http://i.imgur.com/fuD5HkB.jpg

ASUS M5A99FX PRO R2.0, latest BIOS
AMD FX-8350
8GB of RAM (4GB given to the VM)
HD 6870 on both host and guest. Open source radeon driver on host, latest stable catalyst driver on guest.
Guest is Win7 64bit.

qemu incantation:

qemu-system-x86_64 \
-enable-kvm -M q35 -m 4096 -cpu host -smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device ahci,bus=pcie.0,id=ahci -drive file=/mnt/oldarch/windows.img,id=disk,format=raw -device virtio-blk-pci,drive=disk \
-usb -usbdevice host:04d9:0167 -usbdevice host:0461:4d0f -boot menu=on \
-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

Flyser

Member

Registered: 2013-12-19

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Games run, but I get random stalling.

this happens for me as well, but only when the host is swapping (heavily?). I will upgrade to 16GB RAM in the next weeks.

Last edited by Flyser (2014-01-21 08:34:20)

manu.lima

Member

Registered: 2014-01-10

Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for your post. It helped me to succesfully passthrough my vga on a windows 7 x86 guest. My hardware specs are:

AMD FX8320
GIGABYTE GA970-D3 (No issues at all, IOMMU works out of the box)
8 GB RAM
ATI RADEON HD 5570 (on primary pcie slot x16) > attached to windows guest
NVIDIA 7300LE (on secondary pcie x4) > on host using propietary nvidia drivers
TWO SATA DRIVES (One for the host and the other for the guest)

On the software side I'm running stock arch kernel, stock qemu, stock nvidia-304xx drivers, and pulseaudio server (very important, you'll see later).

I managed to setup my vm using virt-manager which uses the old pc-i440fx and pci-assign to passthrough the vga. The only modifications I did, is to add the vga's id to pci-stub on kernel parameters, enable the vfio-pci systemd service (which I think it is not really necessary) and disabled nested page tables. I can restart the VM as many times as I like without having any performance issue. The windows guest evens boots after forcedly killing qemu.

I'm using the VIRTIO drivers on windows which greatly improve performance and set disk cache to "none" on virt-manager.

Now here comes the biggest trouble I had while setting up this vm. The audio was terribly bad. Even after disabling nested pages tables, trying the preempt_voluntary=y and hz=1000 options in the kernel, trying the BFQ schuelder  on linux-ck, setting qemu's priority to realtime and trying both "alsa" and "pa" audio variables.

My solution was to use the PULSEAUDIO SERVER and use this audio variables "QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa". Now the audio works flawlessly.

Thank you for reading my story and hope this helps you solve your audio problems.

PD: Sorry for my english. I'm from Argentina.

alexis_evo

Member

Registered: 2013-07-04

Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Fixed the stalling problems, at least mostly. Noticed a couple, but that may be because of slow disk IO.

Added to modprobe.d confs: options kvm-amd npt=0
Added to bootloader line: iommu=pt iommu=1

Enabled hugepages (I think this is what did it) (the number is amount of RAM given to VM divided by 2 plus a bit extra):
echo 2700 > /proc/sys/vm/nr_hugepages
add "-mem-path /dev/hugepages" to qemu parameters

However... after 20 minutes of playing, I get the same tearing issue I did before. A random amount of the side of the screen is moved to the opposite side of the screen. I had it randomly be fixed once in the past 3-4 hours of playing. A menu popped up in the game telling me what to do next and it snapped back into place. Have not been able to reproduce it, even after playing with the screen torn for an hour.

Would really, really appreciate any help in fixing this problem. The game is running pretty smooth on max settings, really impressive after so many years of barely getting games to run on lowest settings in wine, but... this issue makes games unplayable.

Flyser

Member

Registered: 2013-12-19

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Would really, really appreciate any help in fixing this problem. The game is running pretty smooth on max settings, really impressive after so many years of barely getting games to run on lowest settings in wine, but... this issue makes games unplayable.

You could try to change the screen resolution (ingame or in the operating system).

Last edited by Flyser (2014-01-21 12:54:00)

mbirkis

Member

Registered: 2013-12-12

Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Also, why so many reports of needing to reboot the host system to make the guest work a 2nd time?  This should be solved by the secondary bus reset support.  Could someone experiencing this problem report exactly what kernel and qemu you're using and provide lspci -vvv of the bus with the assigned video card?

Here is some info on my setup. I have it working but i need to reboot the host to make the guest boot a 2nd time. Let me know if i can get you more info and i will sort it.

OP kernel pkgbuild

 % uname -r
3.13.0-2-mainline

Using the qemu from OP along with seabios from OP

 % pacman -Q qemu-git
qemu-git 2.0.r30022.gf976b09-1

My vfio bind devices, the 0000:00:14.2 is onboard sound and works good.

cat /etc/vfio-pci.cfg 
DEVICES="0000:02:00.0 0000:02:00.1 0000:00:14.2"

The 01:00.0 is used on the host, with nvidia proprietary driver and is working. Pci-stub grabs the 01:00.1 because it has the same pci-stub parameter as the 02:00.1 but it is no issue for me as i dont use it for anything.

 % lspci -vv
01:00.0 VGA compatible controller: NVIDIA Corporation GF110 [GeForce GTX 570 Rev. 2] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: eVga.com. Corp. Device 1571
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 76
	Region 0: Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
	Region 1: Memory at d0000000 (64-bit, prefetchable) [size=128M]
	Region 3: Memory at d8000000 (64-bit, prefetchable) [size=32M]
	Region 5: I/O ports at e000 [size=128]
	[virtual] Expansion ROM at f7000000 [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00000  Data: 0000
	Capabilities: [78] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=01
			Status:	NegoPending- InProgress-
	Capabilities: [128 v1] Power Budgeting <?>
	Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Kernel driver in use: nvidia
	Kernel modules: nouveau, nvidia

01:00.1 Audio device: NVIDIA Corporation GF110 High Definition Audio Controller (rev a1)
	Subsystem: eVga.com. Corp. Device 1571
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 7
	Region 0: Memory at f7080000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [78] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM L0s L1 Enabled; RCB 128 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Kernel driver in use: pci-stub
	Kernel modules: snd_hda_intel

02:00.0 VGA compatible controller: NVIDIA Corporation GF110 [GeForce GTX 580] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: Micro-Star International Co., Ltd. Device 2563
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 28
	Region 0: Memory at f4000000 (32-bit, non-prefetchable) [size=16M]
	Region 1: Memory at c0000000 (64-bit, prefetchable) [size=128M]
	Region 3: Memory at c8000000 (64-bit, prefetchable) [size=32M]
	Region 5: I/O ports at d000 [size=128]
	Expansion ROM at f5000000 [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [78] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #1, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=01
			Status:	NegoPending- InProgress-
	Capabilities: [128 v1] Power Budgeting <?>
	Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau, nvidia

02:00.1 Audio device: NVIDIA Corporation GF110 High Definition Audio Controller (rev a1)
	Subsystem: Micro-Star International Co., Ltd. Device 2563
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 11
	Region 0: Memory at f5080000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [78] Express (v1) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #1, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

Edit: did lspci -vv as root to get more info.

Last edited by mbirkis (2014-01-21 14:57:28)

iamtakingiteasy

Member

Registered: 2014-01-21

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello, i am trying to passthrough a GTX 780 Nvidia card. I had my success with Radeon HD 7870 in same setup, but it is way too much overheating, making me worry that it will flood my computer with melting soldering iron :)

(problematic setup)
CPU: Core i7 3770
Motherboard: Asus P8Z77-V LX-2
Guest GPU: Asus GTX 780 in primary (upper, one that closer to CPU) PCI-E slot.
Kernel: 3.13.0-ARCH with SRC and patches from archive in first post, built with supplied x86_64 config (with few changes to support my initrd's lvm-related setup (built-in scsi and sata drivers)).
Kernel params: pci-stub.ids=10de:1004,10de:0e1a intel_iommu=on pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1
qemu: commit 1cf892ca2689c84960b4ce4d2723b6bee453711c from git://git.qemu.org/qemu.git (last available from master branch, also i've failed to see any specific patches or check-outs in qemu-git from the first post, so i am guessing it is untouched upstream)
seabios: commit 19f789bdfd58eba2ed8fe604bbabf8df0fcc0771 from git://git.seabios.org/seabios.git (last available from master branch, built with build function sourced from PKGBUILD with minor change (not to cd to source directory, as i am already standing in there and also setting srcdir=.))

Host is currently operating without own GPU (everything is done over SSH), though i have plans in using either embedded Intel GPU or HD 7870 in secondary PCI-E slot.

I am using this script to launch my Win7 guest:

#!/bin/sh

# some helper function
move_device () {
	BDF=$1; shift
	DRV=$1; shift
	BND=$1; shift
	
	# Unbind a PCI function from its driver as necessary
	[ -e /sys/bus/pci/devices/$BDF/driver/unbind ] && \
		echo -n $BDF > /sys/bus/pci/devices/$BDF/driver/unbind
	
	# Add a new slot to the PCI Backend's list
	vendor=$(sed 's/^0x//' /sys/bus/pci/devices/$BDF/vendor)
	device=$(sed 's/^0x//' /sys/bus/pci/devices/$BDF/device);
	echo -n $vendor $device > /sys/bus/pci/drivers/$DRV/new_id 2>/dev/null
	
	# Now that the backend is watching for the slot, bind to it
	[  "x$BND" = "xtrue" ] && \
		echo -n $BDF > /sys/bus/pci/drivers/$DRV/bind
}

dmesg -C 

modprobe kvm_intel
modprobe pci-stub
modprobe vfio-pci

# primary videocard -- 780 GTX
move_device "0000:01:00.0" "vfio-pci"

# hdmi audio card
move_device "0000:01:00.1" "vfio-pci"

# some crap in the same iommu group as my primary GPU
move_device "0000:00:01.0" "vfio-pci"

qemu-system-x86_64 \
	-enable-kvm -machine type=pc,accel=kvm -enable-kvm -m 8192 -M q35 \
	-cpu host -smp 4,sockets=1,cores=4,threads=1 \
	-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
	-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
	-device piix4-ide,bus=pcie.0,id=ahci \
	-device ide-hd,bus=ahci.0,drive=disk \
	-bios /usr/share/qemu/bios.bin \
	-vnc :1 \
	-drive file=/dev/virt/win7,id=disk,format=raw \
	-net nic,model=rtl8139,macaddr=52:54:00:08:d9:75 \
	-net tap,script=/home/user/bin/asset/qemu-ifup \
	-boot menu=on -vga none

But upon launch this script, screen is just going blank and nothing happens (but win7 is actually booting, when it finished, i could login via RDP and see 43 error on GPU)

I am getting the following output in my dmesg log:

[   35.886341] pci-stub: add 10DE:1004 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[   35.886353] pci-stub 0000:01:00.0: claimed by stub
[   35.886360] pci-stub: add 10DE:0E1A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[   35.907811] VFIO - User Level meta-driver version: 0.3
[   36.091437] tun: Universal TUN/TAP device driver, 1.6
[   36.091440] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[   36.127293] device tap0 entered promiscuous mode
[   36.127986] br0: port 1(tap0) entered forwarding state
[   36.127998] br0: port 1(tap0) entered forwarding state
[   39.077328] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[   40.044509] type=1006 audit(1390315308.533:5): pid=2076 uid=0 old auid=4294967295 new auid=0 old ses=4294967295 new ses=4 res=1
[   41.497683] vfio-pci 0000:01:00.0: Invalid ROM contents
[   41.497799] vfio-pci 0000:01:00.0: Invalid ROM contents

Interesting thing: when i simply replace nvidia with radeon in primary PCI-E slot, everything is launching up just fine without a single change, so i guess it is nvidia-specific problem.

Any hints why it might not work with nvidia and how to possibly fix this?

Thanks.

alexis_evo

Member

Registered: 2013-07-04

Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Would really, really appreciate any help in fixing this problem. The game is running pretty smooth on max settings, really impressive after so many years of barely getting games to run on lowest settings in wine, but... this issue makes games unplayable. :(

You could try to change the screen resolution (ingame or in the operating system).

Nope... doesn't work. Maybe relevant, though: when capturing desktop video via OBS (Open Broadcast Software), it doesn't include the tearing. Tried changing resolution, switching monitors, switching HDMI cables, nothing will fix it short of restarting the VM.

Also, the GPU fails to 'eject' whenever I try to do so to 'safely' shut down the VM. I hear the windows eject sound, and then the entire VM locks up with very corrupted video output.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lauri, mbirkis, what exactly doesn't work that requires you to reboot the host?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

alexis_evo

Member

Registered: 2013-07-04

Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

New problem since restarting after the last screen tear... now the monitor will randomly blank for 2-3 seconds. Seems like the GPU is trying to adjust something on the display, since the On Screen Menu on my monitor will not work/disappears whenever the screen blanks. It happens very often, every 5-10 seconds. After waiting 10-15 minutes, it occasionally stops doing it for 20 minutes, then starts again. Only occurs when a 3D game is at the forefront. Alt tabbing out of the game to the desktop causes it to immediately stop, and it resumes when I tab back in.

Really don't know what could be causing this...

andy123

Member

Registered: 2011-11-04

Posts: 169

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

New problem since restarting after the last screen tear... now the monitor will randomly blank for 2-3 seconds. Seems like the GPU is trying to adjust something on the display, since the On Screen Menu on my monitor will not work/disappears whenever the screen blanks. It happens very often, every 5-10 seconds. After waiting 10-15 minutes, it occasionally stops doing it for 20 minutes, then starts again. Only occurs when a 3D game is at the forefront. Alt tabbing out of the game to the desktop causes it to immediately stop, and it resumes when I tab back in.

Really don't know what could be causing this...

I had a kinda similar issue, but I don't think it was reproducible.
My primary screen (1920x1080 hdmi) "blanked" like you described it and said something about "wrong input/resolution" with Civ5 up, but worked fine when on the desktop. The second screen (1280x1024 vga) still worked on the desktop at the same time.
I think I fixed the issue by messing with the settings and restarting the game. Something like, starting with 1600x900 and setting to 1920x1080 ingame…
In general, I had far less issues in Windows 8(.1) with resolution changes, multiple monitors and so on.

---

i'm sorry for my poor english wirting skills…

mbirkis

Member

Registered: 2013-12-12

Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lauri, mbirkis, what exactly doesn't work that requires you to reboot the host?

If i shutdown the guest, and try to start it back up i cant get the display to work.

here is some relevant dmesg from the failed guest boot:

[41945.924908] vfio-pci 0000:00:14.2: enabling device (0000 -> 0002)
[41949.495172] vfio-pci 0000:02:00.0: Invalid ROM contents

Edit: I actually read the error message, and loaded the rom file as described in the OP. Guest reboot now works for me.

Last edited by mbirkis (2014-01-21 19:38:21)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Lauri, mbirkis, what exactly doesn't work that requires you to reboot the host?

If i shutdown the guest, and try to start it back up i cant get the display to work.

here is some relevant dmesg from the failed guest boot:

[41945.924908] vfio-pci 0000:00:14.2: enabling device (0000 -> 0002)
[41949.495172] vfio-pci 0000:02:00.0: Invalid ROM contents

So it's not actually a guest reboot, it's a guest shutdown and restart?  Why don't you just save off the ROM to a file and use the romfile=option to the vfio-pci device?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

mbirkis

Member

Registered: 2013-12-12

Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You are absolutely right, just as i posted the reply i thought about the romfile thing. It workes flawlesly now. Sorry about that, Guess i needed to post forum replys to get my head working.

Flyser

Member

Registered: 2013-12-19

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Host is currently operating without own GPU (everything is done over SSH), though i have plans in using either embedded Intel GPU or HD 7870 in secondary PCI-E slot.

How do you prevent the GPU from becoming the primary GPU of the host (meaning: showing bios, bootloader and linux boot output)? Because if you don't, that's likely the reason for error 43.

iamtakingiteasy

Member

Registered: 2014-01-21

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Host is currently operating without own GPU (everything is done over SSH), though i have plans in using either embedded Intel GPU or HD 7870 in secondary PCI-E slot.

How do you prevent the GPU from becoming the primary GPU of the host (meaning: showing bios, bootloader and linux boot output)? Because if you don't, that's likely the reason for error 43.

Shouldn't binding to pci-stub/vfio-pci (notice kernel argument line and beginning of the script) deal with it? Besides, Radeon card passes through fine in same setup.

Last edited by iamtakingiteasy (2014-01-21 20:16:11)

Flyser

Member

Registered: 2013-12-19

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Shouldn't binding to pci-stub/vfio-pci (notice kernel argument line and beginning of the script) deal with it? Besides, Radeon card passes through fine in same setup.

As far as I know, no. It definitely wouldn't work on my system.

Val532

Member

Registered: 2013-11-13

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For your testing pleasure (Intel root port ACS):

https://lkml.org/lkml/2014/1/20/453

Also, why so many reports of needing to reboot the host system to make the guest work a 2nd time?  This should be solved by the secondary bus reset support.  Could someone experiencing this problem report exactly what kernel and qemu you're using and provide lspci -vvv of the bus with the assigned video card?

Yeah a new acs patch !!!

But why no x58 or x79 for the moment ?