Arch Linux

ahl

Member

Registered: 2014-03-16

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nvidia xorg module complained that "The NVIDIA device on PCI:1:0:0 is not supported by this driver"

I have no experience with NVIDIA cards but if I understand correctly, it is working exactly as intended by NVIDIA. They desperately want you to buy a NVIDIA Quadro card. You might want the read the last few pages of this thread. Keywords: "qemu-git" and "kvm=off".

...

On the other hand, I'm searching for a new AMD card capable of working perfectly. Any problem that might require rebooting the host is not acceptable.

Any chance that either of these two would meet above criteria?
Asus Radeon R7 260 1GiB (90YV05I0-M0NA00) (UPDATE: NOT GOOD. See below.)
MSI Radeon R9 270 2GiB (R9-270-GAMING-2G)
(Note: Not the "X" versions)

I recently bought a Sapphire HD 5450 512MiB (11166-01-20R), but very surprisingly, it turned out to be trash! It indeed works, but If qemu is terminated without properly shutdowning the guest, the card stops working until next reboot.

So now I'm afraid of buying anything...

For the record, my old Sapphire HD 6670 1GiB (11192-06-20G) works absolutely perfectly. I'm still at 0 host AND 0 guest* crashes after months of use and any game I throw at it Just Works(TM).

(* Actually 2 guest crashes, but it was unrelated.)

Last edited by ahl (2014-07-09 21:29:55)

dwe11er

Member

Registered: 2014-03-18

Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I recently bought a Sapphire HD 5450 512MiB (11166-01-20R), but very surprisingly, it turned out to be trash! It indeed works, but If qemu is terminated without properly shutdowning the guest, the card stops working until next reboot.

I had similar problem with NVIDIA GTS450. Now I've got HD7870 (Pitcarin; same core as R9 270) and it's working flawlessly; unclean shutdowns aren't problems.

heavymetal

Member

Registered: 2014-07-06

Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I've been trying this with my ASUS P8Z77-V LK and Intel Core i5-3550. Both allegedly support VT-d. I have both the IGP bundled with the processor and a Gigabyte NVIDIA GeForce GT 630.

I've installed the kernel proposed in #1 and set the following options in grub.cfg:

intel_iommu=on pci-stub.ids=10de:0f00,10de:0bea i915.enable_hd_vgaarb=1 pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1

I've noticed lots of DMA-related problems when booting this kernel with IOMMU on. The first is this kerneloops shown in dmesg, which was already reported here:

[    0.375724] IOMMU: Setting RMRR:
[    0.375732] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]
[    0.376833] IOMMU: Setting identity map for device 0000:00:14.0 [0xbd08e000 - 0xbd09ffff]
[    0.376849] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbd08e000 - 0xbd09ffff]
[    0.376861] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbd08e000 - 0xbd09ffff]
[    0.376870] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.376875] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.377267] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
[    0.377340] ------------[ cut here ]------------
[    0.377344] WARNING: CPU: 0 PID: 1 at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x65/0x90()
[    0.377345] Modules linked in:
[    0.377348] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.15.1-1-mainline #1
[    0.377349] Hardware name: System manufacturer System Product Name/P8Z77-V LK, BIOS 0908 11/16/2012
[    0.377350]  0000000000000000 0000000081968345 ffff880224453d10 ffffffff81507d91
[    0.377351]  0000000000000000 ffff880224453d48 ffffffff81069acd ffff880224217098
[    0.377353]  ffff880224217000 ffff880224217098 0000000000000000 0000000000000000
[    0.377354] Call Trace:
[    0.377358]  [<ffffffff81507d91>] dump_stack+0x4d/0x6f
[    0.377361]  [<ffffffff81069acd>] warn_slowpath_common+0x7d/0xa0
[    0.377362]  [<ffffffff81069bfa>] warn_slowpath_null+0x1a/0x20
[    0.377364]  [<ffffffff812cfde5>] pci_find_upstream_pcie_bridge+0x65/0x90
[    0.377367]  [<ffffffff813ea07d>] intel_iommu_add_device+0x4d/0x240
[    0.377369]  [<ffffffff813df370>] ? bus_set_iommu+0x50/0x50
[    0.377370]  [<ffffffff813df39a>] add_iommu_group+0x2a/0x60
[    0.377373]  [<ffffffff8138f633>] bus_for_each_dev+0x73/0xc0
[    0.377374]  [<ffffffff813df368>] bus_set_iommu+0x48/0x50
[    0.377377]  [<ffffffff8193d94b>] intel_iommu_init+0x1bf/0x5d2
[    0.377379]  [<ffffffff81900297>] ? memblock_find_dma_reserve+0x143/0x143
[    0.377380]  [<ffffffff819002a9>] pci_iommu_init+0x12/0x3c
[    0.377383]  [<ffffffff81002162>] do_one_initcall+0xf2/0x1b0
[    0.377385]  [<ffffffff8108b47d>] ? parse_args+0x21d/0x490
[    0.377387]  [<ffffffff818f6067>] kernel_init_freeable+0x1a0/0x23c
[    0.377390]  [<ffffffff814fccd0>] ? rest_init+0x90/0x90
[    0.377391]  [<ffffffff814fccde>] kernel_init+0xe/0xf0
[    0.377394]  [<ffffffff815159fc>] ret_from_fork+0x7c/0xb0
[    0.377396]  [<ffffffff814fccd0>] ? rest_init+0x90/0x90
[    0.377399] ---[ end trace 9e3a2ac08a3dd373 ]---
[    0.378939] RAPL PMU detected, hw unit 2^-16 Joules, API unit is 2^-32 Joules, 3 fixed counters 163840 ms ovfl timer

Once my system has booted, I have no wifi. My wireless card is a PCI one, sat on 05:01.0 according to lspci. I get these in dmesg every time:

[ 1855.821855] dmar: DMAR:[DMA Write] Request device [05:00.0] fault addr 0 
DMAR:[fault reason 02] Present bit in context entry is clear
[ 1855.822005] dmar: DRHD: handling fault status reg 3
[ 1855.822012] dmar: DMAR:[DMA Write] Request device [05:00.0] fault addr 0 
DMAR:[fault reason 02] Present bit in context entry is clear
[ 1855.854514] dmar: DRHD: handling fault status reg 3
[ 1855.854521] dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ffffa000 
DMAR:[fault reason 02] Present bit in context entry is clear
[ 1855.886652] ath: phy0: Failed to stop TX DMA, queues=0x001!
[ 1855.897505] dmar: DRHD: handling fault status reg 3
[ 1855.897511] dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr fffd0000 
DMAR:[fault reason 02] Present bit in context entry is clear

The wireless card is the only PCI (not PCI-E) card in my system.

Not being able to reach the Internet through my PCI card, I use a USB one. It works for few minutes, but after a while it stops working, again with DMA problems reported by dmesg.

[  529.166591] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
[  529.306393] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
[  529.725885] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
[  529.865689] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
[  529.879235] ieee80211 phy1: rt2x00usb_watchdog_tx_dma: Warning - TX queue 2 DMA timed out, invoke forced forced reset
[  579.875625] ieee80211 phy1: rt2x00usb_vendor_request: Error - Vendor Request 0x07 failed for offset 0x3040 with error -110

Disconnecting the wireless PCI card does not help.

I don't know if all these DMA problems are hardware or software fault.

Even with all of these, I'm able to boot seabios through the nVidia card with QEMU. I use the following commands:

sudo vfio-bind 0000:01:00.0 0000:01:00.1
sudo chgrp $USER /dev/vfio/13
sudo chmod g+rw /dev/vfio/13
qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 4,sockets=1,cores=4,threads=1 -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

And it works...! The first time. If I press ^C, the Seabios screen persists on my monitor, and running the above command again fails with "Invalid ROM". If I want to run QEMU again I have to reboot the host system.

Adding rombar=0 only yields a blank screen and an emulation error after a while.

I tried searching for my VGA BIOS file at http://www.techpowerup.com/vgabios/, finding 4 possible ROMs for my video card. Three of them made the screen spark characters here and there. None of them made the card work for a second time without rebooting, and usually made my computer hang indefinitely on the second run, needing to hard reset it.

What could be the cause of these problems, specially the DMA ones? Maybe a bug in the BIOS like commented in #4?

ahl

Member

Registered: 2014-03-16

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I had similar problem with NVIDIA GTS450. Now I've got HD7870 (Pitcarin; same core as R9 270) and it's working flawlessly; unclean shutdowns aren't problems.

Thanks for the info. I hope you are right because the above mentioned R7 260 turned out to be yet another paperweight.

A quick search for PAGE_FAULT_IN_NONPAGED_AREA in this thread revealed that people have already reported this new problem of mine numerous times.

Are all of the new Sea Islands cards bad? (BONAIRE, KABINI, KAVERI, HAWAII, HD7790, R7 260(X), R9 290(X))

This is my new problem:

CPU: AMD
Guest GPU: Radeon R7 260
Qemu: git or 2.0.0
Linux: 3.15.4, 3.15.3 or 3.14.10 (all with vga_arbitrer.patch)

Steps to reproduce problem, works 100%:
1. Enter high resolution graphic mode in Win 7 (it works initially)
2. Shutdown guest and start qemu again (OR just soft-reboot the guest)
3. Upon entering high resolution mode again, these errors appear:

Guest:
PAGE_FAULT_IN_NONPAGED_AREA  STOP: 0x50

Host:
Mostly unresponsive, but still working. Qemu won't die even with kill -9. Unlimited number of these flooding very fast:

AMD-Vi: Completion-Wait loop timed out
------------[ cut here ]------------
WARNING: CPU: 6 PID: 3841 at drivers/iommu/amd_iommu.c:1248 __domain_flush_pages+0x1b2/0x1c0()
Modules linked in: radeon cfbfillrect cfbimgblt cfbcopyarea i2c_algo_bit fbcon bitblit softcursor font backlight drm_kms_helper ttm drm fb fbdev pci_stub bridge stp llc loop jc42 it87 hwmon_vid xt_physdev ebt_log ebtable_filter ebtables snd_usb_audio snd_usbmidi_lib snd_hwdep snd_pcm snd_rawmidi snd_seq_device snd_timer snd uas kvm_amd joydev soundcore evdev kvm wmi k10temp i2c_piix4 fam15h_power crct10dif_pclmul xhci_hcd
CPU: 6 PID: 3841 Comm: qemu-system-x86 Not tainted 3.15.3-arbiter #2
Hardware name: To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0, BIOS 2501 04/08/2014
0000000000000009 ffff8802f664bba8 ffffffff815b03fb 0000000000000007
0000000000000000 ffff8802f664bbe8 ffffffff810c0322 000000000000136b
ffff880310605810 00000000fffffffb 8000000000000ffe 0000000000000000
Call Trace:
[<ffffffff815b03fb>] dump_stack+0x46/0x58
[<ffffffff810c0322>] warn_slowpath_common+0x82/0xb0
[<ffffffff810c0365>] warn_slowpath_null+0x15/0x20
[<ffffffff81491e92>] __domain_flush_pages+0x1b2/0x1c0
[<ffffffff81491eba>] domain_flush_tlb_pde+0x1a/0x20
[<ffffffff81492036>] amd_iommu_unmap+0x176/0x190
[<ffffffff81491ec0>] ? domain_flush_tlb_pde+0x20/0x20
[<ffffffff8148f650>] iommu_unmap+0x90/0xe0
[<ffffffff813efdbb>] vfio_remove_dma+0xbb/0x1a0
[<ffffffff813f0526>] vfio_iommu_type1_ioctl+0x3f6/0xa70
[<ffffffffa0064bcc>] ? kvm_set_memory_region+0x3c/0x50 [kvm]
[<ffffffffa0064ef2>] ? kvm_vm_ioctl+0x312/0x770 [kvm]
[<ffffffff813ee1af>] vfio_fops_unl_ioctl+0x7f/0x2b0
[<ffffffff810cee24>] ? do_sigtimedwait+0xb4/0x1e0
[<ffffffff81192a8e>] do_vfs_ioctl+0x7e/0x500
[<ffffffff8111a463>] ? SyS_futex+0x93/0x1a0
[<ffffffff81192f57>] SyS_ioctl+0x47/0x90
[<ffffffff815b6b62>] system_call_fastpath+0x16/0x1b
---[ end trace 4f4b82e9c04d544c ]---

Also these:

AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=04:00.0 address=0x00000004485334f0]

It seems there is no problem when running Win 7 in low resolution mode without amd drivers installed.

Last edited by ahl (2014-07-09 21:32:22)

pascal257

Member

Registered: 2014-01-02

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

VGA-Passthrough running perfectly here. Having ArchLinux host with several headless guests, a windows 7 guest and a linux mint guest. Ejecting the graphics gard on windows 7 works flawlessly and I can reboot the guest without having issues. But sometimes nouveau on the linux guest crashes and while I can still access the system via ssh, I'm unable to eject the gfx, so I have to reboot the whole system.

Is there any way to eject the graphics card in linux like with the windows guest?

btw this is the crash on the linux guest, maybe somehow knows how to fix this:

[15398.752221] nouveau E[   PFIFO][0000:01:00.0] read fault at 0x0000039000 [PAGE_NOT_PRESENT] from PCOPY0/PCOPY0 on channel 0x001fdb2000 [DRM]
[15475.985061] nouveau E[     DRM] GPU lockup - switching to software fbcon
[15491.028017] nouveau E[Xorg[1418]] failed to idle channel 0xcccc0001 [Xorg[1418]]
[15506.028007] nouveau E[Xorg[1418]] failed to idle channel 0xcccc0001 [Xorg[1418]]
[15508.028221] nouveau E[   PFIFO][0000:01:00.0] playlist update failed
[15523.028018] nouveau E[Xorg[1418]] failed to idle channel 0xcccc0000 [Xorg[1418]]
[15538.028019] nouveau E[Xorg[1418]] failed to idle channel 0xcccc0000 [Xorg[1418]]
[15540.028130] nouveau E[   PFIFO][0000:01:00.0] channel 2 [Xorg[1418]] kick timeout
[15540.028448] nouveau W[   PFIFO][0000:01:00.0] INTR 0x00000100: 0x0000000d
[15542.028332] nouveau E[   PFIFO][0000:01:00.0] playlist update failed
[15557.104010] nouveau E[chrome[3057]] failed to idle channel 0xcccc0000 [chrome[3057]]
[15572.104007] nouveau E[chrome[3057]] failed to idle channel 0xcccc0000 [chrome[3057]]
[15574.104188] nouveau E[   PFIFO][0000:01:00.0] channel 5 [chrome[3057]] kick timeout
[15574.104445] nouveau W[   PFIFO][0000:01:00.0] INTR 0x00000100: 0x0000000d
[15576.104311] nouveau E[   PFIFO][0000:01:00.0] playlist update failed
[15591.104009] nouveau E[cinnamon[2055]] failed to idle channel 0xcccc0000 [cinnamon[2055]]
[15606.104010] nouveau E[cinnamon[2055]] failed to idle channel 0xcccc0000 [cinnamon[2055]]
[15608.104091] nouveau E[   PFIFO][0000:01:00.0] channel 4 [cinnamon[2055]] kick timeout
[15608.104343] nouveau W[   PFIFO][0000:01:00.0] INTR 0x00000100: 0x0000000d
[15610.104224] nouveau E[   PFIFO][0000:01:00.0] playlist update failed
[15610.295781] nouveau W[   PFIFO][0000:01:00.0] INTR 0x00000100: 0x0000000d
[15612.295660] nouveau E[   PFIFO][0000:01:00.0] playlist update failed
[15612.310667] nouveau W[   PFIFO][0000:01:00.0] INTR 0x00000100: 0x0000000d
[15614.310542] nouveau E[   PFIFO][0000:01:00.0] playlist update failed

switchphase

Member

Registered: 2014-07-07

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello Guys,
I couldl need some help. My setup worked fine but i had to rip it appart because i needed my HD 6970 for a Triple-Head Setup.
At the time i had the HD6970 and a Nvidia GT 630.
Blacklisted the radeon module and pass it through to the VM. All worked fine.

Now i had a HD6970 and a R9 270x

When i try to start the VM i get

virsh # start GamingVM
error: Failed to start domain GamingVM
error: internal error: early end of file from monitor: possible problem:
2014-07-07T18:55:28.773551Z qemu-kvm: -device vfio-pci,host=06:00.0,bus=root.1,addr=00.1,multifunction=on,x-vga=on: vfio: No available IOMMU models
2014-07-07T18:55:28.773590Z qemu-kvm: -device vfio-pci,host=06:00.0,bus=root.1,addr=00.1,multifunction=on,x-vga=on: vfio: failed to setup container for group 23
2014-07-07T18:55:28.773607Z qemu-kvm: -device vfio-pci,host=06:00.0,bus=root.1,addr=00.1,multifunction=on,x-vga=on: vfio: failed to get group 23
2014-07-07T18:55:28.773627Z qemu-kvm: -device vfio-pci,host=06:00.0,bus=root.1,addr=00.1,multifunction=on,x-vga=on: Device initialization failed.
2014-07-07T18:55:28.773647Z qemu-kvm: -device vfio-pci,host=06:00.0,bus=root.1,addr=00.1,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

IOMMU is enabled in UEFI, Device is bound to pci-stub.

#dmesg | grep pci-stub
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=10714a4e-bd0b-4f6a-8aa4-e349e7cd4c13 rw pci-stub.ids=1002:6810,1002:aab0 iommu=pt iommu=1 vfio_iommu_type quiet
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=10714a4e-bd0b-4f6a-8aa4-e349e7cd4c13 rw pci-stub.ids=1002:6810,1002:aab0 iommu=pt iommu=1 vfio_iommu_type quiet
[    0.971279] pci-stub: add 1002:6810 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.971291] pci-stub 0000:06:00.0: claimed by stub
[    0.971296] pci-stub: add 1002:AAB0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.971303] pci-stub 0000:06:00.1: claimed by stub

#dmesg | grep AMD-Vi  
[    0.843061] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    0.843062] AMD-Vi: Interrupt remapping enabled
[    0.843201] AMD-Vi: Initialized for Passthrough Mode

Also qemu is set to run as root and allowed to access to /dev/vfio/{vfio,23} . So what could be the Problem here?

# grep  "^[^#]" /etc/libvirt/qemu.conf 
user = "root"
group="root"
cgroup_device_acl = [
    "/dev/null", "/dev/full", "/dev/zero",
    "/dev/random", "/dev/urandom",
    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
    "/dev/rtc","/dev/hpet", "/dev/vfio/vfio",
    "/dev/vfio/23"
]
clear_emulator_capabilities = 0

Could this be a Problem? As driver is listed the pci-stub but also the radeon module

06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X] (prog-if 00 [VGA controller])
        Subsystem: ASUSTeK Computer Inc. Device 04a3
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 10
        Region 0: Memory at c0000000 (64-bit, prefetchable) [disabled] [size=256M]
        Region 2: Memory at fe500000 (64-bit, non-prefetchable) [disabled] [size=256K]
        Region 4: I/O ports at b000 [disabled] [size=256]
        Expansion ROM at fe540000 [disabled] [size=128K]
        Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Capabilities: [200 v1] #15
        Capabilities: [270 v1] #19
        Capabilities: [2b0 v1] Address Translation Service (ATS)
                ATSCap: Invalidate Queue Depth: 00
                ATSCtl: Enable+, Smallest Translation Unit: 00
        Capabilities: [2c0 v1] #13
        Capabilities: [2d0 v1] #1b
        Kernel driver in use: vfio-pci
        Kernel modules: radeon


06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
        Subsystem: ASUSTeK Computer Inc. Device aab0
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin B routed to IRQ 11
        Region 0: Memory at fe560000 (64-bit, non-prefetchable) [disabled] [size=16K]
        Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel



06:00.0 0300: 1002:6810
06:00.1 0403: 1002:aab0

ahl

Member

Registered: 2014-03-16

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Could this be a Problem? As driver is listed the pci-stub but also the radeon module

No, that is normal. It's the "in use" that matters.

Perhaps, try this:
https://bbs.archlinux.org/viewtopic.php … 4#p1366134

switchphase

Member

Registered: 2014-07-07

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Perhaps, try this:
https://bbs.archlinux.org/viewtopic.php … 4#p1366134

Thanks, thats it!
The vfio_iommu_type1 module was missing

ahl

Member

Registered: 2014-03-16

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now i had a HD6970 and a R9 270x

Can I ask you, how is your R9 270x working now? Does it work fine even after unclean shutdown of the guest?

My R7 260 is only usable with the "eject" method. Unclean shutdown is fatal. I now put the card on host, but it doesn't work that well on host either (with free drivers: glamor, mesa, latest kernel).

Even worse, when I did this on host:
echo high > /sys/class/drm/card0/device/power_profile

The screen went all funny and locked up the gpu...
Perhaps there is something wrong with the power tables?

UPDATE: power_method "profile" is broken, but the newer "dpm" method works assuming latest radeon firmware is installed.

Last edited by ahl (2014-07-09 21:36:34)

switchphase

Member

Registered: 2014-07-07

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can I ask you, how is your R9 270x working now?

I can't say much about it ATM.
With the 440 Chipset it works like a charm, but the performance is bad. (About 20-30FPS)
With the q35 Chipset the host crashes in 3-4 sec.

It maybe a initialization error. I try to dump the BIOS from the card and see if it works.

dwe11er

Member

Registered: 2014-03-18

Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can I ask you, how is your R9 270x working now?

I can't say much about it ATM.
With the 440 Chipset it works like a charm, but the performance is bad. (About 20-30FPS)
With the q35 Chipset the host crashes in 3-4 sec.

It maybe a initialization error. I try to dump the BIOS from the card and see if it works.

Maybe its AMD's IOMMU problem? Both of you have AMD cpu and mobo. I've got Intel's and never encountered such problems.

switchphase

Member

Registered: 2014-07-07

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Maybe its AMD's IOMMU problem?

Dont think so. Isn't it the same expect the name?
Intel calls it VT-d and AMD calls it IOMMU, but the technology should be the same.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Maybe its AMD's IOMMU problem?

Dont think so. Isn't it the same expect the name?
Intel calls it VT-d and AMD calls it IOMMU, but the technology should be the same.

No, they are completely different IOMMU implementations

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

abdullah

Member

Registered: 2014-06-10

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does the VM's share the CPU cores ? or each VM get it owns core assigned ?

switchphase

Member

Registered: 2014-07-07

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does the VM's share the CPU cores ? or each VM get it owns core assigned ?

I think they are shared amongst the VM's as long as no VCPU pinning is configured

Saved the VBIOS to a file and tried it again. Nothing changed. After a few seconds my screen looks like this
http://imgur.com/G7yT4uZ
After a few seconds the host freezes completly. Output on the passed through card is seen.

I used the modified parameters for tetsting from the first post without disk

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/tma/VBIOS/r9_270x_vbios.rom \
-device vfio-pci,host=06:00.1,bus=root.1,addr=00.1 \
-nographic

Is this VGA Arbiter related?
journalctl tells nothing related to all this.
Is patching Kernel / qemu / seabios anymore needed? (As i heard all related patches are already upstream, but not sure about it)

I've got Intel's and never encountered such problems.

All i can find is that a mainboard with the same chipset as mine are reported to work (990FX)
Buying new Hardware is not an option atm. I spent already to much money on that stuff.

No, they are completely different IOMMU implementations

Didn't know that. Makes pinning down errors not easier with multiple implementations.
Anyone got it working with an ASRock extreme9?

Last edited by switchphase (2014-07-08 20:16:39)

ahl

Member

Registered: 2014-03-16

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can I ask you, how is your R9 270x working now?

I can't say much about it ATM.
With the 440 Chipset it works like a charm, but the performance is bad. (About 20-30FPS)
With the q35 Chipset the host crashes in 3-4 sec.

Ouch.
I'll keep my R7 260 then and reserve it just for the Windows guest. It works mighty fine(UPDATE: see below) there except for that stupid reboot problem. Performance is good.

I must also mention that it runs VERY cool thanks to the properly sized heatsink. It is truly a rarity. At least Asus did one thing right.

Last edited by ahl (2014-07-10 09:13:16)

switchphase

Member

Registered: 2014-07-07

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I must also mention that it runs VERY cool thanks to the properly sized heatsink. It is truly a rarity. At least Asus did one thing right.

True

Had a 9800GTX once.
This thing was unusable. With factory settings it runs hot and powered down the pc in less than 30min.
With custom colling settings (always on 100%) and underclocking I was able to play about 1-2h without running hot.

So no more Nvidia for me

Last edited by switchphase (2014-07-08 20:33:14)

MCMjolnir

Member

Registered: 2014-06-19

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Woohoo! I finally got it working with qemu.

I could do with a couple of tips though, since my setup has its own quirks:
I'm running it as a headless steam-streaming server, which means I won't ordinarily have a KB+M or monitor attached and won't generally interact with the server physically.

I'm launching qemu through ssh with the -nographic argument (or it won't work), and when I do so the ssh tab becomes unresponsive (ctrl+c,x,v don't do anything). I can see the VM boot up fine, but I don't currently have a way of interacting with it. I can't even figure out how to close the VM once it's launched.

Can anyone explain how I set up VNC with the qemu instance, as well as manage qemu guests from the command line? (google is surprisingly unhelpful) Also, can anyone explain how to use a network bridge? Annoyingly, I had both of these working with virt-manager

Here's my qemu profile:

qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host,kvm=off \
-smp 8,sockets=1,cores=4,threads=8 \
-bios /usr/share/qemu/bios.bin -vga none -nographic \
-device piix4-ide,bus=pcie.0,id=piix4-ide \
-drive file=/var/lib/libvirt/images/Win8Steam-clone.qcow2,id=disk,format=qcow2 \
-device ide-hd,bus=piix4-ide.0,drive=disk \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=03:00.1,bus=root.1,addr=00.1

I'm really looking forward to getting everything together and am hoping to do a full write up once it's done

Last edited by MCMjolnir (2014-07-08 21:31:12)

switchphase

Member

Registered: 2014-07-07

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can anyone explain how I set up VNC with the qemu instance, as well as manage qemu guests from the command line?

When your vm is managed via libvirt you can use virsh. Its pretty the same like virt-manager on the shell. The machine/network settings and so on are done via xml files.
If pretty much the same but without VNC/Spice. I did a DHCP reservation and Autologon + Steam Autostart. Firewall rules are set to allow RDP Sessions.

But I'm far away from testing the RDP / Steam In-Home Streaming.

Gratz to your success.

MCMjolnir

Member

Registered: 2014-06-19

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I was using libvirt/virt-manager before, however to use the kvm=off flag to make my Nvidia card play nice, I *have* to use qemu directly

Running list in virsh while the qemu instance is running doesn't come up with anything, although I could be doing it wrong.

ahl

Member

Registered: 2014-03-16

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Maybe its AMD's IOMMU problem? Both of you have AMD cpu and mobo. I've got Intel's and never encountered such problems.

How likely is that?...

I will attempt to test all of my newer cards on Intel platform next week just to see if they behave differently. I have the following hardware:

4 computers: 2x AMD FX, Core i7, AMD A10 APU (HD 7660D)
4-6 gpus: HD3870, HD5450, (HD6450), HD6670, R7 260, (R9 270)

I don't know what to expect.

abdullah

Member

Registered: 2014-06-10

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I got it!
Now, please add AMD HD 4XXX series to the first post.

The guest is running Win8, I had a problem when installing the drivers (Beta ones), and I didn't test the stable driver, Yet, the system works, so.. I think I'm leaving it for now.

I will be editing the google Doc soon.

erganzi

Member

Registered: 2014-07-09

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you for the guide! It finally work after I tried many times, changed many kernel and qemu versions.

So I want to share my experience.

Hardware Info：
Gigabyte Z97-D3H
i5-4570
i915 built-in card for host
Sapphire hd7850 for vm

Software Info:
linux-3.12.4 from kernel.org with following patchs.
pci/iommu: Quirk non-compliant PCIe-to-PCI bridges
i915_fixes[1]
i915_fixes[2]
Enable overrides for missing ACS capabilities

qemu-2.0.0 with patch
vfio-pci: Disallow device from using NoSnoop transactions

Kernel commandline:

linux /vmlinuz-3.12.4-VFIOPassthrough root=UUID=6b2e5409-8b73-40db-ad45-f4ec79cdc0cc quiet
intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream
pci-stub.ids=1002:6819,1002:aab0

/etc/modprobe.d/kvm.conf

options kvm ignore_msrs=1
options kvm allow_unsafe_assigned_interrupts=1
options kvm_intel emulate_invalid_guest_state=0

Qemu commandline:

01:00.0 VGA compatible controller: ATI Technologies Inc Device 6819
01:00.1 Audio device: ATI Technologies Inc Device aab0

/root/vfio-bind 0000:01:00.0 0000:01:00.1

qemu-system-x86_64 -cpu SandyBridge -smp 2,sockets=2,cores=1,threads=1 \
-m 4096 -M q35 -enable-kvm \
-device piix4-ide,bus=pcie.0,id=piix4-ide \
-drive file=/var/lib/libvirt/images/fwq_win7_vfio.img,id=disk,format=raw \
-device ide-hd,bus=piix4-ide.0,drive=disk \
-drive file=/var/lib/libvirt/images/LENOVO_WIN7_64.iso,id=isocd \
-device ide-cd,bus=piix4-ide.1,drive=isocd \
-net nic,model=rtl8139,macaddr=52:54:00:1a:2b:3c \
-net tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
-vga none -display none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=pcie.0 \     /* the audio bus must be pcie.0 or you will be failed. */
-usb -usbdevice host:17ef:6018 -usbdevice host:17ef:600e \
-boot order=cd -monitor stdio

PS: the graphic driver for hd7850 that I use [amd_13_9_win7_win8_64_dd_ccc_whql.exe]

Last edited by erganzi (2014-07-09 02:24:45)

ahl

Member

Registered: 2014-03-16

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The story of Radeon R7 260 "The Problem" continues

Just when I had accepted the inconveniences of the eject workaround, I found out that the card can also freeze the whole system on Windows startup.

The freezes started happening when I increased -smp cores to 6. Annoyed by that I did about a hundred manual reboots in order to find out which settings work and which don't. Unfortunately, the freeze does not happen every time so the results came out inconclusive -- so don't take this too seriously:

FREEZE OCCURS WITH:
-smp sockets=1,cores=6,threads=1

FREEZE DOES NOT OCCUR WITH:
cores=2 OR cores=4 OR cores=8 OR without -smp.

(-cpu value is irrelevant. Tested to crash with: qemu64, host, Opteron_G4, and Nehalem)

It could all be just a coincidence. Perhaps the settings aren't relevant at all?

Since the freeze turned out to be too hard to reproduce, next I did everything in order to get rid of the whole reboot problem (described here). This included: various motherboard bioses, various qemu versions, changing all bios settings, swapping cards, headless host with just one gpu, removing all non-critical devices... nothing helped.

In one instance I did not get the usual BSoD but just a black frozen screen. It happened when I did "-cpu Opteron_G4,kvm=off". Quite a coincidence, huh? But I do believe it was a simple coincidence.

I currently use:
Linux 3.15.4 with vga_arbitrer.patch and qemu git (or 2.0.0).

Host system:
AMD FX 8350, Asus Sabertooth 990FX R2.0 BIOS 2501.

THE GOOD:
The card seems to work fine on Linux host with free drivers. While the older power_method=profile is still broken, the newer power_method=dpm works fine assuming kernel can find the latest radeon firmware blobs.

Summary:
R7 260 on host: Should be okay
R7 260 on guest: Unstable. Only for testing

UPDATE:
After an upgrade to 3.15.5 (which contains kvm and drm fixes) the host freezing problem triggered several times in a row. I was able to refine the above list slightly. Yet my original clue still seems like a strong candidate.

Why would "-smp sockets=1,cores=6,threads=1" make my computer crash?

UPDATE 2:
It took a while, but I finally managed to prove that this card is unstable regardless of qemu settings and thus useless for PCI passthrough. I will use it on host.

The host freeze problem seems to require certain kind of disk and heavy cpu activity or it simply won't occur. It most often occurs right after reboot when caches are empty.

By the way above mostly applies to Qemu 2.0.0. The git version of Qemu seemed even more unstable although I did not experience host crashes.

I will give up for now. Tips and patches are, of course, very welcome.

Last edited by ahl (2014-07-10 09:25:26)

abdullah

Member

Registered: 2014-06-10

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

so... what does -smp intends to do ?

And, how I could block internet connection for one guest and let the other enjoy the internet ?

Last edited by abdullah (2014-07-09 23:32:37)