Arch Linux

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I would go with radeon R9 270 or  better. Nvidia will give you error 43.

The only issue I'm aware of with using nvidia for the host is that the proprietary driver grabs the vga arbiter lock and doesn't release it.  So you either need to avoid vga or hack nvidia's wrapper code.

I think nvidia fixed this since i'm using the stock drivers without issue.

I use IGD for my host graphics, but I just download the latest driver, NVIDIA-Linux-x86_64-346.47  In there I find:

kernel/nv.c:nvidia_probe()
...
#if defined(CONFIG_VGA_ARB)
#if defined(VGA_DEFAULT_DEVICE)
    vga_tryget(VGA_DEFAULT_DEVICE, VGA_RSRC_LEGACY_MASK);
#endif
    vga_set_legacy_decoding(dev, VGA_RSRC_NONE);
#endif

There's no call to vga_put() anywhere.  Still looks broken to me, but maybe this is dead code or there are corner cases.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think nvidia fixed this since i'm using the stock drivers without issue.

I use IGD for my host graphics, but I just download the latest driver, NVIDIA-Linux-x86_64-346.47  In there I find:

kernel/nv.c:nvidia_probe()
...
#if defined(CONFIG_VGA_ARB)
#if defined(VGA_DEFAULT_DEVICE)
    vga_tryget(VGA_DEFAULT_DEVICE, VGA_RSRC_LEGACY_MASK);
#endif
    vga_set_legacy_decoding(dev, VGA_RSRC_NONE);
#endif

There's no call to vga_put() anywhere.  Still looks broken to me, but maybe this is dead code or there are corner cases.

Oh, maybe it is fixed.  Calling vga_set_legacy_decoding() clears the locked resources and does an implicit __vga_put().

EDIT: Embarrassing, it might be me who fixed it http://git.kernel.org/cgit/linux/kernel … 941affd75a

Last edited by aw (2015-03-23 01:28:02)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

asd651651

Member

Registered: 2015-03-23

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi,
I can pass through GPU successfully but after shutdown the VM and restart the VM, the host system stuck. Does anyone have that problem too?
It seems like the problem only occurs when passing through GPU, there is no such problem when passing through the USB controller.

Here’s my system environment:
    Kernel version: 3.18.8 (patched with linux-mainline)
    GPU: GTX 295
    Qemu: 2.2.0

hotfunction

Member

Registered: 2015-02-05

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, I suffer from the booting problem too...
When I use the VFIO GPU passthrough in a VM, shutdown the VM, but I cannot boot the VM unless I restart my host system.
Does anyone know this problem?

dRaiser

Member

From: Poland

Registered: 2013-05-20

Posts: 51

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think nvidia fixed this since i'm using the stock drivers without issue.

I use IGD for my host graphics, but I just download the latest driver, NVIDIA-Linux-x86_64-346.47  In there I find:

kernel/nv.c:nvidia_probe()
...
#if defined(CONFIG_VGA_ARB)
#if defined(VGA_DEFAULT_DEVICE)
    vga_tryget(VGA_DEFAULT_DEVICE, VGA_RSRC_LEGACY_MASK);
#endif
    vga_set_legacy_decoding(dev, VGA_RSRC_NONE);
#endif

There's no call to vga_put() anywhere.  Still looks broken to me, but maybe this is dead code or there are corner cases.

Oh, maybe it is fixed.  Calling vga_set_legacy_decoding() clears the locked resources and does an implicit __vga_put().

EDIT: Embarrassing, it might be me who fixed it http://git.kernel.org/cgit/linux/kernel … 941affd75a

Perhaps I misunderstood what a problem is, thanks for clarifying.

The_Moves

Member

Registered: 2015-01-06

Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I would go with radeon R9 270 or  better. Nvidia will give you error 43.

Look up PCIe 'Function Level Reset'

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

To be honest, i'd totally go check that huge google spreadsheet with stats before changing existing or beginning a new setup.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

dRaiser

Member

From: Poland

Registered: 2013-05-20

Posts: 51

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Duelist, thank you - haven't seen that before. Great resource!

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Duelist, thank you - haven't seen that before. Great resource!

While most of the data is okay, some setups may be semi-working(like my entry). If you're in doubt - you can search that user's posts in this thread using google site: search.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

A short tale about SELinux and VM audio:
QEMU is run as root(which is bad, but i'm too lazy to figure out all the permissions..i should really do this one day), i turned on qemu's VNC audio extension, and SELinux prevents that run-as-root process to access user's pulseaudio. Which is cool.
Setting SELinux mode to permissive(which is bad too) doesn't help too, as root isn't allowed to access user's pulse by something else.
Since i don't understand either SELinux or PulseAudio(when i set up my gentoo system, i didn't use SELinux and pulse audio simply didn't exist), it's easier for me to plug a dedicated sound device into my machine and passthrough it.

BUT!
There are GPU HDMI Audio devices! Does anybody know how do i fetch audio from them?
Do i really-really-really need to use some TV screen with HDMI Audio capability(since it's digital)? Or, maybe, there's some way of transferring audio through DVI, but my monitor doesn't support that?
Maybe there's some weird device which gets only HDMI's audio signal(like video capture device, only for audio) and feeds it into speakers/SPDIF/line in of the sound card?
Also, i remember some older GPUs had SPDIF out on them for that purpose, and you could plug it anywhere you need, including your motherboard. Do those still exist, or "if there's HDMI, then there's no SPDIF"?

Last edited by Duelist (2015-03-23 18:14:19)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

dRaiser

Member

From: Poland

Registered: 2013-05-20

Posts: 51

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Duelist, thank you - haven't seen that before. Great resource!

While most of the data is okay, some setups may be semi-working(like my entry). If you're in doubt - you can search that user's posts in this thread using google site: search.

Well, I'm familiar with Google site search, thank you.

I've added my setup too. It's working very nicely, only it's a little outdated since Radeon 4850 is mostly good only for Blizzard games nowadays.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BUT!
There are GPU HDMI Audio devices! Does anybody know how do i fetch audio from them?
Do i really-really-really need to use some TV screen with HDMI Audio capability(since it's digital)? Or, maybe, there's some way of transferring audio through DVI, but my monitor doesn't support that?
Maybe there's some weird device which gets only HDMI's audio signal(like video capture device, only for audio) and feeds it into speakers/SPDIF/line in of the sound card?
Also, i remember some older GPUs had SPDIF out on them for that purpose, and you could plug it anywhere you need, including your motherboard. Do those still exist, or "if there's HDMI, then there's no SPDIF"?

Using a "TV" is the easiest way to get audio out of HDMI.  If you're using nvidia, be sure to check my blog post about using MSI interrupts.  Nvidia audio works way better with MSI interrupts.  You can also do audio over DVI.  My HD8570 only has DP and DVI connections, my DP->HDMI adapter broke so I temporarily used a DVI->HDMI adapter and was quite surprised when I still had audio.  There are splitters and some (most?) receivers with HDMI input will allow you to channel the audio somewhere other than the TV.  HDCP copy protection comes into play when you start splitting the output, but that's only an issue if you're wanting to send protected content out to the screen.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

tritron4

Member

Registered: 2012-04-14

Posts: 153

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have no issues with my R9 270 I have A88X-pro asus motherboard and options vfio_iommu_type1 disable_hugepages=1 It works great. I can shutdown and restart guest without any issues.
Windows allows to select audio output over hdmi this will work with tv or lcd monitor with speakers. I guess you can use converter to extract audio out of hdmi. Other option is to use usb sound card.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lcd monitor with speakers.

Some screens do not support audio over DVI, but have speakers. Usually, they have 3.5mm line in on their back.
---

Alright, i've got a problem..

Error starting domain: internal error: early end of file from monitor: possible problem:
2015-03-23T20:34:22.276172Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to set iommu for container: Device or resource busy
2015-03-23T20:34:22.276310Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to setup container for group 7
2015-03-23T20:34:22.276368Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to get group 7
2015-03-23T20:34:22.276432Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: Device initialization failed.
2015-03-23T20:34:22.276499Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: Device 'vfio-pci' could not be initialized


Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 89, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 125, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1388, in startup
    self._backend.create()
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 999, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: internal error: early end of file from monitor: possible problem:
2015-03-23T20:34:22.276172Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to set iommu for container: Device or resource busy
2015-03-23T20:34:22.276310Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to setup container for group 7
2015-03-23T20:34:22.276368Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to get group 7
2015-03-23T20:34:22.276432Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: Device initialization failed.
2015-03-23T20:34:22.276499Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: Device 'vfio-pci' could not be initialized

Device or resource busy. But..

03:05.0 Multimedia audio controller: Device 1234:1880 (rev 02)
        Subsystem: Device 1234:2000
        Flags: fast devsel, IRQ 1
        I/O ports at <ignored> [disabled]
        Kernel driver in use: vfio-pci

cgroup_device_acl = [
    "/dev/null", "/dev/full", "/dev/zero",
    "/dev/random", "/dev/urandom",
    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
    "/dev/rtc","/dev/hpet", "/dev/vfio/vfio", "/dev/vfio/1", "/dev/vfio/2", "/dev/vfio/7"
]

And nothing in dmesg.

I bet this is something obvious, but i just can't see it. Any hints where to look?

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

tritron4

Member

Registered: 2012-04-14

Posts: 153

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did you load modules ? I had seen this error when pci-stub was not loaded at boot.

lcd monitor with speakers.

Some screens do not support audio over DVI, but have speakers. Usually, they have 3.5mm line in on their back.
---

Alright, i've got a problem..

Error starting domain: internal error: early end of file from monitor: possible problem:
2015-03-23T20:34:22.276172Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to set iommu for container: Device or resource busy
2015-03-23T20:34:22.276310Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to setup container for group 7
2015-03-23T20:34:22.276368Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to get group 7
2015-03-23T20:34:22.276432Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: Device initialization failed.
2015-03-23T20:34:22.276499Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: Device 'vfio-pci' could not be initialized


Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 89, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 125, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1388, in startup
    self._backend.create()
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 999, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: internal error: early end of file from monitor: possible problem:
2015-03-23T20:34:22.276172Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to set iommu for container: Device or resource busy
2015-03-23T20:34:22.276310Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to setup container for group 7
2015-03-23T20:34:22.276368Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: vfio: failed to get group 7
2015-03-23T20:34:22.276432Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: Device initialization failed.
2015-03-23T20:34:22.276499Z qemu-system-x86_64: -device vfio-pci,host=03:05.0,id=hostdev4,bus=pci.0,addr=0xd: Device 'vfio-pci' could not be initialized

Device or resource busy. But..

03:05.0 Multimedia audio controller: Device 1234:1880 (rev 02)
        Subsystem: Device 1234:2000
        Flags: fast devsel, IRQ 1
        I/O ports at <ignored> [disabled]
        Kernel driver in use: vfio-pci

cgroup_device_acl = [
    "/dev/null", "/dev/full", "/dev/zero",
    "/dev/random", "/dev/urandom",
    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
    "/dev/rtc","/dev/hpet", "/dev/vfio/vfio", "/dev/vfio/1", "/dev/vfio/2", "/dev/vfio/7"
]

And nothing in dmesg.

I bet this is something obvious, but i just can't see it. Any hints where to look?

The_Moves

Member

Registered: 2015-01-06

Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Are you trying to pass through your onboard audio? When I tried doing this, my previous system with either hang or do this. I don't see any of the below in your recent previous posts:

cat /proc/cmdline
cat /etc/default/grub
lspci -nn

Might have this wrong as I don't have access to my comp at the moment.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

That is a dedicated audio device. pci-stub shouldn't be needed, but i'll try adding it anyways.
Before VFIO-PCI driver bound the device, it wasn't bound to anything.
UPD:

BOOT_IMAGE=/vmlinuz-3.19.1-201.fc21.x86_64 root=UUID=86383473-85bd-44d7-9d15-0f15ec76da28 ro vconsole.font=latarcyrheb-sun16 nofb pci-stub.ids=1234:1880,1002:683f,1002:aab0,1002:683f,1002:683f enable_mtrr_cleanup iommu=pt acpi_enforce_resources=lax

Alright, as i thought, adding pci stub doesn't help at all.
IOMMU groups look same way as they look for devices 01:00.0 and 02:00.0 which work fine.
I wonder why there's some disabled IO range indicated..

Anyway, it's just a simple, dumb pci device! Why wouldn't it passthrough..

03:05.0 Multimedia audio controller [0401]: Device [1234:1880] (rev 02)
        Subsystem: Device [1234:2000]
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 1
        Region 0: I/O ports at <ignored> [disabled]
        Kernel driver in use: vfio-pci

If i recall correctly, it's Sound Blaster 16 PCI with rebadged ensoniq chip, made by creative. It's so old, that stock fedora kernel doesn't have drivers for it, however, it seemed to be AC'97-compliant.

00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] FCH PCI Bridge (rev 40) (prog-if 01 [Subtractive decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop+ ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64
        Bus: primary=00, secondary=03, subordinate=03, sec-latency=64
        I/O behind bridge: 0000f000-00000fff
        Memory behind bridge: fff00000-000fffff
        Prefetchable memory behind bridge: fff00000-000fffff
        Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR+
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-

Upstream PCI bridge seems fine to me too.

Last edited by Duelist (2015-03-23 22:19:38)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

tritron4

Member

Registered: 2012-04-14

Posts: 153

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Read this https://lkml.org/lkml/2013/6/28/647

Did you try this ? options vfio_iommu_type1 disable_hugepages=1

http://spica-and-roid.blogspot.com.au/2 … rough.html
7. Necessary permission modifications [3]
According to my experience, you might encounter an error saying:
Device 'pci-assign' could not be initialized.

Before you try any solution, first make sure that modules such as kvm, kvm_intel, pci_stub are loaded. Also make sure that intel_iommu=on is in your grub configuration file (even if you see IOMMU from dmesg, it doesn't ensure that INTEL_IOMMU is enabled).

The command that fixed it was:
# echo 1 > /sys/module/kvm/parameters/allow_unsafe_assigned_interrupts

Last edited by tritron4 (2015-03-24 02:42:27)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Read this https://lkml.org/lkml/2013/6/28/647

Did you try this ? options vfio_iommu_type1 disable_hugepages=1

http://spica-and-roid.blogspot.com.au/2 … rough.html
7. Necessary permission modifications [3]
According to my experience, you might encounter an error saying:
Device 'pci-assign' could not be initialized.

Before you try any solution, first make sure that modules such as kvm, kvm_intel, pci_stub are loaded. Also make sure that intel_iommu=on is in your grub configuration file (even if you see IOMMU from dmesg, it doesn't ensure that INTEL_IOMMU is enabled).

The command that fixed it was:
# echo 1 > /sys/module/kvm/parameters/allow_unsafe_assigned_interrupts

I appreciate your help, but...
IOMMU is 100% online, since devices 01:00.0 and 02:00.0 are getting passed-through fine.
Hugepages aren't used(but i might try forcefully disabling them). and were disabled a long time ago.
QEMU is run as root, clear_emulator_capabilities are set to zero. ACS is working fine, the device is isolated in it's IOMMU group, and since this is an AMD system - there shouldn't be any problems with ACS.
But the idea of rebinding the device is interesting, i might give it a try.

Rebinding a device gives weird issues:

[  821.585416] pci 0000:00:14.4: PCI bridge to [bus 03]
[  821.585427] pci 0000:00:14.4:   bridge window [io  0x1000-0x1fff]
[  821.594894] pci 0000:00:14.4: PCI bridge to [bus 03]
[  821.594903] pci 0000:00:14.4:   bridge window [io  0x1000-0x1fff]
[  821.598754] pci 0000:00:14.4: PCI bridge to [bus 03]
[  821.598762] pci 0000:00:14.4:   bridge window [io  0x1000-0x1fff]
[  821.602439] pci 0000:00:14.4: PCI bridge to [bus 03]
[  821.602446] pci 0000:00:14.4:   bridge window [io  0x1000-0x1fff]
[  821.604906] pci 0000:00:14.4: PCI bridge to [bus 03]
[  821.604914] pci 0000:00:14.4:   bridge window [io  0x1000-0x1fff]

I'm fine with that message, but it's repeated five times. Something is quite not right here.

UPD:

[   71.303135] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270
[   71.334630] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)
[   71.356269] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270
[   71.385082] pci-stub 0000:03:05.0: enabling device (0000 -> 0001)
[   71.638238] virbr0: port 2(vnet0) entered learning state
[   71.868569] pci-stub 0000:03:05.0: kvm assign device
[   73.644474] virbr0: topology change detected, propagating
[   73.644526] virbr0: port 2(vnet0) entered forwarding state
[   74.983914] kvm: zapping shadow pages for mmio generation wraparound

Changed VFIO to KVM pci-assign:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='kvm'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x05' function='0x0'/>
      </source>
      <alias name='hostdev4'/>
      <rom bar='on'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>

And the device is passed-through, showing up in VM's device manager.
It looks bound to pci-stub from host's perspective, however, dmesg clearly shows that it's fine:

03:05.0 Multimedia audio controller [0401]: Device [1234:1880] (rev 02)
        Subsystem: Device [1234:2000]
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 20
        Region 0: I/O ports at 1000 [disabled] [size=128]
        Kernel driver in use: pci-stub

Last edited by Duelist (2015-03-24 11:05:01)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

syshack

Member

Registered: 2015-03-24

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, aw:

my kernel/qemu version info as following:

linux-3.12.5 from kernel.org with following patchs.
pci/iommu: Quirk non-compliant PCIe-to-PCI bridges
i915_fixes[1]
i915_fixes[2]
Enable overrides for missing ACS capabilities

qemu-2.0.0 with patch
vfio-pci: Disallow device from using NoSnoop transactions

Oh jeez, ping me again when you're using something reasonable.  I'd suggest at least 3.16 and QEMU 2.2.  You don't need any patches unless you have ACS issues with IOMMU groups.

OK, I will try.
but yesterday I tried RHEL 7.0, it worked well with GRID K2. and the kernel version is 3.10.0-121.el7.x86_64 , qemu version is 1.5.3-60.el7.
had they work with the patched ?

Can u share your success experience with me?

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I didn't even notice my kernel got updated from 3.18.x to 3.19.1. Thanks redhat folks, everything still works fine.
Some pages ago people asked about 3.19..

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

tritron4

Member

Registered: 2012-04-14

Posts: 153

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So does it work than ?

Read this https://lkml.org/lkml/2013/6/28/647

Did you try this ? options vfio_iommu_type1 disable_hugepages=1

http://spica-and-roid.blogspot.com.au/2 … rough.html
7. Necessary permission modifications [3]
According to my experience, you might encounter an error saying:
Device 'pci-assign' could not be initialized.

Before you try any solution, first make sure that modules such as kvm, kvm_intel, pci_stub are loaded. Also make sure that intel_iommu=on is in your grub configuration file (even if you see IOMMU from dmesg, it doesn't ensure that INTEL_IOMMU is enabled).

The command that fixed it was:
# echo 1 > /sys/module/kvm/parameters/allow_unsafe_assigned_interrupts

I appreciate your help, but...
IOMMU is 100% online, since devices 01:00.0 and 02:00.0 are getting passed-through fine.
Hugepages aren't used(but i might try forcefully disabling them). and were disabled a long time ago.
QEMU is run as root, clear_emulator_capabilities are set to zero. ACS is working fine, the device is isolated in it's IOMMU group, and since this is an AMD system - there shouldn't be any problems with ACS.
But the idea of rebinding the device is interesting, i might give it a try.

Rebinding a device gives weird issues:

[  821.585416] pci 0000:00:14.4: PCI bridge to [bus 03]
[  821.585427] pci 0000:00:14.4:   bridge window [io  0x1000-0x1fff]
[  821.594894] pci 0000:00:14.4: PCI bridge to [bus 03]
[  821.594903] pci 0000:00:14.4:   bridge window [io  0x1000-0x1fff]
[  821.598754] pci 0000:00:14.4: PCI bridge to [bus 03]
[  821.598762] pci 0000:00:14.4:   bridge window [io  0x1000-0x1fff]
[  821.602439] pci 0000:00:14.4: PCI bridge to [bus 03]
[  821.602446] pci 0000:00:14.4:   bridge window [io  0x1000-0x1fff]
[  821.604906] pci 0000:00:14.4: PCI bridge to [bus 03]
[  821.604914] pci 0000:00:14.4:   bridge window [io  0x1000-0x1fff]

I'm fine with that message, but it's repeated five times. Something is quite not right here.

UPD:

[   71.303135] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270
[   71.334630] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)
[   71.356269] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270
[   71.385082] pci-stub 0000:03:05.0: enabling device (0000 -> 0001)
[   71.638238] virbr0: port 2(vnet0) entered learning state
[   71.868569] pci-stub 0000:03:05.0: kvm assign device
[   73.644474] virbr0: topology change detected, propagating
[   73.644526] virbr0: port 2(vnet0) entered forwarding state
[   74.983914] kvm: zapping shadow pages for mmio generation wraparound

Changed VFIO to KVM pci-assign:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='kvm'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x05' function='0x0'/>
      </source>
      <alias name='hostdev4'/>
      <rom bar='on'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>

And the device is passed-through, showing up in VM's device manager.
It looks bound to pci-stub from host's perspective, however, dmesg clearly shows that it's fine:

03:05.0 Multimedia audio controller [0401]: Device [1234:1880] (rev 02)
        Subsystem: Device [1234:2000]
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 20
        Region 0: I/O ports at 1000 [disabled] [size=128]
        Kernel driver in use: pci-stub

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OK, I will try.
but yesterday I tried RHEL 7.0, it worked well with GRID K2. and the kernel version is 3.10.0-121.el7.x86_64 , qemu version is 1.5.3-60.el7.
had they work with the patched ?

Can u share your success experience with me?

RHEL7 supports GPU assignment of K-series Quadro (2000 and up), GRID, and Telsa.  This uses the non-VGA, secondary GPU model.  Configuration is documented here:

http://documentation-devel.engineering. … e-GPU.html

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Tyrewt

Member

Registered: 2014-09-13

Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi,
I can pass through GPU successfully but after shutdown the VM and restart the VM, the host system stuck. Does anyone have that problem too?
It seems like the problem only occurs when passing through GPU, there is no such problem when passing through the USB controller.

Here’s my system environment:
    Kernel version: 3.18.8 (patched with linux-mainline)
    GPU: GTX 295
    Qemu: 2.2.0

Yes, I too experience this and have not found a solution.

tritron4

Member

Registered: 2012-04-14

Posts: 153

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi,
I can pass through GPU successfully but after shutdown the VM and restart the VM, the host system stuck. Does anyone have that problem too?
It seems like the problem only occurs when passing through GPU, there is no such problem when passing through the USB controller.

Here’s my system environment:
    Kernel version: 3.18.8 (patched with linux-mainline)
    GPU: GTX 295
    Qemu: 2.2.0

Yes, I too experience this and have not found a solution.

Any error messages in log ?