You are not logged in.

Pages: 1 … 26 27 28 29 30 … 234

Topic closed

#676 2013-11-04 18:41:41

Unnamed_Hero
Member
Registered: 2012-06-12
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi to all. I have a problem starting test qemu machine.
I'm using kernel 3.11.6, qemu 1.6.1-2

I bind Radeon card (2 ids) with vfio-bind script. (BTW, should I bind Radeon's PCI-E with this script?)

My error message:

qemu-system-x86_64: -device vfio-pci,host=00:01.1,bus=root.1,addr=00.1: vfio: error, group 1 is not viable,
please ensure all devices within the iommu_group are bound to their vfio bus driver


My card list:

01:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 660] (rev a1)
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970]

Nvidia is for Linux host, Radeon is for Windows guest.

My IOMMU list is

lrwxrwxrwx 1 root root 0 ноя  4 21:03 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 <--PCI-E 0 < Nvidia
lrwxrwxrwx 1 root root 0 ноя  4 21:03 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1 <--PCI-E 1 < Radeon
lrwxrwxrwx 1 root root 0 ноя  4 21:03 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 <-- Nvidia
lrwxrwxrwx 1 root root 0 ноя  4 21:03 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1 <-- Nvidia 
lrwxrwxrwx 1 root root 0 ноя  4 21:03 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0 <-- Radeon
lrwxrwxrwx 1 root root 0 ноя  4 21:03 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1 <-- Radeon

So, I can't unbind my system video.

What should I do?

Last edited by Unnamed_Hero (2013-11-04 18:42:24)

Offline

#677 2013-11-04 19:06:09

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Unnamed_Hero wrote:

Hi to all. I have a problem starting test qemu machine.
I'm using kernel 3.11.6, qemu 1.6.1-2

I bind Radeon card (2 ids) with vfio-bind script. (BTW, should I bind Radeon's PCI-E with this script?)

My error message:

qemu-system-x86_64: -device vfio-pci,host=00:01.1,bus=root.1,addr=00.1: vfio: error, group 1 is not viable,
please ensure all devices within the iommu_group are bound to their vfio bus driver


My card list:

01:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 660] (rev a1)
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970]

Nvidia is for Linux host, Radeon is for Windows guest.

My IOMMU list is

lrwxrwxrwx 1 root root 0 ноя  4 21:03 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 <--PCI-E 0 < Nvidia
lrwxrwxrwx 1 root root 0 ноя  4 21:03 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1 <--PCI-E 1 < Radeon
lrwxrwxrwx 1 root root 0 ноя  4 21:03 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 <-- Nvidia
lrwxrwxrwx 1 root root 0 ноя  4 21:03 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1 <-- Nvidia 
lrwxrwxrwx 1 root root 0 ноя  4 21:03 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0 <-- Radeon
lrwxrwxrwx 1 root root 0 ноя  4 21:03 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1 <-- Radeon

So, I can't unbind my system video.

What should I do?

First, things are not going to work with the kernel and qemu versions you have.  See the first post for where to get the right versions.  You will need to build from source.

Next, if your IOMMU group doesn't let you separate devices it's because the root ports do not support PCI ACS and we therefore must assume that peer-to-peer traffic between subordinates of those roots is possible.  The long term fix for this is Intel properly supporting ACS on root ports and providing a white list for existing devices.  We're working with them to do that.  In the interim, several people here are using this patch: https://lkml.org/lkml/2013/5/30/513  This was rejected upstream, so you'll need to patch it in.

Finally, I assume by "PCI-E" you mean the root port the device is attached to (00:01.0 & 00:01.1 in your listing), and no, that should not be bound to vfio-pci.

Also note that if you're using the proprietary nvidia driver in the host it will grab the VGA arbiter lock and never release it.  The result will be that the first VGA access made by the guest for the Radeon device will hang.  There's a patch floating around to modify the nvidia driver wrapper to avoid this, perhaps someone can share a link.

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#678 2013-11-04 19:13:33

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

There's a patch floating around to modify the nvidia driver wrapper to avoid this, perhaps someone can share a link.

That's actually in the first post, too under "ISSUES".

i'm sorry for my poor english wirting skills…

Offline

#679 2013-11-04 19:27:21

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have uploaded kernel 3.12.0 (includes acs override patch + i915 patches) and qemu-git (with NoSnoop patch) on the first post

Last edited by nbhs (2013-11-04 19:27:35)

Offline

#680 2013-11-04 19:55:08

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Looks like some dudes got OSX mavericks working on qemu-kvm on intel hardware http://www.linux.org.ru/gallery/screenshots/9732976, also see: http://www.contrib.andrew.cmu.edu/~somlo/OSXKVM/

Offline

#681 2013-11-04 20:03:07

myweb
Member
Registered: 2013-07-13
Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear all,

Could you please help to solve Nvidia driver issue on the guest (nouveau driver works OK, but I need Nvidia due to hardware video acceleration):

[    42.409] Loading extension NV-GLX
[    44.409] (EE) NVIDIA(GPU-0): WAIT: (E, 0, 0x917d)

I receive the messages when I run startx and when I run xbmc via  "xinit /usr/bin/xbmc-standalone" I receive the following:

[  5709.592] (EE) NVIDIA(0): The NVIDIA kernel module does not appear to be receiving
[  5709.592] (EE) NVIDIA(0):     interrupts generated by the NVIDIA GPU at PCI:1:0:0.
[  5709.592] (EE) NVIDIA(0):     Please see Chapter 8: Common Problems in the README for
[  5709.592] (EE) NVIDIA(0):     additional information.
[  5709.592] (EE) NVIDIA(0): Failed to initialize the NVIDIA graphics device!
[  5709.592] (EE) NVIDIA(0): Failing initialization of X screen 0

lspci -v shows the following (IRQ 16 is shared between GPU and SMBus Controller)

00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
        Subsystem: Red Hat, Inc Device 1100
        Physical Slot: 31
        Flags: fast devsel, IRQ 16
        I/O ports at b100 [size=64]
        Kernel driver in use: i801_smbus
        Kernel modules: i2c_i801

01:00.0 VGA compatible controller: NVIDIA Corporation GK107 [GeForce GT 640] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: ASUSTeK Computer Inc. Device 840e
        Physical Slot: 0
        Flags: bus master, fast devsel, latency 0, IRQ 16
        Memory at fc000000 (32-bit, non-prefetchable) [size=16M]
        Memory at d0000000 (64-bit, prefetchable) [size=256M]
        Memory at e0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at c000 [size=128]
        [virtual] Expansion ROM at fd000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [b4] Vendor Specific Information: Len=14 <?>
        Capabilities: [100] Virtual Channel
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Kernel driver in use: nvidia
        Kernel modules: nouveau, nvidia

Could you please help to "disable" sharing GPU IRQ?

Last edited by myweb (2013-11-04 21:40:31)

Offline

#682 2013-11-04 21:38:54

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:

Dear all,

Could you please help to solve Nvidia driver issue on the guest (nouveau driver works OK, but I need Nvidia due to hardware video acceleration):

[    42.409] Loading extension NV-GLX
[    44.409] (EE) NVIDIA(GPU-0): WAIT: (E, 0, 0x917d)

I receive the messages when I run startx and when I run xbmc via  "xinit /usr/bin/xbmc-standalone" I receive the following:

[  5709.592] (EE) NVIDIA(0): The NVIDIA kernel module does not appear to be receiving
[  5709.592] (EE) NVIDIA(0):     interrupts generated by the NVIDIA GPU at PCI:1:0:0.
[  5709.592] (EE) NVIDIA(0):     Please see Chapter 8: Common Problems in the README for
[  5709.592] (EE) NVIDIA(0):     additional information.
[  5709.592] (EE) NVIDIA(0): Failed to initialize the NVIDIA graphics device!
[  5709.592] (EE) NVIDIA(0): Failing initialization of X screen 0

Have you tried NVreg_EnableMSI=1 as a nvidia module options?  Check /proc/interrupts on the host to see if we're seeing an interrupt there (grep vfio /proc/interrupts).

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#683 2013-11-04 21:48:32

myweb
Member
Registered: 2013-07-13
Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sudo modprobe nvidia NVreg_EnableMSI=1

does not help

grep vfio /proc/interrupts

16:       5513       3621        960        109  IR-IO-APIC-fasteoi   vfio-intx(0000:00:1a.0), vfio-intx(0000:01:00.0)
17:        393        209        261        120  IR-IO-APIC-fasteoi   saa7146 (0), vfio-intx(0000:01:00.1)

0000:00:1a.0 - is USB controller
I run qemu with the following options

qemu-system-x86_64 -M q35  -m 4096  -enable-kvm -cpu host  -vga none -nographic -boot menu=on \
-smp 4,sockets=4,cores=4,threads=4 \
-device ahci,bus=pcie.0,id=ahci \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1 \
-device vfio-pci,host=01:00.1,bus=pcie.0 \
-device vfio-pci,host=00:1a.0,bus=pcie.0 \ 
-drive file=/dev/sda6,id=rootDisk,format=raw -device ide-hd,bus=ahci.0,drive=rootDisk \
-drive file=/dev/sda7,id=swapDisk,format=raw -device ide-hd,bus=ahci.1,drive=swapDisk \
-netdev bridge,br=br0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0

I removed USB controller from qemu options and see the following in Xorg.log

[   176.854] (II) NVIDIA(0): Validated MetaModes:
[   176.854] (II) NVIDIA(0):     "DFP-0:nvidia-auto-select{}"
[   176.854] (II) NVIDIA(0): Virtual screen size determined to be 1920 x 1200
[   176.889] (--) NVIDIA(0): DPI set to (90, 87); computed from "UseEdidDpi" X config
[   176.889] (--) NVIDIA(0):     option
[   176.889] (--) Depth 24 pixmap format is 32 bpp
[   176.890] (II) NVIDIA: Using 3072.00 MB of virtual memory for indirect memory
[   176.890] (II) NVIDIA:     access.
[   176.894] (II) NVIDIA(0): ACPI: failed to connect to the ACPI event daemon; the daemon
[   176.894] (II) NVIDIA(0):     may not be running or the "AcpidSocketPath" X
[   176.894] (II) NVIDIA(0):     configuration option may not be set correctly.  When the
[   176.894] (II) NVIDIA(0):     ACPI event daemon is available, the NVIDIA X driver will
[   176.894] (II) NVIDIA(0):     try to use it to receive ACPI event notifications.  For
[   176.894] (II) NVIDIA(0):     details, please see the "ConnectToAcpid" and
[   176.894] (II) NVIDIA(0):     "AcpidSocketPath" X configuration options in Appendix B: X
[   176.894] (II) NVIDIA(0):     Config Options in the README.
[   176.896] (II) NVIDIA(0): Setting mode "DFP-0:nvidia-auto-select{}"
[   179.920] Loading extension NV-GLX
[   181.920] (EE) NVIDIA(GPU-0): WAIT: (E, 0, 0x917d)

Last edited by myweb (2013-11-04 22:00:19)

Offline

#684 2013-11-04 22:01:45

Norcoen
Member
From: Norway
Registered: 2013-11-02
Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

kaeptnb wrote:

like this:

kernel commandline parameter: pci-stub.ids=1002:6819,1002:aab0

the id's are not the PCI Bus IDs, but the Device Class/Subclass - you get them with "lspci -vn"

# lspci -vnn
03:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) (prog-if 01 [AHCI 1.0])
	Subsystem: ASRock Incorporation Motherboard [1849:0612]
	Flags: bus master, fast devsel, latency 0, IRQ 47
	I/O ports at d050 [size=8]
	I/O ports at d040 [size=4]
	I/O ports at d030 [size=8]
	I/O ports at d020 [size=4]
	I/O ports at d000 [size=32]
	Memory at f0d00000 (32-bit, non-prefetchable) [size=512]
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [78] Power Management version 3
	Capabilities: [80] Express Legacy Endpoint, MSI 00
	Capabilities: [100] Virtual Channel
	Kernel driver in use: ahci
[...]
05:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) (prog-if 01 [AHCI 1.0])
	Subsystem: ASRock Incorporation Motherboard [1849:0612]
	Flags: bus master, fast devsel, latency 0, IRQ 48
	I/O ports at b050 [size=8]
	I/O ports at b040 [size=4]
	I/O ports at b030 [size=8]
	I/O ports at b020 [size=4]
	I/O ports at b000 [size=32]
	Memory at f0b00000 (32-bit, non-prefetchable) [size=512]
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [78] Power Management version 3
	Capabilities: [80] Express Legacy Endpoint, MSI 00
	Capabilities: [100] Virtual Channel
	Kernel driver in use: ahci
[...]
08:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01) (prog-if 01 [AHCI 1.0])
	Subsystem: ASMedia Technology Inc. Device [1b21:1060]
	Flags: bus master, fast devsel, latency 0, IRQ 49
	I/O ports at a050 [size=8]
	I/O ports at a040 [size=4]
	I/O ports at a030 [size=8]
	I/O ports at a020 [size=4]
	I/O ports at a000 [size=32]
	Memory at f0910000 (32-bit, non-prefetchable) [size=512]
	Expansion ROM at f0900000 [disabled] [size=64K]
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [78] Power Management version 3
	Capabilities: [80] Express Legacy Endpoint, MSI 00
	Capabilities: [100] Virtual Channel
	Kernel driver in use: ahci
[...]

So you mean I should not use vendor:device [1b21:0612] but from the ?Subsystem? ID: [1b21:1060] as pci-stub kernelparameter?
Does it matter at this point, that it is a pci-express x1 card, not just pci?

Offline

#685 2013-11-04 22:20:48

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

myweb wrote:
sudo modprobe nvidia NVreg_EnableMSI=1

does not help

grep vfio /proc/interrupts

16:       5513       3621        960        109  IR-IO-APIC-fasteoi   vfio-intx(0000:00:1a.0), vfio-intx(0000:01:00.0)
17:        393        209        261        120  IR-IO-APIC-fasteoi   saa7146 (0), vfio-intx(0000:01:00.1)

You're still using INTx here.  Is that modprobe run from a clean guest boot without the nvidia module previously loaded?  Running modprobe against an already loaded module of course does nothing.  Does a Windows guest with Nvidia drivers work?

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#686 2013-11-04 22:23:17

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:
kaeptnb wrote:

like this:

kernel commandline parameter: pci-stub.ids=1002:6819,1002:aab0

the id's are not the PCI Bus IDs, but the Device Class/Subclass - you get them with "lspci -vn"

So you mean I should not use vendor:device [1b21:0612] but from the ?Subsystem? ID: [1b21:1060] as pci-stub kernelparameter?
Does it matter at this point, that it is a pci-express x1 card, not just pci?

You should use vendor:device, that's what the -n option provides.  The first post is mistaken on what those IDs actually are.  PCIe vs conventional PCI doesn't matter

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#687 2013-11-04 22:34:47

myweb
Member
Registered: 2013-07-13
Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nvidia under Windows works fine. Modprobe run from a clean guest boot without the nvidia module previously loaded.

grep vfio /proc/interrupts
 17:        572        239        284        142  IR-IO-APIC-fasteoi   saa7146 (0), vfio-intx(0000:01:00.1)
 49:          1          0          0          0  IR-PCI-MSI-edge      vfio-msi[0](0000:01:00.0)

Offline

#688 2013-11-05 02:07:52

cryptonymous
Member
Registered: 2013-11-05
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello. I think I'm running into a problem regarding interrupt handling when trying to pass through my NVIDIA GTX 650 card into OS X Mavericks.

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107 [GeForce GTX 650] [10de:0fc6] (rev a1)

Booting OS X with stock QEMU and GTX 650 passed though results in extremly sluggish graphics. Every now and then Finder process gets killed with the following logged into dmesg:

NVDA(OpenGL): Channel exception! Exception type = 0x8 DMA Engine Error (FIFO Error 8)

However, passing through the same GTX 650 into Windows 7 install works fine. I've compared "cat /proc/interrupts | grep vfio" output after booting Windows 7 and OS X:

Windows 7:

 16:     405613          0          0          0          1          0          0          0  IR-IO-APIC-fasteoi   vfio-intx(0000:01:00.0), firewire_ohci

OS X:

 67:          1          0          0          0          0          0          0          0  IR-PCI-MSI-edge      vfio-msi[0](0000:01:00.0)

Also, when running OS X the following backtrace is occasionally seen on the host:

[35849.691253] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[35910.481384] vfio-pci 0000:01:00.0: irq 67 for MSI/MSI-X
[36467.745050] irq 16: nobody cared (try booting with the "irqpoll" option)
[36467.745054] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W    3.12.0+ #8
[36467.745055] Hardware name:
[36467.745056]  0000000000000000 ffffffff81390dff ffff880806f71a00 ffffffff8107d0bb
[36467.745059]  ffff880806f71a00 0000000000000000 ffff880806f71a00 ffffffff8107d463
[36467.745060]  0000000000000000 0000000000000010 ffff880806f71a00 0000000000000000
[36467.745062] Call Trace:
[36467.745063]  <IRQ>  [<ffffffff81390dff>] ? dump_stack+0x41/0x51
[36467.745071]  [<ffffffff8107d0bb>] ? __report_bad_irq+0x2c/0xb7
[36467.745072]  [<ffffffff8107d463>] ? note_interrupt+0x14b/0x1ce
[36467.745074]  [<ffffffff8107b7a1>] ? handle_irq_event_percpu+0x120/0x187
[36467.745076]  [<ffffffff8107b83a>] ? handle_irq_event+0x32/0x4d
[36467.745078]  [<ffffffff8107dce6>] ? handle_fasteoi_irq+0x80/0xb4
[36467.745080]  [<ffffffff8100ebb2>] ? handle_irq+0x18/0x20
[36467.745081]  [<ffffffff8100e82b>] ? do_IRQ+0x40/0x95
[36467.745083]  [<ffffffff8139522d>] ? common_interrupt+0x6d/0x6d
[36467.745084]  <EOI>  [<ffffffff812b5b3e>] ? arch_local_irq_enable+0x7/0x8
[36467.745087]  [<ffffffff812b5e4e>] ? cpuidle_enter_state+0x50/0xa9
[36467.745089]  [<ffffffff812b5f84>] ? cpuidle_idle_call+0xdd/0x132
[36467.745091]  [<ffffffff810140ab>] ? arch_cpu_idle+0x5/0x17
[36467.745093]  [<ffffffff8107af6b>] ? cpu_startup_entry+0x10c/0x165
[36467.745095]  [<ffffffff816b8d33>] ? start_kernel+0x3e8/0x3f3
[36467.745097]  [<ffffffff816b876e>] ? repair_env_string+0x54/0x54
[36467.745099]  [<ffffffff816b8120>] ? early_idt_handlers+0x120/0x120
[36467.745100]  [<ffffffff816b8593>] ? x86_64_start_kernel+0xef/0xf8
[36467.745101] handlers:
[36467.745114] [<ffffffffa0144bdb>] irq_handler [firewire_ohci]
[36467.745115] Disabling IRQ #16

My theory is that although QEMU/VFIO configures the card to use MSI, it continues to use conventional IRQ and OS X drivers never receive any interrupts. I couldn't find any option to disable MSI emulation when using VFIO, so I've tried the most obvious way of commenting out the following lines in hw/misc/vfio.c:

diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c
index fdbb492..459c4ee 100644
--- a/hw/misc/vfio.c
+++ b/hw/misc/vfio.c
@@ -2800,14 +2804,14 @@ static int vfio_add_std_cap(VFIODevice *vdev, uint8_t pos)
 
     switch (cap_id) {
     case PCI_CAP_ID_MSI:
-        ret = vfio_setup_msi(vdev, pos);
+        ret = 0; /* vfio_setup_msi(vdev, pos); */
         break;
     case PCI_CAP_ID_EXP:
         vfio_check_pcie_flr(vdev, pos);
         ret = vfio_setup_pcie_cap(vdev, pos, size);
         break;
     case PCI_CAP_ID_MSIX:
-        ret = vfio_setup_msix(vdev, pos);
+        ret = 0; /* vfio_setup_msix(vdev, pos); */
         break;
     case PCI_CAP_ID_PM:
         vfio_check_pm_reset(vdev, pos);

After rebuilding QEMU with this patch applied interrupt handler switches to vfio-intx instead of vfio-msi and OS X graphics become smooth. Thus, in my case there is something wrong going on when MSI is being utilized.

Offline

#689 2013-11-05 03:14:45

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cryptonymous wrote:

My theory is that although QEMU/VFIO configures the card to use MSI, it continues to use conventional IRQ and OS X drivers never receive any interrupts. I couldn't find any option to disable MSI emulation when using VFIO, so I've tried the most obvious way of commenting out the following lines in hw/misc/vfio.c:

After rebuilding QEMU with this patch applied interrupt handler switches to vfio-intx instead of vfio-msi and OS X graphics become smooth. Thus, in my case there is something wrong going on when MSI is being utilized.

Interesting observation.  By removing those lines you're indeed not exposing MSI or MSI-X capabilities to the guest.  The guest has no choice but to use INTx.  I know I've used MSI successfully with a Quadro card and Linux guest, but I don't think I've tried a Linux guest with MSI on a GeForce (I had a Radeon running with MSI just the other day).  What does 'sudo lspci -vvv' from the host show for the GTX650 with OSX running without your qemu patch?  Random guess; if lspci shows DisINTx- try using setpci to enable it (ex. dev=01:00.0; val=$(setpci -s $dev COMMAND); newval=$(printf %04x $(( 0x$val | 0x400 ))); setpci -s $dev COMMAND=$newval; echo "restore with: setpci -s $dev COMMAND=$val").  If it's DisINTx- and making it DisINT+ makes any difference, perhaps there's a bug in the emulation of that bit.

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#690 2013-11-05 04:48:32

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
cryptonymous wrote:

My theory is that although QEMU/VFIO configures the card to use MSI, it continues to use conventional IRQ and OS X drivers never receive any interrupts. I couldn't find any option to disable MSI emulation when using VFIO, so I've tried the most obvious way of commenting out the following lines in hw/misc/vfio.c:

After rebuilding QEMU with this patch applied interrupt handler switches to vfio-intx instead of vfio-msi and OS X graphics become smooth. Thus, in my case there is something wrong going on when MSI is being utilized.

Interesting observation.  By removing those lines you're indeed not exposing MSI or MSI-X capabilities to the guest.  The guest has no choice but to use INTx.  I know I've used MSI successfully with a Quadro card and Linux guest, but I don't think I've tried a Linux guest with MSI on a GeForce (I had a Radeon running with MSI just the other day).  What does 'sudo lspci -vvv' from the host show for the GTX650 with OSX running without your qemu patch?  Random guess; if lspci shows DisINTx- try using setpci to enable it (ex. dev=01:00.0; val=$(setpci -s $dev COMMAND); newval=$(printf %04x $(( 0x$val | 0x400 ))); setpci -s $dev COMMAND=$newval; echo "restore with: setpci -s $dev COMMAND=$val").  If it's DisINTx- and making it DisINT+ makes any difference, perhaps there's a bug in the emulation of that bit.

Testing a Linux guest w/ a GTX660, I can confirm that things go bad when using the NV_RegEnableMSI=1 option.  This worked fine for me with a Quadro card.  I see a single interrupt and the monitor goes out of sync and never comes back after OS bootup.  I'll try to find some time to dig into it.  I also note that the audio driver specifically checks for nvidia devices and blacklists MSI on snd-hda-intel, so at some point nvidia hasn't played nicely with Linux MSI support.  Perhaps there's a reason why the Linux nvidia driver defaults to INTx as well.

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#691 2013-11-05 07:37:04

myweb
Member
Registered: 2013-07-13
Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

with the following options I get working Nvidia drivers in the guest:

options snd-hda-intel enable_msi=0
options nvidia NVreg_EnableMSI=0

Btw, I found interesting release notice from Nvidia for 325.08 BETA (http://www.nvidia.com/object/linux-disp … river.html)

"Changed the default PCIe interrupt delivery method from virtual-wire to MSI. Note that if the NVIDIA Linux driver fails to initialize with an error indicating that it is not receiving interrupts, MSI can be disabled by setting the module parameter "NVreg_EnableMSI=0" when loading the NVIDIA kernel module. - See more at: http://www.nvidia.com/object/linux-display-ia32-325.08-driver.html#sthash.bXXFjZEK.dpuf"

Last edited by myweb (2013-11-05 09:34:40)

Offline

#692 2013-11-05 11:12:35

cryptonymous
Member
Registered: 2013-11-05
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

What does 'sudo lspci -vvv' from the host show for the GTX650 with OSX running without your qemu patch?  Random guess; if lspci shows DisINTx- try using setpci to enable it (ex. dev=01:00.0; val=$(setpci -s $dev COMMAND); newval=$(printf %04x $(( 0x$val | 0x400 ))); setpci -s $dev COMMAND=$newval; echo "restore with: setpci -s $dev COMMAND=$val").  If it's DisINTx- and making it DisINT+ makes any difference, perhaps there's a bug in the emulation of that bit.

When idling at OS X login screen with unpatched QEMU, 'sudo lspci -vvv -s 01:00.0' shows the following:

01:00.0 VGA compatible controller: NVIDIA Corporation GK107 [GeForce GTX 650] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: Micro-Star International Co., Ltd. Device 2804
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 67
	Region 0: Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
	Region 1: Memory at e0000000 (64-bit, prefetchable) [size=256M]
	Region 3: Memory at f0000000 (64-bit, prefetchable) [size=32M]
	Region 5: I/O ports at e000 [size=128]
	Expansion ROM at f7000000 [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00478  Data: 0000
	Capabilities: [78] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Latency L0 <1us, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest+
	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [128 v1] Power Budgeting <?>
	Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Capabilities: [900 v1] #19
	Kernel driver in use: vfio-pci

So it has DisINTx+ flag. Changing it to DisINTx- via setpci does not make any visible difference. After rebooting the guest without exiting QEMU the flag is back to DisINTx+.

Offline

#693 2013-11-05 20:50:29

SpacePirate
Member
Registered: 2013-09-16
Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After my system ran fine for a week or so now I dont get any sound from the vm at all again.
It starts up with this messages:

Nov 05 21:41:07 host windows-vm[1825]: ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave
Nov 05 21:41:07 host windows-vm[1825]: alsa: Could not initialize DAC
Nov 05 21:41:07 host windows-vm[1825]: alsa: Failed to open `default':
Nov 05 21:41:07 host windows-vm[1825]: alsa: Reason: Device or resource busy
Nov 05 21:41:07 host windows-vm[1825]: ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave
Nov 05 21:41:07 host windows-vm[1825]: alsa: Could not initialize DAC
Nov 05 21:41:07 host windows-vm[1825]: alsa: Failed to open `default':
Nov 05 21:41:07 host windows-vm[1825]: alsa: Reason: Device or resource busy
Nov 05 21:41:07 host windows-vm[1825]: audio: Failed to create voice `dac'

I tried googling the issue but I didnt find anything... Have any of you encountered that problem before?

Offline

#694 2013-11-08 00:30:35

rabcor
Banned
Registered: 2013-02-09
Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You may remember me from earlier in the thread, i tried this but gave up when i couldn't get Qemu to run. I'm back now with the AUR version of qemu-git and linux-mainline as well as the seabios provided here.

I have an Nvidia GTX 670 (01:00.0) and Nvidia GTX 550-Ti (02:00.0)

I have applied this to my cmdline: (The GTX 670 and its Audio)

pci-stub.ids=10de:1189,10de:0e0a

However i'm not actually sure if this is working or not, seeing as the computer is using this GPU when in terminal. When i try to start X if this option is on, i get an error that says the Nvidia Drivers are failing to load the GTX card (PCI 01:00.0), i even set the xorg configuration file to use "PCI:02:00:0" as BusID but that wont work either, and if i try to run the vfio-bind script for the GTX, i get an error just like the one i got the last time i tried to do this.

ls: cannot access /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices: No such file or directory

I think last time setting

CONFIG_INTEL_IOMMU_DEFAULT_ON=y

fixed that specific error(i'm gonna try that again). But how am i to tell the Nvidia drivers to ignore the GTX 670?

Last edited by rabcor (2013-11-08 00:33:38)

Making Logitech G510 Work on Linux!

Offline

#695 2013-11-08 00:47:48

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

You may remember me from earlier in the thread, i tried this but gave up when i couldn't get Qemu to run. I'm back now with the AUR version of qemu-git and linux-mainline as well as the seabios provided here.

I have an Nvidia GTX 670 (01:00.0) and Nvidia GTX 550-Ti (02:00.0)

I have applied this to my cmdline: (The GTX 670 and its Audio)

pci-stub.ids=10de:1189,10de:0e0a

However i'm not actually sure if this is working or not, seeing as the computer is using this GPU when in terminal. When i try to start X if this option is on, i get an error that says the Nvidia Drivers are failing to load the GTX card (PCI 01:00.0), i even set the xorg configuration file to use "PCI:02:00:0" as BusID but that wont work either, and if i try to run the vfio-bind script for the GTX, i get an error just like the one i got the last time i tried to do this.

ls: cannot access /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices: No such file or directory

I think last time setting

CONFIG_INTEL_IOMMU_DEFAULT_ON=y

fixed that specific error(i'm gonna try that again). But how am i to tell the Nvidia drivers to ignore the GTX 670?

Since you're using intel make sure to use the packages i provided instead of the AUR ones, on intel systems you need to boot with intel_iommu=on, add pci-stub to /etc/mkinitcpio.conf in the MODULES section, and rebuild it

mkinitcpio -p linux-mainline

Make sure the Geforce 670 isnt your primary card, and if you're gonna use the nvidia drivers on the host you'll need to patch them (patch on the "issues" section)

Last edited by nbhs (2013-11-08 00:48:25)

Offline

#696 2013-11-08 00:58:33

rabcor
Banned
Registered: 2013-02-09
Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Since you're using intel make sure to use the packages i provided instead of the AUR ones, on intel systems you need to boot with intel_iommu=on, add pci-stub to /etc/mkinitcpio.conf in the MODULES section, and rebuild it

mkinitcpio -p linux-mainline

Make sure the Geforce 670 isnt your primary card, and if you're gonna use the nvidia drivers on the host you'll need to patch them (patch on the "issues" section)

by intel_iommu=on do you mean in the cmdline? something like this?

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on pci-stub.ids=10de:1189,10de:0e0a quiet"

as for mkinitcpio.conf and the command, i already did that. I'll switch to your linux-mainline and qemu packages then.

Also, how do i make sure GeForce 670 isn't my primary card? does it have to do with which PCI port on the motherboard it is placed? or is it something i can configure somewhere? (bios only lets me select between iGPU/PCIe, it doesn't let me specify which PCIe slot is primary i think)

Last edited by rabcor (2013-11-08 01:01:43)

Making Logitech G510 Work on Linux!

Offline

#697 2013-11-08 01:22:48

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:
nbhs wrote:

Since you're using intel make sure to use the packages i provided instead of the AUR ones, on intel systems you need to boot with intel_iommu=on, add pci-stub to /etc/mkinitcpio.conf in the MODULES section, and rebuild it

mkinitcpio -p linux-mainline

Make sure the Geforce 670 isnt your primary card, and if you're gonna use the nvidia drivers on the host you'll need to patch them (patch on the "issues" section)

by intel_iommu=on do you mean in the cmdline? something like this?

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on pci-stub.ids=10de:1189,10de:0e0a quiet"

as for mkinitcpio.conf and the command, i already did that. I'll switch to your linux-mainline and qemu packages then.

Also, how do i make sure GeForce 670 isn't my primary card? does it have to do with which PCI port on the motherboard it is placed? or is it something i can configure somewhere? (bios only lets me select between iGPU/PCIe, it doesn't let me specify which PCIe slot is primary i think)

Yes you have to add intel_iommu=on to your grub config file or whatever bootloader you use, your primary card is the one you see boot messages on

Offline

#698 2013-11-08 01:31:45

rabcor
Banned
Registered: 2013-02-09
Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've switched the slots of the GPUs, 550-Ti is now in the primary slot and 670 in the secondary slot (i see boot messages and stuff on the 550 now, it also seems to play better with my BIOS too, 670 had a resolution problem until official drivers were loaded) if i boot into windows(dualboot) like this strangely enough the drivers get confused (probably because the older/weaker card is in the primary slot)

I still get the same error in X11 though

(PCI 1:0:0 is now the 550-Ti which i want to be using in linux, and 670 is at 2:0:0 which i would like to use in the VM, i didn't tell my system to do anything with the 550)

Failed to initialize the NVIDIA GPU at PCI:1:0:0. Please check yuor system's kernel log for additional error messages...
Screen(s) found, but none have a usable configuration.

it is quite the pain. i tried to install the linux-mainline you uploaded, it compiled but it didn't install for some reason, i'll have to try again hmm

Last edited by rabcor (2013-11-08 01:53:30)

Making Logitech G510 Work on Linux!

Offline

#699 2013-11-08 11:04:14

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

I've switched the slots of the GPUs, 550-Ti is now in the primary slot and 670 in the secondary slot (i see boot messages and stuff on the 550 now, it also seems to play better with my BIOS too, 670 had a resolution problem until official drivers were loaded) if i boot into windows(dualboot) like this strangely enough the drivers get confused (probably because the older/weaker card is in the primary slot)

I still get the same error in X11 though

(PCI 1:0:0 is now the 550-Ti which i want to be using in linux, and 670 is at 2:0:0 which i would like to use in the VM, i didn't tell my system to do anything with the 550)

Failed to initialize the NVIDIA GPU at PCI:1:0:0. Please check yuor system's kernel log for additional error messages...
Screen(s) found, but none have a usable configuration.

it is quite the pain. i tried to install the linux-mainline you uploaded, it compiled but it didn't install for some reason, i'll have to try again hmm

First you have to build it:

makepkg -sf

Then install it:

sudo pacman -U linux-mainline-*

Offline

#700 2013-11-09 16:55:14

cmorrow
Member
Registered: 2013-11-09
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

qemu-git (with and without dsnoop patches) is giving me a Windows Stop Error (0x0000007E) when I boot the DVD (immediately after "Starting Windows" with no O/S installed yet) whenever I use q35 as the machine type. 

Anyone else seeing this?  I downgraded to qemu 1.6.1 and the issue went away.

(This is prior to any vfio passthrough, I just kept running into bluescreens and was trying to whittle down the configuration to barebones)
qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga std \
-boot menu=on \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root \
-device piix4-ide,bus=pcie.0 \
-drive file=/home/windows.img,id=disk,format=raw -device ide-hd,drive=disk \
-drive file=/rpool/Public/ISO/Windows7HomePremium64bit.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd

Offline

Pages: 1 … 26 27 28 29 30 … 234

Topic closed

Board footer

Powered by FluxBB