You are not logged in.

#4951 2015-04-30 01:48:36

Zerqz
Member
Registered: 2015-04-30
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:
Zerqz wrote:

Hey everyone! I have been trying to set this up on my system for hours now and have came to a standstill, lol.

After I add the kernel parameter

"pci-stub.ids=10de:13c0,10de:0fbb" 

my system fails to get passed the "Reached target Graphical Interface" tab while booting. If I remove that parameter it boots fine.

Do you have.. a second GPU that should be left for host?
What distrib, kernel and hardware are you using? You sure that IOMMU is on?


I have 2 gtx980's the first in 01:00.0 and the second in 07:00.0.
and a x64 AMD cpu

Running arch, kernel 4.0.1-1, and iommu=on is in my kernel parameters as well. It is on.

and I have added pci-stub into /etc/mkinitcpio.conf and rebuilt and all that.

Last edited by Zerqz (2015-04-30 02:36:20)

Offline

#4952 2015-04-30 05:21:24

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Zerqz

The vfio messages suggest the IOMMU is not enabled or not present.  Generally the correct option on AMD is amd_iommu=on.  If you have two identical cards then the pci-stub.ids= option is maybe not the best choice for you as it will claim both devices.  This is why you're failing to start graphics on the host, pci-stub owns the device rather than the host graphics driver.  You'll either need to create some scripts to unbind the host device from pci-stub and give it to the host, something like:

echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/unbind
echo 10de:13c0 > /sys/bus/pci/drivers/pci-stub/remove_id
echo 0000:01:00.0 > /sys/bus/pci/drivers_probe
echo 10de:13c0 > /sys/bus/pci/drivers/pci-stub/new_id

(repeat for audio function/ID)  You could actually replace the drivers_probe line with modprobe --ignore-install of the host driver and specify the script as an install option in modprobe.d.  Another option that you'll find used previously in this very long thread is to use the xen-pciback module in place of pci-stub since it has the ability to claim devices based on bus address rather than device IDs.

You also seem to be trying to use VGA-mode assignment (I'd recommend UEFI/OVMF), but you're not specifying x-vga=on as an option for the vfio-pci device.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4953 2015-04-30 05:28:36

Zerqz
Member
Registered: 2015-04-30
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@Zerqz

The vfio messages suggest the IOMMU is not enabled or not present.  Generally the correct option on AMD is amd_iommu=on.  If you have two identical cards then the pci-stub.ids= option is maybe not the best choice for you as it will claim both devices.  This is why you're failing to start graphics on the host, pci-stub owns the device rather than the host graphics driver.  You'll either need to create some scripts to unbind the host device from pci-stub and give it to the host, something like:

echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/unbind
echo 10de:13c0 > /sys/bus/pci/drivers/pci-stub/remove_id
echo 0000:01:00.0 > /sys/bus/pci/drivers_probe
echo 10de:13c0 > /sys/bus/pci/drivers/pci-stub/new_id

(repeat for audio function/ID)  You could actually replace the drivers_probe line with modprobe --ignore-install of the host driver and specify the script as an install option in modprobe.d.  Another option that you'll find used previously in this very long thread is to use the xen-pciback module in place of pci-stub since it has the ability to claim devices based on bus address rather than device IDs.

You also seem to be trying to use VGA-mode assignment (I'd recommend UEFI/OVMF), but you're not specifying x-vga=on as an option for the vfio-pci device.

I thought that pci-stub.ids=10de:13c0,10de:0fbb only pointed to the card in pci slot 7, not slot 1, because the numbers beside the card in slot 1 are all different. That's why I can't just blacklist nvidia, because I still need the nvidia drivers for the first 980.

I did do some command to check if amd-vi was on and it was successful, but I will change to amd_iommu=on to see.

Offline

#4954 2015-04-30 05:34:10

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Zerqz

Then perhaps you don't really have two GTX980s, because they would both be 10de:13c0.  Maybe you could provide us with the actual data you're working from.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4955 2015-04-30 07:05:33

novist
Member
Registered: 2014-03-14
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Since some time ago USB ports on the front of PC stopped working when booted. They work fine when booting though (can boot from usb drive connected to said ports). I noticed this:

On GRUB command line:

pci-stub.ids=1002:4397,1002:4396,1002:6810,1002:aab0
~ % lspci -n | grep 1002:4397
00:12.0 0c03: 1002:4397
00:13.0 0c03: 1002:4397
00:16.0 0c03: 1002:4397
~ % lspci -n | grep 1002:4396
00:12.2 0c03: 1002:4396
00:13.2 0c03: 1002:4396
00:16.2 0c03: 1002:4396
~ % lspci -n | grep 1002:6810
06:00.0 0300: 1002:6810
~ % lspci -n | grep 1002:aab0
06:00.1 0403: 1002:aab0
~ % lspci
...
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
...
00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
...
06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao XT [Radeon R9 270X]
06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
...

How can it be that multiple usb controllers have same id (is it called id?) 1002:4396? It makes me believe assigning 1002:4396 to pci-stub kills my front usb ports as well as card reader. Does anyone have any idea how could i diagnose and possibly fix this? In the past during manual testing i found out that ports i want passed through to VM are on controllers 00:13.0 and 00:13.2.

Offline

#4956 2015-04-30 07:14:13

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:

Hi, all:

my environment:
kernel: 3.18.6 with i915 patches.
qemu: 2.1.3
arch: Intel Xeon series
type: legacy VGA passthrough

There is a problem when I passthrough AMD HD7850 card to VM. the catd can passthrough successfully.
but when I installed driver, the vm crashed with follow error:
pcieport 0000:00:03.0: AER: Multiple Uncorrected (Fatal) error received: id=0000
vfio-pci 0000:04:00.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Unaccessible, id=0400(Unregistered Agent ID)
vfio-pci 0000:04:00.1: PCIe Bus Error: severity=Uncorrected (Fatal), type=Unaccessible, id=0401(Unregistered Agent ID)

qemu-kvm-passthrough: vfio_err_notifier_handler(0000:04:00.1) Unrecoverable error detected.  Please collect any data possible and then kill the guest
qemu-kvm-passthrough: vfio_err_notifier_handler(0000:04:00.0) Unrecoverable error detected.  Please collect any data possible and then kill the guest
qemu-kvm-passthrough: vfio_err_notifier_handler(0000:04:00.1) Unrecoverable error detected.  Please collect any data possible and then kill the guest
qemu-kvm-passthrough: vfio_err_notifier_handler(0000:04:00.0) Unrecoverable error detected.  Please collect any data possible and then kill the guest

If I just passthrough the VGA card, without hdmi audio, the VM worked well with driver. I don't know why, somebody can explain to me, thanks.

Just ignored the HDMI Audio completely (or adding it to pcie.0 bus works as well).

Offline

#4957 2015-04-30 07:14:51

erganzi
Member
Registered: 2014-07-09
Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:

Hi, aw
the AMD Firepro R5000 can passthrough successfully, but when I install vga driver, it restart with BSOD like atikmpag.sys error

my kernel:
linux-3.18.6
qemu:
qemu-2.1.3

commandline:
/usr/libexec/qemu-kvm-passthrough -cpu host -smp 1,sockets=1,cores=1,threads=1 \
-m 8192 -M q35 -enable-kvm \
-rtc base=localtime,clock=host \
-acpitable file=/usr/share/qemu/win-pre-install-with-slic.bin \
-drive file=/var/lib/libvirt/images/fwq_clean_win7_64bit-patched.img,id=virtio-scsi-disk0 \
-device virtio-blk-pci,drive=virtio-scsi-disk0 \
-net nic,model=virtio,macaddr=06:d8:e8:00:00:1b,addr=0x3 \
-net tap,ifname=tap0,script=/root/qemu-ifup \
-vga none -display none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=07:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=06:00.0,bus=root.1,addr=00.2 \
-device vfio-pci,host=06:00.1,bus=root.1,addr=00.3 \
-device vfio-pci,host=06:00.2,bus=root.1,addr=00.4 \
-monitor stdio

just ignored the HDMI Audio completely (or adding it to pcie.0 bus works as well).

Offline

#4958 2015-04-30 08:59:00

fseven
Member
Registered: 2015-04-15
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone found a way to disable/power down the passthrough device when the VM is not in use?
Having an additional graphics card drawing power and producing heat/noise isn't ideal.

Offline

#4959 2015-04-30 14:16:24

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

fseven wrote:

Has anyone found a way to disable/power down the passthrough device when the VM is not in use?
Having an additional graphics card drawing power and producing heat/noise isn't ideal.

There's no way to do this without hardware support for slot power control.  The best we can do generically is put the device into a D3 power state.  The vfio-pci kernel driver will do this automatically for unused device bound to the driver starting in v4.1-rc1.  Don't expect a massive difference, but it saves a few watts.  Unfortunately it doesn't do anything about the fan speed unless the device goes to a lower fan speed in D3.  We would need device specific power management drivers to do anything more effective, but the better alternative to that is probably to make the host graphics drivers more robust with binding and unbinding devices so we can simply give the device back to the host and let the native driver put it into a low power state.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4960 2015-04-30 14:26:53

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:

How can it be that multiple usb controllers have same id (is it called id?) 1002:4396? It makes me believe assigning 1002:4396 to pci-stub kills my front usb ports as well as card reader. Does anyone have any idea how could i diagnose and possibly fix this? In the past during manual testing i found out that ports i want passed through to VM are on controllers 00:13.0 and 00:13.2.

See previous suggestions to Zerqz about pci-stub and multiple devices with the same device ID.  You either need to switch to xen-pciback, use a script to better manage pci-stub, or since these are just usb controllers and manage unbinding and re-binding to host drivers just fine, dynamically move the device to vfio-pci just for the VM, which is what libvirt would do if you're using it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4961 2015-04-30 15:04:49

Zerqz
Member
Registered: 2015-04-30
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@Zerqz

Then perhaps you don't really have two GTX980s, because they would both be 10de:13c0.  Maybe you could provide us with the actual data you're working from.

lspci

01:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 980] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)
02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
04:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
05:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 04)
06:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
07:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 980] (rev a1)
07:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)
0b:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 09)
0c:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
0d:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller

lspci -n

01:00.0 0300: 10de:13c0 (rev a1)
01:00.1 0403: 10de:0fbb (rev a1)
02:00.0 0106: 1b21:0612 (rev 01)
03:00.0 0106: 1b21:0612 (rev 01)
04:00.0 0c03: 1b21:1042
05:00.0 0604: 1b21:1080 (rev 04)
06:04.0 0401: 13f6:8788
07:00.0 0300: 10de:13c0 (rev a1)
07:00.1 0403: 10de:0fbb (rev a1)
0b:00.0 0200: 10ec:8168 (rev 09)

My mistake.

I'll try to unbind it and see what happens soon as I can.

You'll either need to create some scripts to unbind the host device from pci-stub and give it to the host, something like:
echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/unbind
echo 10de:13c0 > /sys/bus/pci/drivers/pci-stub/remove_id
echo 0000:01:00.0 > /sys/bus/pci/drivers_probe
echo 10de:13c0 > /sys/bus/pci/drivers/pci-stub/new_id
(repeat for audio function/ID)  You could actually replace the drivers_probe line with modprobe --ignore-install of the host driver and specify the script as an install option in modprobe.d.  Another option that you'll find used previously in this very long thread is to use the xen-pciback module in place of pci-stub since it has the ability to claim devices based on bus address rather than device IDs.
You also seem to be trying to use VGA-mode assignment (I'd recommend UEFI/OVMF), but you're not specifying x-vga=on as an option for the vfio-pci device.

Offline

#4962 2015-04-30 15:17:02

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Off-topic a bit:
We're steadily going to page #200, and approaching 5000 posts in this thread. We really need to do something about it...

It makes me somewhat envy seeing people with two GTX980 not running SLI for maximum lulz... Would be awesome to quadrify and push them both into the VM.. Sadly, there's no quadro-counterpart yet.

Last edited by Duelist (2015-04-30 15:31:48)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4963 2015-04-30 16:20:09

noctlos
Member
Registered: 2013-08-26
Posts: 36

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I posted about this issue here, but I'm having issues with windows freezing as I'm trying to boot in ovmf. Could anyone help me?

Offline

#4964 2015-04-30 16:43:22

Omar007
Member
Registered: 2015-04-09
Posts: 368

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@noctlos; I'm having the same problem. I think I once read somewhere that it had to do with the Q35 chipset not allowing Windows 8 and later to boot properly but I can't seem to find it so I'm not certain that is still the case or how to solve it.

@Duelist; a physical Q35 chipset doesn't support SLI/CF so I guess you can't use it in the VM even if you do pass through 2 GPU's.. Did anyone try to do this?

Offline

#4965 2015-04-30 17:04:47

noctlos
Member
Registered: 2013-08-26
Posts: 36

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Omar007; I have another image on the same computer, with a similar windows installation. It boots fine under the same arguments. That it won't boot with a new image doesn't compute for me.

Edit: nevertheless, it appears that removing references to q35 chipsets resolved this. I'll do some more experimenting.

Last edited by noctlos (2015-04-30 17:09:56)

Offline

#4966 2015-04-30 21:05:52

chillum
Member
Registered: 2014-07-30
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi! I'm using Linux Sabayon 3.9.0-sabayon

sysfs (/sys/) in this system isn't compatible with the sysfs folder structure used in vfio-bind on the front page; It used to work on my gentoo system, but then I upgraded to Sabayon instead of trying to figure zfs (and many other reasons), now I'm stuck, quite an important bit of my work was done on Windoze, so I'd like it back (and well, games of course), any help? please?

Offline

#4967 2015-04-30 21:17:00

Bronek
Member
From: London
Registered: 2014-02-14
Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw this time different card seem to have received communication intended for FirePro W7100 at 82:00 (see lspci -nnvt output above for my configuration)

[175157.337000] vfio-pci 0000:82:00.0: irq 160 for MSI/MSI-X
[175157.354768] vfio-pci 0000:81:00.0: irq 161 for MSI/MSI-X
[175157.354957] vfio-pci 0000:81:00.0: irq 161 for MSI/MSI-X
[175157.354969] vfio-pci 0000:81:00.0: irq 162 for MSI/MSI-X
[175157.355076] vfio-pci 0000:81:00.0: irq 161 for MSI/MSI-X
[175157.355082] vfio-pci 0000:81:00.0: irq 162 for MSI/MSI-X
[175157.355088] vfio-pci 0000:81:00.0: irq 163 for MSI/MSI-X
[175157.355212] vfio-pci 0000:81:00.0: irq 161 for MSI/MSI-X
[175157.355218] vfio-pci 0000:81:00.0: irq 162 for MSI/MSI-X
[175157.355223] vfio-pci 0000:81:00.0: irq 163 for MSI/MSI-X
[175157.355228] vfio-pci 0000:81:00.0: irq 164 for MSI/MSI-X
[175157.355379] vfio-pci 0000:81:00.0: irq 161 for MSI/MSI-X
[175157.355385] vfio-pci 0000:81:00.0: irq 162 for MSI/MSI-X
[175157.355390] vfio-pci 0000:81:00.0: irq 163 for MSI/MSI-X
[175157.355396] vfio-pci 0000:81:00.0: irq 164 for MSI/MSI-X
[175157.355401] vfio-pci 0000:81:00.0: irq 165 for MSI/MSI-X
[175157.355571] vfio-pci 0000:81:00.0: irq 161 for MSI/MSI-X
[175157.355577] vfio-pci 0000:81:00.0: irq 162 for MSI/MSI-X
[175157.355582] vfio-pci 0000:81:00.0: irq 163 for MSI/MSI-X
[175157.355588] vfio-pci 0000:81:00.0: irq 164 for MSI/MSI-X
[175157.355593] vfio-pci 0000:81:00.0: irq 165 for MSI/MSI-X
[175157.355599] vfio-pci 0000:81:00.0: irq 166 for MSI/MSI-X
[175157.355790] vfio-pci 0000:81:00.0: irq 161 for MSI/MSI-X
[175157.355796] vfio-pci 0000:81:00.0: irq 162 for MSI/MSI-X
[175157.355802] vfio-pci 0000:81:00.0: irq 163 for MSI/MSI-X
[175157.355807] vfio-pci 0000:81:00.0: irq 164 for MSI/MSI-X
[175157.355812] vfio-pci 0000:81:00.0: irq 165 for MSI/MSI-X
[175157.355818] vfio-pci 0000:81:00.0: irq 166 for MSI/MSI-X
[175157.355824] vfio-pci 0000:81:00.0: irq 167 for MSI/MSI-X
[175157.356036] vfio-pci 0000:81:00.0: irq 161 for MSI/MSI-X
[175157.356042] vfio-pci 0000:81:00.0: irq 162 for MSI/MSI-X
[175157.356047] vfio-pci 0000:81:00.0: irq 163 for MSI/MSI-X
[175157.356052] vfio-pci 0000:81:00.0: irq 164 for MSI/MSI-X
[175157.356057] vfio-pci 0000:81:00.0: irq 165 for MSI/MSI-X
[175157.356062] vfio-pci 0000:81:00.0: irq 166 for MSI/MSI-X
[175157.356068] vfio-pci 0000:81:00.0: irq 167 for MSI/MSI-X
[175157.356076] vfio-pci 0000:81:00.0: irq 168 for MSI/MSI-X
[175157.377567] dmar: DRHD: handling fault status reg 2
[175157.382540] dmar: DMAR:[DMA Read] Request device [81:00.0] fault addr 43e39b000
                DMAR:[fault reason 06] PTE Read access is not set
[175159.030150] usb 3-1: reset low-speed USB device number 2 using xhci_hcd
[175159.319282] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff881025d05120
[175159.319287] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff881025d050c0
[175159.319295] usb 3-1: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes
[175159.319299] usb 3-1: ep 0x82 - rounding interval to 64 microframes, ep desc says 80 microframes
[175159.607427] usb 3-1: reset low-speed USB device number 2 using xhci_hcd
[175159.896485] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff881025d05120
[175159.896489] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff881025d050c0
[175159.896495] usb 3-1: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes
[175159.896499] usb 3-1: ep 0x82 - rounding interval to 64 microframes, ep desc says 80 microframes
[175160.184610] usb 3-1: reset low-speed USB device number 2 using xhci_hcd
[175160.472711] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff881025d05120
[175160.472715] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff881025d050c0
[175160.472720] usb 3-1: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes
[175160.472723] usb 3-1: ep 0x82 - rounding interval to 64 microframes, ep desc says 80 microframes
[175160.764503] dmar: DRHD: handling fault status reg 102
[175160.769652] dmar: DMAR:[DMA Write] Request device [82:00.0] fault addr 43de38000
                DMAR:[fault reason 05] PTE Write access is not set

Offline

#4968 2015-04-30 21:21:42

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

chillum wrote:

Hi! I'm using Linux Sabayon 3.9.0-sabayon

sysfs (/sys/) in this system isn't compatible with the sysfs folder structure used in vfio-bind on the front page; It used to work on my gentoo system, but then I upgraded to Sabayon instead of trying to figure zfs (and many other reasons), now I'm stuck, quite an important bit of my work was done on Windoze, so I'd like it back (and well, games of course), any help? please?

Why is sysfs incompatible?  Is Sabayon based on a 3.9 kernel?  That's way too old for VGA assignment.  Have they messed with the sysfs mount point?  If so, they're broken, http://git.kernel.org/cgit/linux/kernel … -rules.txt

- sysfs is always at /sys
  Parsing /proc/mounts is a waste of time. Other mount points are a
  system configuration bug you should not try to solve. For test cases,
  possibly support a SYSFS_PATH environment variable to overwrite the
  application's behavior, but never try to search for sysfs. Never try
  to mount it, if you are not an early boot script.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4969 2015-04-30 21:33:54

chillum
Member
Registered: 2014-07-30
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yes, sorry, I messed about trying to find a kernel that would compile and 3.9.0 was the only one that compiled, problem solved, too old a kernel thanks!

Offline

#4970 2015-05-01 00:19:36

gg
Member
Registered: 2015-03-06
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw (also, thanks @Bronek)

I'm using your patch onto qemu v2.3.0 with kernel v4.1-rc1 (also with libvirt and OVMF) and my R9 290 will reset if I shutdown the machine and start it again, but not restart the machine reliably (see below).

This works both whether the Quadro 6000 is used by pci-stub, or vfio-pci and assigned together with the R9 290. I'm not using the ACS quirk patch for the root PCH.

-[0000:00]-+-00.0  Intel Corporation 4th Gen Core Processor DRAM Controller
           +-01.0-[01]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290]
           |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Device aac8
           +-01.1-[02]--+-00.0  NVIDIA Corporation GF100GL [Quadro 6000]
           |            \-00.1  NVIDIA Corporation GF100 High Definition Audio Controller

Before I installed the Windows drivers for the R9 290 (with only the R9 290 passed through) I was able to the restart the machine once. After installing them and going from a safe resolution to my monitor native resolution (accelerated) and then having the driver crash I haven't been able to restart (but I can shutdown and start the machine again).

After a driver crash,  when I restart I get the 'blinking cursor on black background' as mentioned by @Bronek. I'm not getting messages via dmesg similar to what @Bronek is reporting when he/she also gets the blinking cursor. Per the notes in the patch, might it be the SMC firmware is not running or in some state the patch doesn't work with?

I'm not sure why the driver has been crashing (I'd not had the issue before with the R9 290 passed through but some other variables have changed too). I've been unable to relate the driver crashes to anything yet and I don't know if that's what is preventing restarting. Being unable to restart the machine is a bit inconvenient when upgrading Windows 10 releases. I'll have to try again sometime with Windows 8.1 perhaps.

Just wanted to let you know that being able to shutdown and start the machine has made the R9 290 worlds more practical.

Thank you!!

Offline

#4971 2015-05-01 01:35:45

carterghill
Member
Registered: 2015-04-26
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm still having that code 43 issue on Windows 8.1. Here's my libvirt XML:

<domain type='kvm'>
  <name>Win</name>
  <uuid>f4f827da-ef6c-11e4-a668-8f0730658845</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-utopic'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <kvm>
      <hidden state='off'/>
    </kvm>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>SandyBridge</model>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/win8.1.qcow2'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/carter/Windows_8.1_Pro_X64_Activated.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:a7:85:28'/>
      <source network='default'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Host: Ubuntu 14.10 with GTX 560
Guest: Windows 8.1 GTX 760

Offline

#4972 2015-05-01 02:36:19

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@carterghill

Remove the <graphics> and <video> sections.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4973 2015-05-01 02:40:06

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@gg

I can't really figure out whether the patch is an improvement for you.  There's really not much difference for VM shutdown vs VM reset as far as the types of resets that occur for the device.  My testing on a Bonaire seemed to indicate that resets vs shutdown/restart had about the same success rate, ie. mostly working but occasionally not.  The SMC firmware must be running for the reset to work, otherwise it reverts to previous behavior, which in my experience gives you a black screen rather than bsod.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4974 2015-05-01 02:57:43

carterghill
Member
Registered: 2015-04-26
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@carterghill

Remove the <graphics> and <video> sections.

Sorry I should've clarified, I have tried without them as well. I only had them in just now so that I could actually use windows. Without those tags, there is still no output on my second monitor.

Last edited by carterghill (2015-05-01 04:56:15)

Offline

#4975 2015-05-01 03:48:12

gg
Member
Registered: 2015-03-06
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@gg

I can't really figure out whether the patch is an improvement for you.  There's really not much difference for VM shutdown vs VM reset as far as the types of resets that occur for the device.  My testing on a Bonaire seemed to indicate that resets vs shutdown/restart had about the same success rate, ie. mostly working but occasionally not.  The SMC firmware must be running for the reset to work, otherwise it reverts to previous behavior, which in my experience gives you a black screen rather than bsod.

Before using the patch on qemu 2.3 I was using 2.1 and was not once able to restart or shutdown|start a guest machine without suspending or rebooting the host. It seems from what I'm experiencing that the patch has finally provided a reset (under certain conditions). Is there something else in qemu, the kernel, and the vfio-pci module in particular, that might be making this possible if not the patch you provided for qemu?

Offline

Board footer

Powered by FluxBB