Arch Linux

unixninja92

Member

Registered: 2014-05-26

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is it possible to just use one GPU? For instance, could the host give up the gpu completely and give it to a vm? Then if you needed to access the host you could ssh in.

mbroemme

Member

From: Cologne

Registered: 2014-02-05

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

$ lspci -tv | grep AMD
           +-01.0-[01]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Oland [Radeon HD 8570 / R7 240 OEM]
           |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]

00:01.0 is the upstream bridge.

Read the bridge control register

$ sudo setpci -s 00:01.0 3e.w
0018

OR 0x40 into the result

$ printf %04x $(( 0x0018 | 0x40 ))
0058

Write this to the same register:

$ sudo sudo setpci -s 00:01.0 3e.w=0058

Then restore the original value:

$ sudo sudo setpci -s 00:01.0 3e.w=0018

If the bus reset works, then the screen should have cleared and the monitor gone out of sync.  If nothing happened, then I don't know how we can reset the device.  I think one of my old GeForce cards is also impervious to bus resets.

This works fine with my R9 290X. The monitor is going blank but after starting QEMU a second time nothing happens and kernel logs the following:

[21542.837722] vfio-pci 0000:04:00.0: enabling device (0000 -> 0003)
[21542.837872] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x270
[21542.837879] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1b@0x2d0
[21542.872417] vfio-pci 0000:04:00.1: enabling device (0000 -> 0002)

Right now I'm using QEMU without the vfio_pci_reset_handler patch from the thread. Should I retry with it?

thelamer

Member

Registered: 2014-05-25

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Edit: NM I was able to build the latest qemu from source and it can use this config file, 2.0.0 from the debian repos must have a bug in it. I still get a atikmpag.sys bsod but at least I know I have tried everything. Looks like this R9 270x was not meant to be. Time to try some other cards.

Sorry, I was looking in /etc/qemu/, and it wasn't there, I forgot the connection to libvirtd.
I already have "clear_emulator_capabilities=0" in said qemu.conf, because before finding this board, I followed another tutorial and applied the last paragraph of this site:
http://fedoraproject.org/wiki/How_to_de … assignment

So, I have the same bluescreen error, but your solution somehow does not apply for me :/

//edit:
I found out, the qemu-git I built isn't looking in /etc/libvirt/ but in /etc/qemu, so I copied my qemu.conf to that location, now getting this error:

qemu-system-x86_64:/etc/qemu/qemu.conf:198: no group defined

Line 198 is

user = "root"

So I'm stuck again...

//edit2:
I played around a bit with the config, and the seems to be around the group for the QEMU processes:

# The group for QEMU processes run by the system instance. It can be
# specified in a similar way to user.
group = "root"

If I delete that line, I get a parser error, but if I set it to any group, root, my user, qemu, kvm, I just get the error above, that no group is defined.

I am currently running into the same thing, can you define this options from the command line ? I did not see anything in the man page.

Current config: (/etc/qemu/qemu.conf)

user = "root"
group = "root"
clear_emulator_capabilities = 0

Error :

qemu-system-x86_64:/etc/qemu/qemu.conf:1: no group defined

Version:

QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-4+b1), Copyright (c) 2003-2008 Fabrice Bellard

Last edited by thelamer (2014-05-26 02:41:00)

mbroemme

Member

From: Cologne

Registered: 2014-02-05

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is it possible to just use one GPU? For instance, could the host give up the gpu completely and give it to a vm? Then if you needed to access the host you could ssh in.

Yes but only if the host has not loaded any GPU related drivers (e.g. for AMD/ATI don't load radeon/fglrx and for NVidia don't load nouveau/nvidia). Without loading the drivers you can safely bind the card to vfio and use it.

Last edited by mbroemme (2014-05-26 00:08:27)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This works fine with my R9 290X. The monitor is going blank but after starting QEMU a second time nothing happens and kernel logs the following:

[21542.837722] vfio-pci 0000:04:00.0: enabling device (0000 -> 0003)
[21542.837872] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x270
[21542.837879] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1b@0x2d0
[21542.872417] vfio-pci 0000:04:00.1: enabling device (0000 -> 0002)

Right now I'm using QEMU without the vfio_pci_reset_handler patch from the thread. Should I retry with it?

You need to reboot after doing this, I should have mentioned that.  If a bus reset works, then I don't see why they change to invoke a bus reset regardless of pm-reset support doesn't work.  Can you uncomment the DEBUG_VFIO in hw/misc/vfio.c, rebuild and pastebin the log? /* #define DEBUG_VFIO */ -> #define DEBUG_VFIO  Do that with the reset patch applied.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is it possible to just use one GPU? For instance, could the host give up the gpu completely and give it to a vm? Then if you needed to access the host you could ssh in.

Yes but only if the host has not loaded any GPU related drivers (e.g. for AMD/ATI don't load radeon/fglrx and for NVidia don't load nouveau/nvidia). Without loading the drivers you can safely bind the card to vfio and use it.

I dunno about safely... but it mostly works.  Ideally you also want to detach the GPU from vgacon, which you may be able to google to figure out how to do.  Otherwise any dmesg output is potentially writing to the card while it's in use by the guest.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

mbroemme

Member

From: Cologne

Registered: 2014-02-05

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is it possible to just use one GPU? For instance, could the host give up the gpu completely and give it to a vm? Then if you needed to access the host you could ssh in.

Yes but only if the host has not loaded any GPU related drivers (e.g. for AMD/ATI don't load radeon/fglrx and for NVidia don't load nouveau/nvidia). Without loading the drivers you can safely bind the card to vfio and use it.

I dunno about safely... but it mostly works.  Ideally you also want to detach the GPU from vgacon, which you may be able to google to figure out how to do.  Otherwise any dmesg output is potentially writing to the card while it's in use by the guest.

With safely I meant I've tried it already several times on another machine and it worked very smooth.

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, aw.

I found that not only VM hangs when reboot with 2 VMs working together, but also only 1 VM working.

And I tried to shutdown, not reboot, it can back to tty successfully.
But when I try to start again with same command, it got another error message.

qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:03:00.0
Device option ROM contents are probably invalid (check dmesg).
Skip option ROM probe with rombar=0, or load from file with romfile=

It's may a reason why it hangs when reboot.
But I found that when  I reboot VM while guest OS is Windows 7, it can reboot successfully some times.
So it is very strange for this problem.......

I'll try to use 3.14.4 kernel and see it still a same problem or not.

Same problem here.  I cannot reboot or reissue the qemu command without this error.  Do I need to undo the vfio-bind stuffs?

Fixed.  I needed to download the rom file from http://www.techpowerup.com/vgabios/ .   Then add romfile=/path/to/romfile.rom to the -device line for the GPU.

Also changed the -cpu from type "host" to "qemu64"

Thanks for your solution, looks like some graphic cards need to do this as well.

unixninja92

Member

Registered: 2014-05-26

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes but only if the host has not loaded any GPU related drivers (e.g. for AMD/ATI don't load radeon/fglrx and for NVidia don't load nouveau/nvidia). Without loading the drivers you can safely bind the card to vfio and use it.

I dunno about safely... but it mostly works.  Ideally you also want to detach the GPU from vgacon, which you may be able to google to figure out how to do.  Otherwise any dmesg output is potentially writing to the card while it's in use by the guest.

With safely I meant I've tried it already several times on another machine and it worked very smooth.

My GPU is an Intel 4600. I haven't looked into detaching the GPU from vgacon yet, but I believe I have everything else set up correctly. But I keep getting this error:
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized
when I try running the test qemu stuff from the first post. Dmesg says that Intel IOMMU is enabled, and vfio-pci is loaded. When I boot my pci-stub is set as the driver for my graphics card, and running the vfio-bind script seems to have no effect.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I dunno about safely... but it mostly works.  Ideally you also want to detach the GPU from vgacon, which you may be able to google to figure out how to do.  Otherwise any dmesg output is potentially writing to the card while it's in use by the guest.

With safely I meant I've tried it already several times on another machine and it worked very smooth.

My GPU is an Intel 4600. I haven't looked into detaching the GPU from vgacon yet, but I believe I have everything else set up correctly. But I keep getting this error:
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized
when I try running the test qemu stuff from the first post. Dmesg says that Intel IOMMU is enabled, and vfio-pci is loaded. When I boot my pci-stub is set as the driver for my graphics card, and running the vfio-bind script seems to have no effect.

You don't have VT-d enabled, but it doesn't matter, assignment of an IGD device does not currently work.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

unixninja92

Member

Registered: 2014-05-26

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

My GPU is an Intel 4600. I haven't looked into detaching the GPU from vgacon yet, but I believe I have everything else set up correctly. But I keep getting this error:
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=00:02.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized
when I try running the test qemu stuff from the first post. Dmesg says that Intel IOMMU is enabled, and vfio-pci is loaded. When I boot my pci-stub is set as the driver for my graphics card, and running the vfio-bind script seems to have no effect.

You don't have VT-d enabled, but it doesn't matter, assignment of an IGD device does not currently work.

Ok, good to know. IGD stuff aside, my VT-d is enabled.

Flyser

Member

Registered: 2013-12-19

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just fyi, because I have seen the discussion come up a while back:
In my setup I am running an X server on my nvidia gpu, can stop it, boot the vm, stop the vm and start the X server again. I mainly use it to save power, because the X server can put the GPU in a low power state with no VM running that wastes CPU cycles.

The only "special" thing I had to do was to issue a bus reset (?) after shutting down the VM, which I do by:

qemu-system-x86_64 -name noop -enable-kvm -boot order=c,menu=off,reboot-timeout=0 -machine type=q35,accel=kvm -cpu host -smp 1 -m 32 -k de -vga none -display none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -net none -no-reboot

Everything else should be relatively standard.

Last edited by Flyser (2014-05-26 07:43:13)

sinny

Member

Registered: 2014-05-24

Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok, i managed to overcome my problems (more like stupidity, in fact) - the i915 module i rebuilt had different vermagic, so it failed loading at boot time, which lead to X server not having any devices to operate on. though i never would expect that X server would just freeze like that on no devices instead of just quitting.

now comes another portion of problems - successfully managed to install windows 7 64bit using cirrus vga. after that using only HD 6450 passed through downloaded and installed latest catalyst drivers (excluding CCC). now getting solid BSOD on windows login - will try with restarted host system when i get a chance for this restart.

do catalyst drivers only work on limited set of versions or is it just some random problem with my setup/configuration?

mbroemme

Member

From: Cologne

Registered: 2014-02-05

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This works fine with my R9 290X. The monitor is going blank but after starting QEMU a second time nothing happens and kernel logs the following:

[21542.837722] vfio-pci 0000:04:00.0: enabling device (0000 -> 0003)
[21542.837872] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x270
[21542.837879] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1b@0x2d0
[21542.872417] vfio-pci 0000:04:00.1: enabling device (0000 -> 0002)

Right now I'm using QEMU without the vfio_pci_reset_handler patch from the thread. Should I retry with it?

You need to reboot after doing this, I should have mentioned that.  If a bus reset works, then I don't see why they change to invoke a bus reset regardless of pm-reset support doesn't work.  Can you uncomment the DEBUG_VFIO in hw/misc/vfio.c, rebuild and pastebin the log? /* #define DEBUG_VFIO */ -> #define DEBUG_VFIO  Do that with the reset patch applied.

Okay I tested it now with the following sequence for my system:

#1 Started QEMU to boot Linux VM, load fglrx driver and start X.
#2 Graceful shutdown of VM
#3 Started QEMU and pressed STRG+C during seabios boot.
#4 Executed:

setpci -s 00:0b.0 3e.w
printf %04x $(( 0x0008 | 0x40 ))
setpci -s 00:0b.0 3e.w=0048
setpci -s 00:0b.0 3e.w=0008

After that screen goes blank
#5 Started QEMU again and now it hangs and logfile is here: http://paste2.org/wEay3vFV

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This works fine with my R9 290X. The monitor is going blank but after starting QEMU a second time nothing happens and kernel logs the following:

[21542.837722] vfio-pci 0000:04:00.0: enabling device (0000 -> 0003)
[21542.837872] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x270
[21542.837879] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1b@0x2d0
[21542.872417] vfio-pci 0000:04:00.1: enabling device (0000 -> 0002)

Right now I'm using QEMU without the vfio_pci_reset_handler patch from the thread. Should I retry with it?

You need to reboot after doing this, I should have mentioned that.  If a bus reset works, then I don't see why they change to invoke a bus reset regardless of pm-reset support doesn't work.  Can you uncomment the DEBUG_VFIO in hw/misc/vfio.c, rebuild and pastebin the log? /* #define DEBUG_VFIO */ -> #define DEBUG_VFIO  Do that with the reset patch applied.

Okay I tested it now with the following sequence for my system:

#1 Started QEMU to boot Linux VM, load fglrx driver and start X.
#2 Graceful shutdown of VM
#3 Started QEMU and pressed STRG+C during seabios boot.
#4 Executed:

setpci -s 00:0b.0 3e.w
printf %04x $(( 0x0008 | 0x40 ))
setpci -s 00:0b.0 3e.w=0048
setpci -s 00:0b.0 3e.w=0008

After that screen goes blank
#5 Started QEMU again and now it hangs and logfile is here: http://paste2.org/wEay3vFV

Nothing is expected to work after the setpci secondary bus reset, it's only a test.  Do not attempt to do anything with devices downstream of the bridge until after a reboot.  That said, I do see QEMU reporting success for doing a hot reset on the bus, so I don't see why we're seeing so many reports that reset doesn't work on these cards.

Last edited by aw (2014-05-26 15:22:46)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

mbroemme

Member

From: Cologne

Registered: 2014-02-05

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nothing is expected to work after the setpci secondary bus reset, it's only a test.  Do not attempt to do anything with devices downstream of the bridge until after a reboot.  That said, I do see QEMU reporting success for doing a hot reset on the bus, so I don't see why we're seeing so many reports that reset doesn't work on these cards.

Looks like only folks from AMD/ATI knows the magic for a proper reset.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nothing is expected to work after the setpci secondary bus reset, it's only a test.  Do not attempt to do anything with devices downstream of the bridge until after a reboot.  That said, I do see QEMU reporting success for doing a hot reset on the bus, so I don't see why we're seeing so many reports that reset doesn't work on these cards.

Looks like only folks from AMD/ATI knows the magic for a proper reset.

I hope not.  You've shown that a secondary bus reset seems to reset the card and that QEMU is doing a secondary bus reset, we need to figure out why that's not sufficient.  Does it help to provide the ROM via romfile instead of reading it from the card?  Does playing with the parameters for the reset help?  Will multiple resets help?  Does PCIe link down produce a different result?  We need someone that can debug and experiment with the card in front of them.  It might help to document what guests and what drivers result in the problem.  Does Windows with the Catalyst driver cause it?  Does Linux with the radeon driver cause it?  fglrx?  What exactly goes wrong in each case?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

mbroemme

Member

From: Cologne

Registered: 2014-02-05

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I hope not.  You've shown that a secondary bus reset seems to reset the card and that QEMU is doing a secondary bus reset, we need to figure out why that's not sufficient.  Does it help to provide the ROM via romfile instead of reading it from the card?  Does playing with the parameters for the reset help?  Will multiple resets help?  Does PCIe link down produce a different result?  We need someone that can debug and experiment with the card in front of them.  It might help to document what guests and what drivers result in the problem.  Does Windows with the Catalyst driver cause it?  Does Linux with the radeon driver cause it?  fglrx?  What exactly goes wrong in each case?

I've tried it already with romfile half an hour ago but it doesn't help. The result is the same. About which reset parameters you're talking? Basically I can debug it and I've started already to read instructions on how to use mmio trace as I believe fglrlx did some magic on unloading the module because once I unload the fglrx module before rebooting the VM, everything works fine. So right now I (mabye other as well with R9 series) have the following situation:

#1 Using Linux VM with radeon driver:

- This works fine on several reboots, but I guess the test case is not very meaningful because R9 series has only KMS working and everything on-top (Glamor, X, DRI) is unaccelerated.

#2 Using Linux VM with fglrx driver:

- Works on first boot only if you don't unload fglrx module before VM shutdown/reboot.
- Works on multiple reboots (I've tried around 20 times) if you stop X inside VM, unload fglrx and make VM shutdown/reboot.

#3 Using Windows VM with catalyst:

- Works on first boot only if you don't eject the card before VM shutdown/reboot.
- Works on multiple reboots (also tried several times) if you eject the card inside VM from device manager. This is only workaround for Win8/8.1 as Win7 and earlier require reboot once you remove PCI/PCIe device.

This is what I have for now, maybe others have different experience.

Last edited by mbroemme (2014-05-26 18:07:49)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I hope not.  You've shown that a secondary bus reset seems to reset the card and that QEMU is doing a secondary bus reset, we need to figure out why that's not sufficient.  Does it help to provide the ROM via romfile instead of reading it from the card?  Does playing with the parameters for the reset help?  Will multiple resets help?  Does PCIe link down produce a different result?  We need someone that can debug and experiment with the card in front of them.  It might help to document what guests and what drivers result in the problem.  Does Windows with the Catalyst driver cause it?  Does Linux with the radeon driver cause it?  fglrx?  What exactly goes wrong in each case?

I've tried it already with romfile half an hour ago but it doesn't help. The result is the same. About which reset parameters you're talking? Basically I can debug it and I've started already to read instructions on how to use mmio trace as I believe fglrlx did some magic on unloading the module because once I unload the fglrx module before rebooting the VM, everything works fine. So right now I (mabye other as well with R9 series) have the following situation:

#1 Using Linux VM with radeon driver:

- This works fine on several reboots, but I guess the test case is not very meaningful because R9 series has only KMS working and everything on-top (Glamor, X, DRI) is unaccelerated.

#2 Using Linux VM with fglrx driver:

- Works on first boot only if you don't unload fglrx module before VM shutdown/reboot.
- Works on multiple reboots (I've tried around 20 times) if you stop X inside VM, unload fglrx and make VM shutdown/reboot.

#3 Using Windows VM with catalyst:

- Works on first boot only if you don't eject the card before VM shutdown/reboot.
- Works on multiple reboots (also tried several times) if you eject the card inside VM from device manager. This is only workaround for Win8/8.1 as Win7 and earlier require reboot once you remove PCI/PCIe device.

This is what I have for now, maybe others have different experience.

So both the Linux and Windows drivers from AMD are doing something to the card on shutdown/reboot that they don't do if unbinding the device prior to reboot.  My first guess would be that they put it into D3 power state in the former case, but not the latter.  We do try to wake the device back into D0 before we do a bus reset, but maybe that's not working (it works for most devices).  You could try to hide the PM capability from the guest.  Some drivers will ignore whether we expose the capability and have it hard coded, but it's worth a shot.  IIRC, to hide the capability, find vfio_add_std_cap() in qemu hw/misc/vfio.c, after the "case PCI_CAP_ID_PM:" line add a new line that does "return 0;".  You can also compare the state of the device in the host with lspci -vvv after a shutdown with the driver attached and a shutdown with the device ejected/unbound.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

mbroemme

Member

From: Cologne

Registered: 2014-02-05

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So both the Linux and Windows drivers from AMD are doing something to the card on shutdown/reboot that they don't do if unbinding the device prior to reboot.  My first guess would be that they put it into D3 power state in the former case, but not the latter.  We do try to wake the device back into D0 before we do a bus reset, but maybe that's not working (it works for most devices).  You could try to hide the PM capability from the guest.  Some drivers will ignore whether we expose the capability and have it hard coded, but it's worth a shot.  IIRC, to hide the capability, find vfio_add_std_cap() in qemu hw/misc/vfio.c, after the "case PCI_CAP_ID_PM:" line add a new line that does "return 0;".

Okay I tried this but it doesn't change the behavior. On reboot of Linux VM fglrx still oops and Windows VM get BSOD.

You can also compare the state of the device in the host with lspci -vvv after a shutdown with the driver attached and a shutdown with the device ejected/unbound.

This is the difference between "#1 boot of Linux VM, start X, stop X, poweroff" and "#2 boot of Linux VM, start X, stop X, rmmod fglrx, poweroff". Between both tests I've rebooted the host.

--- lspci.1	2014-05-26 21:48:06.460661219 +0200
+++ lspci.2	2014-05-26 21:50:23.875239015 +0200
@@ -23,7 +23,7 @@
 			ClockPM- Surprise- LLActRep- BwNot-
 		LnkCtl:	ASPM L0s L1 Enabled; RCB 64 bytes Disabled- CommClk+
 			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
-		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
+		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
 		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
 		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
 		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-

lspci.1 is after shutdown of Linux VM in #1
lspci.2 is after shutdown of Linux VM in #2

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So both the Linux and Windows drivers from AMD are doing something to the card on shutdown/reboot that they don't do if unbinding the device prior to reboot.  My first guess would be that they put it into D3 power state in the former case, but not the latter.  We do try to wake the device back into D0 before we do a bus reset, but maybe that's not working (it works for most devices).  You could try to hide the PM capability from the guest.  Some drivers will ignore whether we expose the capability and have it hard coded, but it's worth a shot.  IIRC, to hide the capability, find vfio_add_std_cap() in qemu hw/misc/vfio.c, after the "case PCI_CAP_ID_PM:" line add a new line that does "return 0;".

Okay I tried this but it doesn't change the behavior. On reboot of Linux VM fglrx still oops and Windows VM get BSOD.

You can also compare the state of the device in the host with lspci -vvv after a shutdown with the driver attached and a shutdown with the device ejected/unbound.

This is the difference between "#1 boot of Linux VM, start X, stop X, poweroff" and "#2 boot of Linux VM, start X, stop X, rmmod fglrx, poweroff". Between both tests I've rebooted the host.

--- lspci.1	2014-05-26 21:48:06.460661219 +0200
+++ lspci.2	2014-05-26 21:50:23.875239015 +0200
@@ -23,7 +23,7 @@
 			ClockPM- Surprise- LLActRep- BwNot-
 		LnkCtl:	ASPM L0s L1 Enabled; RCB 64 bytes Disabled- CommClk+
 			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
-		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
+		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
 		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
 		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
 		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-

lspci.1 is after shutdown of Linux VM in #1
lspci.2 is after shutdown of Linux VM in #2

#1 is the case that fails and #2 is the case that works, right?  The link status looks a lot more correct in #2.  The next thing I would try would be testing whether we can poke at the PCIe capability registers on the device to get it to state #2.  So I would look at the LnkSta just after boot, repeat the steps for #1, do the setpci bus reset, then look at the LnkSta for the device again.  Is it any closer to the state just after boot or state #2 (several things should go to reset values after bus reset, concentrate on PCIe status registers)?  If not, we can try to retrain the link.  Here's another setpci sequence, do this after doing the setpci bus reset:

(assuming GPU is 2:00.0, replace with your device address)

$ sudo lspci -v -s 2:00.0 | grep Express
	Capabilities: [58] Express Legacy Endpoint, MSI 00

The PCIe capability is therefore at 0x58, the link control register is offset 0x10 into the PCIe capability, therefore

printf %x $(( 0x58 + 0x10 ))
68

Read the original value of this register

sudo setpci -s 2:00.0 68.w
0040

The retrain bit is bit 5 (0x20)

printf %x $(( 0x40 | 0x20 ))
60

Write this value back

sudo setpci -s 2:00.0 68.w=60

This bit doesn't need to be cleared, re-read LnkSta with lspci and see if anything has changed.  Regardless of whether it has, a reboot is required before trying to use the device, so don't bother trying to start QEMU again.

If it doesn't do anything, then we might need to look at a VFIO debug trace of case #1 to see if we can get any clues what the driver is doing to the device to get it into that state.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

mbroemme

Member

From: Cologne

Registered: 2014-02-05

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#1 is the case that fails and #2 is the case that works, right?

Yes.

The link status looks a lot more correct in #2.

Yes.

The next thing I would try would be testing whether we can poke at the PCIe capability registers on the device to get it to state #2.  So I would look at the LnkSta just after boot, repeat the steps for #1, do the setpci bus reset, then look at the LnkSta for the device again.  Is it any closer to the state just after boot or state #2 (several things should go to reset values after bus reset, concentrate on PCIe status registers)?

Okay I did the setpci bus reset using the following command sequence, valid for my system:

setpci -s 00:0b.0 3e.w
printf %04x $(( 0x0008 | 0x40 ))
setpci -s 00:0b.0 3e.w=0048
setpci -s 00:0b.0 3e.w=0008

I'll hope it was right as you mentioned in previous post "secondary bus reset", after that I get original LnkSta back:

LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

And of course I cannot start the VM via QEMU as mentioned by you.

If not, we can try to retrain the link.  Here's another setpci sequence, do this after doing the setpci bus reset:

(assuming GPU is 2:00.0, replace with your device address)

$ sudo lspci -v -s 2:00.0 | grep Express
	Capabilities: [58] Express Legacy Endpoint, MSI 00

The PCIe capability is therefore at 0x58, the link control register is offset 0x10 into the PCIe capability, therefore

printf %x $(( 0x58 + 0x10 ))
68

Read the original value of this register

sudo setpci -s 2:00.0 68.w
0040

The retrain bit is bit 5 (0x20)

printf %x $(( 0x40 | 0x20 ))
60

Write this value back

sudo setpci -s 2:00.0 68.w=60

This bit doesn't need to be cleared, re-read LnkSta with lspci and see if anything has changed.  Regardless of whether it has, a reboot is required before trying to use the device, so don't bother trying to start QEMU again.

If it doesn't do anything, then we might need to look at a VFIO debug trace of case #1 to see if we can get any clues what the driver is doing to the device to get it into that state.

This was not needed as bus reset above already restored LinkSta.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#1 is the case that fails and #2 is the case that works, right?

Yes.

The link status looks a lot more correct in #2.

Yes.

The next thing I would try would be testing whether we can poke at the PCIe capability registers on the device to get it to state #2.  So I would look at the LnkSta just after boot, repeat the steps for #1, do the setpci bus reset, then look at the LnkSta for the device again.  Is it any closer to the state just after boot or state #2 (several things should go to reset values after bus reset, concentrate on PCIe status registers)?

Okay I did the setpci bus reset using the following command sequence, valid for my system:

setpci -s 00:0b.0 3e.w
printf %04x $(( 0x0008 | 0x40 ))
setpci -s 00:0b.0 3e.w=0048
setpci -s 00:0b.0 3e.w=0008

I'll hope it was right as you mentioned in previous post "secondary bus reset", after that I get original LnkSta back:

LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

And of course I cannot start the VM via QEMU as mentioned by you.

Ok, so do we get the same result with QEMU?  Setup the #1 case again, then start QEMU again with no disk so the VM only boots up to seabios.  Quit QEMU.  What's the lspci state of the device at that point compared to clean boot?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

mbroemme

Member

From: Cologne

Registered: 2014-02-05

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, so do we get the same result with QEMU?  Setup the #1 case again, then start QEMU again with no disk so the VM only boots up to seabios.  Quit QEMU.  What's the lspci state of the device at that point compared to clean boot?

EDIT: Oops I was wrong with diff. I've edited the lspci and updated it!

I created new lspci outputs. This is the diff after this test case:

--- lspci.3	2014-05-27 00:23:49.016539086 +0200
+++ lspci.4	2014-05-27 00:27:07.796718133 +0200
@@ -1,12 +1,12 @@
 04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT [Radeon HD 8970] (prog-if 00 [VGA controller])
 	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0b00
-	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
+	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
 	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
-	Interrupt: pin A routed to IRQ 10
-	Region 0: Memory at c0000000 (64-bit, prefetchable) [disabled] [size=256M]
-	Region 2: Memory at df800000 (64-bit, prefetchable) [disabled] [size=8M]
-	Region 4: I/O ports at ce00 [disabled] [size=256]
-	Region 5: Memory at fdc80000 (32-bit, non-prefetchable) [disabled] [size=256K]
+	Interrupt: pin A routed to IRQ 19
+	Region 0: Memory at c0000000 (64-bit, prefetchable) [size=256M]
+	Region 2: Memory at df800000 (64-bit, prefetchable) [size=8M]
+	Region 4: I/O ports at ce00 [size=256]
+	Region 5: Memory at fdc80000 (32-bit, non-prefetchable) [size=256K]
 	[virtual] Expansion ROM at d0000000 [disabled] [size=128K]
 	Capabilities: [48] Vendor Specific Information: Len=08 <?>
 	Capabilities: [50] Power Management version 3
@@ -44,7 +44,7 @@
 	Capabilities: [270 v1] #19
 	Capabilities: [2b0 v1] Address Translation Service (ATS)
 		ATSCap:	Invalidate Queue Depth: 00
-		ATSCtl:	Enable+, Smallest Translation Unit: 00
+		ATSCtl:	Enable-, Smallest Translation Unit: 00
 	Capabilities: [2c0 v1] #13
 	Capabilities: [2d0 v1] #1b
 	Kernel driver in use: vfio-pci

Last edited by mbroemme (2014-05-26 22:26:26)

mbroemme

Member

From: Cologne

Registered: 2014-02-05

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm... If I extend test case #1 by the following scenario I get another lspci diff:

#1 Test case from original #1
#2 Suspend/resume host
#3 lspci -vvv -s 04:00.0 > lspci.5
#4 Repeat test case from original #1
#5 Start QEMU without any drive, wait until seabios shows "No bootable device." and press STRG+C
#6 lspci -vvv -s 04:00.0 > lspci.6

The diff is the following:

--- lspci.5	2014-05-27 00:34:37.703926234 +0200
+++ lspci.6	2014-05-27 00:35:53.712763395 +0200
@@ -18,7 +18,7 @@
 		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
 			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
 			MaxPayload 128 bytes, MaxReadReq 512 bytes
-		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
+		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
 		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
 			ClockPM- Surprise- LLActRep- BwNot-
 		LnkCtl:	ASPM L0s L1 Enabled; RCB 64 bytes Disabled- CommClk+
@@ -38,7 +38,7 @@
 		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
 		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
 		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
-		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
+		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
 		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
 		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
 	Capabilities: [270 v1] #19