Arch Linux

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It sounds like VGA arbitration isn't working when the AMD card is in use by the host.  What does 'dmesg | grep vgaarb' show?

When I run with Intel as the host:

[    0.333676] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[    0.333680] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none
[    0.333682] vgaarb: device added: PCI:0000:02:00.0,decodes=io+mem,owns=none,locks=none
[    0.333682] vgaarb: loaded
[    0.333683] vgaarb: bridge control possible 0000:02:00.0
[    0.333683] vgaarb: bridge control possible 0000:01:00.0
[    0.333684] vgaarb: no bridge control possible 0000:00:02.0
[    2.855092] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=none
[    3.069294] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io

When I run with AMD as the host:

[    0.330061] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
[    0.330064] vgaarb: device added: PCI:0000:02:00.0,decodes=io+mem,owns=none,locks=none
[    0.330065] vgaarb: loaded
[    0.330065] vgaarb: bridge control possible 0000:02:00.0
[    0.330066] vgaarb: bridge control possible 0000:01:00.0
[    2.609390] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=none

Doesn't seem too much different. Anything catch your eye?

Yep, looks broken to me.  IIRC, without the i915 patch you'd see 00:02.0 setting decodes=none as well.  So my guess to what is happening here is that radeon declares that it opts out of VGA arbitration by setting decodes=none.  We probably then attempt to make use of VGA on the Nvidia card, call into the arbiter to  get the device, it looks at the other registered devices, sees none of them decode VGA space and lets us proceed without actually changing anything.  You might be able to see this in lspci -vvv, look for the bridge with VGA+ set and tell us whether it's the root port above the radeon or nvidia (after you've attempted to start the guest).  This might be an arbiter bug rather than a radeon bug, I'll need to look.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Slabity

Member

Registered: 2013-12-29

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yep, looks broken to me.  IIRC, without the i915 patch you'd see 00:02.0 setting decodes=none as well.  So my guess to what is happening here is that radeon declares that it opts out of VGA arbitration by setting decodes=none.  We probably then attempt to make use of VGA on the Nvidia card, call into the arbiter to  get the device, it looks at the other registered devices, sees none of them decode VGA space and lets us proceed without actually changing anything.  You might be able to see this in lspci -vvv, look for the bridge with VGA+ set and tell us whether it's the root port above the radeon or nvidia (after you've attempted to start the guest).  This might be an arbiter bug rather than a radeon bug, I'll need to look.

Well thank god we finally found the issue, even if I have no idea what the VGA arbiter is.

Here's my relevant lspci:

http://pastebin.com/4YHn4eH0

Thanks again for the quick response. Sorry about being pushy earlier, I'm still a bit put around from surgery yesterday.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yep, looks broken to me.  IIRC, without the i915 patch you'd see 00:02.0 setting decodes=none as well.  So my guess to what is happening here is that radeon declares that it opts out of VGA arbitration by setting decodes=none.  We probably then attempt to make use of VGA on the Nvidia card, call into the arbiter to  get the device, it looks at the other registered devices, sees none of them decode VGA space and lets us proceed without actually changing anything.  You might be able to see this in lspci -vvv, look for the bridge with VGA+ set and tell us whether it's the root port above the radeon or nvidia (after you've attempted to start the guest).  This might be an arbiter bug rather than a radeon bug, I'll need to look.

Well thank god we finally found the issue, even if I have no idea what the VGA arbiter is.

Here's my relevant lspci:

http://pastebin.com/4YHn4eH0

Thanks again for the quick response. Sorry about being pushy earlier, I'm still a bit put around from surgery yesterday.

VGA+ is set on 00:01.0 which is the root port for the AMD card, so VGA accesses are going there, not to the Nvidia card.  I'll take a look.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yep, looks broken to me.  IIRC, without the i915 patch you'd see 00:02.0 setting decodes=none as well.  So my guess to what is happening here is that radeon declares that it opts out of VGA arbitration by setting decodes=none.  We probably then attempt to make use of VGA on the Nvidia card, call into the arbiter to  get the device, it looks at the other registered devices, sees none of them decode VGA space and lets us proceed without actually changing anything.  You might be able to see this in lspci -vvv, look for the bridge with VGA+ set and tell us whether it's the root port above the radeon or nvidia (after you've attempted to start the guest).  This might be an arbiter bug rather than a radeon bug, I'll need to look.

Well thank god we finally found the issue, even if I have no idea what the VGA arbiter is.

Here's my relevant lspci:

http://pastebin.com/4YHn4eH0

Thanks again for the quick response. Sorry about being pushy earlier, I'm still a bit put around from surgery yesterday.

VGA+ is set on 00:01.0 which is the root port for the AMD card, so VGA accesses are going there, not to the Nvidia card.  I'll take a look.

BTW, if you want to make it work in the meantime, try this:

vga-clear.sh

#!/bin/sh
CTRL=$(setpci -s $1 3e.w)
CTRL=$(printf %04x $(( 0x$CTRL & ~0x8 )))
setpci -s $1 3e.w=$CTRL

vga-set.sh

#!/bin/sh
CTRL=$(setpci -s $1 3e.w)
CTRL=$(printf %04x $(( 0x$CTRL | 0x8 )))
setpci -s $1 3e.w=$CTRL

Then before running the guest, do:

sudo vga-clear.sh 00:01.0
sudo vga-set.sh 00:01.1

This is only a workaround and assumes that radeon isn't just lying about no longer needing VGA access.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Slabity

Member

Registered: 2013-12-29

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BTW, if you want to make it work in the meantime, try this:

vga-clear.sh

#!/bin/sh
CTRL=$(setpci -s $1 3e.w)
CTRL=$(printf %04x $(( 0x$CTRL & ~0x8 )))
setpci -s $1 3e.w=$CTRL

vga-set.sh

#!/bin/sh
CTRL=$(setpci -s $1 3e.w)
CTRL=$(printf %04x $(( 0x$CTRL | 0x8 )))
setpci -s $1 3e.w=$CTRL

Then before running the guest, do:

sudo vga-clear.sh 00:01.0
sudo vga-set.sh 00:01.1

This is only a workaround and assumes that radeon isn't just lying about no longer needing VGA access.

Wow, that works perfectly with the test script! Thank you so much!

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Wow, that works perfectly with the test script! Thank you so much!

Does radeon still work ok for the host?

Anyone using radeon on the host (APU users) may want to try this too.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Slabity

Member

Registered: 2013-12-29

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Wow, that works perfectly with the test script! Thank you so much!

Does radeon still work ok for the host?

Anyone using radeon on the host (APU users) may want to try this too.

Yep, my AMD card didn't even noticeably react through the process.

It's a good thing too, my monitor doesn't like the output that the graphics card gives for some reason...

Now I just gotta get this set up with virsh.

HPvirt

Member

Registered: 2014-05-23

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everybody

That post is AWESOME! It fits perfectly in what I was wondering these last years: playing in windows (or other tasks without Linux counterpart, as BluRay reading) and use Linux for everyday tasks without to reboot everytime I need to go from one to the other.

I played with Virtual Box and use it for testing Linux, but gaming is a no way.
I've seen Xen, but it's lack of support for NVidia mainstream videocards slowed me.

KVM seems the way to go for, thanks to you nbhs and this topic.

Before buying some pieces, I would like to have your opinion on them.
I'm planning to buy:
- AsRock Z97 Extreme6
- Core i7-4790 (Haswell)
Both supports VT-d.

I already have 16Go of Ram and a NVidia 680.

My wish:
- Use Linux Mint as host with i7 IGP to display onto a Dell U3014 (DisplayPort) with the Intel LAN
- Use Win7 as guest with GTX 680 to display onto a Dell 3007 (DVI Dual Link) with the Realtek LAN
--> Passthrough for to Win7: GTX 680, Realtek LAN, the ASMedia USB3 controller and the ASMedia SATA Controler.

Linux and Win7 would be installed on 2 different SSDs.

I've read like 25 pages of the topic, and planned to read all. But I've an opportunity to buy these 2 pieces at a good price, so I ask now.

At first glance, does anyone see an impossibility in my project?

I think that I can launch the Win7 VM with more or less memory (playing with -m value of qemu), depending on tasks I planned to do on Win7:
- FullHD Video editing => 12Go or more
- Gaming => 4-8Go
For my Win7 it would be like adding or removing ram. I'm right?
Later, I planned to go to 24 or 32Go Ram if 16Go are not enough.

thanks

(And I beg your pardon for my poor english wording )

Last edited by HPvirt (2014-05-23 20:49:36)

---

Hardware: ASRock Z97 Extreme 6 - Intel Core i7 4790 - NVidia GTX680
Host: Linux Mint 17 Qiana, Kernel 3.15.8 (custom)
Guest: Windows Seven Ultimate SP1

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everybody

That post is AWESOME! It fits perfectly in what I was wondering these last years: playing in windows (or other tasks without Linux counterpart, as BluRay reading) and use Linux for everyday tasks without to reboot everytime I need to go from one to the other.

I played with Virtual Box and use it for testing Linux, but gaming is a no way.
I've seen Xen, but it's lack of support for NVidia mainstream videocards slowed me.

KVM seems the way to go for, thanks to you nbhs and this topic.

Before buying some pieces, I would like to have your opinion on them.
I'm planning to buy:
- AsRock Z97 Extreme6
- Core i7-4790 (Haswell)
Both supports VT-d.

I already have 16Go of Ram and a NVidia 680.

My wish:
- Use Linux Mint as host with i7 IGP to display onto a Dell U3014 (DisplayPort) with the Intel LAN
- Use Win7 as guest with GTX 680 to display onto a Dell 3007 (DVI Dual Link) with the Realtek LAN
--> Passthrough for to Win7: GTX 680, Realtek LAN, the ASMedia USB3 controller and the ASMedia SATA Controler.

Linux and Win7 would be installed on 2 different SSDs.

I've read like 25 pages of the topic, and planned to read all. But I've an opportunity to buy these 2 pieces at a good price, so I ask now.

At first glance, does anyone see an impossibility in my project?

I think that I can launch the Win7 VM with more or less memory (playing with -m value of qemu), depending on tasks I planned to do on Win7:
- FullHD Video editing => 12Go or more
- Gaming => 4-8Go
For my Win7 it would be like adding or removing ram. I'm right?
Later, I planned to go to 24 or 32Go Ram if 16Go are not enough.

thanks

I would not recommend doing device assignment of a Realtek LAN, I've only recently made it work and it's just scary.  IMHO there's not sufficient value in passing LAN or disk controllers to a guest in a home environment.  Virtio LAN and disk controllers will provide you with 90% of the performance with less hassle.  I don't know if you're asking about dynamically changing memory, but you can certainly adjust the memory for a guest between boots.  If you plan to use IGD for the host, you will need the i915 patches.  I don't know the state of Z97 wrt the PCH root port ACS support, so you may need the ACS override patches as well.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Slabity

Member

Registered: 2013-12-29

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

To add what aw said, I recently just got the 4790 when I found out the 4770K did not support Vt-d.

The only reason I got it was because it was on sale for cheaper than the normal 4770. I've only recently compared the two, and honestly it's not worth the extra $15. Hell, it's not even worth an extra $5 in my opinion. The speed is completely unnoticeable, as the only difference is the clock speed. Unless you can find the 4790 someplace cheaper than the 4770, stick with the 4770. I will absolutely guarantee that you will not notice any.

Second, that motherboard is designed for overclocking and high end gaming. The non-k series can not be overclocked, and if you're gaming in a VM then I doubt you'll be taking advantage of the 'gamer features' that the motherboard is flaunting.

I'm not saying that you shouldn't get these things, but there are cheaper options that'll suit your needs perfectly fine.

Last edited by Slabity (2014-05-23 21:07:27)

HPvirt

Member

Registered: 2014-05-23

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I would not recommend doing device assignment of a Realtek LAN, I've only recently made it work and it's just scary.  IMHO there's not sufficient value in passing LAN or disk controllers to a guest in a home environment.  Virtio LAN and disk controllers will provide you with 90% of the performance with less hassle.

Device management under Linux is not the domain I have the most knowledge in. But I'm eager to learn.

If I do device assignment of the Intel LAN, it would be easier?
Performance is a need I have.
But, I will try anyway, for the challenge and to learn.

I don't know if you're asking about dynamically changing memory, but you can certainly adjust the memory for a guest between boots.

It was at boot time. It seems clear it's possible, but I prefer ask and not assume anything.

If you plan to use IGD for the host, you will need the i915 patches.  I don't know the state of Z97 wrt the PCH root port ACS support, so you may need the ACS override patches as well.

Yes the IGP will be for the host. I planned to have 2 keayboards and 2 mouses: my son will be able to play while I do something else. And if the need to have a secondary GPU arises, I'll buy another, but right now, my wallet and my banker would scream if I buy a top GPU.

---

Hardware: ASRock Z97 Extreme 6 - Intel Core i7 4790 - NVidia GTX680
Host: Linux Mint 17 Qiana, Kernel 3.15.8 (custom)
Guest: Windows Seven Ultimate SP1

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I would not recommend doing device assignment of a Realtek LAN, I've only recently made it work and it's just scary.  IMHO there's not sufficient value in passing LAN or disk controllers to a guest in a home environment.  Virtio LAN and disk controllers will provide you with 90% of the performance with less hassle.

Device management under Linux is not the domain I have the most knowledge in. But I'm eager to learn.

If I do device assignment of the Intel LAN, it would be easier?
Performance is a need I have.
But, I will try anyway, for the challenge and to learn.

Yes, Intel NICs are more amenable to device assignment, but really virtio can easily handle gigabit networks.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BTW, if you want to make it work in the meantime, try this:

vga-clear.sh

#!/bin/sh
CTRL=$(setpci -s $1 3e.w)
CTRL=$(printf %04x $(( 0x$CTRL & ~0x8 )))
setpci -s $1 3e.w=$CTRL

vga-set.sh

#!/bin/sh
CTRL=$(setpci -s $1 3e.w)
CTRL=$(printf %04x $(( 0x$CTRL | 0x8 )))
setpci -s $1 3e.w=$CTRL

Then before running the guest, do:

sudo vga-clear.sh 00:01.0
sudo vga-set.sh 00:01.1

This is only a workaround and assumes that radeon isn't just lying about no longer needing VGA access.

Wow, that works perfectly with the test script! Thank you so much!

Slabity, if you'd be willing to test it, I think this might resolve the problem (kernel patch):  http://pastebin.com/2ukyQbkG

I'll need to swap around some hardware to make my AMD card the default before I can test it (curses to Asus and their BIOS that doesn't let me select _which_ PCIe graphics card to use)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

mv0lnicky

Member

Registered: 2014-03-05

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

(curses to Asus and their BIOS that doesn't let me select _which_ PCIe graphics card to use)

Is there even such a thing? I've yet to see more options than PCI or PEG (and AGP back in the days).

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

(curses to Asus and their BIOS that doesn't let me select _which_ PCIe graphics card to use)

Is there even such a thing? I've yet to see more options than PCI or PEG (and AGP back in the days).

I've seen it.  I can't remember where, but I'm pretty sure at least one of my systems lets me do it (which means it's either Gigabyte or Lenovo).

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Slabity

Member

Registered: 2013-12-29

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity, if you'd be willing to test it, I think this might resolve the problem (kernel patch):  http://pastebin.com/2ukyQbkG

Sure! I'll try it out immediately!

I'll need to swap around some hardware to make my AMD card the default before I can test it (curses to Asus and their BIOS that doesn't let me select _which_ PCIe graphics card to use)

Blargh, I know what you mean. I always loved ASUS years ago when I was just into overclocking. But now it feels like their boards have no utility. And their technical support is hell. But I'm stuck with them for the time being.

Last edited by Slabity (2014-05-23 21:54:40)

HPvirt

Member

Registered: 2014-05-23

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

To add what aw said, I recently just got the 4790 when I found out the 4770K did not support Vt-d.

The only reason I got it was because it was on sale for cheaper than the normal 4770. I've only recently compared the two, and honestly it's not worth the extra $15. Hell, it's not even worth an extra $5 in my opinion. The speed is completely unnoticeable, as the only difference is the clock speed. Unless you can find the 4790 someplace cheaper than the 4770, stick with the 4770. I will absolutely guarantee that you will not notice any.

I'll have the 4790 cheaper than a 4770

Second, that motherboard is designed for overclocking and high end gaming. The non-k series can not be overclocked, and if you're gaming in a VM then I doubt you'll be taking advantage of the 'gamer features' that the motherboard is flaunting.
I'm not saying that you shouldn't get these things, but there are cheaper options that'll suit your needs perfectly fine.

During my search for the MB, I've seen that VT-d support is random for Asus, Gigabyte, MSI as it's not officialy supported. And when it's working, it's mostly haphazardly. Updating a bios can remove the support, which come back with the next update... or not.
So AsRock was the way to go.
As I want to passthrough Sata & LAN & USB, I need 2 controllers of each type if I've understand how the passthrough works (my flaw is perhaps here), and I don't want to buy LAN and SATA card. So choices are limited. "Gamer features" are not the reason I choose this MB.

But, I take into account all advices, so if you know a MB compliant with my needs, I'll check it and see if I can buy it in France.

Go back to the topic: I'll repost here during the configuration of my system, for help (most probably), or, if it works flawlessly, to record my conf as working

Last edited by HPvirt (2014-05-23 22:02:06)

---

Hardware: ASRock Z97 Extreme 6 - Intel Core i7 4790 - NVidia GTX680
Host: Linux Mint 17 Qiana, Kernel 3.15.8 (custom)
Guest: Windows Seven Ultimate SP1

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Slabity, if you'd be willing to test it, I think this might resolve the problem (kernel patch):  http://pastebin.com/2ukyQbkG

Sure! I'll try it out immediately!

Hmm, it's not working.  I can reproduce it now, so I'll keep poking at it.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tried in these days that I want to make CUDA works in guest OS.
I've tested many possible reasons cause CUDA can't work if guest OS is Ubuntu.

If I use Windows 7 as guest, it can work perfectly.
But if I use Ubuntu as guest, it can install graphic driver, can install CUDA SDK, can compile sample codes, but when execute CUDA application, it still gets error.

At beginnig, I thought it's driver compatible problem, so I changed to older version, and still didn't work.
Now I'm try to upgrade guest's kernel and see it work or not.

But I think out there is another problem may cause it happen.
Because when we tried to launch QEMU, KVM is required. But my KVM is installed from Ubuntu ( sudo apt-get install kvm ) , and install KVM first, then QEMU 2.0 .
From Ubuntu official version, this modules may not compatible with newer QEMU or our settings.
Is that possible?

Slabity

Member

Registered: 2013-12-29

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm, it's not working.  I can reproduce it now, so I'll keep poking at it.

Nope, it didn't work for me either. I added those scripts to the vfio-bind service though, so it seems to work perfectly fine. My Windows VM allows me to to run the passthrough and view the results on another monitor.

However, I'd like to know if it's possible to connect to the VM using virt-manager. I have my configuration file as such:

http://pastebin.com/L2Exj8bR

I see other users online have been able to achieve this sort of thing, but they do not explain how. Essentially I'd like it to run a Spice display so that I could view it from my host, but still use the passthrough card for the actual processing.

Any ideas of where to look?

Last edited by Slabity (2014-05-24 03:31:22)

Destroy

Member

Registered: 2014-05-05

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I see other users online have been able to achieve this sort of thing, but they do not explain how. Essentially I'd like it to run a Spice display so that I could view it from my host, but still use the passthrough card for the actual processing.

Any ideas of where to look?

Could you point us where did you looked that?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

However, I'd like to know if it's possible to connect to the VM using virt-manager. I have my configuration file as such:

http://pastebin.com/L2Exj8bR

I see other users online have been able to achieve this sort of thing, but they do not explain how. Essentially I'd like it to run a Spice display so that I could view it from my host, but still use the passthrough card for the actual processing.

Any ideas of where to look?

No, it's not possible with current code.  When you use GPU assignment, the guest own the display, the host has no visibility to it.  Things like Spice are out-of-band, host driven protocols.  Once you assign a GPU you're left with in-band, guest driven protocols.  You have the same ability to get to the display remotely as you do a separate physical system, which does not currently involved libvirt/virt-manager.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Slabity

Member

Registered: 2013-12-29

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I see other users online have been able to achieve this sort of thing, but they do not explain how. Essentially I'd like it to run a Spice display so that I could view it from my host, but still use the passthrough card for the actual processing.

Any ideas of where to look?

Could you point us where did you looked that?

Hmm, well it seems like I may have been sorta wrong. Here's the original video I was looking at:

https://www.youtube.com/watch?v=Qi1LdFkRzIs&html5=1

It seems he used a combination of a spice display and a VNC server to run this.

This one:

https://www.youtube.com/watch?v=37D2bRsthfI&html5=1

Used some xrandr magic and synergy to synchronize his desktops.

There were 2 more videos that I can't seem to find right now. But I don't know what sort of setup would be useful for this.

So I guess that sort of rephrases my question (since I don't think Linux can grab the video output of a passthrough card). What would you do if you were trying to accomplish this sort of thing? I'm gonna try and see if VNC is fast enough for 60 FPS output, but I suppose I'll see.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I see other users online have been able to achieve this sort of thing, but they do not explain how. Essentially I'd like it to run a Spice display so that I could view it from my host, but still use the passthrough card for the actual processing.

Any ideas of where to look?

Could you point us where did you looked that?

Hmm, well it seems like I may have been sorta wrong. Here's the original video I was looking at:

https://www.youtube.com/watch?v=Qi1LdFkRzIs&html5=1

It seems he used a combination of a spice display and a VNC server to run this.

No spice, clearly running tigervnc in the host as the view and a vnc server in the guest.  You can see the icon in the toolbar.

This one:

https://www.youtube.com/watch?v=37D2bRsthfI&html5=1

Used some xrandr magic and synergy to synchronize his desktops.

Yep, no remote display at all, xrandr releases the display, which is then assigned to a guest.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

sinny

Member

Registered: 2014-05-24

Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

i know that it is kinda wrong to bring my problems to arch linux boards, but i am pretty desperate at that point )

MB: asrock z87 pro4
CPU: intel i7 4770
OS: fedora 20

host vga: IGD (details below)

kernel (with i915 module being recompiled with i915_314.patch from linux-mainline.tar.gz in first post):

[s@localhost ~]$ uname -r
3.14.4-200.fc20.x86_64
[s@localhost ~]$ modinfo i915 | head -n1
filename:       /lib/modules/3.14.4-200.fc20.x86_64/updates/i915.ko
[s@localhost ~]$ 

couldnt have any success at all for 5 days while trying different variations, completely out of ideas for now on what may be wrong - too much outdated info out there

target guest os is win7

when trying to passthrough HD 6450:
- [test command from first post] host screen usually gets screwed (looking like pallete was overwritten randomly, i doubt X is running in indexed mode though) - fixable by switching to tty console and then back to X
- [virt-manager/kvm] Code 10 (device cannot be started) in device manager, latest catalyst drivers setup terminates while trying to detect hardware

when trying to passthrough GT 430:
- [test command from first post] seen this:

KVM internal error. Suberror: 1
emulation failure
EAX=0000b422 EBX=0000b4b4 ECX=00005dd7 EDX=000003d4
ESI=00009e48 EDI=000018d2 EBP=0000d494 ESP=000018ca
EIP=0000b4b4 EFL=00010082 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 ffffffff 00809300
CS =b4b4 000b4b40 ffffffff 00809b00
SS =0000 00000000 ffffffff 00809300
DS =0000 00000000 ffffffff 00809300
FS =0000 00000000 ffffffff 00809300
GS =0000 00000000 ffffffff 00809300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     000f6c48 00000037
IDT=     00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=20 07 20 07 20 07 20 07 20 07 20 07 20 07 20 07 20 07 20 07 <20> 07 20 07 20 07 20 07 20 07 20 07 55 aa 6d eb 4b 37 34 30 30 e9 4c 19 77 cc 56 49 44 45

- [virt-manager/kvm] Code 43 (windows stopped the device) in device manager

if someone could point me to the direction to look at, that would be great
thanks in advance
some environment info:

[s@localhost ~]$ lspci -nn | grep VGA
00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] (rev 06)
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM] [1002:6779]
05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF108 [GeForce GT 430] [10de:0de1] (rev a1)
[s@localhost ~]$

[s@localhost ~]$ sudo lspci -s0:2.0 -k
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
	Subsystem: ASRock Incorporation Device 0412
	Kernel driver in use: i915
	Kernel modules: i915
[s@localhost ~]$ sudo lspci -s1:0.0 -k
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM]
	Subsystem: Gigabyte Technology Co., Ltd Device 2549
	Kernel driver in use: vfio-pci
	Kernel modules: radeon
[s@localhost ~]$ sudo lspci -s5:0.0 -k
05:00.0 VGA compatible controller: NVIDIA Corporation GF108 [GeForce GT 430] (rev a1)
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau
[s@localhost ~]$ 

[s@localhost ~]$ readlink /sys/bus/pci/devices/0000\:01\:00.0/iommu_group
../../../../kernel/iommu_groups/1
[s@localhost ~]$ readlink /sys/bus/pci/devices/0000\:05\:00.0/iommu_group
../../../../kernel/iommu_groups/9
[s@localhost ~]$ ls /dev/vfio/
total 0
drwxr-xr-x  2 root root      100 May 24 11:22 .
drwxr-xr-x 22 root root     3780 May 24 12:31 ..
crw-------  1 root root 247,   0 May 24 11:22 1
crw-------  1 root root 247,   1 May 24 11:22 9
crw-rw-rw-  1 root root  10, 196 May 24 11:22 vfio
[s@localhost ~]$ 

[s@localhost ~]$ cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-3.14.4-200.fc20.x86_64 root=UUID=588911d3-23f7-4544-9d46-0b06b85f287c ro nouveau.modeset=0 rd.driver.blacklist=nouveau video=vesa:off pci-stub.ids=1002:6779,1002:aa98,10de:0de1,10de:0bea intel_iommu=on
[s@localhost ~]$ 

[s@localhost ~]$ tail -n100  /etc/modprobe.d/*
==> /etc/modprobe.d/kvm.conf <==
options kvm ignore_msrs=1

==> /etc/modprobe.d/pci-stub.conf <==
blacklist nouveau
blacklist radeon

==> /etc/modprobe.d/vfio_iommu_type1.conf <==
options vfio_iommu_type1 allow_unsafe_interrupts=1
[s@localhost ~]$ 

[s@localhost ~]$ sudo grep -v "^#" /etc/libvirt/qemu.conf | grep -v "^[ \t]*$"
user = "root"
group = "root"
clear_emulator_capabilities = 0
[s@localhost ~]$ 

[s@localhost ~]$ rpm -qa | grep "qemu\|seabios\|virt" | sort
ipxe-roms-qemu-20140303-1.gitff1e7fc7.fc20.noarch
libgovirt-0.3.0-3.fc20.x86_64
libvirt-client-1.2.4-1.fc20.x86_64
libvirt-daemon-1.2.4-1.fc20.x86_64
libvirt-daemon-driver-interface-1.2.4-1.fc20.x86_64
libvirt-daemon-driver-network-1.2.4-1.fc20.x86_64
libvirt-daemon-driver-nodedev-1.2.4-1.fc20.x86_64
libvirt-daemon-driver-nwfilter-1.2.4-1.fc20.x86_64
libvirt-daemon-driver-qemu-1.2.4-1.fc20.x86_64
libvirt-daemon-driver-secret-1.2.4-1.fc20.x86_64
libvirt-daemon-driver-storage-1.2.4-1.fc20.x86_64
libvirt-daemon-kvm-1.2.4-1.fc20.x86_64
libvirt-glib-0.1.7-2.fc20.x86_64
libvirt-python-1.2.4-1.fc20.x86_64
qemu-common-2.0.0-4.fc20.x86_64
qemu-img-2.0.0-4.fc20.x86_64
qemu-kvm-2.0.0-4.fc20.x86_64
qemu-system-x86-2.0.0-4.fc20.x86_64
seabios-bin-1.7.4-5.fc20.noarch
virt-install-1.0.1-3.fc20.noarch
virt-manager-1.0.1-3.fc20.noarch
virt-manager-common-1.0.1-3.fc20.noarch
virt-top-1.0.8-7.fc20.x86_64
virt-viewer-0.6.0-1.fc20.x86_64
[s@localhost ~]$