Arch Linux

Calrama

Member

From: Berlin

Registered: 2011-10-14

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Mostly perfomance, but I had some trouble with OVMF not recognising non-scsi devices, so I have everything on scsi now.

That's strange: I had the opposite problem.
If I want to use SCSI, will I also need "-device virtio-scsi-pci,id=scsi" option?

As far as I know, yes. This option puts a virtual SCSI bus onto the PCI bus.
After you have created this SCSI bus, you can then add devices to it like this:

-drive file=${WINDOWS_INSTALL_DISK},id=iso_install,if=none \
-device scsi-cd,bus=scsi.0,drive=iso_install

or this:

-drive file=/path/to/device,if=none,id=disk_primary \
-device scsi-hd,bus=scsi.0,drive=disk_primary

Sorry if this sound condescending, but are you sure you're on the serial console and not the compat monitor? That's what the compat monitor should be showing, not the serial console.
You should have three views to choose from (if your qemu is compiled with GTK and SDL at least):
View -> compatmonitor0
View -> serial0                       <---- This is the one
View -> parallel0

My qemu is compiled without SDL nor GTK.
The explanation of this is that qemu reside on a headless pc (Xorg not installed at all) and all the work is done through ssh.

If you don't mind me asking this, what is the use-case for this? VNC (which IIRC is what you're using) isn't even fast enough to use it for high-quality video transmission inside one PC (I tried) from guest to host. Not to mention the latency and spikes.

EDIT: just found this, so I can say I'm accessing the qemu monitor console (hope it can be useful).

Well, you need to see the output of the serial console. One way to do this would be to compile qemu-git like I said above (WITH sdl and gtk) and then start qemu over your ssh connection with X11 forwarding. That way, you should see the qemu window on your SSH client (with these three views) and be able to access the serial console. Read the "X11 forwarding" section of this wiki page,

Ansa89

Member

Registered: 2014-08-30

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After you have created this SCSI bus, you can then add devices to it like this:

-drive file=${WINDOWS_INSTALL_DISK},id=iso_install,if=none \
-device scsi-cd,bus=scsi.0,drive=iso_install

or this:

-drive file=/path/to/device,if=none,id=disk_primary \
-device scsi-hd,bus=scsi.0,drive=disk_primary

Is exactly what I did, but after initial installation screen, Windows said "cannot find installation media" (probably due to lack of necessary drivers); so I ended up using IDE.

If you don't mind me asking this, what is the use-case for this? VNC (which IIRC is what you're using) isn't even fast enough to use it for high-quality video transmission inside one PC (I tried) from guest to host. Not to mention the latency and spikes.

You right about VNC performance, but I my idea was to use nVidia GameStream and/or Steam HomeStreaming (they had been created for this situation).

Well, you need to see the output of the serial console. One way to do this would be to compile qemu-git like I said above (WITH sdl and gtk) and then start qemu over your ssh connection with X11 forwarding. That way, you should see the qemu window on your SSH client (with these three views) and be able to access the serial console. Read the "X11 forwarding" section of this wiki page,

Isn't there a way to access serial0 without SDL/GTK (for example "-serial stdio" option)?
I'm asking this because SDL has a ton of dependencies .

EDIT: quote from this (very outdated) thread:

When I try the Ctrl+Alt+2 thing in VNC, I get a screen with a display "serial0", with no "(qemu)" prompt as was implied in documentation. From what I gather from the aforementioned thread, the Ctrl+Alt+2 thing is disabled by default, won't get you to a QEMU Monitor.

I tried Ctrl+Atl+2 and the console0 switched to (or at lest become) a black screen without the "(qemu)" prompt (however can't see "serial0" written anywhere).

Last edited by Ansa89 (2014-10-18 14:41:19)

Calrama

Member

From: Berlin

Registered: 2011-10-14

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After you have created this SCSI bus, you can then add devices to it like this:

-drive file=${WINDOWS_INSTALL_DISK},id=iso_install,if=none \
-device scsi-cd,bus=scsi.0,drive=iso_install

or this:

-drive file=/path/to/device,if=none,id=disk_primary \
-device scsi-hd,bus=scsi.0,drive=disk_primary

Is exactly what I did, but after initial installation screen, Windows said "cannot find installation media" (probably due to lack of necessary drivers); so I ended up using IDE.

Yes, that is normal. That's not OVMF not finding your device, though (which would mean that the device does not show up on the UEFI shell), that's (as you suspected) Windows not having any Virtio drivers by default.
What guide did you follow? You need VirtIO drivers for Windows if you want to use VirtIO devices in a Windows guest, see here. If you want a direct download, you can grab the - currently - latest Fedora-compiled, signed VirtIO drivers for Windows here. That is an iso file, so you have to add it to your machine without VirtIO, i.e. as a normal IDE drive, so Windows can find it (Adding "-cdrom /path/to/virtio-win-0.1-81.iso" to your qemu command should do it). Once you booted from the Windows installation disc and got to the part where the installer "cannot find installation media" you should be able to select "Browse" or something similar (cannot remember exactly what it is called), at which point you select the Win8 directory on the drive that corresponds to the virtio driver iso (should be the only CDROM drive at this point) and select all the VirtIO drivers that he shows you for installation. Afterwards, you should be able to continue the Windows installation normally.

If you don't mind me asking this, what is the use-case for this? VNC (which IIRC is what you're using) isn't even fast enough to use it for high-quality video transmission inside one PC (I tried) from guest to host. Not to mention the latency and spikes.

You right about VNC performance, but I my idea was to use nVidia GameStream and/or Steam HomeStreaming (they had been created for this situation).

And in my experience they have not succeeded yet by a long shot. I've tried Steam InhomeStreaming from the guest VM directly to the Gentoo host (no real network card involved) with the Guest using a VirtIO network card (which is currently the best in performance AFAIK).
I had my left monitor show the actual output of the guest VM (connected to the passed-through graphics card) and the right monitor be the Host's Steam Inhome-Streaming client with the following results at 1080p@60Hz:
If the screen content stays nearly the same or is very slow moving, you're not going to see a difference on average. But I saw a significant difference with fast-moving content, There were many skipped frames, the FPS count plummeted from 60 down do 30. Also, regardless of whether it's fast or slow-moving content, I had spikes which had the Inhome-Streaming client lag as far behind as one second (the missing frames were then completly skipped to get to the current frames).
I have no experience with nVidia GameStream, but I doubt they have a solution for network spikes. If the above is something you can live with, then definitely go for it, but for me this was unbearable.

But regardless of the above two paragraphs, you might really want to get your passthrough to work on its own, before moving on to that stuff, because right now you're trying two things at the same time without having confirmed the first thing to be working properly.

Well, you need to see the output of the serial console. One way to do this would be to compile qemu-git like I said above (WITH sdl and gtk) and then start qemu over your ssh connection with X11 forwarding. That way, you should see the qemu window on your SSH client (with these three views) and be able to access the serial console. Read the "X11 forwarding" section of this wiki page,

Isn't there a way to access serial0 without SDL/GTK (for example "-serial stdio" option)?
I'm asking this because SDL has a ton of dependencies .

EDIT: quote from this (very outdated) thread:

When I try the Ctrl+Alt+2 thing in VNC, I get a screen with a display "serial0", with no "(qemu)" prompt as was implied in documentation. From what I gather from the aforementioned thread, the Ctrl+Alt+2 thing is disabled by default, won't get you to a QEMU Monitor.

I tried Ctrl+Atl+2 and the console0 switched to (or at lest become) a black screen without the "(qemu)" prompt (however can't see "serial0" written anywhere).

Sorry, I have no further information about that. Although imho you should really get the passthrough to work properly first and then move on to accessing the VM via network. What's the problem with having the SDL dependencies, though. If your machine is even remotely close to good enough to be used for virtualised gaming, you should be able to give the host a few more MB. It's not like they'll slow down your host? Anyway, what I can tell you is that when you start QEMU with OVMF, the serial console will show the EFI shell, which you can then use to boot the Windows installer or the installed windows should it in fact be stuck there. Once the VM has booted an EFI program (like Windows bootloader) it will become black. Anyway, I would suggest to try qemu-git first, before doing anything else.

Last edited by Calrama (2014-10-18 16:41:44)

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I use in-home streaming and with option FAST + Unlimited bandwith (peaks about 90+mbps) = 0 frame drops at 1200p 60fps, no matter how fast screen moves....
If u install geforce experience  and turn on / off shadowplay, that enables NVFBC for full desktop/better capture for streaming.
http://i.imgur.com/D2bWvrR.jpg u can see at bottom that display latency is below 20ms and that screen is pretty much worst case scenario, lots of action, cpu suffers

Last edited by slis (2014-10-18 17:10:28)

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anyone had this issue where nVidia's Windows drivers crash every 20 minutes , and produce artifacts on pretty much everything that moves on the screen ?

I'm using 340.52 .

Removing ALL "hv_enhancements" EXCEPT "hv_time" seems to solve the issue + it finally allowed me to reboot the VM without crashing it .

And it seems that I didn't lose any performance too .

I will keep it running for 48 hours and see if the artifacts issues are indeed gone .

EDIT :

Nope , artifacts and horizontal tears still appear in videos . I want to update the drivers , but the new ones require the removal of "hv_time" . Will I lose THAT much of CPU performance by removing it ?

Last edited by Denso (2014-10-18 21:41:20)

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

That sounds more like hardware problem, too high oc/temp.

You will lose about 10-20% cpu performance by removing it.

Maybe u load wrong bios with romfile option?

Last edited by slis (2014-10-19 08:27:56)

MichaelLong

Member

Registered: 2014-01-06

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anyone had this issue where nVidia's Windows drivers crash every 20 minutes , and produce artifacts on pretty much everything that moves on the screen ?

I'm using 340.52 .

Removing ALL "hv_enhancements" EXCEPT "hv_time" seems to solve the issue + it finally allowed me to reboot the VM without crashing it .

And it seems that I didn't lose any performance too .

I will keep it running for 48 hours and see if the artifacts issues are indeed gone .

EDIT :

Nope , artifacts and horizontal tears still appear in videos . I want to update the drivers , but the new ones require the removal of "hv_time" . Will I lose THAT much of CPU performance by removing it ?

Hi Denso,

over time I've tested my VM with three different graphic cards:

• AMD 5850

• NVIDIA GTX 560 TI

• NVIDIA GTX 980.

With all of them I've got vertical colored polygon-shaped artifacs, they flicker with different intensity and often related to the direction in which I'm looking when playing a Battlefield 4 map. The problem occurs only after a few hours and can be "solved" be restarting the game. I would also rule out overheating problems because the mentioned cards are very different regarding their cooling solutions and I don't have those artifacts when playing on the same physical machine with the same hardware.

Also I found no relation to hv_*-parameters. There is no doubt a performance loss without hv_*-parameters. But my impression is that the host, more specifically the qemu-process produces more load.

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

That sounds more like hardware problem, too high oc/temp.

You will lose about 10-20% cpu performance by removing it.

Maybe u load wrong bios with romfile option?

This is a normal slim GT610 , no exotic thing and no OCing , just a small stock fan for web browsing .

And I don't specify a ROM file at all , it just "works" for passthrough .

Hi Denso,

over time I've tested my VM with three different graphic cards:

    AMD 5850

    NVIDIA GTX 560 TI

    NVIDIA GTX 980.

With all of them I've got vertical colored polygon-shaped artifacs, they flicker with different intensity and often related to the direction in which I'm looking when playing a Battlefield 4 map. The problem occurs only after a few hours and can be "solved" be restarting the game. I would also rule out overheating problems because the mentioned cards are very different regarding their cooling solutions and I don't have those artifacts when playing on the same physical machine with the same hardware.

Also I found no relation to hv_*-parameters. There is no doubt a performance loss without hv_*-parameters. But my impression is that the host, more specifically the qemu-process produces more load.

Exactly what I found today . There is no relation between my issues and hv_ parameters .

I have 2 exact VMs , one with a GTX770 and it works with no issues at all , I can reboot it , shutdown and launch it again , no issues at all .

The second one has a GT610 for my web browsing , it also works good except for this stupid reboot issue , it reboots ok until Windows logo finishes then it hangs forever . The host and other VMs are not affected though .

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dump bios from gt610 and use it to solve reboot issue. I think that is 5xx series rebranded and for me 560ti rebooting only worked with dumped bios.

Last edited by slis (2014-10-19 11:43:50)

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dump bios from gt610 and use it to solve reboot issue. I think that is 5xx series rebranded and for me 560ti rebooting only worked with dumped bios.

Well , a GPU-Z dump from the VM result in the exact same issue . Also , trying the way mentioned in Alex's blog results in input/output error (Invalid ROM contents) .

Any other way to obtain non-initialized ROM file ?

TechPowerUp's ROMs do not work at all and result in passthrough failure .

By the way , I tried dumping the ROM from the card while it wasn't touched by anything else other than PCI-STUB . VFIO was disabled , and VM wasn't started yet . So I don't know what was really wrong .

Last edited by Denso (2014-10-19 13:51:11)

Calrama

Member

From: Berlin

Registered: 2011-10-14

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I use in-home streaming and with option FAST + Unlimited bandwith (peaks about 90+mbps) = 0 frame drops at 1200p 60fps, no matter how fast screen moves....
If u install geforce experience  and turn on / off shadowplay, that enables NVFBC for full desktop/better capture for streaming.
http://i.imgur.com/D2bWvrR.jpg u can see at bottom that display latency is below 20ms and that screen is pretty much worst case scenario, lots of action, cpu suffers

Wow, thank you for that bit of information:
After enabling NVFBC as the encoder on the host  AND VAAPI as the decoder on the guest (the latter of which took some fiddling on Gentoo), I only notice a slight lag between input and output on the client side (not noticeable on the monitor connected to the guest graphics card), which is still too much for FPS games. For everything else, though, this should be pretty good.
So, I have to retract my previous statement of them not getting it to work by a long shot, you just need to use a hardware encoder and a hardware decoder.

Last edited by Calrama (2014-10-19 16:06:39)

Calrama

Member

From: Berlin

Registered: 2011-10-14

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dump bios from gt610 and use it to solve reboot issue. I think that is 5xx series rebranded and for me 560ti rebooting only worked with dumped bios.

Also , trying the way mentioned in Alex's blog results in input/output error (Invalid ROM contents) .

By the way , I tried dumping the ROM from the card while it wasn't touched by anything else other than PCI-STUB . VFIO was disabled , and VM wasn't started yet . So I don't know what was really wrong .

Just for reference: I cannot access my GTK 660 TIs firmware that way either. Even in pristine pci-stub stub right after booting. I'm not sure, but I think that's a thing with certain NVIDIA cards and hybrid firmware. My card came with only a legacy BIOS as its firmware, so for OVMF I had to flash it with the official ASUS tool. Before flashing - that is, when it had the legacy bios as firmware - I could dump the firmware like described in the blog you referenced, but after flashing the new firmware I get get the input/output error.

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dump bios from gt610 and use it to solve reboot issue. I think that is 5xx series rebranded and for me 560ti rebooting only worked with dumped bios.

Also , trying the way mentioned in Alex's blog results in input/output error (Invalid ROM contents) .

By the way , I tried dumping the ROM from the card while it wasn't touched by anything else other than PCI-STUB . VFIO was disabled , and VM wasn't started yet . So I don't know what was really wrong .

Just for reference: I cannot access my GTK 660 TIs firmware that way either. Even in pristine pci-stub stub right after booting. I'm not sure, but I think that's a thing with certain NVIDIA cards and hybrid firmware. My card came with only a legacy BIOS as its firmware, so for OVMF I had to flash it with the official ASUS tool. Before flashing - that is, when it had the legacy bios as firmware - I could dump the firmware like described in the blog you referenced, but after flashing the new firmware I get get the input/output error.

Thank you . It wasn't surprising , as nvidia will do their best to fight this usage scenario of their cards .

This exact card was working fine on my previous system (AMD FX-8350 + 990FXA) , but with this X99 + i7-5930k it's not .

How about using a thin client + SPICE protocol instead of GPU passthrough , will it give a near-native performance ?

Calrama

Member

From: Berlin

Registered: 2011-10-14

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you . It wasn't surprising , as nvidia will do their best to fight this usage scenario of their cards .

Suprising, no. Assholly, yes. But it works for me without artifacts so I'm happy for now. Soon enough most new AAA games will have Linux support anyway with the Unreal 4 engine and several others having official support for it, so to me all of this is mostly for legacy-game support and bridging the intermission time.

This exact card was working fine on my previous system (AMD FX-8350 + 990FXA) , but with this X99 + i7-5930k it's not .

I'm using this processor on this mainboard. Warning, [OT]Never saw any real use-case for the *k processors or the i7 series for desktop systems (or overclocking in general for that matter)[/OT]. One thing, though: I know that Intel made the recent *K models Vt-d capable (previous generations were not, only the non-*K variants IIRC), but maybe there's an issue there? Seems somewhat unlikely, but you never know with all these black boxes.

How about using a thin client + SPICE protocol instead of GPU passthrough , will it give a near-nativthin cliente performance ?

Not sure what you mean by that, sorry. Would you care to elaborate on that?

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Not sure what you mean by that, sorry. Would you care to elaborate on that?

Umm , think of it as a VNC brother , only faster as I read . I've done a little search about it , and it seems that it's still a bit laggy behind an actual GPU passthrough .

Calrama

Member

From: Berlin

Registered: 2011-10-14

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Not sure what you mean by that, sorry. Would you care to elaborate on that?

Umm , think of it as a VNC brother , only faster as I read . I've done a little search about it , and it seems that it's still a bit laggy behind an actual GPU passthrough .

No, I  know what SPICE is. But I don't understand what you want to do. SPICE is a protocol for remote control (like VNC), it has - AFAIK - nothing to do with your graphics card itself. You still need to either pass a graphics card to a virtualised Windows, or use a native Windows, so I don't get why you would bring SPICE into this?

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Not sure what you mean by that, sorry. Would you care to elaborate on that?

Umm , think of it as a VNC brother , only faster as I read . I've done a little search about it , and it seems that it's still a bit laggy behind an actual GPU passthrough .

No, I  know what SPICE is. But I don't understand what you want to do. SPICE is a protocol for remote control (like VNC), it has - AFAIK - nothing to do with your graphics card itself. You still need to either pass a graphics card to a virtualised Windows, or use a native Windows, so I don't get why you would bring SPICE into this?

The idea of using SPICE with a low-end thin client (maybe a thin client that costs less than a dedicated GPU) sounds good enough for web browsing and watching YouTube . It can also be a solution for those who don't have enough PCI-E lanes for more GPUs . But I watched it in action on a YT video , and it seems to stutter/lag . So it won't be a good replacement for GPU passthrough .

Calrama

Member

From: Berlin

Registered: 2011-10-14

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Umm , think of it as a VNC brother , only faster as I read . I've done a little search about it , and it seems that it's still a bit laggy behind an actual GPU passthrough .

No, I  know what SPICE is. But I don't understand what you want to do. SPICE is a protocol for remote control (like VNC), it has - AFAIK - nothing to do with your graphics card itself. You still need to either pass a graphics card to a virtualised Windows, or use a native Windows, so I don't get why you would bring SPICE into this?

The idea of using SPICE with a low-end thin client (maybe a thin client that costs less than a dedicated GPU) sounds good enough for web browsing and watching YouTube . It can also be a solution for those who don't have enough PCI-E lanes for more GPUs . But I watched it in action on a YT video , and it seems to stutter/lag . So it won't be a good replacement for GPU passthrough .

Forgive me for being so blunt, but from my point of view you're still not answering my question about why you think about bringing in SPICE as a replacement for a graphics card passthrough. Please consider the following:

Setup A:

- One hardware box
- Linux runs natively on that box
- Windows runs as a VM on that box and gets a graphics card passed through from the Linux host

Setup B:

- One hardware box
- Linux runs natively on that box
- Windows runs as a VM on that box with an emulated graphics card (for QEMU that would be any of std,cirrus,qxl,vmware,...)

Setup C:

- Two separate hardware boxes, connected by some network (e.g. 1Gbit Ethernet, Infiniband, whatever)
- Box 1 has a native Linux running
- Box 2 has a native Windows running

These are essentially your choices (very abstract). The statement "SPICE with a low-end thin client" does not specify which, though, since the native Linux in all three setups could be such a thin client.
SPICE, VNC, or any such protocol is about how you control - in this context - the Windows machine (regardless of whether you have one or two boxes); the Windows machine (regardless of whether it is being virtualised or not) will still need access to a real, dedicated graphics card (setups A and C). I know of no way aroung that. In the far future when XenGT has become something that can actually be done with NVIDIA and AMD graphics cards, we will have a way around that by paravirtualising our graphics card analogous to how Vt-x/AMD-V paravirtualise your processor. But that's still quite a bit away.

SPICE is by definition not a replacement for a graphics card passthrough, it's about remote control and can be done regardless of whether you're doing such a passthrough, which is why I was - and still am - so confused by your posts about this.

From what I can gather from your posts, I believe - but am not certain - that what you meant was "Setup B using SPICE for controlling the Windows VM from the Linux machine". Is that correct? If so, then you won't be able to use that for any GPU-heavy applications, since an emulated graphics card won't be even close to good enough to what you need for that, regardless of how good or bad SPICE is for streaming the video to you.

Last edited by Calrama (2014-10-20 12:44:41)

mouton_sanglant

Member

Registered: 2014-10-20

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone, & thanks nbhs for this tutorial.

I I tried to set up a working VGA-Passthrough on my laptop for a few days now, I wish I could do it by myself with this excellent tutorial but it seems I'm doing some things wrong. Now I'm stuck, and I call for your help, it will be very appreciated.
I know this is experimental and there's a 50/50 chance that it work for my configuration but since I'm not 100% sure I didn't forgot or misunderstood something, I prefer to ask before giving up.

I plugged in a screen with HDMI. (got VGA cable too)

Here is my setup:

model:
   Acer Aspire 7745G
proc:
   intel i5-540M (not the original one, I changed to get VT-d support, intel's KB tells it is supported: http://ark.intel.com/fr/products/43544/Intel-Core-i5-540M-Processor-3M-Cache-2_53-GHz)
chipset:
   hm55 (still refering to intel's kb, should support VT-d too: http://ark.intel.com/products/43183/Intel-BD82HM55-PCH)
host gpu (IGP):
   Intel® HD Graphics
guest gpu:
   ATI Mobility Radeon HD5650

In order to enable VT-d in BIOS, I had to flash it with a modded BIOS enabling Intel's hidden menu. VT-d was "disabled" ans I put it "enabled"

Question 0: First, I got a strange message in dmesg about DMAR:

# dmesg | grep -e DMAR -e IOMMU
[    0.000000] ACPI: DMAR 0x00000000937EF000 0000B8 (v01 INTEL  CP_DALE  00000001 INTL 00000001)
[    0.000000] Intel-IOMMU: enabled
[    0.033446] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c9008020e30272 ecap 1000
[    0.033451] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c0000020230272 ecap 1000
[    0.033458] dmar: IOMMU 2: reg_base_addr fed93000 ver 1:0 cap c9008020630272 ecap 1000
-->[    0.348199] DMAR: BIOS has allocated no shadow GTT; disabling IOMMU for graphics
[    0.452912] DMAR: No ATSR found
[    0.452968] IOMMU 0 0xfed90000: using Register based invalidation
[    0.452969] IOMMU 2 0xfed93000: using Register based invalidation
[    0.452971] IOMMU: Setting RMRR:
[    0.452992] IOMMU: Setting identity map for device 0000:00:1a.0 [0x936e9000 - 0x936fffff]
[    0.453055] IOMMU: Setting identity map for device 0000:00:1d.0 [0x936e9000 - 0x936fffff]
[    0.453087] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.453108] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

I can't find any understandable informations about this "shadow GTT", every google search leads me to techy stuff. Seems to be something important, I dont't want to disable IOMMU for graphics, right ? Is it important ? Or is it just a message to tell "your IGP can't be passed" ? That would be ok since I don't want to pass the IGP but the discret gpu, right ?
I couldn't find anything about this GTT in the BIOS, I suppose it should be somewhere because I unlocked the hidden intel menus, maybe with an alternative name ?

Since I couldn't find anything about this issue, I tried to go further.

What I have done:
- install a fresh arch linux, the downloaded qemu from the main repository
- compile linux-mainstream with patches

# uname -r
3.17.0.1-mainline

- set kernel parameters in bootloader

# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=d28d848b-32ae-4fe9-b74e-3978972c840e rw intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=1002:68c1,1002:aa60 quiet

- blacklist radeon
- configure pci-stub

# dmesg | grep pci-stub
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=d28d848b-32ae-4fe9-b74e-3978972c840e rw intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=1002:68c1,1002:aa60 quiet
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=d28d848b-32ae-4fe9-b74e-3978972c840e rw intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=1002:68c1,1002:aa60 quiet
[    0.694456] pci-stub: add 1002:68C1 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.694473] pci-stub 0000:01:00.0: claimed by stub
[    0.694481] pci-stub: add 1002:AA60 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.694490] pci-stub 0000:01:00.1: claimed by stub

- setting up vfio & KVM modules

# cat /etc/modprobe.d/vfio_iommu_type1.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

- enable service to automatically bind the devices

- test it with the following command

/usr/bin/qemu-system-x86_64 \
	-enable-kvm \
	-m 2048 \
	-M q35 \
	-bios /usr/share/qemu/bios.bin \
	-cpu host \
	-vga none \
	-device 'ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1' \
	-device 'vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/mouton/GPUPT/vgabios.bin' \
	-device 'vfio-pci,host=01:00.1,bus=root.1,addr=00.1' \
	-cdrom /home/mouton/GPUPT/win7.iso 

That's where I'm stuck: qemu launches but laptop's screen show "compat_monitor0 console from qemu". I can see there is cpu-usage so I suppose emulation works fine but I just can't see the results screen.

Question 1: Removing "-vga none", it is functionning correctly, but from what I understood, the GPU is not passed then. Is it wrong ?

I tried with and without "romfile=..." result is the same. I dumped the rom from my graphic card with a liveCD, rom-parser returns the following

Valid ROM signature found @0h, PCIR offset 40h
	PCIR: type 0, vendor: 8086, device: 0046, class: 030000
	PCIR: revision 0, vendor revision: 0
	Last image

Question 2: This is the first things which surprise me: vendor/device is 8086:0046 which correspond to my IGP. So, this is the vgabios from my IGP. Is it the correct vbios for "romfile" option or should I find a way to get the passthroughed GPU vbios ? (Which sounds logical to me, but I don't get the whole picture yet, so many confusion right now...)

Question 3: I'm actually using qemu version 2.1.2 from official arch repository... I couldn't find a link in the thread for nbhs' qemu. In the tutorial, it is advised not to use the qemu-git one so I didn't. Is it important ? Where can I find nbhs' qemu and Seabios ? (couldn't find it too, all links are broken)

Question 4: Since it's my first time with vga-passthrough, I'm not sure about the cable configuration. Actually I plugged a secondary screen on the HDMI connector of my laptop. Screen is not duplicated. However, when I plug a cable in the VGA connector, screen is duplicated. Should I use VGA instead or stay on HDMI ? (I assumed, since we pass the hdmi audio device, I should use HDMI)

What makes me persevere is I achieved to send some junk datas to the secondary screen with the "echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind " command. It worked once but I am not able to reproduce it.

Question 5: Last, I got so much confused with all the threads I read on the topic that I can't understand if I should or should not install the passed-through gpu drivers ?!

If anyone could guide me in order to make my configuration work, I would greatly apreciate it.

PureTryOut

Member

Registered: 2014-09-23

Posts: 64

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OK so after letting it rest for a while, and then starting to fiddle with it again, I have found what part in the command line ruined the booting (just entering "QEMU 2.1.2 monitor")...
Whenever I remove the -vga none option it boots up fine. But whenever I add it back it gives me the console again.

What is going wrong here?

Calrama

Member

From: Berlin

Registered: 2011-10-14

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

host gpu (IGP):
   Intel® HD Graphics
guest gpu:
   ATI Mobility Radeon HD5650

Are you sure that's even feasible? In a normal PC, every graphics card has its own output ports (the IGP's ports are on the mainboard and every other graphics card has its on its own board).
In laptops - at least to the best of my knownledge - the dedicated graphics cards usually don't have any external ports, but instead operate as "calculation slaves" below the IGP, meaning that any output the dedicated graphics card produces is forwarded to the IGP through shared memory, which in turn forwards it to its own output ports (which is either the laptop's screen or the IGP's external ports). Even if you did pass through the dedicated card correctly, how would the guest OS know about using it correctly (since this card doesn't seem to be designed to be used on its own)? This is pure speculation on my part, of course, but I somewhat doubt the AMD drivers inside the guest system would be able to function properly (though I would be happy to be proven wrong).

In order to enable VT-d in BIOS, I had to flash it with a modded BIOS enabling Intel's hidden menu. VT-d was "disabled" ans I put it "enabled"

Question 0: First, I got a strange message in dmesg about DMAR:

[...]
-->[    0.348199] DMAR: BIOS has allocated no shadow GTT; disabling IOMMU for graphics
[...]

I can't find any understandable informations about this "shadow GTT", every google search leads me to techy stuff.

Personally I have not heard of "shadow GTT" before (speculation: it might be related to the shared memory I mentioned above; it could be that in order for the host to be able to pass through the dedicated graphics card, the bios needs to have allocated some extra memory, because the usual shared memory is in use by the host OS video driver), but a quick Google search turned up this commit, which seems to imply that if this happens, your bios contains a bug. You might want to try asking the committer (who seems to be an Intel developer, so he should really know about this) for an explanation.

That's where I'm stuck: qemu launches but laptop's screen show "compat_monitor0 console from qemu". I can see there is cpu-usage so I suppose emulation works fine but I just can't see the results screen.

Where would you expect the passed through graphics card's output to be shown? The external ports belong to the IGP (unless of course that is different for your specific laptop?).

Question 1: Removing "-vga none", it is functionning correctly, but from what I understood, the GPU is not passed then. Is it wrong ?

qemu-system-x86_64 --help | grep -A1 vga

returns

-vga [std|cirrus|vmware|qxl|xenfb|tcx|cg3|none]
                select video card type

That is the graphics card you want QEMU to emulate. In other words, if you set "std" here, the VM will get an emulated graphics card of the type "std". If you don't set this argument at all, the default (which is one of these, don't remember which) will be selected. If you select "none", the VM will not have any emulated graphics card, which is what you usually want when you passthrough a real graphics card because you don't have any use for the emulated one. But the emulated graphics card should not interfere with the passed through graphics card in any way other then it being selected as the primary card. It should not have any effect on whether your card gets passed through correctly.

I tried with and without "romfile=..." result is the same. I dumped the rom from my graphic card with a liveCD, rom-parser returns the following

Valid ROM signature found @0h, PCIR offset 40h
	PCIR: type 0, vendor: 8086, device: 0046, class: 030000
	PCIR: revision 0, vendor revision: 0
	Last image

Question 2: This is the first things which surprise me: vendor/device is 8086:0046 which correspond to my IGP. So, this is the vgabios from my IGP. Is it the correct vbios for "romfile" option or should I find a way to get the passthroughed GPU vbios ? (Which sounds logical to me, but I don't get the whole picture yet, so many confusion right now...)

If it is the bios of the IGP, then no, it's not the correct one to be set for your dedicated card. If you want to set a romfile option for the dedicated card you need a firmware compatible with the dedicated card.

Question 4: Since it's my first time with vga-passthrough, I'm not sure about the cable configuration. Actually I plugged a secondary screen on the HDMI connector of my laptop. Screen is not duplicated. However, when I plug a cable in the VGA connector, screen is duplicated. Should I use VGA instead or stay on HDMI ? (I assumed, since we pass the hdmi audio device, I should use HDMI)

See my reasoning above. The external ports should all belong to the IGP so I don't see how you could see any output even if you passed through your card correctly. You could try installing a VNC on the guest OS while using an emulated graphics card and see if you can connect when not using one (and passing through your dedicated card).

What makes me persevere is I achieved to send some junk datas to the secondary screen with the "echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind " command. It worked once but I am not able to reproduce it.

I cannot explain this one without reasonable doubt and it seems to contradict what I speculated about above, so take the above with a grain of salt. You should really try to get in touch with that Intel developer.

Question 5: Last, I got so much confused with all the threads I read on the topic that I can't understand if I should or should not install the passed-through gpu drivers ?!

In general you want to install the host's graphic card's driver in the host and the guest's graphics card's driver in the guest.

gyrfalco

Member

Registered: 2014-09-30

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well, FWIW I now own an XFX HD7790 card.  All I can confirm so far is that it's quite broken.  VM reboots work well with the standard Windows VGA driver, but once the Catalyst drivers are loaded a reboot results

SAPPHIRE Radeon HD 7790 1GB (non UEFI)
QEMU emulator version 2.1.50
seabios-1.7.5
q35 / -vga none

I succeeded to get a result: http://gpuz.techpowerup.com/14/10/20/2tg.png

But only once, after uninstalling Catalyst in -vga qxl mode and starting machine in -vga none. Then i decided to install fresh system and repeat that procedure, but never been able to achieve the same result. 

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

No, I  know what SPICE is. But I don't understand what you want to do. SPICE is a protocol for remote control (like VNC), it has - AFAIK - nothing to do with your graphics card itself. You still need to either pass a graphics card to a virtualised Windows, or use a native Windows, so I don't get why you would bring SPICE into this?

The idea of using SPICE with a low-end thin client (maybe a thin client that costs less than a dedicated GPU) sounds good enough for web browsing and watching YouTube . It can also be a solution for those who don't have enough PCI-E lanes for more GPUs . But I watched it in action on a YT video , and it seems to stutter/lag . So it won't be a good replacement for GPU passthrough .

Forgive me for being so blunt, but from my point of view you're still not answering my question about why you think about bringing in SPICE as a replacement for a graphics card passthrough. Please consider the following:

Setup A:

- One hardware box
- Linux runs natively on that box
- Windows runs as a VM on that box and gets a graphics card passed through from the Linux host

Setup B:

- One hardware box
- Linux runs natively on that box
- Windows runs as a VM on that box with an emulated graphics card (for QEMU that would be any of std,cirrus,qxl,vmware,...)

Setup C:

- Two separate hardware boxes, connected by some network (e.g. 1Gbit Ethernet, Infiniband, whatever)
- Box 1 has a native Linux running
- Box 2 has a native Windows running

These are essentially your choices (very abstract). The statement "SPICE with a low-end thin client" does not specify which, though, since the native Linux in all three setups could be such a thin client.
SPICE, VNC, or any such protocol is about how you control - in this context - the Windows machine (regardless of whether you have one or two boxes); the Windows machine (regardless of whether it is being virtualised or not) will still need access to a real, dedicated graphics card (setups A and C). I know of no way aroung that. In the far future when XenGT has become something that can actually be done with NVIDIA and AMD graphics cards, we will have a way around that by paravirtualising our graphics card analogous to how Vt-x/AMD-V paravirtualise your processor. But that's still quite a bit away.

SPICE is by definition not a replacement for a graphics card passthrough, it's about remote control and can be done regardless of whether you're doing such a passthrough, which is why I was - and still am - so confused by your posts about this.

From what I can gather from your posts, I believe - but am not certain - that what you meant was "Setup B using SPICE for controlling the Windows VM from the Linux machine". Is that correct? If so, then you won't be able to use that for any GPU-heavy applications, since an emulated graphics card won't be even close to good enough to what you need for that, regardless of how good or bad SPICE is for streaming the video to you.

Sorry for my lack of words (English isn't my native language ) , and yes , I meant setup B . Where I get to interact with the VM directly from the host or through an external client . I was wondering whether SPICE can offer a big performance improvement over regular VNC or RDP , and from what I saw on YT it's still somewhat laggy . Maybe this was off topic when it comes to GPU passthrough , but it can be a solution for web surfing , printing documents , and watching videos . (Especially when you lack PCI-E lanes to add more cards as I stated previously) .

Thank you

Calrama

Member

From: Berlin

Registered: 2011-10-14

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

From what I can gather from your posts, I believe - but am not certain - that what you meant was "Setup B using SPICE for controlling the Windows VM from the Linux machine". Is that correct? If so, then you won't be able to use that for any GPU-heavy applications, since an emulated graphics card won't be even close to good enough to what you need for that, regardless of how good or bad SPICE is for streaming the video to you.

Sorry for my lack of words (English isn't my native language ) , and yes , I meant setup B . Where I get to interact with the VM directly from the host or through an external client . I was wondering whether SPICE can offer a big performance improvement over regular VNC or RDP , and from what I saw on YT it's still somewhat laggy . Maybe this was off topic when it comes to GPU passthrough , but it can be a solution for web surfing , printing documents , and watching videos . (Especially when you lack PCI-E lanes to add more cards as I stated previously) .

Thank you

No problem, it isn't my first language either. Alright, setup B it is. Do you mind my asking what the reason is? Web surfing, printing, watching videos, and pretty much anything else that does not expicitly depend on Windows-exclusive programs can be done on the Linux host, as well. And Intel's IGP are a lot more powerful than some people think, so even 1080p videos should be no problem at all. However, if you really want to do all of that on Windows, why even install Linux as the host? Just have Windows as the host and Linux in a VM (if you need the Linux at all, that is).

In either case, I don't think you should see stuttering from host-side if you use the "std" emulated graphics card (unless you intend to watch something really heavy, like a 1080p video). You don't need any extra VNC, or RDP there either, just use the normal monitor window that opens when you start the VM (I think it uses VNC internally, but I'm not sure). If you want to connect from a different, separate box, however, you might want to look into Steam Inhome-streaming. I've read on several pages that you can highjack it to get back at the desktop (i.e. have the Windows-desktop streamed to the client) and then you can just use any application you want through that. See this discussion for some ways to do that.

mouton_sanglant

Member

Registered: 2014-10-20

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello Calrama ! Thank you a lot for you suggestions !

I finally got it working !
The host display is on the laptop screen and the guest screen on the secondary screen. It works fine but I had some other duties to do so I couldn't finish my configuration right now but I will soon.

So the key issue was this vga bios misunderstanding. It became obvious as long as I was writting my request. So I completly disabled the IGP from the bios, then launched a live-cd, there I could dump the discrete gpu vbios. I specified it to qemu and it worked !

About the outputs ports, it's actually working on the hdmi. I will try on the VGA on next try.

The DMAR message is still there, but it doesn't seem to be an issue. Actually, there is something relative to memory in the BIOS: in the "Video (Intel IGD) Control Sub-Menu", there is a parameter "Total Graphics Memory" which can be set to 128MB, 256MB or MaxDVMT. Informations says: "Select the amount of Total Graphics Memory - Pre-Allocated + Fixed + DVMT for use by the Intel Graphics Device" As I'm actually working with OpenGL, I thought it was the framebuffer memory and nothing related to the DMA Remapping.
Well, I'm still wondering what that really means...

I'll try to get in touch with this Intel guy, he would certainly know about it.

I tried with and without "romfile=..." result is the same. I dumped the rom from my graphic card with a liveCD, rom-parser returns the following

Valid ROM signature found @0h, PCIR offset 40h
	PCIR: type 0, vendor: 8086, device: 0046, class: 030000
	PCIR: revision 0, vendor revision: 0
	Last image

Question 2: This is the first things which surprise me: vendor/device is 8086:0046 which correspond to my IGP. So, this is the vgabios from my IGP. Is it the correct vbios for "romfile" option or should I find a way to get the passthroughed GPU vbios ? (Which sounds logical to me, but I don't get the whole picture yet, so many confusion right now...)

If it is the bios of the IGP, then no, it's not the correct one to be set for your dedicated card. If you want to set a romfile option for the dedicated card you need a firmware compatible with the dedicated card.

Yep, that's right, it bothered me a lot !

Anyway, thank you a lot for your time. I'll post again tomorrow to tell you if it's stable but the big part is done, know I'm very satisfied and proud of my work. Gonna play with my new baby, hehe !

Have a nice day since then !