Arch Linux

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thanks for the quick answer. Getting the card rom helped to eliminate the errors regarding the "Invalid ROM contents". However, the situation didn't change - still no signal on the second monitor. Instead of enabling my Titan card /should be something like " vfio-pci 0000:01:00.0: enabling device (0000 -> 0001)" like for the audio 01:00.1/ it drops a memory reservation error:

[  179.361851] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[  179.363102] vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0x98000000-0x99ffffff 64bit pref]
[  179.365305] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)

Furthermore starting Qemu says: "VFIO 0000:01:00.0 BAR 3 mmap unsupported. Performance may be slow"

Any idea? Thx.

I believe Duelist has figured out that this typically means you have something like vesafb claiming resources of the card you're trying to assign, which isn't going to work.  Look in /proc/iomem for the driver using pieces of your GPU and prevent it from happening.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Child_of_Sun

Member

Registered: 2014-07-16

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I bought a new Graphics Card and tried Primary Passthrough with it and it works perfect :-)

It's a Sapphire Radeon R9 280X Tri-X OC and it replaces both Radeon HD7770 and brings slightly better Graphics Performance.

My Sartup command is now:

until /usr/bin/sudo /usr/bin/nice -n 0 /usr/bin/qemu-system-x86_64 -M q35 -enable-kvm -monitor stdio -nographic \
-balloon none -mem-path /dev/hugepages -mem-prealloc -m 10240 -k de -cpu host -smp 8,sockets=1,cores=8,threads=1 \
-bios /usr/share/qemu/bios.bin-1.7.5 -realtime mlock=on -vga none -D /var/log/qemu-out.log -boot menu=on -usb \
-usbdevice host:046d:c517 -usbdevice host:093a:2510 -usbdevice host:1a2c:0023 \
-device vfio-pci,host=00:11.0,bus=pcie.0,addr=1c.0,multifunction=on,bootindex=0 \
-device vfio-pci,host=00:14.2,bus=pcie.0,addr=1c.1,multifunction=on \
-device ioh3420,bus=pcie.0,addr=1c.2,multifunction=on,port=1,chassis=1,id=root.0 \
-device vfio-pci,host=01:00.0,bus=root.0,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=pcie.0,multifunction=on \
-device vfio-pci,host=03:00.0,bus=pcie.0,addr=1c.6,multifunction=on \
-device vfio-pci,host=04:00.0,bus=pcie.0,addr=1c.7,multifunction=on \
-drive id=bitlocker_keys,file=/dev/loop0,if=none -device usb-storage,drive=bitlocker_keys \
-netdev type=tap,id=guest0,vhost=on,ifname="${IFACE}" -device virtio-net-pci,netdev=guest0,mac="${macaddr}" ${options} ; do
 echo "Qemu crashed with exit code $?.  Respawning.." >&2
    sleep 5
done

*EDIT* Since the Card have a switchable Bios (UEFI/Compatible) i wanted to tell you all that both settings work.

Only the Performance is bad :-(, so i switched back to native Windows for gaming, before with the HD7770s Unigine Heaven 3.01 (Standard Settings) had a score of ~1422 in the VM and ~1522 native, the faster Card had ~822 in the VM and ~864 native, that was good for me, because the Performance loss wasn't great.

Now with the 280x i have ~1522 Points in the VM and ~2880 Points native (Tested with the UEFI Bios from the Card), i know there are too many Variables to find the "Bottleneck" by myself, so i only wanted to tell it here in the Thread.

anickname

Member

Registered: 2014-01-07

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I bought a new Graphics Card and tried Primary Passthrough with it and it works perfect :-)

It's a Sapphire Radeon R9 280X Tri-X OC and it replaces both Radeon HD7770 and brings slightly better Graphics Performance.

Only the Performance is bad :-(, so i switched back to native Windows for gaming, before with the HD7770s Unigine Heaven 3.01 (Standard Settings) had a score of ~1422 in the VM and ~1522 native, the faster Card had ~822 in the VM and ~864 native, that was good for me, because the Performance loss wasn't great.

Now with the 280x i have ~1522 Points in the VM and ~2880 Points native (Tested with the UEFI Bios from the Card), i know there are too many Variables to find the "Bottleneck" by myself, so i only wanted to tell it here in the Thread.

I just run the test on my VM using UEFI, video card Radeon 270X. The score was:
Heaven Benchmark v3.0 Basic

FPS: 102.9
Scores: 2592
Min FPS: 41.3
Max FPS: 189.9

Hardware

Binary:
Windows 32bit Visual C++ 1600 Release Mar 7 2012

Operating system:
Windows NT 6.2 (build 9200) 64bit

CPU model:
Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz

CPU flags:
3399MHz MMX SSE SSE2 SSE3 SSSE3 SSE41 SSE42 HTT

GPU model: AMD Radeon R9 200 Series 14.301.1001.0 2048Mb

Settings
Render: direct3d11
Mode: 1680x1050 fullscreen
Shaders: high
Textures: high
Filter: trilinear
Anisotropy: 4x
Occlusion: enabled
Refraction: enabled
Volumetric: enabled
Tessellation: disabled

The performance is good.

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You need to make sure you are using virtio disk in xml, then add ide cd-rom with virtio-drivers... like here

<disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/hdd2/win8.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/hdd/INSTALL/en_windows_8.1_professional_vl_with_update_x64_dvd_4065194.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/hdd/INSTALL/virtio-win-0.1-81.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>

At windows setup, when it asks where to install, choose load driver and then point it to mounted cd-rom with viritio drivers.

Hi there! Thanks for posting part of your XML file. Unfortunately, it didn't work. Not only did it not ask for the virtio Drivers, it also doesn't find my storage - a physical device. This is part of my XML file where I made some changes:

<domain type='kvm'>
  <name>win8.1</name>
  <uuid>502f5be4-0126-4a32-8d63-3f8d06a1c80a</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
    <loader type='pflash'>/var/lib/libvirt/images/win8.1-OVMF.fd</loader>
  </os>
....
   <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/mapper/VG-kvmwin'/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/root/windows.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='3'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/root/virtio.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='1' target='0' unit='1'/>
    </disk>

Can you please have a look? Is there something missing or wrong? Thank you very much!

Does anyone know why it's not finding the virtio drivers and my storage device? I also use lvm ...

any help is greatly appreciated!

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's normal to not detect any hdd's (storage drives in windows setup), cause windows doesn't have viritio drivers, thats why u mount viritio drivers as ide cdrom, so when windows shows select where to install screen, u got there option LOAD DRIVERS, click on it, select virtio cdrom, win8, drivers then select Red Hat Virtio Storage Driver and it should show your virtio disk.

dakabali

Member

Registered: 2014-11-11

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I believe Duelist has figured out that this typically means you have something like vesafb claiming resources of the card you're trying to assign, which isn't going to work.  Look in /proc/iomem for the driver using pieces of your GPU and prevent it from happening.

All right, I changed the Gigabyte Dual UEFI BIOS setting "legacy only" to "legacy first" regarding the priorities of my NVIDIA cards. This change seems to select my GT730 (I suppose it doesn't support UEFI anyway) as primary video card over the Titan and this helped to eliminate the BAR error. However, still no signal on the second monitor and having started the vm dmsg delivers the following one line:

[ 9650.055230] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900

So I managed to decrease the number of error lines to one without coming any closer to solve the problem... "great" :-(  Any idea?

EDIT: Solved! The above described BIOS setting change did the trick. I had to remove the card bios from the configuration and everything works just fine. Summary: it has to explicitly be set in the BIOS which video card is the primary, "auto" might not be enough. Thanks for your kind help!

Last edited by dakabali (2014-11-12 15:13:44)

Child_of_Sun

Member

Registered: 2014-07-16

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I bought a new Graphics Card and tried Primary Passthrough with it and it works perfect :-)

It's a Sapphire Radeon R9 280X Tri-X OC and it replaces both Radeon HD7770 and brings slightly better Graphics Performance.

Only the Performance is bad :-(, so i switched back to native Windows for gaming, before with the HD7770s Unigine Heaven 3.01 (Standard Settings) had a score of ~1422 in the VM and ~1522 native, the faster Card had ~822 in the VM and ~864 native, that was good for me, because the Performance loss wasn't great.

Now with the 280x i have ~1522 Points in the VM and ~2880 Points native (Tested with the UEFI Bios from the Card), i know there are too many Variables to find the "Bottleneck" by myself, so i only wanted to tell it here in the Thread.

I just run the test on my VM using UEFI, video card Radeon 270X. The score was:
Heaven Benchmark v3.0 Basic

FPS: 102.9
Scores: 2592
Min FPS: 41.3
Max FPS: 189.9

Hardware

Binary:
Windows 32bit Visual C++ 1600 Release Mar 7 2012

Operating system:
Windows NT 6.2 (build 9200) 64bit

CPU model:
Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz

CPU flags:
3399MHz MMX SSE SSE2 SSE3 SSSE3 SSE41 SSE42 HTT

GPU model: AMD Radeon R9 200 Series 14.301.1001.0 2048Mb

Settings
Render: direct3d11
Mode: 1680x1050 fullscreen
Shaders: high
Textures: high
Filter: trilinear
Anisotropy: 4x
Occlusion: enabled
Refraction: enabled
Volumetric: enabled
Tessellation: disabled

The performance is good.

As i told before, i don't want to search for the Problem. I quit my testing with Qemu-KVM-Vfio not only because of the Performance, Powersaving is another Reason for it, don't want to  blow 2000 kw/h per Anno for my private PC .

My Actual Configuration is:
Graka: Radeon R9 280X Tri-X OC
Mainboard: Asrock 970 Extreme4
HDDs: Sandisk 64GB SSD, Seagate Barracuda 7200.12 1TB, Seagate Barracuda LP 1TB
CPU: Amd FX-8350
Burners: Optiarc DVD-RW, LG Blueray
Ram: 4x4 GB Transcend 1333MHz

The Devices i had passthrough to the VM:
-usbdevice host:046d:c517 -usbdevice host:093a:2510 -usbdevice host:1a2c:0023 \ Keyboard and Mouse ( 2 Different Keyboards, only 1 at a time)
-device vfio-pci,host=00:11.0,bus=pcie.0,addr=1c.0,multifunction=on,bootindex=0 \ (Onboard AHCI Controller, only works in AHCI Mode without Bios Option Rom)
-device vfio-pci,host=00:14.2,bus=pcie.0,addr=1c.1,multifunction=on \ (Onboard Audio Controller)
-device ioh3420,bus=pcie.0,addr=1c.2,multifunction=on,port=1,chassis=1,id=root.0 \ (Graphics Card Port)
-device vfio-pci,host=01:00.0,bus=root.0,addr=00.0,multifunction=on,x-vga=on \ (Graphics Card)
-device vfio-pci,host=01:00.1,bus=pcie.0,multifunction=on \ (Graphics Card Audio)
-device vfio-pci,host=03:00.0,bus=pcie.0,addr=1c.6,multifunction=on \ (Usb 3.0 Controller 1)
-device vfio-pci,host=04:00.0,bus=pcie.0,addr=1c.7,multifunction=on \ (Usb 3.0 Controller 2)

The first thing i would have tested, is  a different niceness, since i set it from 10 to 0 when i changed the Cards (setting it back to 10).

Maybe i test it another Time (I have backed up my Linux and repartitioned the Drives to use them all under Windows [except for the root drive, my second ssd, i wiped it and added it to my Mailserver as root Device, because of it's poor Performance]), but for the moment i am Happy with using Windows Native :-)

*EDIT* Some Corrections.

Last edited by Child_of_Sun (2014-11-12 16:56:36)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

As i told before, i don't want to search for the Problem. I quit my testing with Qemu-KVM-Vfio not only because of the Performance, Powersaving is another Reason for it, don't want to  blow 2000 kw/h per Anno for my private PC .

devianceluka and I were talking on irc yesterday and we think that AMD ZeroCore works for devices assigned with vfio.  We still have all the same reasons that it doesn't always work on bare-metal though, in my case probably that I started using the hdmi audio.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Child_of_Sun

Member

Registered: 2014-07-16

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

As i told before, i don't want to search for the Problem. I quit my testing with Qemu-KVM-Vfio not only because of the Performance, Powersaving is another Reason for it, don't want to  blow 2000 kw/h per Anno for my private PC .

devianceluka and I were talking on irc yesterday and we think that AMD ZeroCore works for devices assigned with vfio.  We still have all the same reasons that it doesn't always work on bare-metal though, in my case probably that I started using the hdmi audio.

It's mainly because of the second Graphics card for the Host (since i now have a second PC as Mailserver, i wanted to save as much as Possible [I haven't got a Onboard Graphics device] and the second Monitor. :-)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

EDIT: Solved! The above described BIOS setting change did the trick. I had to remove the card bios from the configuration and everything works just fine. Summary: it has to explicitly be set in the BIOS which video card is the primary, "auto" might not be enough. Thanks for your kind help!

Well, that's simple.
Your distributive has kernel mode setting or some other kind of framebuffer enabled.
It goes like this:
@ the system boot, one of the cards gets assigned as a primary one, and legacy VGA output methods are tied to it. Then the kernel boots up, claims those resources as aw said. Then you start a VM, it tries to pin the whole device, but fails, outputting you that first error line with BAR3.

Gigabyte motherboards (ASUS, for example, doesn't have this in their UEFI) allow you to manually choose which GPU will be the primary, instead of assuming that "first 16x pci-e port is primary GPU".

Actually, swapping the cards physically would also help and it would be required to do so if you wouldn't have that UEFI setting. Or you could do what aw said: look up /proc/iomem, see some vesafb or nouveau or some other KMS/framebuffer stuff sitting on VGA MMIO range of the card in question, and go disable it. Oh noes, your system would usually lack a graphical loading screen(like plymouth). I think we can sacrifice that.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

[ 9650.055230] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900

BTW, this is just informational, not an error.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

As i told before, i don't want to search for the Problem. I quit my testing with Qemu-KVM-Vfio not only because of the Performance, Powersaving is another Reason for it, don't want to  blow 2000 kw/h per Anno for my private PC .

devianceluka and I were talking on irc yesterday and we think that AMD ZeroCore works for devices assigned with vfio.  We still have all the same reasons that it doesn't always work on bare-metal though, in my case probably that I started using the hdmi audio.

Well i can confirm that!
My crossfire setup has ULPS(ultra-low-power-state, actually it is zerocore itself) active, and it actually poweroffs the second, rarely used GPU. However, the first gpu doesn't change it's power state and doesn't adjust it's clocks, but that may be the metering issue here, as it requires some quirky way to check them.
The CPU also scales it's frequency nicely, without any problems so far. The variating frequency is seen inside the VM and on the host, yet cpu-z on VM tries to access some MSRs, which qemu doesn't pass, even when using -cpu host. That results in false multiplier and system bus clock meters, also i can't see the voltage and temperature. Neither i can do it on the host, thank you, AMD, for having such a quirky realization of thermometer, that it took A YEAR to fix native windows AMD OverDrive software.

Also, when using the whole block device(like hdd) or maybe even scsi/sata controller passed through, it is real to cast a spindown(and TRIM also) command to the block device, putting it in it's power saving mode.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When you rebooted the host you saw this error, or do you mean after you rebooted with the new quirk you ran into this problem.  The description doesn't make much sense to me.  The error indicates that both devices tried to register with no flags (00000000), which I think means exclusive IRQ, which means the INTx disable masking test failed.  By using nointxmask=1, you're telling vfio to use the same path that it should have been using based on those flags, so I'm not sure why it makes a difference.

It happened before and after I applied the new quirk . Sorry for not mentioning that

It works with nointxmask=1 , but with sound being choppy and slow .

Also this might be related :

...

Sorry for my lack of knowledge , but I see that both devices use some identical IRQs . Maybe that's the issue here ?

EDIT :

One side note , I use these modules options /etc/modprobe.d/kvm.conf :

options kvm ignore_msrs=1
options intel_kvm emulate_invalid_guest_state=0
options intel_kvm nested=1
options intel_kvm enable_shadow_vmcs=1
options intel_kvm enable_apicv=1
options intel_kvm ept=1
options vfio_pci nointxmask=1
options vfio_iommu_type1 disable_hugepages=1
options vfio_iommu_type1 allow_unsafe_interrupts=1

So I kept digging !

And I found the problem . the BIOS does a "Full Initialization" of the USB card upon booting , which causes all these issues . Setting "USB Initialization" to "Partial Initalization" in BIOS makes everything working like a charm again .

Hope this helps !

Last edited by Denso (2014-11-12 22:37:42)

dakabali

Member

Registered: 2014-11-11

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Your distributive has kernel mode setting or some other kind of framebuffer enabled.
It goes like this:
@ the system boot, one of the cards gets assigned as a primary one, and legacy VGA output methods are tied to it. Then the kernel boots up, claims those resources as aw said. Then you start a VM, it tries to pin the whole device, but fails, outputting you that first error line with BAR3.

I see. I find it confusing that Gigabyte calls this option "Legacy only"/"Legacy first" in their BIOS - I had no clue that this setting has something to do with the video card priorities... Trial and error has worked, anyway

Gigabyte motherboards (ASUS, for example, doesn't have this in their UEFI) allow you to manually choose which GPU will be the primary, instead of assuming that "first 16x pci-e port is primary GPU".

Right. However, the available GPU slots are called  "PEG11" "PEG12" and "PEG60" -  I haven't found anything in the motherboard manual what they stand for. Btw any of these was selected and saved, after the next reboot it set back to "Auto"...

Actually, swapping the cards physically would also help and it would be required to do so if you wouldn't have that UEFI setting.

No way. I've got a microATM and my Titan card has got literally no space in the 2nd PIC-e

Something else: can anyone tell me which is the latest working Nvidia driver pack for the passthrough setup? The latest avalable 344.65 installation breaks and falls back to 1024x768 after reboot. My Asus PG278Q monitor has come with 340.52 which works okay, but the Titan card will be shown as "Unknown PCI device" in the Win8.1 device manager.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Your distributive has kernel mode setting or some other kind of framebuffer enabled.
It goes like this:
@ the system boot, one of the cards gets assigned as a primary one, and legacy VGA output methods are tied to it. Then the kernel boots up, claims those resources as aw said. Then you start a VM, it tries to pin the whole device, but fails, outputting you that first error line with BAR3.

I see. I find it confusing that Gigabyte calls this option "Legacy only"/"Legacy first" in their BIOS - I had no clue that this setting has something to do with the video card priorities... Trial and error has worked, anyway

"Legacy only vs Legacy first" sounds more like preference of PCI option ROM execution, not ordering of VGA device.  A PCI option ROM can contain multiple images, often one for legacy BIOS and one for UEFI.  That option sounds like it's selecting which one to prefer.  Using a similar option on my Asus board is how I determined my R7790 card came from XFX with a broken UEFI ROM.  The bare metal system boots fine with Legacy First, but if I switch it to opt for the UEFI ROM the screen never turns on.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

winie

Member

Registered: 2014-03-01

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there a way to fix the new nvidia drivers looking for hyper-v parameters? possibly in future releases of qemu or from inside windows?

Are you asking for a solution beyond simply not using hyper-v extensions?  You might want to revert back to an old nvidia driver and test whether it would be worthwhile.  I don't know of anyone looking a further solutions.  If you need hyper-v extensions, nvidia professional series cards or AMD cards might be a better option.

I have been thinking about the whole nvidia situation and came up with two possible solutions. However I don't know enough about the details of PCI-E and emulation in general to tell if they are feasible or not.

The first idea is to "softmod" a GeForce to a Quadro by tapping in the vfio driver or pcie subsystem of the linux kernel. So instead of hardmodding the pci id with a soldering iron, you provide a fake pci id to the guest.

The second idea is mostly a hack. Since the nvidia driver checks the cpuid for hv_* features, would it be possible to patch qemu to change the cpuid during runtime? So during bootup when the nvidia driver is initialized, the hv_* features are hidden and only enabled after two minutes or via the qemu console.

Maybe @aw or others could comment if one of these approaches could work and which component of the stack needs to be patched.

What about hacking the nvidia drivers?

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You guys may already know the news about microsoft getting .net core opensource and giving out free visual studio in 2015. That visual studio, due to xamarin framework and stuff, comes with an android emulator! The android emulator in question is based on HyperV.

So, it gets even more interesting to get the nested virtualization running. Windows 7 does not support HyperV, so to test the nested - you'll need to upgrade.
BUT.
Nvidia?
As it emulates the droid, we'll definitely need to run at least opengl1.1(they don't have opengl 2.0 support yet, but it isn't released in the public either), and that means we must have hardware acceleration. That means we need the drivers working.
Is it really possible to passthrough an nvidia card into the VM and have nested hyperv virtualization support without buying you-know-what?

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Flyser

Member

Registered: 2013-12-19

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm going to tread lightly here, but I will confirm that PCI config space of assigned devices is a combination of direct passthrough, emulation, and virtualization.  PCI SR-IOV devices, by the spec, don't actually have PCI vendor and device IDs, it's up to system software/VMM to provide them.  Likewise, it's a trivial matter to virtualize the PCI config space IDs - http://fpaste.org/149722/72280914/  In the vast majority of cases, modifying the vendor ID is a bad idea, but I include it to be complete.

On a completely separate note, I'll mention that PCI device IDs are also stored in the PCI ROM and the ROM includes a checksum.  In the unlikely event that you want to expose the ROM on a modified device, the device ID and checksum need to be updated.

IT WORKS!
I set the Quadro k4000 pci ids for my GTX660, supplied the VBIOS on the qemu cmdline and the latest nvidia quadro drivers work like a charm

Also note that supported Quadro devices operate in a secondary graphics mode, ie. in addition to emulated VGA.  I've also never had much luck getting them to work on Q35 machine types, but it's on my todo list to look further into the Code 12 they get there.  They work fine on a 440FX machine.

Thanks for the hint as I would have given up on the initial black screen^^. But why is that? From my understanding real quadro devices work as primary graphics devices. Is it possible to use them as such in qemu as well?

Last edited by Flyser (2014-11-13 08:23:25)

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I set the Quadro k4000 pci ids for my GTX660

Can you explain how did you achieve that without soldering iron ?

Thank you .

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I set the Quadro k4000 pci ids for my GTX660

Can you explain how did you achieve that without soldering iron ?

Thank you .

aw wrote:
I'm going to tread lightly here, but I will confirm that PCI config space of assigned devices is a combination of direct passthrough, emulation, and virtualization.  PCI SR-IOV devices, by the spec, don't actually have PCI vendor and device IDs, it's up to system software/VMM to provide them.  Likewise, it's a trivial matter to virtualize the PCI config space IDs - http://fpaste.org/149722/72280914/  In the vast majority of cases, modifying the vendor ID is a bad idea, but I include it to be complete.
On a completely separate note, I'll mention that PCI device IDs are also stored in the PCI ROM and the ROM includes a checksum.  In the unlikely event that you want to expose the ROM on a modified device, the device ID and checksum need to be updated.

Also note that supported Quadro devices operate in a secondary graphics mode, ie. in addition to emulated VGA.  I've also never had much luck getting them to work on Q35 machine types, but it's on my todo list to look further into the Code 12 they get there.  They work fine on a 440FX machine.

I have a quadrified geforce 470gtx --> quadro 5000 and i have no problems either using it as primary on q35 (i do get boot messages using romfile=) or secondary on 440fx

Q35 primary:

440FX secondary:

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hmmm .. changing the device IDs to Quadros + keep Hyper-V enhancements , but is it REALLY worth it ?  I did try to disable Hyper-V enhancements and I don't see any performance hit in any real world workload (Gaming , surfing & multimedia) . So I think I'll stick with kvm=off and keep my nVidia drivers up to date .

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a quadrified geforce 470gtx --> quadro 5000 and i have no problems either using it as primary on q35 (i do get boot messages using romfile=) or secondary on 440fx

Huh?
Quadro 5000 core config: 352:44:40
GTX470 core config: 448:56:40.

Ummm.. Ummmm... like who gives a damn as long as the chip architecture is the same.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

dwe11er

Member

Registered: 2014-03-18

Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a quadrified geforce 470gtx --> quadro 5000 and i have no problems either using it as primary on q35 (i do get boot messages using romfile=) or secondary on 440fx

Huh?
Quadro 5000 core config: 352:44:40
GTX470 core config: 448:56:40.

Ummm.. Ummmm... like who gives a damn as long as the chip architecture is the same.

Now wait for NVIDIA to check core configuration in their driver...

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's normal to not detect any hdd's (storage drives in windows setup), cause windows doesn't have viritio drivers, thats why u mount viritio drivers as ide cdrom, so when windows shows select where to install screen, u got there option LOAD DRIVERS, click on it, select virtio cdrom, win8, drivers then select Red Hat Virtio Storage Driver and it should show your virtio disk.

thank you! Finally I got it So I successfully installed Windows 8.1 after some hiccups I managed to install the "Assassin's Creed Unity"-ready Nvidia drivers (344.65) without any error code (43).

BUT then:

After a simple host reboot I got the CODE 43 back and I couldn't make it work again.

This is what I changed in the XML:

<os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
    <loader type='pflash'>/var/lib/libvirt/images/win8.1-OVMF.fd</loader>
  </os>

<features>
    <acpi/>
    <apic/>
    <pae/>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>

<cpu mode='host-passthrough'>
  </cpu>
  <clock offset='utc'/>

after that I ran virt-manager and added Nvidia GPU and Audio.

I use latest linux-mainline and provided by NBHS, ovmf-git, qemu-git and libvirt ...

this is how I start arch:

BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=07c30ed1-3204-4c6a-8gd1-22r8ae54t1f5 rw i915.enable_hd_vgaarb=1 intel_iommu=on pcie_acs_override=downstream quiet

Is there something I forgot or got wrong? I am this close to game in 4k in a virtual machine, please help

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there something I forgot or got wrong? I am this close to game in 4k in a virtual machine, please help

Assuming they haven't added anything new, any of the following will cause a Code 43:

  <features>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>

  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
  </clock>

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users