Arch Linux

avengerf12

Member

Registered: 2014-05-30

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone, first of all thanks to everyone who contributed to this great forum post, it has been immensely useful to configure my setup.

Second, I'm having a bit of a problem with the CPU configuration in a Windows 8.1 VM.
I'm currently assigning 6 vcpus to my real cpu using libvirt and when i start the VM for the first time everything works flawlessly, but if I shutdown the VM and start it again without rebooting the computer it freezes the whole host. Searching through the thread about this issue only results in a few posts suggesting to use the "isolcpus" argument in the kernel commandline to avoid any conflict. Unfortunately it didn't work and so I was hoping if someone here had any idea of what the problem could be.

This is my current config

<domain type='kvm'>
  <name>win8.1</name>
  <uuid>f3aa271e-8184-4a56-9bf5-ffbacc2f1511</uuid>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <vcpu placement='static' cpuset='6-11'>6</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='6'/>
    <vcpupin vcpu='1' cpuset='7'/>
    <vcpupin vcpu='2' cpuset='8'/>
    <vcpupin vcpu='3' cpuset='9'/>
    <vcpupin vcpu='4' cpuset='10'/>
    <vcpupin vcpu='5' cpuset='11'/>
    <emulatorpin cpuset='6-11'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
    <loader type='pflash'>/usr/share/ovmf/ovmf_x64.bin</loader>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-passthrough'>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='dca'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='invtsc'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/media/Linux-Drive/VM/Windows/Windows8Pro.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/media/Linux-Drive/VM/Windows/virtio.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/media/Linux-Drive/VM/Windows/Windows8Pro.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/media/vm-drives/WBoot.img'/>
      <target dev='sda' bus='scsi'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sda1'/>
      <target dev='sdb' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:fb:c5:e7'/>
      <source network='default'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </memballoon>
  </devices>
</domain>

I'm using a Gigabyte X79 UD3 with an i7 3930k, a r7 270x on an Arch Linux host and a gtx 680 on a Windows 8.1 VM.

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone, first of all thanks to everyone who contributed to this great forum post, it has been immensely useful to configure my setup.

Second, I'm having a bit of a problem with the CPU configuration in a Windows 8.1 VM.
I'm currently assigning 6 vcpus to my real cpu using libvirt and when i start the VM for the first time everything works flawlessly, but if I shutdown the VM and start it again without rebooting the computer it freezes the whole host. Searching through the thread about this issue only results in a few posts suggesting to use the "isolcpus" argument in the kernel commandline to avoid any conflict. Unfortunately it didn't work and so I was hoping if someone here had any idea of what the problem could be.

This is my current config

<domain type='kvm'>
  <name>win8.1</name>
  <uuid>f3aa271e-8184-4a56-9bf5-ffbacc2f1511</uuid>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <vcpu placement='static' cpuset='6-11'>6</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='6'/>
    <vcpupin vcpu='1' cpuset='7'/>
    <vcpupin vcpu='2' cpuset='8'/>
    <vcpupin vcpu='3' cpuset='9'/>
    <vcpupin vcpu='4' cpuset='10'/>
    <vcpupin vcpu='5' cpuset='11'/>
    <emulatorpin cpuset='6-11'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
    <loader type='pflash'>/usr/share/ovmf/ovmf_x64.bin</loader>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-passthrough'>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='dca'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='invtsc'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/media/Linux-Drive/VM/Windows/Windows8Pro.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/media/Linux-Drive/VM/Windows/virtio.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/media/Linux-Drive/VM/Windows/Windows8Pro.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/media/vm-drives/WBoot.img'/>
      <target dev='sda' bus='scsi'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sda1'/>
      <target dev='sdb' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:fb:c5:e7'/>
      <source network='default'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </memballoon>
  </devices>
</domain>

I'm using a Gigabyte X79 UD3 with an i7 3930k, a r7 270x on an Arch Linux host and a gtx 680 on a Windows 8.1 VM.

You probably do not need "<emulatorpin cpuset='6-11'/>", that will keep all IO for guest to the same CPUs which are used as well. It's probably better (for guest performance) to allow this IO to spread over all available CPUs - just remove the line. However, I do not expect this to fix your immediate problem since it seems something deeper than just performance issue. It is possible that selected CPU type is causing problems , see if replacing "  <cpu mode='host-passthrough'> ... </cpu>" with specific model helps. FWIW I had to downgrade my guests CPU to Penryn for stable Windows 7 guest, and it still occasionally drops to "paused" state when booting it up (no problem shutting down, though). My host CPUs are IvyBridge E5-2630v2 .

avengerf12

Member

Registered: 2014-05-30

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You probably do not need "<emulatorpin cpuset='6-11'/>", that will keep all IO for guest to the same CPUs which are used as well. It's probably better (for guest performance) to allow this IO to spread over all available CPUs - just remove the line. However, I do not expect this to fix your immediate problem since it seems something deeper than just performance issue. It is possible that selected CPU type is causing problems , see if replacing "  <cpu mode='host-passthrough'> ... </cpu>" with specific model helps. FWIW I had to downgrade my guests CPU to Penryn for stable Windows 7 guest, and it still occasionally drops to "paused" state when booting it up (no problem shutting down, though). My host CPUs are IvyBridge E5-2630v2 .

Thanks for your reply. I removed the emulatorpin line (even though I thought it was only an override for the cpuset variable) and changed my cpu mode to <cpu mode='custom' match='exact'>. It still hasn't crashed and has a bonus it even improved performance as far as I can tell so your suggestions were definitely helpful, thanks again.
Now I'll just to wait and see, even though I think it was "cpu-passthrough"'s fault.

Edit: In the exact moment I inserted again cpu-passthrough the whole machine froze. I think my problem is solved.

Edit2: Can someone tell me if the isolcpus option is necessary in my case?

Last edited by avengerf12 (2015-02-16 16:59:04)

jaeger

Member

Registered: 2015-02-16

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Greetings, all. Is it currently possible to get VGA passthrough working with a haswell onboard Intel HD host GPU and a discrete nvidia 900-series guest GPU or is this a unicorn at the moment?

I've pieced together various bits of info from posts in this thread and other places online but no matter what I try I always run into the code 43 error after installing the nvidia drivers in the windows guest. For reference, the hardware is:

ASRock H97m-Pro4
Intel i7-4790K (onboard GPU is Intel HD 4600)
MSI GeForce GTX 970

I'm running a CRUX host with kernel 3.18.7 (also tried 3.14.x) and qemu 2.2.0 (also tried straight from git). The guest is Windows 8.1 Enterprise 64-bit. I'm using OVMF to avoid the VGA arbitration issues with Intel onboard graphics.

I'm using qemu on the command line rather than libvirt and disabling the kvm CPU flag. I'm able to successfully install and boot the windows VM as well as see its display on the monitor connected directly to the discrete nvidia GPU, but I still can't get past the code 43 error. I see quite often that kvm and various hyper-v flags should be disabled, and as mentioned I disable the kvm flag already. When using qemu's command line rather than libvirt is it necessary to explicitly hide/disable the hyper-v flags? Is it documented somewhere how to do that? Plenty of examples out there for libvirt but none that I've seen for command line. If I've simply missed it and someone has a link, please do tell.

Thanks in advance, hope someone's got a similar setup working so I can compare.

edit: Forgot to add that the iommu groups containing the onboard and discrete GPUs do NOT overlap.

Last edited by jaeger (2015-02-16 22:37:40)

The_Moves

Member

Registered: 2015-01-06

Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Greetings, all. Is it currently possible to get VGA passthrough working with a haswell onboard Intel HD host GPU and a discrete nvidia 900-series guest GPU or is this a unicorn at the moment?

I've pieced together various bits of info from posts in this thread and other places online but no matter what I try I always run into the code 43 error after installing the nvidia drivers in the windows guest. For reference, the hardware is:

ASRock H97m-Pro4
Intel i7-4790K (onboard GPU is Intel HD 4600)
MSI GeForce GTX 970

I'm running a CRUX host with kernel 3.18.7 (also tried 3.14.x) and qemu 2.2.0 (also tried straight from git). The guest is Windows 8.1 Enterprise 64-bit. I'm using OVMF to avoid the VGA arbitration issues with Intel onboard graphics.

I'm using qemu on the command line rather than libvirt and disabling the kvm CPU flag. I'm able to successfully install and boot the windows VM as well as see its display on the monitor connected directly to the discrete nvidia GPU, but I still can't get past the code 43 error. I see quite often that kvm and various hyper-v flags should be disabled, and as mentioned I disable the kvm flag already. When using qemu's command line rather than libvirt is it necessary to explicitly hide/disable the hyper-v flags? Is it documented somewhere how to do that? Plenty of examples out there for libvirt but none that I've seen for command line. If I've simply missed it and someone has a link, please do tell.

Thanks in advance, hope someone's got a similar setup working so I can compare.

edit: Forgot to add that the iommu groups containing the onboard and discrete GPUs do NOT overlap.

I believe the HyperV part is particular to libvirt and KVM, not qemu. What emulator are you using? Please paste your qemu command.

dwe11er

Member

Registered: 2014-03-18

Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Greetings, all. Is it currently possible to get VGA passthrough working with a haswell onboard Intel HD host GPU and a discrete nvidia 900-series guest GPU or is this a unicorn at the moment?

I've pieced together various bits of info from posts in this thread and other places online but no matter what I try I always run into the code 43 error after installing the nvidia drivers in the windows guest. For reference, the hardware is:

ASRock H97m-Pro4
Intel i7-4790K (onboard GPU is Intel HD 4600)
MSI GeForce GTX 970

I'm running a CRUX host with kernel 3.18.7 (also tried 3.14.x) and qemu 2.2.0 (also tried straight from git). The guest is Windows 8.1 Enterprise 64-bit. I'm using OVMF to avoid the VGA arbitration issues with Intel onboard graphics.

I'm using qemu on the command line rather than libvirt and disabling the kvm CPU flag. I'm able to successfully install and boot the windows VM as well as see its display on the monitor connected directly to the discrete nvidia GPU, but I still can't get past the code 43 error. I see quite often that kvm and various hyper-v flags should be disabled, and as mentioned I disable the kvm flag already. When using qemu's command line rather than libvirt is it necessary to explicitly hide/disable the hyper-v flags? Is it documented somewhere how to do that? Plenty of examples out there for libvirt but none that I've seen for command line. If I've simply missed it and someone has a link, please do tell.

Thanks in advance, hope someone's got a similar setup working so I can compare.

edit: Forgot to add that the iommu groups containing the onboard and discrete GPUs do NOT overlap.

They shouldn't be enabled by default, so problem is somewhere else. Still, you can try and disable/remove hv_time, hv_relaxed, hv_vapic and hv_spinlocks on your -cpu section.

Last edited by dwe11er (2015-02-17 01:04:48)

mauorrizze

Member

Registered: 2014-08-22

Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@jaeger:
There must be no video device defined (vga/qxl etc.) to ensure that the nvidia is the primary display device.
In -cpu kvm=off should be explecitly set and the hv_ items should be missing/removed as already said. If this doesn't help you can try to enforce hv_time=off.
That's all I can think of.

And yes, please paste your existing qemu command.

jaeger

Member

Registered: 2015-02-16

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I believe the HyperV part is particular to libvirt and KVM, not qemu. What emulator are you using? Please paste your qemu command.

Not sure what you mean by "what emulator", can you clarify?

The qemu command is this:

sudo /usr/bin/qemu-system-x86_64 \
    -name "Windows 8.1" \
    -nographic \
    -enable-kvm \
    -cpu host,kvm=off \
    -m 8192 \
    -smp cores=4,threads=1,sockets=1 \
    -vga none -nodefconfig \
    -drive if=pflash,format=raw,readonly,file=/home/jaeger/ovmf-x64/OVMF_CODE-pure-efi.fd \
    -drive if=pflash,format=raw,file=/home/jaeger/ovmf-x64/OVMF_VARS-pure-efi.fd \
    -device vfio-pci,host=01:00.0,multifunction=on \
    -device vfio-pci,host=01:00.1 \
    -drive file="/home/jaeger/VMs/win81/win81.img",format=raw,if=virtio,id=disk0 \
    -drive file="/home/jaeger/VMs/win81/win81ent.iso",format=raw,id=cd0,if=none -device ide-cd,bus=ide.1,drive=cd0 \
    -netdev bridge,id=net0 \
    -device virtio-net-pci,netdev=net0 \
    -usbdevice host:04d9:0183 \
    -usbdevice host:1038:1369

01:00.0 and 01:00.1 are:

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)

They shouldn't be enabled by default, so problem is somewhere else. Still, you can try and disable/remove hv_time, hv_relaxed, hv_vapic and hv_spinlocks on your -cpu section.

OK, I sorta wondered if that were the case. When using the command line and not libvirt/virt-manager, though, how do you disable them? Adding hv_time=off to the cpu line seems ok but the others it complains about no matter what I pass.

edit: I was a bit premature in saying it complained "no matter what I pass", sorry for that. Passing hv_time=off,hv_relaxed=off causes no problems but doesn't fix the issue. Passing hv_spinlocks wants a numeric value so I wasn't sure what to try there, and hv_apic is unrecognized.

edit 2: I looked in the qemu source for the hv* options, I was mistyping hv-vapic as hv-apic. Still have a code 43, though.

Last edited by jaeger (2015-02-17 03:16:03)

jaeger

Member

Registered: 2015-02-16

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Playing with the various hv_options on the command line hasn't fixed my code 43 yet but I did get some new output in the qemu window when attempting to install the nvidia drivers. Lots of this sort of thing:

qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0xa60, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0xa64, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0xa68, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0x488, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0x144, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x144, 0xffffff30, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0x658, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x658, 0xfffffff2, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x624, 0x1c94e307, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x62c, 0x4ce8f40f, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x634, 0x35b4f7ca, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x63c, 0xa33431b5, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0x41c, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x624, 0x0, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x62c, 0x0, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x634, 0x0, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x63c, 0x0, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0x41c, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x484, 0x0, 0x4) failed: Bad address

Have I missed something obvious here?

Len

Member

Registered: 2015-01-06

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I managed to get some things working, thanks guys!
But right now I'm facing other problems:

This is my running script

qemu-system-x86_64 \
-enable-kvm -vga none -display none \
-cpu host -m 2048 -smp 4,cores=4,threads=1,sockets=1 \
-drive if=pflash,format=raw,readonly,file=/home/len/win/bios/ovmf_code_x64.bin \
-drive if=pflash,format=raw,file=/home/len/win/bios/ovmf_vars_x64.bin \
-device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1 \
-drive if=none,file="/dev/sdc",format=raw,id=disk0 -device ide-hd,bus=ide.0,drive=disk0 \
-drive if=none,file="/home/len/win/win7.iso",format=raw,id=cd0 -device ide-cd,bus=ide.1,drive=cd0 \
-usbdevice host:046d:c317

It works, well in some way.. OVMF starts, then I can boot from CD, but installer freeze on: "Starting Windows" (~15min still nothing).
I wanted mount it using AHCI (or I should?), but OVMF is not finding it, dunno why.

Second problem is, how can I make possible mounting /dev/sdc for normal user?
I managed to allow non-root user, take care of vfio-pci devices using chown (-__-) and some limits.conf edits.

Btw, is that normal that sometimes after running qemu, host graphic gots glithes? (fixed by switching tty).

jonascj

Member

Registered: 2014-02-28

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there still (as of 2015-02-17) a problem with the i915 patch disabling DRI in XORG?

Is the i915 patch still needed for Intel iGPU as host gpu, or are have the patches made it upstream?

Will things still work without any patches if you use AMD/Radeon GPU's only (host and guest)?

Thanks, JonasCJ

Thank you, aw, for that very insightful reply.

I have a couple of follow-up questions and/or need a few clarifications.

The first patch above fixes this, making i915 play correctly with VGA arbitration and the second patch cleans up some ordering to avoid a big white box drawn on the screen.  It works and makes everyone happy... except Xorg on the host.  The DRI code in Xorg, so I'm told, wants to mmap(2) VGA MMIO space.  Doing so is incompatible with the VGA arbiter because Xorg would need to hold the VGA arbitration lock for the life of that mmap, effectively a permanent lock.  At that point VGA arbitration is deadlocked.  Rather than do that, Xorg detects that there are multiple VGA devices and disables DRI. The result is that making i915 not lie about its VGA usage kills desktop performance for anyone with multiple graphics cards.  This is the reason the two patches were reverted.

Solutions to this are:
a) don't use VGA space in the guest (DanaGoyette attempted to do this using UEFI/OVMF in place of Seabios, also accomplished on some cards using assigned graphics as a secondary device)
b) create a VGA arbiter interface that would allow mmaps (I have some prototype code to do this, but I don't have time to figure out how to make Xorg use it)
c) figure out why DRI wants to mmap VGA space and avoid it
d) your idea here

Regarding what is marked in red:

So applying the the first patch (which makes i915 work correctly with VGA arbitration) kills overall performance (because it makes Xorg  disable DRI which forces everyone to make indirect rendering) and because of that that patch was removed / reverted upstream?

Does this mean that kernels with this patch reapplied (OP's kernel for example, or etcet's core repo kernel with i915 patch) suffer from this problem (i.e. bad overall performance)?

Regarding the text marked in blue:
Solutions to which problem? Solutions to the i915 arbiter problem - i.e. an alternative to the current patch?

In both cases yes, when multiple VGA devices are present (and possibly only on the i915 devices - radeon and nouveau appear to participate in VGA arbitration just fine).

Sorry for being so thick, but this means that everyone using the i915 module/driver suffer from disabled DRI while they use kernels patched with this patch? I.e. everyone using Intel IGPU for their host while having a dedicated gfx passed through to a KVM guest have DRI disabled on their host.

Yes

mauorrizze

Member

Registered: 2014-08-22

Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Len: are you trying to install Windows 7 or Windows 8+?

@jonascj: AFAIK the i915 arbitration patch can't be improved without breaking things in Xorg or elsewhere. So there'd be no way that this became mainstream.
On the other hand if you install Windows 8 or newer, use OVMF, no need for VGA arbitration. AMD drivers don't need any patches, work with both VGA and OVMF.

jonascj

Member

Registered: 2014-02-28

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@jonascj: AFAIK the i915 arbitration patch can't be improved without breaking things in Xorg or elsewhere. So there'd be no way that this became mainstream.
On the other hand if you install Windows 8 or newer, use OVMF, no need for VGA arbitration. AMD drivers don't need any patches, work with both VGA and OVMF.

So with AMD R9 270X as the intended guest card, and an Intel iGPU (from an i7-whatever) I would go with a unpatched kernel if I go the OVMF route?

I found this by rereading post #1: http://vfio.blogspot.dk/2014/08/primary … t-vga.html

Do you know of any other good starting points for diving into this.

Thanks, a lot!

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I managed to get some things working, thanks guys!
But right now I'm facing other problems:

This is my running script

qemu-system-x86_64 \
-enable-kvm -vga none -display none \
-cpu host -m 2048 -smp 4,cores=4,threads=1,sockets=1 \
-drive if=pflash,format=raw,readonly,file=/home/len/win/bios/ovmf_code_x64.bin \
-drive if=pflash,format=raw,file=/home/len/win/bios/ovmf_vars_x64.bin \
-device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1 \
-drive if=none,file="/dev/sdc",format=raw,id=disk0 -device ide-hd,bus=ide.0,drive=disk0 \
-drive if=none,file="/home/len/win/win7.iso",format=raw,id=cd0 -device ide-cd,bus=ide.1,drive=cd0 \
-usbdevice host:046d:c317

It works, well in some way.. OVMF starts, then I can boot from CD, but installer freeze on: "Starting Windows" (~15min still nothing).
I wanted mount it using AHCI (or I should?), but OVMF is not finding it, dunno why.

both AHCI and IDE are incredibly slow, I found that the best way to install Windows on kvm is to attach second CD with iso downloaded from http://www.linux-kvm.org/page/WindowsGu … ad_Drivers, define your C drive as virtio one. During system installation you will select virtio drivers from the attached 2nd CDROM (i.e. iso you downloaded from above URL) and then carry on with installation which will be much faster

Len

Member

Registered: 2015-01-06

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I managed to get some things working, thanks guys!
But right now I'm facing other problems:

This is my running script

qemu-system-x86_64 \
-enable-kvm -vga none -display none \
-cpu host -m 2048 -smp 4,cores=4,threads=1,sockets=1 \
-drive if=pflash,format=raw,readonly,file=/home/len/win/bios/ovmf_code_x64.bin \
-drive if=pflash,format=raw,file=/home/len/win/bios/ovmf_vars_x64.bin \
-device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1 \
-drive if=none,file="/dev/sdc",format=raw,id=disk0 -device ide-hd,bus=ide.0,drive=disk0 \
-drive if=none,file="/home/len/win/win7.iso",format=raw,id=cd0 -device ide-cd,bus=ide.1,drive=cd0 \
-usbdevice host:046d:c317

It works, well in some way.. OVMF starts, then I can boot from CD, but installer freeze on: "Starting Windows" (~15min still nothing).
I wanted mount it using AHCI (or I should?), but OVMF is not finding it, dunno why.

both AHCI and IDE are incredibly slow, I found that the best way to install Windows on kvm is to attach second CD with iso downloaded from http://www.linux-kvm.org/page/WindowsGu … ad_Drivers, define your C drive as virtio one. During system installation you will select virtio drivers from the attached 2nd CDROM (i.e. iso you downloaded from above URL) and then carry on with installation which will be much faster

Thanks for reply, I'll try it right now.

So I'm assuming that after installation, I should still use 'virtio' drivers for my drives?
I'm looking for ones with best performance for SSD drive, (have both SSD, for linux and windows).

@mauorrizze Windows 7

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@mauorrizze Windows 7

It won't work with OVMF without some hacking , you can search for some posts about installing it from FAT32 USB drive .

Also , I think its better to use SCSI rather than AHCI . My drives defined as follows :

-drive file=/VMs/Win_Main.img,cache=writeback,format=raw,if=none,id=drive0,aio=threads \
-device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \
-device virtio-scsi-pci,id=scsi \
-drive file=/VMs/Win8.iso,id=iso_install,if=none \
-device scsi-cd,drive=iso_install \
-cdrom /VMs/virtio.iso \

Good luck !

Last edited by Denso (2015-02-17 12:00:37)

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for reply, I'll try it right now.

So I'm assuming that after installation, I should still use 'virtio' drivers for my drives?

that's right

Len

Member

Registered: 2015-01-06

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@mauorrizze Windows 7

It won't work with OVMF without some hacking , you can search for some posts about installing it from FAT32 USB drive .

Also , I think its better to use SCSI rather than AHCI . My drives defined as follows :

-drive file=/VMs/Win_Main.img,cache=writeback,format=raw,if=none,id=drive0,aio=threads \
-device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \
-device virtio-scsi-pci,id=scsi \
-drive file=/VMs/Win8.iso,id=iso_install,if=none \
-device scsi-cd,drive=iso_install \
-cdrom /VMs/virtio.iso \

Good luck !

No luck even with scsi device, still stuck at installing stage,
I was trying to boot in OVMF with usb device too (pendrive created using dd), but no luck yet.

Last edited by Len (2015-02-17 18:04:06)

sl44x

Member

Registered: 2015-02-17

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everbody. Just leaving my personal tests with my desktop hardware, from the last couple of days. This maybe can save some time for someone with similar hardware:

Gigabyte GA-H61M-D2-B3
Intel i5 2500 @ 3300 Mhz
Intel IGP
NVidia GeForce 9600GT (to passthrough)

I tryed with default Arch Kernel (latest) with no success and then try with custom patches from topic author, also, with no success.
In last, tryed with some other kernel parameters posted here like ACS override and vfio_iommu_type1.allow_unsafe_interrupts, but no success.

Working:

- Only can enable the Intel IOMMU on Kernel and get only this one message on dmesg: ([0.000000] Intel-IOMMU: enabled)

Issues:

- No mapping DMAR or others IOMMU related messages on dmesg. Just the above.
- The "/sys/kernel/iommu_groups" is always empty.
- The device to passthrough dont have a iommu_group link in "/sys/bus/pci/devices/$dev"
- In case to ignore all these "issues" and continue with vfio-bind command and open the qemu receive a already waited "vfio: error no iommu_group for device".

P.S:

- All tests done with NVidia driver on blacklist at "/etc/modprobe.d/blacklist.conf"
- The MOBA has no IOMMU/VT-d related configs on BIOS, but a generic "Enable Virtualization Features" (obviously made all tests with this enabled).
- I have seen some people reporting successuly VGA passthrough with H61 chipset from Intel, but with other MOBOs with same chipset (all reports I read are from non-Gigabyte). But with this from Gigabyte, I can't.

Final consideration:

- I assume that this MOBO (Gigabyte GA-H61M-D2-B3) don't support the VT-d (IOMMU) feature.

Someone else have any suggestion to try something more? Thanks.

Last edited by sl44x (2015-02-17 18:50:27)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Final consideration:

- I assume that this MOBO (Gigabyte GA-H61M-D2-B3) don't support the VT-d (IOMMU) feature.

Someone else have any suggestion to try something more? Thanks.

Check for BIOS updates, my ASUS mobo didn't support IOMMU from two revisions ago, and there was a release with broken IOMMU support.

Last edited by Duelist (2015-02-17 19:26:37)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

sl44x

Member

Registered: 2015-02-17

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Final consideration:

- I assume that this MOBO (Gigabyte GA-H61M-D2-B3) don't support the VT-d (IOMMU) feature.

Someone else have any suggestion to try something more? Thanks.

Check for BIOS updates, my ASUS mobo didn't support IOMMU from two revisions ago, and there was a release with broken IOMMU support.

Yes, sadly forgot to say that update the BIOS was one of the first things that I tryed. My BIOS already is in the latest version available. Thanks anyways.

mauorrizze

Member

Registered: 2014-08-22

Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Len: you can try to drop -smp and use only one core and/or force -cpu hv_time=off
According to https://bugzilla.redhat.com/show_bug.cgi?id=1185253 it's a interference bug between ovmf, smp and hyperv-timer but in my experience it's sadly a more general problem with ovmf and window 7 at the moment. Maybe try different versions of qemu and ovmf.

@All:
If both graphic cards are in the same IOMMU group, then I can only pass one or both of them to a common guest VM, not each to different guests, is that correct?

deniv

Member

Registered: 2013-10-16

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Recently I bought a sound card because I was fed up with laggy sound in VMs. The card works. I can pass it through... but only by soft-removing my main usb2.0 controller. All because the controller and the card share the same IRQ. I can live without that controller (already swapped my keyboard, mouse, etc to available usb3.0 ports), but it still sucks when you can't use 4 out of 6 available ports on a motherboard. I don't mind losing the controller while the VM is online. Therefore I've been using soft-remove (echo 1 > /sys/bus/pci/devices/***/remove) . But the biggest issue is that I have to reboot every time to get the controller back online. I also tried assigning both the controller and the sound card to the VM, but it doesn't work. What else I can do?

This is the error in dmesg when I try to assign the sound card alone or both the controller and the card:

genirq: Flags mismatch irq 16. 00000080 (vfio-intx(0000:00:1a.0)) vs. 00000000 (vfio-intx(0000:08:04.0))

And here is the relevant lspci output (the sound card is behind a bridge, which I don't assign):

00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) (prog-if 20 [EHCI])
        Subsystem: Gigabyte Technology Co., Ltd 7 Series/C210 Series Chipset Family USB Enhanced Host Controller
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at f7a18000 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Debug port: BAR=1 offset=00a0
        Capabilities: [98] PCI Advanced Features
                AFCap: TP+ FLR+
                AFCtrl: FLR-
                AFStatus: TP-
        Kernel driver in use: ehci-pci

07:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 04) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 16
        Bus: primary=07, secondary=08, subordinate=08, sec-latency=64
        I/O behind bridge: 0000c000-0000cfff
        Memory behind bridge: fff00000-000fffff
        Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
        Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [78] Power Management version 3
                Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [80] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ BrConfRtry-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <2us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
        Capabilities: [c0] Subsystem: Device 0000:0000
        Capabilities: [100 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=01
                        Status: NegoPending- InProgress-

08:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
        Subsystem: ASUSTeK Computer Inc. CMI8788 [Oxygen HD Audio]
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 16
        Region 0: I/O ports at c000 [disabled] [size=256]
        Capabilities: [c0] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

deniv

Member

Registered: 2013-10-16

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

P.S. if anyone needs UEFI-combatible VBIOS file for ASUS HD7750(silent, 800mhz one, but i think it's possible to edit it for any card) - send me an e-mail, because i seem to be unable to get it officially from support anymore.

Sorry, can I request that vbios file here (i.e. without sending you an email)? Also, sorry if you've already answered such a question. I haven't read all the posts after that message.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alright, something is very broken here.

sudo qemu-system-x86_64 \
-d out_asm,in_asm,op,op_opt,int,exec,cpu,pcall,cpu_reset,ioport,unimp,guest_errors \
-D /mnt/hdd/qemu/log-`date +%d-%m-%y-%T`.log \
-enable-kvm \
-M pc \
-m 4096 \
-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \
-smp 4,sockets=1,cores=4,threads=1 \
-vga none \
-usb \
-boot menu=on \
-soundhw hda \
-device vfio-pci,host=01:00.0,x-vga=off,multifunction=on \
-device vfio-pci,host=01:00.1 \

...various stuff...

-device qxl \
-drive if=pflash,format=raw,readonly,file=/mnt/hdd/qemu/OVMF_CODE.fd \
-drive if=pflash,format=raw,file=/mnt/hdd/qemu/OVMF_VARS.fd

And...
http://pastebin.com/MVaMNp7K
100% cpu load, nothing on any output.
Pure nothing!
WTF?

When booting bare metal, i can see the setup menu and even attempt to boot into windows7. But windows7 fails to output video signal after the starting logo.

UPD:
Alright, i've connected to qemu via gdb(had to set the mode manually), and..
I'm no low-level specialist, but something hints me that this won't work:

(gdb) info reg  
rax            0x0      0
rbx            0x810248 8454728
rcx            0x402    1026
rdx            0x402    1026
rsi            0x10     16
rdi            0xbfed1060       3219984480
rbp            0xbff62ad0       0xbff62ad0
rsp            0xbff62ac0       0xbff62ac0
r8             0x1      1
r9             0x0      0
r10            0x36     54
r11            0xd7     215
r12            0x0      0
r13            0x0      0
r14            0x0      0
r15            0x0      0
rip            0xbed9d9df       0xbed9d9df
eflags         0x246    [ PF ZF IF ]
cs             0x28     40
ss             0x8      8
ds             0x8      8
es             0x8      8
fs             0x8      8
gs             0x8      8
(gdb) x/10i $rip
=> 0xbed9d9df:  mov    -0x8(%rbp),%rax
   0xbed9d9e3:  test   %rax,%rax
   0xbed9d9e6:  je     0xbed9d9df
   0xbed9d9e8:  leaveq 
   0xbed9d9e9:  retq   
   0xbed9d9ea:  push   %rbp
   0xbed9d9eb:  mov    %rsp,%rbp
   0xbed9d9ee:  sti    
   0xbed9d9ef:  pop    %rbp
   0xbed9d9f0:  retq

I don't recall mov being able to do stuff with MINUS, but let it be.
So it seems like we're trying to set RBP(Stack Base pointer register Holds the base address of the stack) to zero(RAX) MINUS 0x8.

I'm not familiar with GDB at all, but this looks strange as hell.
I guess i'll have to start qemu with -S and run the VM step-by-step.
Something hints me that the ROM on the device is broken, since this kind of stuff happens only after i add the ROM(currently it's flashed on the hardware).
Damn, forgot that it's AT&T syntax, so we're writing RBP-0x8 to RAX, and previously(at 0xbed9d9d7, not shown in the post) we did " movl $0x0,-0x8(%rbp)" and we have our zero in RAX. Why did it stop..
UPD2:
topkek. It moves zero to RAX, compares RAX with RAX, sets ZF to 1 and does JE(if ZF=1 then jump) back to that MOV. Alright, now i know that something is deeply borked here. Now i must determine where did this come from..

Last edited by Duelist (2015-02-18 00:25:16)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.