Arch Linux

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
Thanks again for explanations. Now it gives me, when the driver for this card is pci-stub (lspci -kn)

cat rom > /tmp/image.rom                                               /sys/bus/pci/devices/0000:03:00.0
cat: rom: Input/output error

Maybe I just try with the latest bios on http://www.techpowerup.com/ ...

walkindude@Desktop> ./rom-parser ~/Sapphire.HD6850.1024.110620.rom                                        ~/rom-parser
Valid ROM signature found @0h, PCIR offset 1f4h
    PCIR: type 0, vendor: 1002, device: 6739, class: 030000
    PCIR: revision 0, vendor revision: d0c
    Last image

latest from this site.. meanging no UEFI with this card?

and if I disable OVMF, and boot with seabios, should I see anything on screen which is connected to hd 6850 by hdmi? For now, it's nothing there.

Yep, no UEFI support in that ROM.  If you use seabios then you need x-vga=on, but search back a couple weeks and find the wrapper script method of adding it, don't use the <qemu:arg> abomination.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

walkindude

Member

Registered: 2012-03-04

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw

tried with

<domain type='kvm'>
  <name>windows</name>
  <uuid>c8bc2373-6271-4810-991e-8618a0ee01d2</uuid>
  <memory unit='KiB'>3145728</memory>
  <currentMemory unit='KiB'>3145728</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-wrapper</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/walkindude/Downloads/JM1_CCSA_X64FRE_EN-US_DV5.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/walkindude/virtio-win-0.1-94.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/walkindude/brasero.iso'/>
      <target dev='hdd' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='1'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/vm1.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='ide' index='1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='scsi' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x0cf3'/>
        <product id='0x9271'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none' model='none'/>
</domain>

modified wrapper to

#!/bin/sh
exec qemu-system-x86_64 `echo "\$@" | sed 's|03:00.0|03:00.0,x-vga=on|g'`

03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts PRO [Radeon HD 6850]

start the VM, switch to hmdi which connected to HD 6850 - no connection

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

start the VM, switch to hmdi which connected to HD 6850 - no connection

Do you get the "Invalid ROM contents" message in dmesg when starting the VM?  If so, then you need to pass a rom with the <rom file="$FILE"/> option (double check that syntax)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

walkindude

Member

Registered: 2012-03-04

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have:

[    4.355351] radeon 0000:01:00.0: Invalid ROM contents

but this is for

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao PRO [Radeon R9 270]

which works nice in linux with the latest mesa-git drivers

hmmm, sometimes it shows Seabios on screen, sometimes screen is completely dark.. but with no "no connection".. it's like it's waiting for something

in dmesg I have:

[  740.315287] vfio-pci 0000:03:00.0: enabling device (0400 -> 0403)

[  744.276043] kvm: zapping shadow pages for mmio generation wraparound

Stupid me: should add tag bootmenu to os

Now I need to passthrough my second, ps2 keyboard to actually press F12. How?)

Last edited by walkindude (2015-01-11 23:07:15)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now I need to passthrough my second, ps2 keyboard to actually press F12. How?)

QEMU doesn't support ps/2 passthrough afaik.  Probably too much BIOS involved in ps/2.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

walkindude

Member

Registered: 2012-03-04

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
Damn, and my other, bluetooth keyboard doesn't work too. Not detected before any system has been booted.
Time to borrow some from friend)

Last edited by walkindude (2015-01-12 00:37:39)

DanaGoyette

Member

Registered: 2014-01-03

Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It really is a bummer that the E3-12xx series lacks isolation on the CPU root ports.
Motherboards that use PLX chips (which have ACS) should help a lot.

http://www.anandtech.com/show/6170/four … s-and-evga

The Gigabyte diagram shows all CPU lanes connected directly into the switch, which is the ideal case.
The EVGA diagram shows the first x16 slot directly connected to the CPU, and all the other slots behind the switch.
In that situation, is it possible for the PLX switch to provide isolation between that directly-connected slot and all the other slots?  I'd hope it would be able to force all transactions to go directly upstream.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It really is a bummer that the E3-12xx series lacks isolation on the CPU root ports.
Motherboards that use PLX chips (which have ACS) should help a lot.

http://www.anandtech.com/show/6170/four … s-and-evga

The Gigabyte diagram shows all CPU lanes connected directly into the switch, which is the ideal case.
The EVGA diagram shows the first x16 slot directly connected to the CPU, and all the other slots behind the switch.
In that situation, is it possible for the PLX switch to provide isolation between that directly-connected slot and all the other slots?  I'd hope it would be able to force all transactions to go directly upstream.

Nope, switches downstream of root ports don't help, the full topology must support ACS.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

anthonym52

Member

Registered: 2014-12-07

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

does anyone have any tips on reducing audio latency?

winie

Member

Registered: 2014-03-01

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

does anyone have any tips on reducing audio latency?

best option is to not use an emulated sound device in geust.

instead pasthrough a cheap usb sound card. you can find one for a few dollars.
or passthrough the graphic cards built in hdmi audio device.

Last edited by winie (2015-01-12 05:35:37)

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 119

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

does anyone have any tips on reducing audio latency?

best option is to not use an emulated sound device in geust.

instead pasthrough a cheap usb sound card. you can find one for a few dollars.
or passthrough the graphic cards built in hdmi audio device.

or passthrough ODAC USB device (like this http://www.amazon.co.uk/Epiphany-Acoust … B00G36MTT0 - that's mine BTW) or even better, whole USB controller to which such device is attached.

also, use libvirt vcpupin or some other means of pinning VCPU.

finally, only if the computer is dedicated  to running virtual machines, you can use isolcpus kernel option to ensure that some cores are left by kernel scheduler for exclusive use of the virtual machine.

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 119

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yep, no UEFI support in that ROM.  If you use seabios then you need x-vga=on, but search back a couple weeks and find the wrapper script method of adding it, don't use the <qemu:arg> abomination.

... or only two pages back  : https://bbs.archlinux.org/viewtopic.php … 1#p1491451

walkindude

Member

Registered: 2012-03-04

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek
Already done that, see https://bbs.archlinux.org/viewtopic.php … 7#p1492217 at the end. Just forgot to set bootmenu timeout.
Tried you wrapper: no connection.

But after I had seen Seabios load screen, pressed F12, tried to boot from Windows DVD / Arch Live CD, see "Booting from CD/DVD" or "ISOLINUX ..." caption for about a second and then screen goes dark. Tried to wait several minutes: nothing, just blackness.

My current xml is:

<domain type='kvm'>
  <name>windows</name>
  <uuid>c8bc2373-6271-4810-991e-8618a0ee01d2</uuid>
  <memory unit='KiB'>3145728</memory>
  <currentMemory unit='KiB'>3145728</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
    <loader type='rom'>/usr/share/qemu/bios.bin</loader>
    <bootmenu enable='yes' timeout='60000'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-wrapper</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/walkindude/Downloads/JM1_CCSA_X64FRE_EN-US_DV5.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='directsync' io='threads'/>
      <source file='/var/lib/libvirt/images/windows.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <boot order='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='ide' index='1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='scsi' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <rom bar='on'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x09da'/>
        <product id='0x0260'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x09da'/>
        <product id='0x000e'/>
      </source>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

@Bronek
Of course, I know why your script didn't work for me: didn't load rom file for my card.

But when I try this:

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <rom file='/home/walkindude/hd6850.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x15' function='0x0'/>
    </hostdev>

and it's definitely there:

walkindude@Desktop> ls /home/walkindude/hd6850.rom                                                                                                                           ~
/home/walkindude/hd6850.rom

it gives me

walkindude@Desktop> sudo virsh start windows                                                                                                                                 ~
error: Failed to start domain windows
error: internal error: early end of file from monitor: possible problem:
2015-01-12T20:25:41.102985Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on,id=hostdev0,bus=pci.0,addr=0x15,romfile=/home/walkindude/hd6850.rom: pci_add_option_rom: failed to find romfile "/home/walkindude/hd6850.rom"
2015-01-12T20:25:41.235373Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on,id=hostdev0,bus=pci.0,addr=0x15,romfile=/home/walkindude/hd6850.rom: Device initialization failed.
2015-01-12T20:25:41.235447Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on,id=hostdev0,bus=pci.0,addr=0x15,romfile=/home/walkindude/hd6850.rom: Device 'vfio-pci' could not be initialized

Last edited by walkindude (2015-01-12 20:26:27)

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 119

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek
Of course, I know why your script didn't work for me: didn't load rom file for my card.

But when I try this:

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <rom file='/home/walkindude/hd6850.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x15' function='0x0'/>
    </hostdev>

and it's definitely there:

walkindude@Desktop> ls /home/walkindude/hd6850.rom                                                                                                                           ~
/home/walkindude/hd6850.rom

it gives me

walkindude@Desktop> sudo virsh start windows                                                                                                                                 ~
error: Failed to start domain windows
error: internal error: early end of file from monitor: possible problem:
2015-01-12T20:25:41.102985Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on,id=hostdev0,bus=pci.0,addr=0x15,romfile=/home/walkindude/hd6850.rom: pci_add_option_rom: failed to find romfile "/home/walkindude/hd6850.rom"
2015-01-12T20:25:41.235373Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on,id=hostdev0,bus=pci.0,addr=0x15,romfile=/home/walkindude/hd6850.rom: Device initialization failed.
2015-01-12T20:25:41.235447Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on,id=hostdev0,bus=pci.0,addr=0x15,romfile=/home/walkindude/hd6850.rom: Device 'vfio-pci' could not be initialized

ok so my script uses "vga.rom" suffix in the filename to detect that VGA ROM is being loaded and by this it "knows" to append x-vga=on. If your filename suffix is different, or if you do not need to load VGA ROM (e.g because your computer's BIOS alread did this) then you can use some other distinguishing feature of VGA PCI device. Just run "ps lfxww | grep qemu"  to find what your qemu options look like and then you will know what the script should be looking for

Last edited by Bronek (2015-01-12 22:47:01)

walkindude

Member

Registered: 2012-03-04

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek

walkindude@Desktop> ps lfxww | grep qemu                                                                             ~
0  1000  5944  1447   5   0  14256  2240 pipe_w S+   pts/0      0:00  |       \_ grep qemu

No qemu options there...

If I look in /var/log/libvirt/qemu/windows.log, I see

2015-01-12 23:01:43.682+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-wrapper -name windows -S -machine pc-i440fx-2.2,accel=kvm,usb=off -bios /usr/share/qemu/bios.bin -m 3072 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid c8bc2373-6271-4810-991e-8618a0ee01d2 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/windows.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot menu=on,splash-time=60000,strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device lsi,id=scsi0,bus=pci.0,addr=0x8 -device ahci,id=ahci0,bus=pci.0,addr=0x9 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/home/walkindude/Downloads/JM1_CCSA_X64FRE_EN-US_DV5.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -drive file=/var/lib/libvirt/images/windows.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=directsync,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x15 -device usb-host,hostbus=4,hostaddr=2,id=hostdev1 -device usb-host,hostbus=4,hostaddr=3,id=hostdev2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=o

As I wrote before, I use:

#!/bin/sh
exec qemu-system-x86_64 `echo "\$@" | sed 's|03:00.0|03:00.0,x-vga=on|g'`

It indeed shows Seabios info on desired screen, I can choose boot device (so keyboard also working), but then turns black.

Japanlinux

Member

Registered: 2010-05-18

Posts: 173

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've tried this process several times without the kernel+patches mentioned in the OP and I have determined that I do need the ACS patch to make this work. I then proceeded to download the kernel, makepkg and pacman -U the mainline, kernel, and headers xz files that it generated 2 hours later. However, I do not have that kernel installed (still using the same default one with no option to use the new during boot). So I must have done something wrong here. I'd appreciate some help.

walkindude

Member

Registered: 2012-03-04

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Japanlinux
Maybe you should do

grub-mkconfig -o /boot/grub/grub.cfg

?

Or something similar, if you have different bootloader

Last edited by walkindude (2015-01-13 02:32:56)

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 119

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek

walkindude@Desktop> ps lfxww | grep qemu                                                                             ~
0  1000  5944  1447   5   0  14256  2240 pipe_w S+   pts/0      0:00  |       \_ grep qemu

No qemu options there...

If I look in /var/log/libvirt/qemu/windows.log, I see

2015-01-12 23:01:43.682+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-wrapper -name windows -S -machine pc-i440fx-2.2,accel=kvm,usb=off -bios /usr/share/qemu/bios.bin -m 3072 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid c8bc2373-6271-4810-991e-8618a0ee01d2 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/windows.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot menu=on,splash-time=60000,strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device lsi,id=scsi0,bus=pci.0,addr=0x8 -device ahci,id=ahci0,bus=pci.0,addr=0x9 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/home/walkindude/Downloads/JM1_CCSA_X64FRE_EN-US_DV5.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -drive file=/var/lib/libvirt/images/windows.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=directsync,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x15 -device usb-host,hostbus=4,hostaddr=2,id=hostdev1 -device usb-host,hostbus=4,hostaddr=3,id=hostdev2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=o

As I wrote before, I use:

#!/bin/sh
exec qemu-system-x86_64 `echo "\$@" | sed 's|03:00.0|03:00.0,x-vga=on|g'`

It indeed shows Seabios info on desired screen, I can choose boot device (so keyboard also working), but then turns black.

I suppose you do not see all processes in your ps lfxww command because you are not running it as root. Make it "ps alfxww" to see your effective qemu command. Above shows that the call to wrapper is correct and I checked your wrapper, it should work. So you should see "qemu-system-x86_64 . . . -device vfio-pci,host=03:00.0,x-vga=on,id=hostdev0,bus=pci.0,addr=0x15 . . ." reported by ps

Looking at what you wrote earlier I would hazard a guess that you are not setting up permissions of libvirt relaxed enough to access all the devices it needs and the ROM file as well - this could explain "pci_add_option_rom: failed to find romfile "/home/walkindude/hd6850.rom" . See https://bbs.archlinux.org/viewtopic.php … 6#p1438076

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I have difficuties enabling hugepages. I followed the arch wiki.

First step ist to add this line to /etc/fstab:

hugetlbfs       /dev/hugepages  hugetlbfs       mode=1770,gid=78        0 0

First Problem. It says,

gid must match that of the kvm group

. How do I find out if the gids match? I have no clue ...

Then I mounted /dev/hugepages. No problem.

grep Hugepagesize /proc/meminfo shows:

[root@archkvm ~]# grep Hugepagesize /proc/meminfo
Hugepagesize:       2048 kB

Now: I want my guest Windows 8.1 to run with 10 GB RAM. If I undersood correctly, I should add half to /etc/sysctl.d/40-hugepage.conf. So I added this:

vm.nr_hugepages = 5120

I am not sure about this though. My thought was: 10 GB equals 10*1024=10240. Half of it is 5120.

But when I start Windows and use the command "grep HugePages /proc/meminfo" it shows this:

[root@archkvm ~]# grep HugePages /proc/meminfo
AnonHugePages:    366592 kB
HugePages_Total:    5120
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0

I think it's pretty obvious I made a mistake. I think the allocation is wrong. Why does it show 0 for HugePages_Free?

oh, and this is what I added to my XML:

<memoryBacking>
    <hugepages/>
    <nosharepages/>
  </memoryBacking>

I am also not sure about those line. Especially the nosharepages.

Can someone please help me?

thank you very much.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

...long story short...
I've finally flashed my cards with UEFI-compatible "secret"(i can't get it anywhere anymore) firmware, but appears like windows7 doesn't really works right with it. I turn off CSM, boot up the windows, it shows the splash logo and then image freezes. Yay for windows7. So, OVMF/UEFI and windows 7 is possible but troublesome, so most folks shouldn't bother about it.

Have you had any further success with UEFI Windows 7? I'm stuck at this flash screen as well, unfortunately.

No errors or messages, just a pulsing "Starting Windows" logo indefinitely on both the installation disk and a boot disk.

My test Win7 VM booted and worked on OVMF-pure-efi image with emulated graphics.
But when i try to passthrough my GPU with romfile option, it doesn't work.

Maybe I sould have said this earlier, AFAIK Windows 7 boots in UEFI mode, however doesn't support UEFI vbios. I remenber getting a boot error on bare metal when disabling CSM even If i was booting using UEFI mode.
If you can't use seabios or ovmf with csm (like me), use Win8+ 

Windows 7 boots and shows me that "starting windows" logo, BUT when it's time for some high-res video output through GOP, the image freezes on my bare metal hardware(with CSM disabled, of course) while system continues to boot. I can't make any valuable conclusions from that because i haven't tested booting Win8 on that hardware and my vbios is "illegal" - you can't get it anywhere anymore(yay for asus). But hang on a second, i've got an idea..

There should be no "boot errors", it's just the image output part.

It may be not so obvious, but to install windows on UEFI, you need to boot the UEFI-setup and use GPT on your disk. Only that way the windows installation cd will provide you EFI loader.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I have difficuties enabling hugepages. I followed the arch wiki.

First step ist to add this line to /etc/fstab:

hugetlbfs       /dev/hugepages  hugetlbfs       mode=1770,gid=78        0 0

First Problem. It says,

gid must match that of the kvm group

. How do I find out if the gids match? I have no clue ...

Then I mounted /dev/hugepages. No problem.

I doubt the above is necessary, but then I don't use arch, so I don't know for sure.  On Fedora hugetlbfs is already mounted and works, all that's necessary is to make sure there are some available.

grep Hugepagesize /proc/meminfo shows:

[root@archkvm ~]# grep Hugepagesize /proc/meminfo
Hugepagesize:       2048 kB

Now: I want my guest Windows 8.1 to run with 10 GB RAM. If I undersood correctly, I should add half to /etc/sysctl.d/40-hugepage.conf. So I added this:

vm.nr_hugepages = 5120

I am not sure about this though. My thought was: 10 GB equals 10*1024=10240. Half of it is 5120.

But when I start Windows and use the command "grep HugePages /proc/meminfo" it shows this:

[root@archkvm ~]# grep HugePages /proc/meminfo
AnonHugePages:    366592 kB
HugePages_Total:    5120
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0

I think it's pretty obvious I made a mistake. I think the allocation is wrong. Why does it show 0 for HugePages_Free?

Looks like what I would expect.  You have 5120 hugepages, 0 are free, therefore you've configured the system for exactly the number you need.  Congrats.  If you want further confirmation, look in top.  The resident memory for the qemu process should be much smaller now since it doesn't include hugepages.

oh, and this is what I added to my XML:

<memoryBacking>
    <hugepages/>
    <nosharepages/>
  </memoryBacking>

I am also not sure about those line. Especially the nosharepages.

Matches what I have.

Last edited by aw (2015-01-13 19:28:15)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, that's it then? Great, I didn't know how to check. Very happy then

thank you!

walkindude

Member

Registered: 2012-03-04

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek

Did as you'd told me. Now it doesn't complain about rom file, but black screen after "Booting from CD/DVD" remains.

walkindude@Desktop> sudo ps alfxww | grep qemu                                                                       ~
0  1000  6602  1492   4   0  14256  2244 pipe_w S+   pts/0      0:00          |   |   \_ grep qemu
6     0  5830     1   1   0 4001136 3175140 poll_s SLl ?        0:52 qemu-system-x86_64 -name windows -S -machine pc-i440fx-2.2,accel=kvm,usb=off -bios /usr/share/qemu/bios.bin -m 3072 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid c8bc2373-6271-4810-991e-8618a0ee01d2 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/windows.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot menu=on,splash-time=60000,strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device lsi,id=scsi0,bus=pci.0,addr=0x8 -device ahci,id=ahci0,bus=pci.0,addr=0x9 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/home/walkindude/Downloads/JM1_CCSA_X64FRE_EN-US_DV5.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -drive file=/var/lib/libvirt/images/windows.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=directsync,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device vfio-pci,host=03:00.0,x-vga=on,id=hostdev0,bus=pci.0,addr=0x15,romfile=/home/walkindude/hd6850.rom -device usb-host,hostbus=4,hostaddr=2,id=hostdev1 -device usb-host,hostbus=4,hostaddr=3,id=hostdev2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on

in dmesg

[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA

[  450.266194] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)

Tried with rom which I got from running Atiflash and also some roms from techPowerUp. Every time: black screen after Seabios and boot device choosing.

It's like I'm missing something stupid or something big, dunno

Last edited by walkindude (2015-01-13 20:57:01)

Nanori

Member

Registered: 2015-01-13

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone,

I follow this topic for a long time in order to manage a VGA pass-through on an Nvidia GTX 760.

At this time, I'm able to start my VM without errors from qemu but my screen seems to receve no signal from the card... :'(
I've perform a successful try with an ATI Radeon 5770 so I hope that isn't mean my GTX 760 doesn't support pass-through

Here is my configuration :
   Host software configuraton (Fedora 21) :
         kernel 3.17-300 without any patch (I've try 3.18 with i915 patch but I've get the save result)
         cmdline :

         GRUB_CMDLINE_LINUX="rd.lvm.lv=fedora/swap rd.lvm.lv=fedora/root rhgb quiet intel_iommu=on \
         vfio_iommu_type1.allow_unsafe_interrupts=1 \
         kvm_intel.enable_apicv=1 \
         kvm_intel.emulate_invalid_guest_state=0 \
         kvm.allow_unsafe_assigned_interrupts=1 \
         pci-stub.ids=10de:1187,10de:0e0a"

         qemu 2.2.0

   Host GPU (IGD):

          00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller

   Guest GPU and attached audio device (GTX 760) :

          01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 760]
          01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)

To run my VM, I use those commands :

./vfio-bind 0000:01:00.0 0000:01:00.1

qemu-system-x86_64 \
-enable-kvm -M q35 \
-vga none \
-m 4096 \
-boot menu=on \
-cpu host -smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device piix4-ide,bus=pcie.0,id=piix4-ide \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=${HOME}/MSI.GTX760.2048.140225.rom \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-hda ${HOME}/windowsDisk"

After running this command, dmesg display those messages :

[ 3843.261116] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[ 3849.124426] kvm: zapping shadow pages for mmio generation wraparound

Can someone give me a hand ?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

/me beats head on desk...

If you're using Intel host graphics and VGA mode for the device, you need the i915 patch.  If it didn't work for you, it's probably because you didn't enable it via the kernel commandline.

Most of your current kernel options are unnecessary.

And... since you have a Windows guest, don't use q35!

Better yet, use Windows8+ and OVMF.

Doesn't seem like you've been following the thread very carefully...

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users