Arch Linux

SchaduwBlink

Member

Registered: 2007-01-07

Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You should be using 'file'  as the disk type for '/dev/mordor/EmmaPC'. This does not look like a valid block device, like /dev/sda2. Also you will want to change the target bus from SATA to IDE, and SCSI to IDE. Here is mine:

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/home/user/virtual_machines/disks/GamingMachineOS.img'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/user/Disk_Images/Windows10_TechnicalPreview_x64_EN-US_9926.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>

It's a ZFS ZVol, so I'm installing to a real hard drive partition, with the extras that come along with ZFS. The drives are mapped to /dez/zd*, but I don't know what numbers and I was afraid they'd change on reboot, but I guess they don't.

I'll make the other changes you said and see if that works.

But I heard you can use virtio disks to install and OVMF was supposed to pick them up.

Thanks.

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've used roms from other manufacturers (equal hardware specs; gpu/#core's/clock)

My process:
1. new win7 VM (base, sytem updates) for every rom file
2. shutdown; make copy of VM on host
3. boot copy; install video drivers (start with most recent, work back)
4. working? rename and file

I threw in the towel because I couldn't take advantage of any power saving features; VM's wouldn't recover from sleep or monitor wake. Force close through host, reboot to a corrupted VM image (every time, unrecoverable), and rollback to step 3.

Careful though: I have a wireless hdmi 460ti that lost it's wireless hdmi capability through this type of tinkering (i think). Can't find the card-specific rom for it so I gambled

Haven't tried with win8

You have been playing with fire. If you have to use romfile, use the one which is for your specific card, even better which you have read yourself from your card. But most of the time you do not need romfile anyway, if the card can actually work with GPU passthrough.

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

My OVMF q35 booting is going strait to the UEFI command line. It doesn't see my drives. This is part of the libvirt xml:

    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/mordor/EmmaPC'/>
      <target dev='sda' bus='sata'/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/media/Storage/OS/en-gb_windows_8.1_enterprise_with_update_x64_dvd_4048611.iso'/>
      <target dev='sdb' bus='scsi'/>
      <readonly/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>

This is with me trying the most basic of settings for the drive. I'm using the latest git source for virt-manager and libvirt.

Hope you can help because I really want to get this set up with q35 because of the better handling of AMD cards.

whut??? AMD cards have no problem with FX440 at all, unless you are unlucky to own a model which is not suited for GPU passthrough at all. In which case Q35 is not going to help you anyway.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

100% cpu load, nothing on any output.
..deep OVMF problems..

Seems like GPU's UEFI oprom is broken. Deep debugging to be done... or not. Maybe i'll just sack the cards and buy something MORE working. Or try poking ASUS. Or something else. But seems like i'm damned with those cards and shall continue digging.

http://www.insanelymac.com/forum/topic/ … ati-cards/
Based on this thread, i've created my own, handmade ROM from legacy HD7750's ROM and HD6450's GOP EFI part.
And it boots with OVMF. Creepy stuff.

UPD: Win7 installer froze after boot logo animation, win8 goes fine.

UPD2: Driver install went well. Benchmarking time!
If you want my ROM - I can send it.

Last edited by Duelist (2015-02-20 15:56:48)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

hotfunction

Member

Registered: 2015-02-05

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If 440fx and -nographic don't help and you're looking at the right output from the GPU, I'd next be suspicious of the ROM file.  It looks like one from techpowerup.  Does it actually match your card?  Can you dump the ROM from your card instead?  Do you even need a dump of the ROM?

hi sir aw.. i tried downloading various ROMs from other sites too, but none of them seems to get me working, they still gave me black screen output on the sscreen... and it seemed to be hard to find nvidia's ROM corresponding on that... so is it okay if i change into another GPU card? will it help me to work?

That black window is the sdl graphics head for the VM which appears because you haven't used the -nographic option.  The GPU output should be on a monitor connected to the assigned GPU.  I'd also recommend, like I almost always do, using 440fx rather than q35 for the VM.  It looks like you're using a Linux guest, but afaik, the nvidia driver does not have an issue with PCIe topology in Linux (until you get to lots of GPUs and that may already be fixed too).

and also about this part... i actually still confused, after some reading and stuff, i redone the steps from nbhs's page one, after making the single script and others.. i still see the black qemu command line window when adding -vga none... and also without a rom setting, i also tried rombar=0 option and -nographic still didnt give me output.. can you give me further explanation on this.. ?? i am very confused about the black qemu command line, and how does people use -vga none without -nographics to go straight to the VMS without the black qemu command line?

Edit : i tried adding -curses instead of -nographics, and its still giving me blank output, which i can type "quit" and i can go back to the qemu, which is that i am guessing that is still the black qemu command line output again.. removing that -vga none boots -curses to the VM.. so is the black qemu command line from the -vga none output is showing because of my GPU rom incorrect? or am i missing something..?

Any help on this anyone?

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

100% cpu load, nothing on any output.
..deep OVMF problems..

Seems like GPU's UEFI oprom is broken. Deep debugging to be done... or not. Maybe i'll just sack the cards and buy something MORE working. Or try poking ASUS. Or something else. But seems like i'm damned with those cards and shall continue digging.

http://www.insanelymac.com/forum/topic/ … ati-cards/
Based on this thread, i've created my own, handmade ROM from legacy HD7750's ROM and HD6450's GOP EFI part.
And it boots with OVMF. Creepy stuff.

UPD: Win7 installer froze after boot logo animation, win8 goes fine.

UPD2: Driver install went well. Benchmarking time!
If you want my ROM - I can send it.

Congratulations. You can call your GPU "Frankenstein"

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Congratulations. You can call your GPU "Frankenstein"

I can call my whole system a "Frankenstein" now, my host gpu is tied to the case in front of the CPU cooler, connected via riser card. There's 1-1.5cm of space between steel brace and 1800rpm 120mm fan.

BTW, check out my virtual disk subsystem, it's awesome.

It's a raw image residing on now-six-years-old-750gb-wd-black-louder-than-a-CNC-mill connected to the VM via virtio-blk-pci.

Astonishing read results. I assume there's some hardcore caching involved.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

jaeger

Member

Registered: 2015-02-16

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tested some more today with Fedora 21 and everything works fine there. When I boot back into the CRUX installation I still can't get things working. I even tried dumping the exact qemu command line from the working libvirt and I see the same results, these same errors:

qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0x41c, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x624, 0x0, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x62c, 0x0, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x634, 0x0, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x63c, 0x0, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_read_config(0000:01:00.0, 0x41c, 0x4) failed: Bad address
qemu-system-x86_64: vfio_pci_write_config(0000:01:00.0, 0x484, 0x0, 0x4) failed: Bad address

At this point I'm thinking it has to be a kernel configuration issue but I haven't been able to pinpoint it so far. Searching online hasn't revealed the answer, there aren't many hits for these particular errors. Does anyone at all have an idea where I should be looking? It's getting pretty frustrating at this point, not making any progress.

For reference, my kernel config already contains these:

CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y
CONFIG_IOMMU_SUPPORT=y
CONFIG_INTEL_IOMMU=y
# CONFIG_INTEL_IOMMU_DEFAULT_ON is not set
CONFIG_KVM_VFIO=y

pci-stub.ids and intel_iommu=on are passed on the grub2 command line.

-----

EDIT: After this bit of whining I decided to try Fedora 21's kernel config on my CRUX install. Lo and behold, the GTX 970 works properly in the VM after that! So now I have to figure out exactly which option I'm missing, since the Fedora kernel includes a LOT more stuff than mine.

Now that I know I can make it work in CRUX I feel a lot better, sorry for the downer tone in the post.

Last edited by jaeger (2015-02-21 00:59:10)

Mysingen

Member

Registered: 2015-02-18

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

EDIT: After this bit of whining I decided to try Fedora 21's kernel config on my CRUX install. Lo and behold, the GTX 970 works properly in the VM after that! So now I have to figure out exactly which option I'm missing, since the Fedora kernel includes a LOT more stuff than mine.

Now that I know I can make it work in CRUX I feel a lot better, sorry for the downer tone in the post.

Have you diffed the .config files for both kernels? If that doesn't reveal anything of interest, you should check for distribution specific kernel patches. At least you shouldn't have to look anywhere else in the systems since the "adopted" Fedora kernel worked for you in CRUX.

jaeger

Member

Registered: 2015-02-16

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Have you diffed the .config files for both kernels? If that doesn't reveal anything of interest, you should check for distribution specific kernel patches. At least you shouldn't have to look anywhere else in the systems since the "adopted" Fedora kernel worked for you in CRUX.

That's what I'll be doing next, got distracted by trying to pass a usb gamepad into the VM, which is failing. For some reason I can't get it to play nicely with either my Logitech F710 or Sony DualShock 3. One thing at a time, I guess. :) As far as the kernel config goes, I didn't apply any of the Fedora patches, just used the .config with vanilla 3.18.7 and things work. So yeah, I'll be combing through that to figure out which option is the magic one.

deniv

Member

Registered: 2013-10-16

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

http://www.insanelymac.com/forum/topic/ … ati-cards/
Based on this thread, i've created my own, handmade ROM from legacy HD7750's ROM and HD6450's GOP EFI part.
And it boots with OVMF. Creepy stuff.

UPD: Win7 installer froze after boot logo animation, win8 goes fine.

UPD2: Driver install went well. Benchmarking time!
If you want my ROM - I can send it.

Thanks a lot for this message! I've managed to make my own rom. Since I liked the latest vbios for my 7750, which is a bit overclocked compared to the original, I decided to replace only the buggy GOP part. And it even works! I can boot systemrescuecd in pure EFI mode. Using DRI freezes the VM, though (not that I care at this moment).

The biggest issue now is that I can't run windows install. OVMF doesn't support UDF filesystem that is used by the install image. It refuses to detect an actual USB pendrive created by rufus. My last resort is a fat32 partition. OVMF detects it, and even starts (I get, "Press any key to boot from CD/DVD..."), just to die the moment I press a key. So I've turned on the debug console. It looks like the problem is here:

Booting EFI USB Device
 BlockSize : 512 
 LastBlock : 7FFFFF 
 Valid efi partition table header
 Valid efi partition table header
 Valid primary and Valid backup partition table
 Partition entries read block success
 Number of partition entries: 128
 start check partition entries
 End check partition entries
 Index : 0
 Start LBA : 800
 End LBA : 7FFFDE
 Partition size: 7FF7DF
 Start : 100000 End : FFFFBC00
Prepare to Free Pool
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B 7F307A40
Loading driver at 0x00010000000 EntryPoint=0x000100061C0 cdboot.efi
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF 7EFD8A18
Error: Image at 00010000000 start failed: No mapping
Image Return Status = No mapping
PlatformBdsBootFail

Wish I knew what "no mapping" means. Can somebody help me?

P.S. Installing Windows on UEFI is such a pain. :x

Last edited by deniv (2015-02-21 18:40:48)

Len

Member

Registered: 2015-01-06

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm can someone help me, with running my windows 8 rootless, using virsh (libvirt)?
I'm in libvirt group, still it can't attach my vfio-pci devices.
Also I don't know how can I handle mounting /dev/sd* disks on normal user.

Btw, there is fully working XML (on root, Windows 8 + GTX 970 + SSD as primary + 2 x HDD + mouse, keyboard + some other devices),
But maybe someone have any hints, to tune performance? My CPU is lacking on guest, don't have HT on i5-4690K, and 3 cores on host is not good enough for me : /

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>Win81</name>
  <uuid>df602579-a4f2-41b9-bb2f-6e23d65a24bd</uuid>
  <title>Windows 8.1</title>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/home/len/win/bios/ovmf_code_x64.bin</loader>
    <nvram template='/home/len/win/bios/ovmf_vars_x64.bin'>/var/lib/libvirt/qemu/nvram/Win81_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <pmu state='on'/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Haswell</model>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <on_lockfailure>poweroff</on_lockfailure>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sdb'/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' io='native'/>
      <source dev='/dev/sdc'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' io='native'/>
      <source dev='/dev/sdd'/>
      <target dev='vdc' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='scsi' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='33:45:1c:fc:6d:cd'/>
      <source bridge='br0'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </interface>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc317'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc068'/>
      </source>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/random</backend>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </rng>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='Haswell,kvm=off'/>
  </qemu:commandline>
</domain>

And some minor case, I have Sound Card, but I can't attach it using vfio-pci, don't know why...

05:00.0 Multimedia audio controller [0401]: Creative Labs CA0106 Soundblaster [1102:0007]
        Subsystem: Creative Labs SB0570 [SB Audigy SE] [1102:100a]
        Flags: bus master, medium devsel, latency 32, IRQ 19
        I/O ports at c000 [size=32]
        Capabilities: [dc] Power Management version 2
        Kernel driver in use: snd_ca0106
        Kernel modules: snd_ca0106

Any clues?

If there is someone whos trying to install Windows 8, there are working qemu args for this XML

  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='virtio-scsi-pci,id=scsi'/>
    <qemu:arg value='-drive'/>
    <qemu:arg value='file=/home/len/win/win81.iso,id=cd0,if=none'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='scsi-cd,drive=cd0'/>
    <qemu:arg value='-drive'/>
    <qemu:arg value='file=/home/len/win/virtio-win-0.1-100.iso,id=virtiocd,if=none'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='ide-cd,bus=ide.1,drive=virtiocd'/>
  </qemu:commandline>

Also weird case, I have new versions of qemu and libvirt, but

    <kvm>
      <hidden state='on'/>
    </kvm>

is not working for me, so I have use

  <qemu:commandline>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='Haswell,kvm=off'/>
  </qemu:commandline>

Oh, and thanks for help!

Last edited by Len (2015-02-21 19:03:48)

deniv

Member

Registered: 2013-10-16

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

But maybe someone have any hints, to tune performance? My CPU is lacking on guest, don't have HT on i5-4690K, and 3 cores on host is not good enough for me : /

Hugepages, vcpu pinning, iothreads for virtio-blk, and virtio for network. AFAIK, io='native' is good if you want the best latency, otherwise cache='writeback' is better because it uses the host RAM for caching.
Another option is to install Windows on baremetal and use Linux in a VM. Seriously, you want all cores and 3 full disks in your Windows VM. Again, three full disks! It looks like you're planning to use the Windows VM as your primary machine.

    <rng model='virtio'>
      <backend model='random'>/dev/random</backend>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </rng>

AFAIK, there are no virtio-rng drivers for Windows. That device is not used.

And some minor case, I have Sound Card, but I can't attach it using vfio-pci, don't know why...

05:00.0 Multimedia audio controller [0401]: Creative Labs CA0106 Soundblaster [1102:0007]
        Subsystem: Creative Labs SB0570 [SB Audigy SE] [1102:100a]
        Flags: bus master, medium devsel, latency 32, IRQ 19
        I/O ports at c000 [size=32]
        Capabilities: [dc] Power Management version 2
        Kernel driver in use: snd_ca0106
        Kernel modules: snd_ca0106

Any clues?

Post logs and how you attach it.

If there is someone whos trying to install Windows 8, there are working qemu args for this XML

  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='virtio-scsi-pci,id=scsi'/>
    <qemu:arg value='-drive'/>
    <qemu:arg value='file=/home/len/win/win81.iso,id=cd0,if=none'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='scsi-cd,drive=cd0'/>
    <qemu:arg value='-drive'/>
    <qemu:arg value='file=/home/len/win/virtio-win-0.1-100.iso,id=virtiocd,if=none'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='ide-cd,bus=ide.1,drive=virtiocd'/>
  </qemu:commandline>

One question. Why? Why do you use qemu:arg to add disks? It's so much better to use libvirt defines if you're using libvirt in the first place.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For your testing consideration: https://github.com/awilliam/linux-vfio.git tags/vfio-misc-test-20150221

A few vfio-pci things I've been playing with over the last couple days:

- Add an ids module option like pci-stub, hopefully self explanatory
- Add a vgaoptout=1 module option, also enabled by disabling CONFIG_VFIO_PCI_VGA.  This is useful if you're using OVMF because it disables VGA region access to devices bound to vfio-pci and marks them as not participating in VGA arbitration.  If you have a config like me (i915 + nvidia + amd) then i915 opts-out of VGA arbitration on the host (but not really), but DRI2 still gets disabled because there are 2 other VGA arbitration participants.  With this and OVMF, there are none and DRI2 works in the host (seems to fix a screensaver bug in the host failing to render the unlock dialog too).
- Put unused devices bound to vfio-pci in D3 power state.  This seems to save 5-10 watts on my system.  If you have a kill-a-watt or other means to measure power, please provide before/after figures.  Anecdotal fan noise might be interesting too.

This is for testing only, I intend to try to get these upstream, so please don't ask for refreshes of this series for newer/older kernels.  Thanks.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Len

Member

Registered: 2015-01-06

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for reply so...

Hugepages, vcpu pinning, iothreads for virtio-blk, and virtio for network. AFAIK, io='native' is good if you want the best latency, otherwise cache='writeback' is better because it uses the host RAM for caching.

Right now, I don't have much RAM (only 8GB), so I'll stay with native io.
Should I use hugepages, or it's not needed right now?
Disks performance is very good in my opinion, so I'll stay with this configuration.
I have switched my driver for network, thanks for hint!
Thanks for info about rnd.

Another option is to install Windows on baremetal and use Linux in a VM. Seriously, you want all cores and 3 full disks in your Windows VM. Again, three full disks! It looks like you're planning to use the Windows VM as your primary machine.

This option is not for me, don't get me wrong.. I'm trying to switch FROM Windows TO Linux+Windows. I was using very long time windows, and linux for servers (only). So those 3 disks are still using NTFS partitions, but it gonna change later.

Post logs and how you attach it.

Group 8 is added to /etc/libvirt/qemu.conf.
Stubbed '1102:0007', binded 0000:05:00.0 to vfio-pci and this is log

2015-02-22 00:10:14.074+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -name Win81 -S -machine pc-i440fx-2.2,accel=kvm,usb=off -cpu Haswell,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,kvm=off,pmu=on -drive file=/home/len/win/bios/ovmf_code_x64.bin,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/Win81_VARS.fd,if=pflash,format=raw,unit=1 -m 6144 -realtime mlock=off -smp 4,sockets=1,cores=4,threads=1 -uuid df602579-a4f2-41b9-bb2f-6e23d65a24bd -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Win81.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device lsi,id=scsi0,bus=pci.0,addr=0x2 -drive file=/dev/sdb,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/dev/sdc,if=none,id=drive-virtio-disk1,format=raw,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/dev/sdd,if=none,id=drive-virtio-disk2,format=raw,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xa,drive=drive-virtio-disk2,id=virtio-disk2 -netdev tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:fc:a7:cd,bus=pci.0,addr=0xb -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,multifunction=on,addr=0x4 -device vfio-pci,host=01:00.1,id=hostdev1,bus=pci.0,addr=0x4.0x1 -device vfio-pci,host=00:1b.0,id=hostdev2,bus=pci.0,addr=0x5 -device vfio-pci,host=05:00.0,id=hostdev3,bus=pci.0,addr=0x7 -device usb-host,hostbus=3,hostaddr=3,id=hostdev4 -device usb-host,hostbus=3,hostaddr=2,id=hostdev5 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -cpu Haswell,kvm=off -device virtio-scsi-pci,id=scsi -drive file=/home/len/win/win81.iso,id=cd0,if=none -device scsi-cd,drive=cd0 -drive file=/home/len/win/virtio-win-0.1-100.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd -msg timestamp=on
Domain id=3 is tainted: high-privileges
Domain id=3 is tainted: custom-argv
2015-02-22T00:10:14.975176Z qemu-system-x86_64: -device vfio-pci,host=05:00.0,id=hostdev3,bus=pci.0,addr=0x7: vfio: error, group 8 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
2015-02-22T00:10:14.975208Z qemu-system-x86_64: -device vfio-pci,host=05:00.0,id=hostdev3,bus=pci.0,addr=0x7: vfio: failed to get group 8
2015-02-22T00:10:14.975219Z qemu-system-x86_64: -device vfio-pci,host=05:00.0,id=hostdev3,bus=pci.0,addr=0x7: Device initialization failed.
2015-02-22T00:10:14.975232Z qemu-system-x86_64: -device vfio-pci,host=05:00.0,id=hostdev3,bus=pci.0,addr=0x7: Device 'vfio-pci' could not be initialized
2015-02-22 00:10:15.096+0000: shutting down

One question. Why? Why do you use qemu:arg to add disks? It's so much better to use libvirt defines if you're using libvirt in the first place.

Got many problems witch 'translating' this commands to libvirt XML format, so I just leaved it like that.
And if you guess about -cpu host,kvm=off... I have to, otherwise that hidden=on from libvirt, is not working for me

Oh, and what should I do to run it as normal user, any hints? Right now I have...

virsh start Win81u
error: Failed to start domain Win81u
error: Requested operation is not valid: PCI device 0000:01:00.0 is not assignable

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Len, what version of libvirt are you using?  You should not need to touch /etc/libvirt/qemu.conf and <qemu:arg> options are nothing but trouble.  They're often a prime cause of not being able to run a VM w/o screwing with permissions.  Requiring <qemu:arg> for kvm=off is a good indication that something is wrong there.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Len

Member

Registered: 2015-01-06

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Len, what version of libvirt are you using?  You should not need to touch /etc/libvirt/qemu.conf and <qemu:arg> options are nothing but trouble.  They're often a prime cause of not being able to run a VM w/o screwing with permissions.  Requiring <qemu:arg> for kvm=off is a good indication that something is wrong there.

Hello aw,

libvirtd (libvirt) 1.2.12
QEMU emulator version 2.2.0, Copyright (c) 2003-2008 Fabrice Bellard

Ops, I was going through this: https://bbs.archlinux.org/viewtopic.php … 6#p1438076
How should I fix it?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Len, what version of libvirt are you using?  You should not need to touch /etc/libvirt/qemu.conf and <qemu:arg> options are nothing but trouble.  They're often a prime cause of not being able to run a VM w/o screwing with permissions.  Requiring <qemu:arg> for kvm=off is a good indication that something is wrong there.

Hello aw,

libvirtd (libvirt) 1.2.12
QEMU emulator version 2.2.0, Copyright (c) 2003-2008 Fabrice Bellard

Ops, I was going through this: https://bbs.archlinux.org/viewtopic.php … 6#p1438076
How should I fix it?

Your libvirt version looks ok.  How are you determining that the hidden='true' option isn't working?  You should just be able to look at ps aux | grep qemu.  All of the modifications to qemu.conf in that link are wrong.  Practically all uses of <qemu:args> is also wrong.  Maybe you want to reset to a default qemu.conf, you might have broken something trying to follow those instructions.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Len

Member

Registered: 2015-01-06

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Your libvirt version looks ok.  How are you determining that the hidden='true' option isn't working?  You should just be able to look at ps aux | grep qemu.  All of the modifications to qemu.conf in that link are wrong.  Practically all uses of <qemu:args> is also wrong.  Maybe you want to reset to a default qemu.conf, you might have broken something trying to follow those instructions.

I have removed all qemu args from xml, except kvm=off.

I have run my vm without qemu:arg kvm=off, and I'm getting Error Code 43.
After checking logs for qemu command, one difference is https://www.diffchecker.com/an4xdivl this, and yet it's declared before... still don't know why it's not working.

My old qemu.conf: http://wklej.org/id/1642973/
and libvirtd.conf: http://wklej.org/id/1642974/

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Your libvirt version looks ok.  How are you determining that the hidden='true' option isn't working?  You should just be able to look at ps aux | grep qemu.  All of the modifications to qemu.conf in that link are wrong.  Practically all uses of <qemu:args> is also wrong.  Maybe you want to reset to a default qemu.conf, you might have broken something trying to follow those instructions.

I have removed all qemu args from xml, except kvm=off.

I have run my vm without qemu:arg kvm=off, and I'm getting Error Code 43.
After checking logs for qemu command, one difference is https://www.diffchecker.com/an4xdivl this, and yet it's declared before... still don't know why it's not working.

My old qemu.conf: http://wklej.org/id/1642973/
and libvirtd.conf: http://wklej.org/id/1642974/

From your diff:

-cpu Haswell,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,kvm=off,pmu=on
...
-cpu Haswell,kvm=off

Tell me again how hidden='on' isn't working...  Of course you're still getting code 43, you still have hyperv extensions enabled.  This is covered in the faq

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Len

Member

Registered: 2015-01-06

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tell me again how hidden='on' isn't working...  Of course you're still getting code 43, you still have hyperv extensions enabled.  This is covered in the faq

I guess, my first cpu declaration

-cpu Haswell,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,kvm=off,pmu=on

was ignored due to second

-cpu Haswell,kvm=off

Okay, it's working with hidden=on, sorry for problems.

About running libvirt on normal user, I'm doing something wrong?

[len@holo ~]$ virsh start Win81
error: Failed to start domain Win81
error: Requested operation is not valid: PCI device 0000:01:00.0 is not assignable

libvirtd.conf: http://wklej.org/id/1642998/
qemu.conf: http://wklej.org/id/1643001/
xml: http://wklej.org/id/1643004/

Last edited by Len (2015-02-22 01:32:58)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tell me again how hidden='on' isn't working...  Of course you're still getting code 43, you still have hyperv extensions enabled.  This is covered in the faq

I guess, my first cpu declaration

-cpu Haswell,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,kvm=off,pmu=on

was ignored due to second

-cpu Haswell,kvm=off

Okay, it's working with hidden=on, sorry for problems.

About running libvirt on normal user, I'm doing something wrong?

[len@holo ~]$ virsh start Win81
error: Failed to start domain Win81
error: Requested operation is not valid: PCI device 0000:01:00.0 is not assignable

libvirtd.conf: http://wklej.org/id/1642998/
qemu.conf: http://wklej.org/id/1643001/
xml: http://wklej.org/id/1643004/

You can find a default qemu.conf here: http://libvirt.org/git/?p=libvirt.git;a … e3;hb=HEAD

Reboot to make sure it takes effect.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Len

Member

Registered: 2015-01-06

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You can find a default qemu.conf here: http://libvirt.org/git/?p=libvirt.git;a … e3;hb=HEAD

Reboot to make sure it takes effect.

Okay got it, so safely what should I change?

My old qemu.conf

security_driver = "none"
security_default_confined = 0
user = "+0"
group = "+0"
dynamic_ownership = 1
cgroup_device_acl = [
    "/dev/null", "/dev/full", "/dev/zero",
    "/dev/random", "/dev/urandom",
    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
    "/dev/rtc","/dev/hpet", "/dev/vfio/vfio",
    "/dev/vfio/1", "/dev/vfio/7", "/dev/vfio/8"
]
hugetlbfs_mount = "/dev/hugepages"
clear_emulator_capabilities = 0
relaxed_acs_check = 1

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You should not need to touch /etc/libvirt/qemu.conf

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Len

Member

Registered: 2015-01-06

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You should not need to touch /etc/libvirt/qemu.conf

Without touching user, group:

$ virsh start Win81
error: Failed to start domain Win81
error: Requested operation is not valid: PCI device 0000:01:00.0 is not assignable

# virsh start Win81
error: Failed to start domain Win81
error: internal error: process exited while connecting to monitor: Could not access KVM kernel module: Permission denied
failed to initialize KVM: Permission denied

After changing user and group to root, I can use my vm as root, on user still same error.

Btw, forgot to ask... I have 'The drivers for this device are not installed.' on SCSI Controller in Windows 8, all the time...
PCI\VEN_1000&DEV_0012&SUBSYS_10000000&REV_00
Where can I get this drivers?

Can I catch you somewhere on IRC, aw?

Last edited by Len (2015-02-22 03:32:34)