Arch Linux

PureTryOut

Member

Registered: 2014-09-23

Posts: 64

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now when I run the VM script it gives the following error:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,x-vga=on: vfio: error no iommu_group for device

That means the IOMMU is not enabled.  Maybe it's not enabled in the kernel config.

Well that would be a bad thing, considering this kernel is made to enable pci passthrough... Well i'll report it in the AUR then.

dante82

Member

Registered: 2015-03-31

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

And, aw, wasn't there some problem wioth one of hyper-v extensions and nvidia?

I didn't feel like looking it up, I assume they're using the version that detects KVM, but not Hyper-V

First of all, thanks for the quick replies!

I tried both variants, with and without hv-extensions. With latest driver version I get the error 43, with the 340.52 a horribly slow desktop experience.

Any messages in dmesg maybe? Like IO_PAGE_FAULT?
What motherboard and CPU do you have?
And, aw, wasn't there some problem with one of hyper-v extensions and nvidia?

Nothing unusual in dmesg. I'm currently changing graphics against an old 8800GTS I have spare here. Will post the dmesg output and results shortly

In the meantime further data that I can provide:
Mainboard: Intel DH87RL
CPU: Intel 4570
RAM: 32 GB

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@dante82

Does the lag go away without the audio function assigned?  If so: http://vfio.blogspot.com/2014/09/vfio-i … ndows.html

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now when I run the VM script it gives the following error:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,x-vga=on: vfio: error no iommu_group for device

That means the IOMMU is not enabled.  Maybe it's not enabled in the kernel config.

Well that would be a bad thing, considering this kernel is made to enable pci passthrough... Well i'll report it in the AUR then.

It's unlikely its disabled in the kernel config since its based on -ARCH kernel:

try this and check if its enabled:

zcat /proc/config.gz | grep -i iommu

if not, check if its enabled in your bios

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

In the meantime further data that I can provide:
Mainboard: Intel DH87RL
CPU: Intel 4570
RAM: 32 GB

Well, your hardware should support IOMMU, H87 chipset supports vt-d and intel i5-4570(i assume you have that CPU) supports vt-d.
That's a good thing. Try following aw's advice.

Last edited by Duelist (2015-03-31 21:56:40)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

dante82

Member

Registered: 2015-03-31

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys,

to answer your comments first:

Does the lag go away without the audio function assigned?  If so: http://vfio.blogspot.com/2014/09/vfio-i … ndows.html

From what I see in the qemu call, I don't have audio assigned. Correct me if I'm wrong there, it's already late here.
By Audio, do you mean the Audio device of the GeForce (PCI ID 1:00.1)?

Well, your hardware should support IOMMU, H87 chipset supports vt-d and intel i5-4570(i assume you have that CPU) supports vt-d.

That's correct and I tried to choose my HW setup wisely considering the idle power consumption as well. Otherwise I would have gone with an Asrock ;-)

here are the results. With the GF 8800 GTS I experience the same lagging with the drivers and dmesg gives me the following output:

[  815.599961] device tap0 entered promiscuous mode
[  815.600081] br0: port 2(tap0) entered forwarding state
[  815.600090] br0: port 2(tap0) entered forwarding state
[  815.950618] vfio-pci 0000:01:00.0: enabling device (0400 -> 0403)
[  815.978917] vfio-pci 0000:03:00.0: enabling device (0500 -> 0502)
[  816.545577] irq 16: nobody cared (try booting with the "irqpoll" option)
[  816.546138] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.19.2-1-vfio #1
[  816.546139] Hardware name:                  /DH87RL, BIOS RLH8710H.86A.0327.2014.0924.1645 09/24/2014
[  816.546140]  0000000000000000 1e87b86f7f3a999a ffff88082fa83e28 ffffffff8155d0cf
[  816.546142]  0000000000000000 ffff88080a6dac00 ffff88082fa83e58 ffffffff810ce6a6
[  816.546143]  ffff88080a0b4b50 ffff88080a6dac00 0000000000000000 0000000000000010
[  816.546145] Call Trace:
[  816.546146]  <IRQ>  [<ffffffff8155d0cf>] dump_stack+0x4c/0x6e
[  816.546155]  [<ffffffff810ce6a6>] __report_bad_irq+0x36/0xd0
[  816.546157]  [<ffffffff810cea47>] note_interrupt+0x257/0x2a0
[  816.546159]  [<ffffffff810cbfae>] handle_irq_event_percpu+0xae/0x1f0
[  816.546161]  [<ffffffff810cc131>] handle_irq_event+0x41/0x70
[  816.546162]  [<ffffffff810cefd6>] handle_fasteoi_irq+0x86/0x140
[  816.546166]  [<ffffffff81018252>] handle_irq+0x22/0x40
[  816.546169]  [<ffffffff815654bf>] do_IRQ+0x4f/0xf0
[  816.546171]  [<ffffffff8156346d>] common_interrupt+0x6d/0x6d
[  816.546172]  <EOI>  [<ffffffff81421ac2>] ? poll_idle+0x42/0x80
[  816.546176]  [<ffffffff81421519>] cpuidle_enter_state+0x49/0x190
[  816.546177]  [<ffffffff81421747>] cpuidle_enter+0x17/0x20
[  816.546179]  [<ffffffff810b49c8>] cpu_startup_entry+0x388/0x410
[  816.546182]  [<ffffffff8104b55f>] start_secondary+0x1af/0x1f0
[  816.546183] handlers:
[  816.546384] [<ffffffffa004da70>] usb_hcd_irq [usbcore]
[  816.546817] [<ffffffffa047a780>] vfio_intx_handler [vfio_pci]
[  816.547298] Disabling IRQ #16
[  818.531373] irq 16: nobody cared (try booting with the "irqpoll" option)
[  818.531875] CPU: 3 PID: 501 Comm: qemu-system-x86 Not tainted 3.19.2-1-vfio #1
[  818.531876] Hardware name:                  /DH87RL, BIOS RLH8710H.86A.0327.2014.0924.1645 09/24/2014
[  818.531877]  0000000000000000 00000000d16d99f1 ffff88082fb83e28 ffffffff8155d0cf
[  818.531879]  0000000000000000 ffff88080a6dac00 ffff88082fb83e58 ffffffff810ce6a6
[  818.531880]  ffff88080a0b4b50 ffff88080a6dac00 0000000000000000 0000000000000010
[  818.531882] Call Trace:
[  818.531883]  <IRQ>  [<ffffffff8155d0cf>] dump_stack+0x4c/0x6e
[  818.531892]  [<ffffffff810ce6a6>] __report_bad_irq+0x36/0xd0
[  818.531894]  [<ffffffff810cea47>] note_interrupt+0x257/0x2a0
[  818.531896]  [<ffffffff810cbfae>] handle_irq_event_percpu+0xae/0x1f0
[  818.531897]  [<ffffffff810cc131>] handle_irq_event+0x41/0x70
[  818.531899]  [<ffffffff810cefd6>] handle_fasteoi_irq+0x86/0x140
[  818.531902]  [<ffffffff81018252>] handle_irq+0x22/0x40
[  818.531904]  [<ffffffff815654bf>] do_IRQ+0x4f/0xf0
[  818.531906]  [<ffffffff8156346d>] common_interrupt+0x6d/0x6d
[  818.531907]  <EOI>  [<ffffffff813c9f28>] ? intel_iommu_unmap+0x1a8/0x1e0
[  818.531911]  [<ffffffff813c9eb4>] ? intel_iommu_unmap+0x134/0x1e0
[  818.531913]  [<ffffffff813baedc>] iommu_unmap+0xac/0x1c0
[  818.531919]  [<ffffffffa040fa13>] vfio_remove_dma+0xc3/0x1b0 [vfio_iommu_type1]
[  818.531921]  [<ffffffffa0410204>] vfio_iommu_type1_ioctl+0x464/0xaae [vfio_iommu_type1]
[  818.531926]  [<ffffffffa05d08aa>] ? kvm_set_memory_region+0x3a/0x50 [kvm]
[  818.531929]  [<ffffffffa05d0d05>] ? kvm_vm_ioctl+0x445/0x790 [kvm]
[  818.531931]  [<ffffffffa0404fc7>] vfio_fops_unl_ioctl+0x77/0x2c0 [vfio]
[  818.531935]  [<ffffffff811e4e48>] do_vfs_ioctl+0x2f8/0x500
[  818.531937]  [<ffffffff811ef592>] ? __fget+0x72/0xb0
[  818.531939]  [<ffffffff811e50d1>] SyS_ioctl+0x81/0xa0
[  818.531941]  [<ffffffff81562909>] system_call_fastpath+0x12/0x17
[  818.531941] handlers:
[  818.532119] [<ffffffffa004da70>] usb_hcd_irq [usbcore]
[  818.532505] [<ffffffffa047a780>] vfio_intx_handler [vfio_pci]
[  818.532936] Disabling IRQ #16
[  822.210072] kvm: zapping shadow pages for mmio generation wraparound
[  830.651754] br0: port 2(tap0) entered forwarding state

With re-attaching the GF 750TI I got an error message from KVM:

KVM internal error. Suberror: 1
emulation failure
RAX=0000000000000170 RBX=00000000003c2000 RCX=0000000000010008 RDX=00000000003c2000
RSI=0000000080000005 RDI=fffff80004560180 RBP=0000000000000001 RSP=ffffd000232e72e8
R8 =0000000000000000 R9 =0000000000001000 R10=ffffd000232e7490 R11=0000000000000000
R12=0000000000000000 R13=0000000000000002 R14=ffffd000232e7488 R15=0000000000000001
RIP=ffffffffffd03000 RFL=00010246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
FS =0053 0000000066076000 00003c00 0040f300 DPL=3 DS   [-WA]
GS =002b fffff80004560000 ffffffff 00c0f300 DPL=3 DS   [-WA]
LDT=0000 0000000000000000 ffffffff 00c00000
TR =0040 fffff80005f01080 00000067 00008b00 DPL=0 TSS64-busy
GDT=     fffff80005f00000 0000007f
IDT=     fffff80005f00080 00000fff
CR0=80050031 CR2=000000001026efd8 CR3=000000016bb4a000 CR4=001506f8
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <0f> 01 c1 c3 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

After a restart of the VM, the 340 driver shows an error 43 message (which didn't come up before).

BUT: After reinstalling the driver and rebooting the VM again, the lagging has gone.
I don't understand why but here's typical output I see from dmesg:

[  144.160027] device tap0 entered promiscuous mode
[  144.160181] br0: port 2(tap0) entered forwarding state
[  144.160189] br0: port 2(tap0) entered forwarding state
[  144.522094] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e@0x258
[  144.522107] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[  144.525417] vfio-pci 0000:03:00.0: enabling device (0500 -> 0502)
[  150.145059] kvm: zapping shadow pages for mmio generation wraparound
[  159.222102] br0: port 2(tap0) entered forwarding state

I'm happy for now but the stability of the complete setup still worries me. Any further ideas on where to investigate?

Last edited by dante82 (2015-03-31 22:19:37)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@dante82:

-device vfio-pci,host=01:00.1 \

That's your line, and it passes your gpu's audio function.
First dmesg error just asking for aw's fix about MSI - IRQ #16 happened, but nobody cared. Something hints me that this irq 16 belong to 01:00.1 host device.

The solutions are:
1.Ditch out host device 01:00.1
2.Follow aw's advice on forcing it use MSI
3.Ditch the software part of driver installation doing anything with audio.
4.Get the IRQ #16 disabled via kernel error everytime.

Oh, and what the hell is host device 03:00.0? Usb controller? Studying your logs further gives me some idea about usb.

Last edited by Duelist (2015-03-31 22:21:08)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

dante82

Member

Registered: 2015-03-31

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@dante82:

-device vfio-pci,host=01:00.1 \

That's your line, and it passes your gpu's audio function.
First dmesg error just asking for aw's fix about MSI - IRQ #16 happened, but nobody cared. Something hints me that this irq 16 belong to 01:00.1 host device.

The solutions are:
1.Ditch out host device 01:00.1
2.Follow aw's advice on forcing it use MSI
3.Ditch the software part of driver installation doing anything with audio.
4.Get the IRQ #16 disabled via kernel error everytime.

I just had the same idea (see my edited post). ;-)

Thanks a lot for your advices! I will investigate further tomorrow and see how it goes.

dante82

Member

Registered: 2015-03-31

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Oh, and what the hell is host device 03:00.0? Usb controller? Studying your logs further gives me some idea about usb.

Yeah right, I forgot to mention that small detail:

03:00.0 USB controller: VIA Technologies, Inc. Device 3483 (rev 01) (prog-if 30 [XHCI])
	Subsystem: VIA Technologies, Inc. Device 3483
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 33
	Region 0: Memory at f7800000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: <access denied>
	Kernel driver in use: vfio-pci
	Kernel modules: xhci_pci

PureTryOut

Member

Registered: 2014-09-23

Posts: 64

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's unlikely its disabled in the kernel config since its based on -ARCH kernel:

try this and check if its enabled:

zcat /proc/config.gz | grep -i iommu

if not, check if its enabled in your bios

Well it seemed my BIOS got reset somehow without mentioning me, probably when I plugged the host GPU in again.
Vt-d was disabled, enabled it, and it worked again. Thanks for your help guys!

erganzi

Member

Registered: 2014-07-09

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, aw:

I use linux-3.18.6 and qemu-2.1.3 and can passthrough AMD R5000 successful.
but I can not see the VM bios startup message on screen, it direct go to the windows 7 startup process.

give me some idea, thanks.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@erganzi

-boot menu=on[,splash-time=$timeout_ms]  ?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

darkstego

Member

Registered: 2014-08-19

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys,

I was able to get Windows 8.1 running with VFIO and OVMF, but I have an odd issue.

After the windows install, OVMF will not boot windows if I remove the drive with the install media

Here is my running script.

sudo qemu-system-x86_64 \
  -L $HOME/Desktop/ovmf -bios bios.bin \
  -enable-kvm -m 8192 -smp 4,cores=2,threads=2,sockets=1 \
  -cpu host,kvm=off \
  -device virtio-scsi-pci,id=scsi \
  -drive file=$HOME/windows8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install \
  -drive file=/dev/sdb,id=disk_primary,format=raw,if=none -device scsi-hd,drive=disk_primary \
  -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \
  -device vfio-pci,host=01:00.1 \
  -vga none \
  -rtc base=localtime,clock=host \
  -soundhw hda \
  -device vfio-pci,host=05:00.0

The trouble is if I omit the

  -drive file=$HOME/windows8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install 

line or move it after the disk drive the EFI will not find anything to boot.

I am sure the windows installed in EFI mode (GPT partition and everything), but I can't seem to convince OVMF to boot into it without the iso present.

I would appreciate any advice on the matter.

erganzi

Member

Registered: 2014-07-09

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@erganzi

-boot menu=on[,splash-time=$timeout_ms]  ?

Try:
-boot menu=on,splash-time=20000

It still display nothing, just black screen, then turned into windows startup.

Naruni

Member

Registered: 2010-08-05

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Any idea why this command:

virt-install --name=win7 --os-type=windows --disk path=/dev/mapper/VMssd-win7 --cdrom=/vm/win7x64.iso --graphics spice --ram=2048 --boot loader_type=pflash,loader_ro=yes,loader=/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd,nvram_template=/vm/win7_VARS-pure-efi.fd

Results in this:

ERROR    Unknown options ['loader_ro', 'nvram_template', 'loader_type']

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Any idea why this command:

virt-install --name=win7 --os-type=windows --disk path=/dev/mapper/VMssd-win7 --cdrom=/vm/win7x64.iso --graphics spice --ram=2048 --boot loader_type=pflash,loader_ro=yes,loader=/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd,nvram_template=/vm/win7_VARS-pure-efi.fd

Results in this:

ERROR    Unknown options ['loader_ro', 'nvram_template', 'loader_type']

Maybe it's /etc/libvirtd/qemu.conf error, and there's no OVMF line there as mentioned in OVMF whitepaper?

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Naruni

Member

Registered: 2010-08-05

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nvram = [ "/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd:/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd" ]

That's in /etc/libvirt/qemu.conf

It almost seems like the error is telling me that virt-install isnt capable of those options (loader_ro, nvram_template and loader_type) even though it is documented here and here.

I think you are right duelist, because I have noticed in vart-manager I am lacking the option for setting OVMF instead of BIOS in overview.

Edit: virt-install not virsh

Even using the whitepaper as an example, I get the same error:

LDR="loader=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,loader_ro=yes,loader_type=pflash"
virt-install \ 
>       --name fedora20 \
>       --memory 2048 \
>       --vcpus 2 \
>       --os-variant fedora20 \
>       --boot hd,cdrom,$LDR \
>       --disk size=20 \
>       --disk path=Fedora-Live-Xfce-x86_64-20-1.iso,device=cdrom,bus=scsi

ERROR    Unknown options ['loader_ro', 'loader_type']

Yes, I have a problem:

$ virt-install --boot=?
--boot options:
  arch
  cdrom
  clearxml
  domain_type
  dtb
  emulator
  fd
  hd
  init
  initargs
  initrd
  kernel
  kernel_args
  loader
  machine
  menu
  network
  os_type
  useserial

I should have loader, loader_ro, loader_type, nvram and nvram_template...

Last edited by Naruni (2015-04-01 17:59:01)

tritron4

Member

Registered: 2012-04-14

Posts: 153

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well libvirt git from aur lets you select ovmf. I am using ovmf pure from rpm and virt-manager works fine with ovmf

nvram = [ "/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd:/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd" ]

That's in /etc/libvirt/qemu.conf

It almost seems like the error is telling me that virt-install isnt capable of those options (loader_ro, nvram_template and loader_type) even though it is documented here and here.

I think you are right duelist, because I have noticed in vart-manager I am lacking the option for setting OVMF instead of BIOS in overview.

Edit: virt-install not virsh

Even using the whitepaper as an example, I get the same error:

LDR="loader=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,loader_ro=yes,loader_type=pflash"
virt-install \ 
>       --name fedora20 \
>       --memory 2048 \
>       --vcpus 2 \
>       --os-variant fedora20 \
>       --boot hd,cdrom,$LDR \
>       --disk size=20 \
>       --disk path=Fedora-Live-Xfce-x86_64-20-1.iso,device=cdrom,bus=scsi

ERROR    Unknown options ['loader_ro', 'loader_type']

Yes, I have a problem:

$ virt-install --boot=?
--boot options:
  arch
  cdrom
  clearxml
  domain_type
  dtb
  emulator
  fd
  hd
  init
  initargs
  initrd
  kernel
  kernel_args
  loader
  machine
  menu
  network
  os_type
  useserial

I should have loader, loader_ro, loader_type, nvram and nvram_template...

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nvram = [ "/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd:/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd" ]

That's in /etc/libvirt/qemu.conf

It almost seems like the error is telling me that virt-install isnt capable of those options (loader_ro, nvram_template and loader_type) even though it is documented here and here.

Y'sure that your software is updated? Seems like old libvirtd on host to me.

[user@crossfire ~]$ yum info libvirt
Loaded plugins: langpacks
Available Packages
Name        : libvirt
Arch        : x86_64
Version     : 1.2.9.2
Release     : 1.fc21
Size        : 43 k
Repo        : updates/21/x86_64

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

bierbauch

Member

Registered: 2015-03-28

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,
it's me again ... :'(
I got the 4790k today and have a fresh arch on a drive but sadly no success with my setup - I have no Idea what to try next, can't figure out how to create the iommu group other than with the vfio-bind script provided by the op.

This is the provided error msg:
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: Device 'vfio-pci' could not be initialized

// journalctl:
Apr 01 22:23:36 teststation kernel: Intel-IOMMU: enabled
Apr 01 22:23:36 teststation kernel: pci-stub: add 10DE:13C2 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
Apr 01 22:23:36 teststation kernel: pci-stub 0000:01:00.0: claimed by stub
Apr 01 22:23:36 teststation kernel: pci-stub: add 10DE:0FBB sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
Apr 01 22:23:36 teststation kernel: pci-stub 0000:01:00.1: claimed by stub
Apr 01 22:24:04 teststation kernel: vfio-pci: probe of 0000:01:00.0 failed with error -22
Apr 01 22:24:04 teststation kernel: vfio-pci: probe of 0000:01:00.0 failed with error -22
Apr 01 22:24:04 teststation kernel: vfio-pci: probe of 0000:01:00.1 failed with error -22

A short Rundown of my configuration:

i7 4970K
Nvidia GTX 970
Asus Sabertooth Z87
2 SSD Mirrors (one for arch, one for windows)
16 GB RAM

Monitor1 --> Intel HD4600 HDMI
Monitor2 --> Intel HD4600 DisplayPort
Monitor3 --> GTX 970 Displayport

$ lspci -nnv | grep NVIDIA
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1) (prog-if 00 [VGA controller])
    Subsystem: NVIDIA Corporation Device [10de:1116]
01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
    Subsystem: NVIDIA Corporation Device [10de:1116]

$ sed -n 4p /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet pcie_acs_override=downstream i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:13c2,10de:0fbb"

$ sed -n 7p /etc/mkinitcpio.conf
MODULES="pci-stub"

$lsmod|egrep -i "vfio|pci_stub|kvm"
vfio_pci               35525  0
vfio_iommu_type1       17118  0
vfio                   18477  2 vfio_iommu_type1,vfio_pci
kvm_intel             143417  0
kvm                   435299  1 kvm_intel
pci_stub               12429  0

If you need any further Info on my system, pls let me know ...
Regards.

Edit: Added System Log.

Last edited by bierbauch (2015-04-01 20:51:16)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This is the provided error msg:
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: Device 'vfio-pci' could not be initialized

Look back a page or two; VT-d is not enabled, probably in you BIOS.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

bierbauch

Member

Registered: 2015-03-28

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This is the provided error msg:
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: Device 'vfio-pci' could not be initialized

Look back a page or two; VT-d is not enabled, probably in you BIOS.

Damn  ...
You are right. The only good thing is, I figured it out before I read your post

MAAAAN I feel so dumb ^^ - The CPU swap disabled the feature and I forgot to recheck!

Got it working. Let the fun begin!

marcosonoio

Member

Registered: 2014-04-14

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,
thanks to all for this useful topic.
in these days i'm trying to have a working VM with windows 8.1, but i have some problems.
with this config i can install windows, boot up and use windows, but when i try to install nvidia driver (340.52), i received a BSOD (with a video driver's error):

qemu-system-x86_64 -enable-kvm -m 2048 -cpu host,kvm=off -smp 4 \
-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd \
-device vfio-pci,host=01:00.0,romfile=/home/marco/image.rom,multifunction=on \
-device vfio-pci,host=01:00.1 \
-device virtio-scsi-pci,id=scsi \
-drive file=/home/marco/it_windows_8.1_with_update_x64_dvd_4048528.iso,id=isocd,if=none -device scsi-cd,drive=isocd \
-drive file=/home/marco/Scaricati/virtio-win-0.1-100.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \
-drive file=/home/marco/win81.raw,id=disk,format=raw,if=none -device scsi-hd,drive=disk \
-rtc base=localtime,clock=host \
-soundhw hda \
-vga none

so i want to try with the q25 chipset, i can't boot up windows, but i can install the system with the "-vga std" option:

qemu-system-x86_64 -enable-kvm -m 2048 -cpu host,kvm=off -smp 4 -M q35 \
-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,romfile=/home/marco/image.rom,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device virtio-scsi-pci,id=scsi \
-drive file=/home/marco/it_windows_8.1_with_update_x64_dvd_4048528.iso,id=isocd,if=none -device scsi-cd,drive=isocd \
-drive file=/home/marco/Scaricati/virtio-win-0.1-100.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \
-drive file=/home/marco/win81.raw,id=disk,format=raw,if=none -device scsi-hd,drive=disk \
-rtc base=localtime,clock=host \
-soundhw hda \
-vga none

when i start the VM i find this in the dmesg output(always):

[   47.485338] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb000 flags=0x0010]
[   47.485345] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb3c0 flags=0x0010]
[   47.485347] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb040 flags=0x0010]
[   47.485348] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb400 flags=0x0010]
[   47.485350] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb440 flags=0x0010]
[   47.485351] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb080 flags=0x0010]
[   47.485353] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb680 flags=0x0010]
[   47.485354] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb0c0 flags=0x0010]
[   47.485355] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb480 flags=0x0010]
[   47.485356] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb100 flags=0x0010]
[   47.485358] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb4c0 flags=0x0010]
[   47.485359] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb140 flags=0x0010]
[   47.485360] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb500 flags=0x0010]
[   47.485361] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb180 flags=0x0010]
[   47.485363] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb540 flags=0x0010]
[   47.485364] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb1c0 flags=0x0010]
[   47.485365] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb580 flags=0x0010]
[   47.485367] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb200 flags=0x0010]

my system:

amd a10 7850k
gigabyte GA-F2A88XN-WIFI
2x4gb ddr3
ssd 128gb 840pro
nvidia gtx 750 ti
updated fedora 21 + libvirt preview repo

i want to install the 340.52 to not have problem with the hyperv(nvidia related).
what's wrong?

thanks you

Last edited by marcosonoio (2015-04-02 20:49:14)

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Update:  so the -smp answers were helpful, thanks.  Apparently Win 7 under q35 and 440fx deals somewhat differently re number of processors.  First I had looked under Computer->Properties, then under Device Manager->Processors.   Both setups show 6 processors under Device Manager, which I guess is the important thing.  Under Computer->Properties the q35 just mentions the processor name without a number; presumably that means 1 physical processor 6 virtual cores. Under the 440fx setup it says 2 processors; presumably 2 "physical" with 3 virtual each.  Weird and probably unimportant.  If I get a chance I'll see if it changes the "windows experience" processor score.

As far as the USB xhci driver vs passthrough: I haven't had a chance to experiment much yet, but thanks for the extra parameters to try.  I'm pretty sure Win 8/8.1/server 2012 and the rest of the UEFI aware OSes are probably not a good indicator of success with Win 7.

Naruni

Member

Registered: 2010-08-05

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yep, arch's current libvirt is version 1.2.14-1-x86_64
I wonder if I'll go to AUR libvirt...