You are not logged in.

#4701 2015-04-03 12:54:57

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

marcosonoio wrote:

Hi,
thanks to all for this useful topic.
in these days i'm trying to have a working VM with windows 8.1, but i have some problems.
with this config i can install windows, boot up and use windows, but when i try to install nvidia driver (340.52), i received a BSOD (with a video driver's error):

qemu-system-x86_64 -enable-kvm -m 2048 -cpu host,kvm=off -smp 4 \
-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd \
-device vfio-pci,host=01:00.0,romfile=/home/marco/image.rom,multifunction=on \
-device vfio-pci,host=01:00.1 \
..snip..
-vga none

so i want to try with the q25 chipset, i can't boot up windows, but i can install the system with the "-vga std" option:

qemu-system-x86_64 -enable-kvm -m 2048 -cpu host,kvm=off -smp 4 -M q35 \
-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,romfile=/home/marco/image.rom,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
..snip..
-vga none

when i start the VM i find this in the dmesg output(always):

[   47.485338] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb000 flags=0x0010]
[   47.485345] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb3c0 flags=0x0010]
..snip..
[   47.485367] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0022 address=0x000000007efcb200 flags=0x0010]

my system:

amd a10 7850k
gigabyte GA-F2A88XN-WIFI
2x4gb ddr3
ssd 128gb 840pro
nvidia gtx 750 ti
updated fedora 21 + libvirt preview repo

i want to install the 340.52 to not have problem with the hyperv(nvidia related).
what's wrong?

thanks you

First hint where to look:
Your CPU has bugs in IOMMU, they're listed there if i recall your CPU family number right.
That, i believe(haven't red IOMMU architectural guide yet), yelds those IO_PAGE_FAULT errors.
Also, seems like you're using your integrated card as the host gpu, there may be problems with that, but there's none i've heard of.
And you shouldn't absolutely need x-vga option when using OVMF, as stated in aw's presentation.
Try disabling x-vga.

The -vga std option just connects a virtual, emulated graphics card, making your GPU secondary in windows but not in OVMF setup menus.

EDIT:
Forgot to add a reassuring note: i have amd athlon x4 750k, which is based on a similar architecture as yours(and opteron 6300 series too), and asus F2A55 motherboard, and it works with OVMF only.

Last edited by Duelist (2015-04-03 14:22:05)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4702 2015-04-03 14:55:07

larrabee
Member
Registered: 2014-08-14
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello. I have troubles with vfio.
I have:
Motherboard ASRock Z87 Extreme4
CPU Intel core i7 4771
video AMD 6450
video IGD
SAS adapter Adaptec 7805h
OS: Arch
kernel: stock  (3.19.2-1)
I connect video to vfio and power up VM, but gave error «vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.»

lspci output:

00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 04)
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 04)
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4)
00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4)
00:1c.6 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d4)
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 04)
00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 04)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series]
02:00.0 Serial Attached SCSI controller: Adaptec PMC-Sierra PM8018 SAS HBA [Series 7H] (rev 06)
04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
05:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03)

lspci -n output:

00:00.0 0600: 8086:0c00 (rev 06)
00:01.0 0604: 8086:0c01 (rev 06)
00:01.1 0604: 8086:0c05 (rev 06)
00:02.0 0300: 8086:0412 (rev 06)
00:03.0 0403: 8086:0c0c (rev 06)
00:14.0 0c03: 8086:8c31 (rev 04)
00:16.0 0780: 8086:8c3a (rev 04)
00:19.0 0200: 8086:153b (rev 04)
00:1a.0 0c03: 8086:8c2d (rev 04)
00:1b.0 0403: 8086:8c20 (rev 04)
00:1c.0 0604: 8086:8c10 (rev d4)
00:1c.3 0604: 8086:8c16 (rev d4)
00:1c.6 0604: 8086:244e (rev d4)
00:1d.0 0c03: 8086:8c26 (rev 04)
00:1f.0 0601: 8086:8c44 (rev 04)
00:1f.2 0106: 8086:8c02 (rev 04)
00:1f.3 0c05: 8086:8c22 (rev 04)
01:00.0 0300: 1002:6779
01:00.1 0403: 1002:aa98
02:00.0 0107: 9005:8088 (rev 06)
04:00.0 0200: 8086:1539 (rev 03)
05:00.0 0604: 1b21:1080 (rev 03)

cmdline:

initrd=\intel-ucode.img initrd=\initramfs-linux.img root=/dev/sda2 rw intel_iommu=on iommu=on elevator=noop pci-stub.ids=1002:6779,1002:aa98

ll /sys/bus/pci/devices/*/iommu_group:

lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:00:00.0/iommu_group -> ../../../kernel/iommu_groups/0/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:00:01.0/iommu_group -> ../../../kernel/iommu_groups/1/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:00:01.1/iommu_group -> ../../../kernel/iommu_groups/1/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:00:02.0/iommu_group -> ../../../kernel/iommu_groups/2/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:00:03.0/iommu_group -> ../../../kernel/iommu_groups/3/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:00:14.0/iommu_group -> ../../../kernel/iommu_groups/4/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:00:16.0/iommu_group -> ../../../kernel/iommu_groups/5/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:00:19.0/iommu_group -> ../../../kernel/iommu_groups/6/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:00:1a.0/iommu_group -> ../../../kernel/iommu_groups/7/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:00:1b.0/iommu_group -> ../../../kernel/iommu_groups/8/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:00:1c.0/iommu_group -> ../../../kernel/iommu_groups/9/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:00:1c.3/iommu_group -> ../../../kernel/iommu_groups/10/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:00:1c.6/iommu_group -> ../../../kernel/iommu_groups/11/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:00:1d.0/iommu_group -> ../../../kernel/iommu_groups/12/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:00:1f.0/iommu_group -> ../../../kernel/iommu_groups/13/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:00:1f.2/iommu_group -> ../../../kernel/iommu_groups/13/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:00:1f.3/iommu_group -> ../../../kernel/iommu_groups/13/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:01:00.0/iommu_group -> ../../../../kernel/iommu_groups/1/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:01:00.1/iommu_group -> ../../../../kernel/iommu_groups/1/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:02:00.0/iommu_group -> ../../../../kernel/iommu_groups/1/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:04:00.0/iommu_group -> ../../../../kernel/iommu_groups/14/
lrwxrwxrwx 1 root root 0 апр  3 10:57 /sys/bus/pci/devices/0000:05:00.0/iommu_group -> ../../../../kernel/iommu_groups/11/

Video card and SAS adapter in same iommu group (1)

I try:
Use kernel with ACS override patch
Use option vfio_iommu_type1 allow_unsafe_interrupts=1
It doesnot help.

How i can fix this error, please help. Thank you.

Offline

#4703 2015-04-03 15:35:58

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

larrabee wrote:

Hello. I have troubles with vfio.
I have:
Motherboard ASRock Z87 Extreme4
CPU Intel core i7 4771
video AMD 6450
video IGD
SAS adapter Adaptec 7805h
OS: Arch
kernel: stock  (3.19.2-1)
I connect video to vfio and power up VM, but gave error «vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.»

..snip..
Video card and SAS adapter in same iommu group (1)

I try:
Use kernel with ACS override patch
Use option vfio_iommu_type1 allow_unsafe_interrupts=1
It doesnot help.

How i can fix this error, please help. Thank you.

You sure you enabled your acs override mode? There's a kernel command line parameter for that:

+	pcie_acs_override =
+			[PCIE] Override missing PCIe ACS support for:
+		downstream
+			All downstream ports - full ACS capabilties
+		multifunction
+			All multifunction devices - multifunction ACS subset
+		id:nnnn:nnnn
+			Specfic device - full ACS capabilities
+			Specified as vid:did (vendor/device ID) in hex

And are you sure you need that "allow_unsafe_interrupts=1"?
Did you read aw's blog post about it?(question 8)

Last edited by Duelist (2015-04-03 15:36:11)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4704 2015-04-03 17:53:01

apex8
Member
Registered: 2014-03-29
Posts: 60

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi together,

I managed to create an OVMF Windows 7 VM. After patching and installing drivers I have two problems left.
1: Starting the VM isn't very reliable. Right after the "Windows is starting" screen it can happen that the VM turns off. Qemu just displays a message that the index moved from 0 to <higher number>. After some attempts the VM starts.
2: It appears that I can passthrough my three devices (USB 3.0 controller, graphics adapter, sound card) only the first time I start the VM. When the VM turns off as described in [1] the devices are missing in the VM. The VM is still accessible via RDP, but in the device manager the devices aren't listed.

Is this related to the OVMF Setup? Do I need to release the devices in vfio explicitly?

Thanks in advance!

taskset 0x0000003F \
qemu-system-x86_64 \
-nodefaults \
-name win7uefi \
-enable-kvm \
-machine type=pc,accel=kvm \
-cpu host,kvm=off \
-smp 6,sockets=1,cores=3,threads=2 \
-m 4096 \
-mem-path /dev/hugepages \
-mem-prealloc \
-balloon none \
-k de \
-drive if=pflash,format=raw,readonly,file=ovmf_code_x64.bin \
-drive if=pflash,format=raw,file=ovmf_vars_x64.bin \
-drive if=none,file=/dev/sdb3,id=vdisk1,cache=none,format=raw,aio=native \
-device virtio-blk-pci,scsi=off,config-wce=off,drive=vdisk1,id=disk1,x-data-plane=on \
-drive if=none,file=/dev/sdb7,id=vdisk2,cache=none,format=raw,aio=native \
-device virtio-blk-pci,scsi=off,config-wce=off,drive=vdisk2,id=disk2,x-data-plane=on \
-device vfio-pci,host=01:00.0 \
-device vfio-pci,host=01:00.1 \
-device vfio-pci,host=03:00.0 \
-device vfio-pci,host=07:00.0 \
-net nic,macaddr=52:54:00:12:34:57,model=virtio -net bridge,br=kvmbr0 \
-spice port=5930,disable-ticketing \
-device virtio-serial-pci \
-device virtserialport,chardev=spicechannel0,name=com.redhat.spice.0 \
-chardev spicevmc,id=spicechannel0,name=vdagent \
-rtc base=utc \
-monitor unix:user.sock,server,nowait \
-daemonize

Offline

#4705 2015-04-03 18:06:38

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

Hi together,

I managed to create an OVMF Windows 7 VM. After patching and installing drivers I have two problems left.
1: Starting the VM isn't very reliable. Right after the "Windows is starting" screen it can happen that the VM turns off. Qemu just displays a message that the index moved from 0 to <higher number>. After some attempts the VM starts.
2: It appears that I can passthrough my three devices (USB 3.0 controller, graphics adapter, sound card) only the first time I start the VM. When the VM turns off as described in [1] the devices are missing in the VM. The VM is still accessible via RDP, but in the device manager the devices aren't listed.

Is this related to the OVMF Setup? Do I need to release the devices in vfio explicitly?

Thanks in advance!

taskset 0x0000003F \
qemu-system-x86_64 \
-nodefaults \
-name win7uefi \
-enable-kvm \
-machine type=pc,accel=kvm \
-cpu host,kvm=off \
-smp 6,sockets=1,cores=3,threads=2 \
-m 4096 \
-mem-path /dev/hugepages \
-mem-prealloc \
-balloon none \
-k de \
-drive if=pflash,format=raw,readonly,file=ovmf_code_x64.bin \
-drive if=pflash,format=raw,file=ovmf_vars_x64.bin \
-drive if=none,file=/dev/sdb3,id=vdisk1,cache=none,format=raw,aio=native \
-device virtio-blk-pci,scsi=off,config-wce=off,drive=vdisk1,id=disk1,x-data-plane=on \
-drive if=none,file=/dev/sdb7,id=vdisk2,cache=none,format=raw,aio=native \
-device virtio-blk-pci,scsi=off,config-wce=off,drive=vdisk2,id=disk2,x-data-plane=on \
-device vfio-pci,host=01:00.0 \
-device vfio-pci,host=01:00.1 \
-device vfio-pci,host=03:00.0 \
-device vfio-pci,host=07:00.0 \
-net nic,macaddr=52:54:00:12:34:57,model=virtio -net bridge,br=kvmbr0 \
-spice port=5930,disable-ticketing \
-device virtio-serial-pci \
-device virtserialport,chardev=spicechannel0,name=com.redhat.spice.0 \
-chardev spicevmc,id=spicechannel0,name=vdagent \
-rtc base=utc \
-monitor unix:user.sock,server,nowait \
-daemonize

Errr, did you use -device qxl-vga as stated in OVMF whitepaper for installing the drivers stage?

I'm currently using libvirt, my VM starts, showing me "starting windowns" screen on QXL graphics in a window on host, then, after some black screen time, it fires up the main GPUs and they start outputting the video signal to the screen properly.

And what do you mean "patching the drivers"?


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4706 2015-04-03 18:19:41

apex8
Member
Registered: 2014-03-29
Posts: 60

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes I needed qxl-vga during the setup. But after I installed the nvidia drivers I removed this device. This is working so far. The last thing I added to the startup script was adding the second disk partition. I removed that one and now I got a fine startup with all devices working well. Is something wrong with that parameter, maybe bus related?
Anyway I do not know if this is a oneshot or if that "guest moved index from 0 to ..." issue still persists.

EDIT: I meant patching windows (updating) and installing device drivers.

EDIT2: Sorry, I had a Seabios VM before and thus I might got something wrong. I was sure that removing the qxl-vga was intended after the drivers of the dedicated graphics device are installed. But I think what you wanted to tell me, that they always better should be present in the VM. Right?

Last edited by apex8 (2015-04-03 18:45:01)

Offline

#4707 2015-04-03 18:51:00

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

Yes I needed qxl-vga during the setup. But after I installed the nvidia drivers I removed this device. This is working so far. The last thing I added to the startup script was adding the second disk partition. I removed that one and now I got a fine startup with all devices working well. Is something wrong with that parameter, maybe bus related?
Anyway I do not know if this is a oneshot or if that "guest moved index from 0 to ..." issue still persists.

EDIT: I meant patching windows (updating) and installing device drivers.

EDIT2: Sorry, I had a Seabios VM before and thus I might got something wrong. I was sure that removing the qxl-vga was intended after the drivers of the dedicated graphics device are installed. But I think what you wanted to tell me, that they always better should be present in the VM. Right?

Well, even if nothing works, qxl-vga still works. Windows turns off QXL for me. For some reason, i couldn't make my VM output "starting windows" message on the main GPU, it works only on qxl-vga. I don't know why, but let it be. Well, it's intended to remove the QXL.

Try adding -device virtio-blk-pci for the second partition and see if that makes it work. Seems like i'm unable help you anymore.

Last edited by Duelist (2015-04-03 18:51:13)


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4708 2015-04-04 12:17:01

apex8
Member
Registered: 2014-03-29
Posts: 60

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is it normal for windows guests, that the red hat QXL graphics device is reported with code 43 in the device manager? However the dedicated nvidia device is working normally.

Offline

#4709 2015-04-04 13:24:30

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

Is it normal for windows guests, that the red hat QXL graphics device is reported with code 43 in the device manager? However the dedicated nvidia device is working normally.

It's strange, but totally fine.
Actually, you can just disconnect it totally.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4710 2015-04-04 14:42:56

lersek
Member
Registered: 2015-03-15
Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

darkstego wrote:

Hey guys,

I was able to get Windows 8.1 running with VFIO and OVMF, but I have an odd issue.

After the windows install, OVMF will not boot windows if I remove the drive with the install media

Here is my running script.

sudo qemu-system-x86_64 \
  -L $HOME/Desktop/ovmf -bios bios.bin \
  ...
  -drive file=$HOME/windows8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install \
  -drive file=/dev/sdb,id=disk_primary,format=raw,if=none -device scsi-hd,drive=disk_primary \
  ...

The trouble is if I omit the

  -drive file=$HOME/windows8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install 

line or move it after the disk drive the EFI will not find anything to boot.

Please stop using -L and -bios. Refer to the wiki or the example in the whitepaper wrt. the pflash drives instead.

Please capture the debug output with "-debugcon file:debug.log -global isa-debugcon.iobase=0x402"; if there's a problem with processing QEMU's "bootorder" fw_cfg file, the debug log should help us understand it.

Also, how recent is your OVMF build?

Offline

#4711 2015-04-04 14:46:21

lersek
Member
Registered: 2015-03-15
Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Naruni wrote:

Any idea why this command:

virt-install --name=win7 --os-type=windows --disk path=/dev/mapper/VMssd-win7 --cdrom=/vm/win7x64.iso --graphics spice --ram=2048 --boot loader_type=pflash,loader_ro=yes,loader=/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd,nvram_template=/vm/win7_VARS-pure-efi.fd

Results in this:

ERROR    Unknown options ['loader_ro', 'nvram_template', 'loader_type']

Your virt-manager (or virt-install) package (or build) is too old.

Offline

#4712 2015-04-04 14:57:07

lersek
Member
Registered: 2015-03-15
Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:
aw wrote:

@erganzi

-boot menu=on[,splash-time=$timeout_ms]  ?

Try:
-boot menu=on,splash-time=20000

It still display nothing, just black screen, then turned into windows startup.

Please describe the symptoms in more detail: command line or domain XML, OVMF debug log (see capture notes above), what displays are configured, which one is the display that you see stuff on, etc.

Also, I doubt qemu 2.1.3 is recent enough (Alex will know better); as far as I remember, for my Nvidia GTX750 it was too old. I needed commit fe08275d, which implies 2.2.

Other than that, OVMF indeed supports '-boot menu=on[,splash-time=$timeout_ms]'. The behavior is not identical to that of SeaBIOS (as the qemu manual explains, these options have firmware-specific behavior), but it should be a close approximation. See https://github.com/tianocore/edk2/commit/9253c14d and its surrounding commits.

Offline

#4713 2015-04-04 15:12:52

larrabee
Member
Registered: 2014-08-14
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

You sure you enabled your acs override mode? There's a kernel command line parameter for that:
...
And are you sure you need that "allow_unsafe_interrupts=1"?
Did you read aw's blog post about it?(question 8)

Duelist, thank you.
Option pcie_acs_override=downstream resolve it. Now SAS and video card in different groups.
Thank you.

Offline

#4714 2015-04-04 15:14:21

lersek
Member
Registered: 2015-03-15
Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

Hi together,

I managed to create an OVMF Windows 7 VM. After patching and installing drivers I have two problems left.
1: Starting the VM isn't very reliable. Right after the "Windows is starting" screen it can happen that the VM turns off. Qemu just displays a message that the index moved from 0 to <higher number>. After some attempts the VM starts.

This should "never happen", bar guest bugs. The QEMU error message reports a guest bug in the virtio communication. I've encountered the error message once (it was related to an obscure virtio-scsi bug in SeaBIOS), but you really shouldn't be seeing it.

... Any particular reason for using virtio-blk with dataplane enabled?

Offline

#4715 2015-04-04 15:25:37

lersek
Member
Registered: 2015-03-15
Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

i couldn't make my VM output "starting windows" message on the main GPU

If you mean the colorful animation: I think that's presented by the windows 7 / windows server 2008 r2 boot loader, which is a UEFI application (or else, it could be early runtime OS code too, but it certainly inherits the video framebuffer from one of the GOPs (graphics output protocols) installed before ExitBootServices()).

It looks like this animation code only uses the first GOP that it finds. The order in which it finds GOPs is unspecified. You could experiment with replacing the guest-visible PCI addresses that are assigned to QXL and to the passthru card with each other on the qemu command line. That might invert the order in which these devices are enumerated in the firmware, and consequently the order in which their respective drivers install their GOPs.

I never tried this -- as you say there's some "blank screen time" to endure on the passthru display, but it isn't overly annoying.

Offline

#4716 2015-04-04 16:19:00

Duelist
Member
Registered: 2014-09-22
Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:
Duelist wrote:

i couldn't make my VM output "starting windows" message on the main GPU

If you mean the colorful animation: I think that's presented by the windows 7 / windows server 2008 r2 boot loader, which is a UEFI application (or else, it could be early runtime OS code too, but it certainly inherits the video framebuffer from one of the GOPs (graphics output protocols) installed before ExitBootServices()).

It looks like this animation code only uses the first GOP that it finds. The order in which it finds GOPs is unspecified. You could experiment with replacing the guest-visible PCI addresses that are assigned to QXL and to the passthru card with each other on the qemu command line. That might invert the order in which these devices are enumerated in the firmware, and consequently the order in which their respective drivers install their GOPs.

I never tried this -- as you say there's some "blank screen time" to endure on the passthru display, but it isn't overly annoying.

Well, i kinda knew that it grabs the first GPU it finds. Nice idea of switching the addresses that way real GPUs would be presented first.


The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4717 2015-04-04 17:52:36

apex8
Member
Registered: 2014-03-29
Posts: 60

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:
apex8 wrote:

Hi together,

I managed to create an OVMF Windows 7 VM. After patching and installing drivers I have two problems left.
1: Starting the VM isn't very reliable. Right after the "Windows is starting" screen it can happen that the VM turns off. Qemu just displays a message that the index moved from 0 to <higher number>. After some attempts the VM starts.

This should "never happen", bar guest bugs. The QEMU error message reports a guest bug in the virtio communication. I've encountered the error message once (it was related to an obscure virtio-scsi bug in SeaBIOS), but you really shouldn't be seeing it.

... Any particular reason for using virtio-blk with dataplane enabled?

Yes, I use it because it is recommended for best IO performance and I havn't had any problems when I was using a seabios VM. If dataplane would be the cause of this problem in the OVMF VM, how much would performance decrease if I switch it off?

Offline

#4718 2015-04-04 19:24:14

lersek
Member
Registered: 2015-03-15
Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:
lersek wrote:
apex8 wrote:

Hi together,

I managed to create an OVMF Windows 7 VM. After patching and installing drivers I have two problems left.
1: Starting the VM isn't very reliable. Right after the "Windows is starting" screen it can happen that the VM turns off. Qemu just displays a message that the index moved from 0 to <higher number>. After some attempts the VM starts.

This should "never happen", bar guest bugs. The QEMU error message reports a guest bug in the virtio communication. I've encountered the error message once (it was related to an obscure virtio-scsi bug in SeaBIOS), but you really shouldn't be seeing it.

... Any particular reason for using virtio-blk with dataplane enabled?

Yes, I use it because it is recommended for best IO performance and I havn't had any problems when I was using a seabios VM. If dataplane would be the cause of this problem in the OVMF VM, how much would performance decrease if I switch it off?

I have no reason to imply that dataplane was the "culprit" (or just "trigger") here; I only raised it because it was the only "unusual" option to my eye, and because I vaguely recall the dataplane code having/being a separate "virtio server" implementation. (But, this may not be true any longer, and it may not have been true ever.)

I never used dataplane myself (and I certainly never tried to test it with any remotely "scientific" method), so I can't give you numbers, unfortunately.

Anyway it shouldn't be hard to test your setup with dataplane off.

Offline

#4719 2015-04-05 22:21:29

logix
Member
Registered: 2013-11-04
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Need some help. I have read most of the thread; however, cannot shake this.

The error: vfio: error no iommu_group for device

# qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 -device vfio-pci,host=02:00.0,x-vga=on -device vfio-pci,host=02:00.1 -vga none
qemu-system-x86_64: -device vfio-pci,host=02:00.0,x-vga=on: vfio: error no iommu_group for device

# dmesg | grep probe
[  478.084005] vfio-pci: probe of 0000:02:00.0 failed with error -22

1. VT-d is supported and enabled in my bios [ASUS P9x79 LE (bios 4801/4701) i7-3930k (C2 stepping)].
2. Intel io_mmu is enabled.
3. pci-stub has blocked nouveau from taking control of the card
4. I am using pcie_acs_override=downstream

# dmesg | grep -e DMAR -e IOMMU
[    0.000000] Intel-IOMMU: enabled
[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
#dmesg | grep pci-stub
[    0.700460] pci-stub 0000:02:00.0: claimed by stub
[    0.700501] pci-stub 0000:02:00.1: claimed by stub
# dmesg | grep -i DMAR
#

Using the linux-vfio kernel out of the AUR: BOOT_IMAGE=/boot/vmlinuz-linux-vfio
I believe my kernel boot flags are correct:

intel_iommu=on pcie_acs_override=downstream pci-stub.ids=10de:1184,10de:0e0a emulate_invalid_guest_state=0

I have checked the following modules have loaded:

# cat /etc/modules-load.d/vga-passthrough.conf 
pci_stub
vfio
vfio_iommu_type1
vfio_pci
kvm
kvm-intel

I am not using the i915 pach because I do not have Intel graphics. I am using a GTX 560ti as primary, trying to pass through a GTX 770.

I have read through most of the thread and tons of pages on the internet, and almost everyone who has seen this problem either 1) cpu/MB did not support vt-d or 2) forgot to enable it in bios or via kernel option. I have tried many, many variations of fixes; however, nothing.

What am I missing here?

Last edited by logix (2015-04-05 22:52:38)

Offline

#4720 2015-04-05 23:15:37

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@logix

The IOMMU is not enabled, your first grep should produce lots more output.  Does /sys/firmware/acpi/tables/DMAR exist on your system?  A DMAR reference should have shown in the grep.  Without a DMAR table, VT-d cannot be configured, which suggests a BIOS issue.

EDIT: I've seen cases where a hard power cycle is required after enabling VT-d in the BIOS to make it actually take effect.

Last edited by aw (2015-04-05 23:17:36)


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4721 2015-04-05 23:25:39

logix
Member
Registered: 2013-11-04
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

@logix

The IOMMU is not enabled, your first grep should produce lots more output.  Does /sys/firmware/acpi/tables/DMAR exist on your system?  A DMAR reference should have shown in the grep.  Without a DMAR table, VT-d cannot be configured, which suggests a BIOS issue.

EDIT: I've seen cases where a hard power cycle is required after enabling VT-d in the BIOS to make it actually take effect.

I do not have a DMAR reference

$ sudo ls -al /sys/firmware/acpi/tables/DMAR
ls: cannot access /sys/firmware/acpi/tables/DMAR: No such file or directory

I had a feeling it was a problem with the firmware, and I guess that confirms it. I have hard-reset many times. In fact, disabling and enabling VT-d does a hard reset automatically. I will try downgrading the bios-- as there is a post with it working for old bios.

Thanks!

Edit: Tried 4 different bios dating from 1/2013 to most current. Cannot get DMAR tables to appear. Sad.

Last edited by logix (2015-04-06 14:45:01)

Offline

#4722 2015-04-06 07:03:10

apex8
Member
Registered: 2014-03-29
Posts: 60

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:
apex8 wrote:
lersek wrote:

This should "never happen", bar guest bugs. The QEMU error message reports a guest bug in the virtio communication. I've encountered the error message once (it was related to an obscure virtio-scsi bug in SeaBIOS), but you really shouldn't be seeing it.

... Any particular reason for using virtio-blk with dataplane enabled?

Yes, I use it because it is recommended for best IO performance and I havn't had any problems when I was using a seabios VM. If dataplane would be the cause of this problem in the OVMF VM, how much would performance decrease if I switch it off?

I have no reason to imply that dataplane was the "culprit" (or just "trigger") here; I only raised it because it was the only "unusual" option to my eye, and because I vaguely recall the dataplane code having/being a separate "virtio server" implementation. (But, this may not be true any longer, and it may not have been true ever.)

I never used dataplane myself (and I certainly never tried to test it with any remotely "scientific" method), so I can't give you numbers, unfortunately.

Anyway it shouldn't be hard to test your setup with dataplane off.

I switched off x-data-plane...

-drive if=none,file=/dev/sdb3,id=vdisk1,cache=none,format=raw,aio=native \
-device virtio-blk-pci,scsi=off,config-wce=off,drive=vdisk1,id=disk1,x-data-plane=off \

...but the result was the same.

qemu-system-x86_64: Guest moved used index from 14 to 0
dmesg
[  139.746997] device tap0 entered promiscuous mode
[  139.747123] kvmbr0: port 2(tap0) entered listening state
[  139.747131] kvmbr0: port 2(tap0) entered listening state
[  139.747490] qemu-system-x86: sending ioctl 5326 to a partition!
[  139.747523] qemu-system-x86: sending ioctl 80200204 to a partition!
[  140.179757] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[  140.179889] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[  140.181864] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)
[  140.183020] vfio-pci 0000:03:00.0: enabling device (0100 -> 0102)
[  144.698904] kvm: zapping shadow pages for mmio generation wraparound
[  153.360553] kvmbr0: port 2(tap0) entered disabled state
[  153.360595] device tap0 left promiscuous mode
[  153.360601] kvmbr0: port 2(tap0) entered disabled state

Is it possible to gather some more data to find out whats happening in the background?

Offline

#4723 2015-04-06 15:38:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:
qemu-system-x86_64: Guest moved used index from 14 to 0

I don't think this is as unusual as Laszlo seems to think it is.  I was seeing it roughly one in every three boots, but it doesn't seem like I can reproduce it anymore.  I'd try updating kernel and QEMU.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4724 2015-04-06 15:42:10

darkstego
Member
Registered: 2014-08-19
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lersek wrote:
darkstego wrote:

Hey guys,

I was able to get Windows 8.1 running with VFIO and OVMF, but I have an odd issue.

After the windows install, OVMF will not boot windows if I remove the drive with the install media

Here is my running script.

sudo qemu-system-x86_64 \
  -L $HOME/Desktop/ovmf -bios bios.bin \
  ...
  -drive file=$HOME/windows8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install \
  -drive file=/dev/sdb,id=disk_primary,format=raw,if=none -device scsi-hd,drive=disk_primary \
  ...

The trouble is if I omit the

  -drive file=$HOME/windows8.iso,id=iso_install,if=none -device scsi-cd,drive=iso_install 

line or move it after the disk drive the EFI will not find anything to boot.

Please stop using -L and -bios. Refer to the wiki or the example in the whitepaper wrt. the pflash drives instead.

Please capture the debug output with "-debugcon file:debug.log -global isa-debugcon.iobase=0x402"; if there's a problem with processing QEMU's "bootorder" fw_cfg file, the debug log should help us understand it.

Also, how recent is your OVMF build?

Thanks,

The reason I am using the -L and -bios options is it happened to be the only one I could get to work for the install.

I have now switched back to pflash and it seems the missing part was  "bootindex=0" option added to the primary disk.

Everything works well now.

Offline

#4725 2015-04-06 17:03:11

walkindude
Member
Registered: 2012-03-04
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can't use keyboard before windows begins to load (after I can use it without problems), but I need it to press button to load from DVD.

Nevermind, with qemu (not -git) works ok

Last edited by walkindude (2015-04-06 17:11:18)

Offline

Board footer

Powered by FluxBB