Arch Linux

apex8

Member

Registered: 2014-03-29

Posts: 60

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can't use keyboard before windows begins to load (after I can use it without problems), but I need it to press button to load from DVD.

Have a look at the arch wiki pages for qemu and look for information about setting up a spice server. You will get an empty black window, which makes you able to press keys in the VM. The other option is to use VNC in your setup and connect to the VM via a VNC client.

walkindude

Member

Registered: 2012-03-04

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@apex8
Don't want to use VNC. Besides it's working for now, so I guess I stick to qemu (without -git)
But more serious problem remains: sometimes it cannot boot at all, just seabios screen -> black for 1-2 second -> seabios screen -> black -> seabios... If I then use hard reset, and then turn VM on again windows boots as expected. It happening on 2.2.1 and on git version.

zir_blazer

Member

Registered: 2013-12-12

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@logix

The IOMMU is not enabled, your first grep should produce lots more output.  Does /sys/firmware/acpi/tables/DMAR exist on your system?  A DMAR reference should have shown in the grep.  Without a DMAR table, VT-d cannot be configured, which suggests a BIOS issue.

EDIT: I've seen cases where a hard power cycle is required after enabling VT-d in the BIOS to make it actually take effect.

I do not have a DMAR reference

$ sudo ls -al /sys/firmware/acpi/tables/DMAR
ls: cannot access /sys/firmware/acpi/tables/DMAR: No such file or directory

I had a feeling it was a problem with the firmware, and I guess that confirms it. I have hard-reset many times. In fact, disabling and enabling VT-d does a hard reset automatically. I will try downgrading the bios-- as there is a post with it working for old bios.

Thanks!

Edit: Tried 4 different bios dating from 1/2013 to most current. Cannot get DMAR tables to appear. Sad.

I think that this info is wrong. If I recall correctly, ACPI DMAR Tables are for the AMD IOMMU implementation, AMD-Vi. Intel ACPI tables are called IVRS. Basically, you should not have DMAR on Intel or IVRS on AMD, it makes no sense.

About your vfio: error no iommu_group for device, if my common sense is right, you should check if you have populated the /sys/kernel/iommu_groups location. Try this:

find /sys/kernel/iommu_groups/

It should return lots of results, something like...

/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.1

If your iommu_groups directory is not populated, maybe you require something else in your Boot Loader besides intel_iommu=on. There were a few more parameters that some guys required to have the iommu_groups directory populated, one was related to unsafe interrupts or something like that.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think that this info is wrong. If I recall correctly, ACPI DMAR Tables are for the AMD IOMMU implementation, AMD-Vi. Intel ACPI tables are called IVRS. Basically, you should not have DMAR on Intel or IVRS on AMD, it makes no sense.

You have this completely backwards.

If your iommu_groups directory is not populated, maybe you require something else in your Boot Loader besides intel_iommu=on. There were a few more parameters that some guys required to have the iommu_groups directory populated, one was related to unsafe interrupts or something like that.

allow_unsafe_interrupts has absolutely no bearing on whether iommu_groups is populated.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

zir_blazer

Member

Registered: 2013-12-12

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think that this info is wrong. If I recall correctly, ACPI DMAR Tables are for the AMD IOMMU implementation, AMD-Vi. Intel ACPI tables are called IVRS. Basically, you should not have DMAR on Intel or IVRS on AMD, it makes no sense.

You have this completely backwards.

It seems you're right. Intel mentions DMAR and AMD IVRS, I got it backwards.

https://www.kernel.org/doc/Documentatio … -IOMMU.txt
http://ubuntuforums.org/showthread.php?t=2254677

SoftWinds

Member

Registered: 2015-04-02

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've managed to passthrough my nVidia 970 card to my Windows guest using OVMF, and everything's working perfectly except for one weird thing: my mainboard is a Supermicro X10SAE, and I've configured it to use my integrated GPU (Intel P4600) as the main GPU, and pci-stub to stub out the GTX 970. This all works fine if I don't have the HDMI cable connect to the nVidia card when the computer boots up; however if it's connected, weird thing starts happening - it seems that nVidia "hijacks" the boot process somehow - GRUB doesn't display the entries correctly, and Arch starts showing on the HDMI output even though it shouldn't. The OS can't even get to show the display manager in this case, it just stays in the console. Does anyone have any suggestions? Thanks!

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've managed to passthrough my nVidia 970 card to my Windows guest using OVMF, and everything's working perfectly except for one weird thing: my mainboard is a Supermicro X10SAE, and I've configured it to use my integrated GPU (Intel P4600) as the main GPU, and pci-stub to stub out the GTX 970. This all works fine if I don't have the HDMI cable connect to the nVidia card when the computer boots up; however if it's connected, weird thing starts happening - it seems that nVidia "hijacks" the boot process somehow - GRUB doesn't display the entries correctly, and Arch starts showing on the HDMI output even though it shouldn't. The OS can't even get to show the display manager in this case, it just stays in the console. Does anyone have any suggestions? Thanks!

Maybe it's your HDMI audio function? Is it stubbed too?

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

SoftWinds

Member

Registered: 2015-04-02

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've managed to passthrough my nVidia 970 card to my Windows guest using OVMF, and everything's working perfectly except for one weird thing: my mainboard is a Supermicro X10SAE, and I've configured it to use my integrated GPU (Intel P4600) as the main GPU, and pci-stub to stub out the GTX 970. This all works fine if I don't have the HDMI cable connect to the nVidia card when the computer boots up; however if it's connected, weird thing starts happening - it seems that nVidia "hijacks" the boot process somehow - GRUB doesn't display the entries correctly, and Arch starts showing on the HDMI output even though it shouldn't. The OS can't even get to show the display manager in this case, it just stays in the console. Does anyone have any suggestions? Thanks!

Maybe it's your HDMI audio function? Is it stubbed too?

Yes it is stubbed as well. I also use it to output audio in the VM so I know it's passed through correctly.

dhiru1602

Member

Registered: 2015-01-28

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone managed to get SLI working by passing through 2 cards?

I have a spare AMD graphic card that I would like to use for my host and passthrough both my identical 9600GT to the guest OS so that I can create an SLI. I have blacklisted both the cards and assigned them to the VM using i440fx. However, I was able to see both the cards listed in the Device Manager but Nvidia Control Panel was unable to detect any SLI capability. After some reading, I came to a conclusion that Nvidia drivers required SLI certified motherboard to enable SLI configuration.

I tried using the q35 chipset hoping that it was SLI certified. However, my 2nd GPU shows up in the device manager with error "Code 12". Came across a similar issue in the qemu-devel mailing list.

https://lists.gnu.org/archive/html/qemu … 00753.html

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone managed to get SLI working by passing through 2 cards?

I have a spare AMD graphic card that I would like to use for my host and passthrough both my identical 9600GT to the guest OS so that I can create an SLI. I have blacklisted both the cards and assigned them to the VM using i440fx. However, I was able to see both the cards listed in the Device Manager but Nvidia Control Panel was unable to detect any SLI capability. After some reading, I came to a conclusion that Nvidia drivers required SLI certified motherboard to enable SLI configuration.

I tried using the q35 chipset hoping that it was SLI certified. However, my 2nd GPU shows up in the device manager with error "Code 12". Came across a similar issue in the qemu-devel mailing list.

https://lists.gnu.org/archive/html/qemu … 00753.html

Maybe an interesting endeavor, if your time and electricity are free.  A 9600GT is not an impressive card anymore.  Conservatively, a GT740 has 2x the passmark score of a 9600GT, runs about 2/3 the power of a single 9600GT, and can be purchased for under $100.  For not much more, something like a GTX750 doubles the performance again, at a lower power point and better performance per dollar margin.  It's fun to use the hardware you've got laying around, but there's all sorts of unknown proprietary bits of Nvidia SLI and in this case it just doesn't seem worth it.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

dhiru1602

Member

Registered: 2015-01-28

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone managed to get SLI working by passing through 2 cards?

I have a spare AMD graphic card that I would like to use for my host and passthrough both my identical 9600GT to the guest OS so that I can create an SLI. I have blacklisted both the cards and assigned them to the VM using i440fx. However, I was able to see both the cards listed in the Device Manager but Nvidia Control Panel was unable to detect any SLI capability. After some reading, I came to a conclusion that Nvidia drivers required SLI certified motherboard to enable SLI configuration.

I tried using the q35 chipset hoping that it was SLI certified. However, my 2nd GPU shows up in the device manager with error "Code 12". Came across a similar issue in the qemu-devel mailing list.

https://lists.gnu.org/archive/html/qemu … 00753.html

Maybe an interesting endeavor, if your time and electricity are free.  A 9600GT is not an impressive card anymore.  Conservatively, a GT740 has 2x the passmark score of a 9600GT, runs about 2/3 the power of a single 9600GT, and can be purchased for under $100.  For not much more, something like a GTX750 doubles the performance again, at a lower power point and better performance per dollar margin.  It's fun to use the hardware you've got laying around, but there's all sorts of unknown proprietary bits of Nvidia SLI and in this case it just doesn't seem worth it.

I totally agree with you. I stopped gaming after I moved permanently to linux few years ago and I haven't considered upgrading my GPU. Thanks to your great work with vfio, I was able to passthrough my GPU to a virtual machine and combined with a monitor and keyboard mouse pair, I was able to create a second standalone PC for family use and save electricity.

Putting together a PC with a beefy hardware for linux finally makes sense. Thanks to you

Last edited by dhiru1602 (2015-04-07 21:13:21)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone managed to get SLI working by passing through 2 cards?

I have a spare AMD graphic card that I would like to use for my host and passthrough both my identical 9600GT to the guest OS so that I can create an SLI. I have blacklisted both the cards and assigned them to the VM using i440fx. However, I was able to see both the cards listed in the Device Manager but Nvidia Control Panel was unable to detect any SLI capability. After some reading, I came to a conclusion that Nvidia drivers required SLI certified motherboard to enable SLI configuration.

I tried using the q35 chipset hoping that it was SLI certified. However, my 2nd GPU shows up in the device manager with error "Code 12". Came across a similar issue in the qemu-devel mailing list.

https://lists.gnu.org/archive/html/qemu … 00753.html

Maybe an interesting endeavor, if your time and electricity are free.  A 9600GT is not an impressive card anymore.  Conservatively, a GT740 has 2x the passmark score of a 9600GT, runs about 2/3 the power of a single 9600GT, and can be purchased for under $100.  For not much more, something like a GTX750 doubles the performance again, at a lower power point and better performance per dollar margin.  It's fun to use the hardware you've got laying around, but there's all sorts of unknown proprietary bits of Nvidia SLI and in this case it just doesn't seem worth it.

I totally agree with you. I stopped gaming after I moved permanently to linux few years ago and I haven't considered upgrading my GPU. Thanks to your great work with vfio, I was able to passthrough my GPU to a virtual machine and combined with a monitor and keyboard mouse pair, I was able to create a second standalone PC for family use and save electricity.

Putting together a PC with a beefy hardware for linux finally makes sense. Thanks to you

Using two 9600GTs you could... do three PCs in one, but you'll probably will run out of memory or CPU power or some other IOMMU-related resource like iommu groups.

Also, seems like amd crossfire works overall better than nvidia SLI, considering the fact that back in the day you would have to install some hypervisor-like software called HyperSLI on a real hardware platform, let alone virtual machine.

Offtopic:
just accidentally found out that spice or VNC passes through ACPI power button event. Seems like my virtual PC works so well and integrated, i began "blurring the edge between virtuality and reality".

Last edited by Duelist (2015-04-07 22:30:10)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

mutiny

Member

Registered: 2015-02-15

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just curious, is it possible to have a "passed through" physical hard drive without any virtual disk drive/image? Basically, instead of having an OS raw or qcow2 image file on the host's hard disk/filesystem, or instead of even passing an entire block device (drive /dev/sda for example), can we attach a physical hard drive to a motherboard's 2nd SATA controller, and pass through the SATA controller to the guest and have all disk operations performed by the guest alone through that controller? Will OVMF and the guest see the hard disk and be able to install OS, boot off it, etc?

SoftWinds

Member

Registered: 2015-04-02

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Actually I do have another question, but I'm not really sure if it's related to GPU passthrough or just general KVM/QEMU: When I start the VM the first time everything's fine, but if I shut down the VM and try to start it again I run into the error:

qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory

Here's my qemu command:

qemu-system-x86_64 -enable-kvm -m 10240 -cpu host,kvm=off -smp 8,sockets=1,cores=4,threads=2 \
	-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \
	-drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \
	-device vfio-pci,host=01:00.0 -device vfio-pci,host=01:00.1 \
	-device virtio-scsi-pci,id=scsi \
	-drive file=/dev/mapper/VolGroup00-vm,id=disk,format=raw,if=none \
	-device scsi-hd,drive=disk \
	-net nic -net bridge,br=br0 \
	-rtc base=localtime,clock=host \
	-vga none

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Actually I do have another question, but I'm not really sure if it's related to GPU passthrough or just general KVM/QEMU: When I start the VM the first time everything's fine, but if I shut down the VM and try to start it again I run into the error:

qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory

Sounds like you're hitting a memory leak, what kernel/qemu versions are you using?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just curious, is it possible to have a "passed through" physical hard drive without any virtual disk drive/image? Basically, instead of having an OS raw or qcow2 image file on the host's hard disk/filesystem, or instead of even passing an entire block device (drive /dev/sda for example), can we attach a physical hard drive to a motherboard's 2nd SATA controller, and pass through the SATA controller to the guest and have all disk operations performed by the guest alone through that controller? Will OVMF and the guest see the hard disk and be able to install OS, boot off it, etc?

Yes, you can theoretically pass through any PCI endpoint device.  How well it works depends on the device.  Whether it's supported as a pre-boot device depends on whether there's a boot ROM for the device or built-in support or standard compatibility in the VM firmware.  IOW, try it and find out.  OTOH, my personal opinion is that assigning PCI HBAs often provides a marginal performance improvement and significantly more headaches vs paravirtual solutions.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

SoftWinds

Member

Registered: 2015-04-02

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Actually I do have another question, but I'm not really sure if it's related to GPU passthrough or just general KVM/QEMU: When I start the VM the first time everything's fine, but if I shut down the VM and try to start it again I run into the error:

qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory

Sounds like you're hitting a memory leak, what kernel/qemu versions are you using?

I'm running qemu 2.2.1-2 on Arch with kernel 3.19.2-1. I think it's a memory leak as well, since I checked all the process in htop and nothing seems out of the ordinary, and when I sum the memory usage up it's nowhere near enough to fill up my RAM. Is there anything I can do to debug this?

By the way, thanks aw for your work on this!

Last edited by SoftWinds (2015-04-08 00:05:23)

Slabity

Member

Registered: 2013-12-29

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm trying to figure out whether or not my Nvidia 780 supports EFI. I would think it would, but when I try to dump the rom:

$ cd /sys/bus/pci/devices/000:02:00.0
$ echo 1 > rom
$ cat rom
cat: rom: Input/output error
$ dmesg | tail
...
[ 4850.407367] pci-stub 0000:02:00.0: Invalid ROM contents

The card has been taken as a stub:

$ dmesg | grep stub
[    0.435187] pci-stub: add 10DE:1004 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.435192] pci-stub 0000:02:00.0: claimed by stub
[    0.435195] pci-stub: add 10DE:0E1A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.435199] pci-stub 0000:02:00.1: claimed by stub

My relevant kernel boot parameters:

$ cat /proc/cmdline
intel_iommu=on pci-stub.ids=10de:1004,10de:0e1a pcie_acs_override=downstream

And my lspci:

$ lspci
00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06)
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 04)
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 04)
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4)
00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 (rev d4)
00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4)
00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d4)
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 04)
00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 04)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series]
02:00.0 VGA compatible controller: NVIDIA Corporation GK110 [GeForce GTX 780] (rev a1)
02:00.1 Audio device: NVIDIA Corporation GK110 HDMI Audio (rev a1)
04:00.0 Network controller: Qualcomm Atheros AR9462 Wireless Network Adapter (rev 01)
05:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03)
06:04.0 Multimedia audio controller: C-Media Electronics Inc CMI8788 [Oxygen HD Audio]
07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

In the past I've been able to successfully do the hacky VGA-based passthrough, but I'd like to use the OVMF/EFI method instead.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm trying to figure out whether or not my Nvidia 780 supports EFI. I would think it would, but when I try to dump the rom:

There's no harm in just trying it and it seems likely that it would.  If you get the TianoCore bootsplash, the answer is yes.

Last edited by aw (2015-04-08 20:05:37)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series]
02:00.0 VGA compatible controller: NVIDIA Corporation GK110 [GeForce GTX 780] (rev a1)
02:00.1 Audio device: NVIDIA Corporation GK110 HDMI Audio (rev a1)

In the past I've been able to successfully do the hacky VGA-based passthrough, but I'd like to use the OVMF/EFI method instead.

I'd say, both of your GPUs support UEFI.
EFI GOP driver part of my VBIOS is taken from barts GPU vbios.

There are some problems with nvidia ROMs, like pre-POST/post-POST ROMs and stuff. You could actually try lurking in TPU's VBIOS database or try fetching the VBIOS from windows using GPU-z software, i've heard recent versions of it can handle stuff well.

Last edited by Duelist (2015-04-08 22:22:46)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

mutiny

Member

Registered: 2015-02-15

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, you can theoretically pass through any PCI endpoint device.  How well it works depends on the device.  Whether it's supported as a pre-boot device depends on whether there's a boot ROM for the device or built-in support or standard compatibility in the VM firmware.  IOW, try it and find out.  OTOH, my personal opinion is that assigning PCI HBAs often provides a marginal performance improvement and significantly more headaches vs paravirtual solutions.

I gave this a shot today just to experiment. I connected a spare 1TB SATA drive to the motherboard's Marvell SATA controller (non-chipset 2nd controller) and passed that through qemu to play around with. OVMF shows no signs of seeing a drive/controller (not sure if I should even be seeing anything really), but booting off a Windows install ISO shows the hard drive perfectly fine, and allows me to install to it. However, after the install completes and the guest system reboots, OVMF just dumps back at the Shell> prompt. OVMF's Boot Manager shows a "Windows Boot Manager" entry with device path: HD(2,GPT,UUID#)/\EFI\Microsoft\Boot\bootmgfw.efi so it seems like Windows at least installed boot information to the drive and to the firmware VARS I think? Not sure what to do from here or if it is a dead end. Maybe others have experimented with trying something like this (this is just all out of curiosity's sake). Probably has something to do with HD index showing as being at 2, wonder if this could be manually edited.

Last edited by mutiny (2015-04-09 04:44:03)

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I encountered small problem when reinstalling my hypervisor OS recently, hope someone can point me to most robust solution.

My setup is basically headless host running two guest Windows 7 instances, each with own GPU and other peripherals (including native sound). One of guest instances takes primary GPU and the other takes second; hypervisor is controlled via serial console and/or ssh. The problem seems to be that "basically headless" is not as good as (planned) "actually headless".

What seems to be happening is that pci-stub is unable to attach itself to primary GPU because it is used by vgaarb early on kernel startup. Because pci-stub is unable to attach itself to primary GPU, snd_hda_intel comes along later and grabs HDMI audio output of this card (readeon is blacklisted), and this leads to all sort of trouble when guest starts up. Of course simple configuration tweak is to blacklist this module (which I did) but I'd prefer it pci-stub just did what it was configured to do in the first place i.e. prevent any driver from attaching itself to primary GPU (e.g. just in case some other driver came along in newer Linux version).

I think for this I need to disable vgaarb or set configuration such that vgaarb "knows" that the machine is meant to be running headless and no VGA output is available to host OS. Solutions coming to my mind are:
* disable CONFIG_VGA_CONSOLE in configuration and recompile kernel
* or disable CONFIG_VGA_ARB (same drawback as above)
* or set "install vgaarb /bin/false" in /etc/modprobe.d/ (assuming vgaarb can be built as module?)

I wonder what is the right way to approach the problem; ideally I'd rather use configuration thus making my setup less dependent on correct set of configuration switches when building the kernel. If only I could somehow set kernel parameters such as to force screen_info.orig_video_isVGA=0 ... perhaps boot parameter "earlyprintk=serial,..." might help?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek

vgaarb is not a "device driver" and cannot be built as a module.  It's a kernel service.  It's not consuming resources of the device and won't prevent pci-stub from attaching.  Have you tried building CONFIG_PCI_STUB=y?  If you build pci-stub as a module then you need to go through extra effort to make sure it loads before other modules.  See the modprobe.d softdep option for example.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek

vgaarb is not a "device driver" and cannot be built as a module.  It's a kernel service.  It's not consuming resources of the device and won't prevent pci-stub from attaching.  Have you tried building CONFIG_PCI_STUB=y?  If you build pci-stub as a module then you need to go through extra effort to make sure it loads before other modules.  See the modprobe.d softdep option for example.

Thanks aw; yes I build with CONFIG_PCI_STUB=y and yet somehow option to claim my primary GPU is ignored by pci-stub while all other PCI resources are claimed with no problems (USB cards, secondary GPU, motherboard sound, motherboard USB). I can use primary GPU for passthrough despite it, but this does not seem very robust. From dmesg I can see that vgaarb makes this GPU its default and I thought this is related. My kernel is 3.18.11 , I will post more details later today when I get home. At this moment I'm tempted to build without VGA_ARB but I'd prefer runtime option instead if there is one ...

Child_of_Sun

Member

Registered: 2014-07-16

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, you can theoretically pass through any PCI endpoint device.  How well it works depends on the device.  Whether it's supported as a pre-boot device depends on whether there's a boot ROM for the device or built-in support or standard compatibility in the VM firmware.  IOW, try it and find out.  OTOH, my personal opinion is that assigning PCI HBAs often provides a marginal performance improvement and significantly more headaches vs paravirtual solutions.

I gave this a shot today just to experiment. I connected a spare 1TB SATA drive to the motherboard's Marvell SATA controller (non-chipset 2nd controller) and passed that through qemu to play around with. OVMF shows no signs of seeing a drive/controller (not sure if I should even be seeing anything really), but booting off a Windows install ISO shows the hard drive perfectly fine, and allows me to install to it. However, after the install completes and the guest system reboots, OVMF just dumps back at the Shell> prompt. OVMF's Boot Manager shows a "Windows Boot Manager" entry with device path: HD(2,GPT,UUID#)/\EFI\Microsoft\Boot\bootmgfw.efi so it seems like Windows at least installed boot information to the drive and to the firmware VARS I think? Not sure what to do from here or if it is a dead end. Maybe others have experimented with trying something like this (this is just all out of curiosity's sake). Probably has something to do with HD index showing as being at 2, wonder if this could be manually edited.

I try the "same" thing with my Motherbord Sata Controller it's a SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode]. The Windows 8.1 DVD won't boot and the Windows 10 Tech Preview freezes.

Ovmf has no boot option for this controller, even when i use uefi-raid mode.

I hope someone can help with that.