Arch Linux

sinny

Member

Registered: 2014-05-24

Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone .

Can anyone who uses OVMF successfully post his entire qemu launch command ?

OVMF might be the answer to all my hangs issues on VMs' reboots , but it still doesn't detect any of my drives .

I don't use libvirt , just plain shell scripts .

Much appreciated .

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/sbin/qemu-system-x86_64 -name windows -S -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu host,hv_time,-kvm_pv_eoi,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -drive file=/usr/share/ovmf/x64/ovmf_code_x64.bin,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/windows_VARS.fd,if=pflash,format=raw,unit=1 -m 8192 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu -realtime mlock=off -smp 6,sockets=1,cores=3,threads=2 -uuid 09d81f35-51a4-4c7d-8db0-46597c7233ba -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/windows.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,clock=vm,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x4.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x4 -drive file=/dev/storage/windows-ovmf,if=none,id=drive-virtio-disk0,format=raw,aio=native -device virtio-blk-pci,ioeventfd=on,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/dev/storage/steam,if=none,id=drive-virtio-disk1,format=raw,aio=native -device virtio-blk-pci,ioeventfd=on,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:c6:b5:45,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device vfio-pci,host=00:1b.0,id=hostdev0,bus=pci.0,addr=0x2 -device vfio-pci,host=01:00.0,id=hostdev1,bus=pci.0,multifunction=on,addr=0x7,rombar=0,romfile=/var/lib/libvirt/firmware/HD7870_EFI.rom -device vfio-pci,host=01:00.1,id=hostdev2,bus=pci.0,addr=0x7.0x1 -device usb-host,hostbus=2,hostaddr=17,id=hostdev3 -device usb-host,hostbus=2,hostaddr=3,id=hostdev4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -set device.virtio-disk0.scsi=off -set device.virtio-disk0.x-data-plane=on -set device.virtio-disk1.scsi=off -set device.virtio-disk1.x-data-plane=on -msg timestamp=on

It's how libvirt launches my vm, but should work anyway with few adjustments.

Thank you very much !

I'm going to adapt these settings to my existing WIndows VM , however I couldn't find the file "windows_VARS.fd" , and Google points me to your post , if it isn't any trouble can you tell me what is it for ? is it important ? if so , how can I create one ?

Sorry for this barge of questions , I want to learn .

@"windows_VARS.fd":
i suspect you should read up every post at aw's blog (search in his forum sig) - especially the ones related to ovmf, even more especially for ovmf split image support

it's a bit strange that you try to find it somewhere externally even though this command line parameter declaration looks almost identical to guest's hard drive info

mauorrizze

Member

Registered: 2014-08-22

Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I couldn't find the file "windows_VARS.fd"

To summarize the infos from aw's blog: if you use splitted OVMF images, libvirt copies for you the vars-file, for qemu you should do it manually because this file might get modified. The original file is part or compiled of your OVMF-package. http://vfio.blogspot.de/2014/09/ovmf-sp … pport.html

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you both !

My OVMF is exactly 2MB and according to Alex's blog post , it is a unified package so specifying a VARS file is not necessary I guess .

I don't know what is wrong with my shell script , but OVMF always drops me to the shell with no disks detected . Been trying for weeks ...

qemu-system-x86_64 -name gaming -nographic \
-enable-kvm -m 8192 -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000,kvm=off -smp 4,sockets=1,cores=4,threads=1 \
-pflash /VMs/ovmf_x64.bin
-device vfio-pci,host=02:00.0,bus=bus=pci.0,addr=addr=0x7,multifunction=on \
-device vfio-pci,host=02:00.1,bus=bus=pci.0,0x7.0x1 \
-drive file=/VMs/Win_Gaming.qed,cache=writeback,if=none,id=drive0,aio=native \
-device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \
-drive file=/VMs/Gaming_External.qed,cache=none,if=none,id=drive1,aio=native \
-device virtio-blk-pci,drive=drive1,ioeventfd=on \
-net nic,model=virtio,macaddr=64:25:5E:87:0E:5C -net bridge,br=br0 \
-usb -usbdevice host:1c4f:0034 -usbdevice host:1c4f:0002 -usbdevice host:0a12:0001 \
-localtime \
-monitor unix:/tmp/vm_gaming,server,nowait &

sinny

Member

Registered: 2014-05-24

Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you both !

My OVMF is exactly 2MB and according to Alex's blog post , it is a unified package so specifying a VARS file is not necessary I guess .

I don't know what is wrong with my shell script , but OVMF always drops me to the shell with no disks detected . Been trying for weeks ...

qemu-system-x86_64 -name gaming -nographic \
-enable-kvm -m 8192 -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000,kvm=off -smp 4,sockets=1,cores=4,threads=1 \
-pflash /VMs/ovmf_x64.bin
-device vfio-pci,host=02:00.0,bus=bus=pci.0,addr=addr=0x7,multifunction=on \
-device vfio-pci,host=02:00.1,bus=bus=pci.0,0x7.0x1 \
-drive file=/VMs/Win_Gaming.qed,cache=writeback,if=none,id=drive0,aio=native \
-device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \
-drive file=/VMs/Gaming_External.qed,cache=none,if=none,id=drive1,aio=native \
-device virtio-blk-pci,drive=drive1,ioeventfd=on \
-net nic,model=virtio,macaddr=64:25:5E:87:0E:5C -net bridge,br=br0 \
-usb -usbdevice host:1c4f:0034 -usbdevice host:1c4f:0002 -usbdevice host:0a12:0001 \
-localtime \
-monitor unix:/tmp/vm_gaming,server,nowait &

it always dropped me there whenever i was trying to boot something that is not uefi-aware (ISO's mostly)
if you try to switch already installed os - i guess you should reinstall (not that aware of details but it seems to me at least windows-installed boot stuff/layout is different depending on whether it's bios or uefi)

Drakulix

Member

Registered: 2014-10-10

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi there!
I have some question on this subject and got already pretty far on my own, although I have a very experimental hardware configuration, but I am stuck now and need some help.

First about my hardware setup. Please don't tell me right away, that VGA Passthough will not work on my setup. I know that some people might have mixed feeling, when reading the next sentence.

So I have a Macbook Pro Retina 15" with an Intel HD4000 and an nVidia GeForce 650M and yes I am trying to Passthough a Mobile Graphics Card.
My CPU (Intel(R) Core(TM) i7-3720QM) does support VT-d, I have no idea, what my mainbroad might support. It is from mid-2012, so I have at least a chance, I guess.

I am running Linux Kernel 3.17 with the acs override patch, the i915 vga arbiter fixes and some other patches. (4 cherry picks from 3.18.  First to fix broken HDMI info frames with nouveau, and some about my battery and thunderbold hotplug support on apple devices. So nothing else that should matter.)

I know that my HDMI connector is directly attached to my graphics card. My kernel also contains a self-made patch that allows me to switch my mux directly (bypassing vga_switcheroo and the ability to switch without any graphic drivers loaded). So I can decide, if my internal screen is attached to my intel card or the nvidia card. (works nicely, if someone needs help on that subject, feel free to pm me. the wiki article is not very helpful with this).

I can currently boot a Windows 7 SP1 (64-Bit) VM with VGA Passthough either on my HDMI Monitor (allowing my to use linux meanwhile) or on my internal screen (swapping the mux beforehand and after running qemu via a little bash script). The graphics card does get detected, I can install the nVidia drivers, but I get error 43 in the Device Manager.

Some more details:

I have the following in my kernel boot parameters:

modprobe.blacklist=nouveau i915.enable_hd_vgaarb=1 pci-stub.ids=10de:0fd5,10de:0e1b,8086:1547 intel_iommu=on

I am normally running the with nouveau drivers, the nvidia drivers are not installed.
The last device:vendor-id pair is from my thunderbolt connectors, which appear to be connected to the same iommu_group my graphics adapter is (probably for displayport output).
I need to bind them to vfio as well, otherwise I get an error in qemu (iommu_group not viable).
Everything shows up as loaded by pci-stub correctly, before I bind to vfio-pci.

So this is what my binding looks like:

sudo ./vfio-bind 0000:01:00.0 0000:01:00.1 0000:05:00.0 0000:06:00.0 0000:06:03.0 0000:06:04.0 0000:06:05.0 0000:06:06.0 0000:07:00.0

The first is my graphics card, the seconds is HDMI audio, the rest is thunderbolt.

I also have the following set via configs in modprobe.d:

options kvm ignore_msrs=1
options vfio_iommu_type1 allow_unsafe_interrupts=1

I need the second option, at least thats what qemu tells me, when I try to boot without it.

Last but not atleast I run qemu like this:

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 2048 -cpu host -smp 4,sockets=1,cores=4,threads=1 \
 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=vbios.rom \
 -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -drive file=QEMU/windows.img,id=disk,format=qcow2 \
 -device ide-hd,bus=ide.0,drive=disk -device qxl

I need to pass the romfile, otherwise qemu complains and dmesg says something like "Invalid VBIOS Contents". I am using -device qxl, since I do want to use my keyboard and trackpad.

I am using the qemu package from the arch repos, which is version 2.1.2. The qemu-git package failed for some reason, but I can try to get it to work, if that would help anything.

If you need any more informations, please feel free to ask.
I have not much ideas left, to fix the error 43. I already removed all hv_* functions and added kvm=off to my qemu cpu options, still not any better.

Mardok45

Member

Registered: 2014-08-10

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've done a lot of switching between Xen and QEMU when researching this topic and have ultimately decided to stick with QEMU.  I have a desktop that's running both Windows 7 and Gentoo and it works almost perfectly.

There's still one problem that I haven't solved yet.  My Windows VM has a Nvidia GTX 260 exposed to it and I am able to play Dota 2.  However, Dota 2 was having framerate problems.  I know for a fact that on bare-metal, my GTX 260 chews right through Dota 2 with no problems.  I expect a performance degredation, but this is unusually slow. 

I looked at the CPU usage, and all four CPU's that were exposed to it were running at 50%.  So the CPU isn't the problem.  I looked at the memory usage and 2 gigabytes were used out of the 3 that were exposed to it.  So the memory isn't the problem.

After a lot of head-scratching, I noticed that the GTX 260 was really quiet through all this.  Usually when playing a game like Dota 2, the GPU fan gets really loud.  I think the problem is the GTX 260 isn't increasing its clock speed.

I tried a QEMU config using the Q35 machine model and the 440FX model and both have the same problem.

I tried turning on maximum performance in the NVidia control panel and Windows settings, but neither solved the problem.  Does anyone know what the problem might be?

Last edited by Mardok45 (2014-10-11 00:19:59)

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've done a lot of switching between Xen and QEMU when researching this topic and have ultimately decided to stick with QEMU.  I have a desktop that's running both Windows 7 and Gentoo and it works almost perfectly.

There's still one problem that I haven't solved yet.  My Windows VM has a Nvidia GTX 260 exposed to it and I am able to play Dota 2.  However, Dota 2 was having framerate problems.  I know for a fact that on bare-metal, my GTX 260 chews right through Dota 2 with no problems.  I expect a performance degredation, but this is unusually slow. 

I looked at the CPU usage, and all four CPU's that were exposed to it were running at 50%.  So the CPU isn't the problem.  I looked at the memory usage and 2 gigabytes were used out of the 3 that were exposed to it.  So the memory isn't the problem.

After a lot of head-scratching, I noticed that the GTX 260 was really quiet through all this.  Usually when playing a game like Dota 2, the GPU fan gets really loud.  I think the problem is the GTX 260 isn't increasing its clock speed.

I tried a QEMU config using the Q35 machine model and the 440FX model and both have the same problem.

I tried turning on maximum performance in the NVidia control panel and Windows settings, but neither solved the problem.  Does anyone know what the problem might be?

Try different cpu configs, here works best with 2 or more cores 1 thread each, and hv-time option that doesn't work with drivers after 340.xx..

Last edited by slis (2014-10-11 01:10:06)

Mardok45

Member

Registered: 2014-08-10

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've done a lot of switching between Xen and QEMU when researching this topic and have ultimately decided to stick with QEMU.  I have a desktop that's running both Windows 7 and Gentoo and it works almost perfectly.

There's still one problem that I haven't solved yet.  My Windows VM has a Nvidia GTX 260 exposed to it and I am able to play Dota 2.  However, Dota 2 was having framerate problems.  I know for a fact that on bare-metal, my GTX 260 chews right through Dota 2 with no problems.  I expect a performance degredation, but this is unusually slow. 

I looked at the CPU usage, and all four CPU's that were exposed to it were running at 50%.  So the CPU isn't the problem.  I looked at the memory usage and 2 gigabytes were used out of the 3 that were exposed to it.  So the memory isn't the problem.

After a lot of head-scratching, I noticed that the GTX 260 was really quiet through all this.  Usually when playing a game like Dota 2, the GPU fan gets really loud.  I think the problem is the GTX 260 isn't increasing its clock speed.

I tried a QEMU config using the Q35 machine model and the 440FX model and both have the same problem.

I tried turning on maximum performance in the NVidia control panel and Windows settings, but neither solved the problem.  Does anyone know what the problem might be?

Try different cpu configs, here works best with 2 or more cores 1 thread each, and hv-time option that doesn't work with drivers after 340.xx..

qemu-system-x86_64 \
-nodefconfig \
-monitor stdio -serial none -parallel none \
-enable-kvm \
-rtc base=localtime \
-M q35 \
-smp 4,sockets=1,cores=4,threads=1 -cpu host,kvm=off \
-m 3072 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-netdev bridge,br=br0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:12:34:56:78:97 \
-drive file=/dev/system/windows,cache=none,if=none,id=drive0,aio=threads,format=raw -device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=2 \
-drive file=/root/win7.iso,if=none,id=cd0,aio=threads,format=raw -device ide-cd,drive=cd0,bootindex=1 \
-drive file=/root/qemu/virtio.iso,if=none,id=cd1,aio=threads,format=raw -device ide-cd,drive=cd1,bus=ide.1 \
-device vfio-pci,host=00:12.0,bus=pcie.0 \
-device vfio-pci,host=00:12.2,bus=pcie.0 \
-device vfio-pci,host=00:14.2,bus=pcie.0 \
-device vfio-pci,host=01:00.0,multifunction=on,x-vga=on,bus=pcie.0 \
-vga none \
-nographic \

I'm running the 334 drivers.  I removed hv-time and it doesn't seem to help.  I'm not sure why the GPU wouldn't increase its clock speed.

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

well i got here native like perfomance in dota2, with hv-time and 2+ cores, remove that smp option maybe?

http://pastie.org/9639320

i'm using this config for streaming on headless server for last 2-3 months and with 340.82 driver it works without problem.

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

it always dropped me there whenever i was trying to boot something that is not uefi-aware (ISO's mostly)
if you try to switch already installed os - i guess you should reinstall (not that aware of details but it seems to me at least windows-installed boot stuff/layout is different depending on whether it's bios or uefi)

Then how can one install Windows if the installation media could not be detected , I'm using Windows 8.1 ISO ?

cryptonymous

Member

Registered: 2013-11-05

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can anyone who have tried using OVMF confirm that initial OVMF bootup messages and EFI shell are shown on a monitor connected to a passed-through video card? I can seem to get even this basic thing to work with OVMF.

Command line I've tried:

qemu-system-x86_64 -enable-kvm -machine pc-i440fx-2.1,accel=kvm,usb=off -device vfio-pci,host=01:00.0,bus=pci.0,addr=7,multifunction=on,romfile=/home/user/NV280MS1-EFI.rom -device vfio-pci,host=01:00.1,bus=pci.0,addr=7.1 -drive if=pflash,format=raw,readonly,file=/tmp/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd -vga none -nographic

All messages, including EFI shell, are shown on QEMU's curses console. Nothing is shown on a passed-though video card. Does it mean EFI ROM that performs video card initialization doesn't run properly? If I type `pci' command in EFI shell, I see that correct VID/PID pair is listed for the video card, thus it is clearly detected.

The ROM file I use (NV280MS1-EFI.rom) does have a EFI part:

Valid ROM signature found @400h, PCIR offset 190h
	PCIR: type 0, vendor: 10de, device: 0fc6, class: 030000
	PCIR: revision 0, vendor revision: 1
Valid ROM signature found @10000h, PCIR offset 1ch
	PCIR: type 3, vendor: 10de, device: 0fc6, class: 030000
	PCIR: revision 3, vendor revision: 0
		EFI: Signature Valid
	Last image

Video card in question is a NVidia GTX 650 (MSI N650 PE 1GD5/OC). Initially it came with hybrid (EFI/Legacy) BIOS flashed, later update was just Legacy. I'm trying to use original ROM I've fortunately backed up before reflashing.

sinny

Member

Registered: 2014-05-24

Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

it always dropped me there whenever i was trying to boot something that is not uefi-aware (ISO's mostly)
if you try to switch already installed os - i guess you should reinstall (not that aware of details but it seems to me at least windows-installed boot stuff/layout is different depending on whether it's bios or uefi)

Then how can one install Windows if the installation media could not be detected , I'm using Windows 8.1 ISO ?

_right_ (that's the key part here) innstallation media should work without problems
what processor are you emulating (probably 64 bit)?
what iso are you trying? does it match processor bitness (it may matter)? are you sure it's uefi-bootable?

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

_right_ (that's the key part here) innstallation media should work without problems
what processor are you emulating (probably 64 bit)?
what iso are you trying? does it match processor bitness (it may matter)? are you sure it's uefi-bootable?

1 - Processor is passed-through (-cpu host) which is an i7-5930k .

2 - I'm using the Windows 8.1 Enterprise ISO (MSDN with Update) .

I don't see why shouldn't it be UEFI-compatible , when I switch machine type to 440FX , it detects the ISO , but refuses to boot it . I tried to manually execute the file located at /EFI/BOOT/BOOTX64.EFI  but it hangs the VM for 3 seconds then return to the shell .

Mardok45

Member

Registered: 2014-08-10

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

well i got here native like perfomance in dota2, with hv-time and 2+ cores, remove that smp option maybe?

http://pastie.org/9639320

i'm using this config for streaming on headless server for last 2-3 months and with 340.82 driver it works without problem.

I copied and edited that config and it seemed to help a bit.  It's still not running anywhere near native performance, but Dota 2 is at least playable.  I'm running a 6-core Bulldozer CPU, so maybe I just need to throw better hardware at it.  The VM is running the 334 Nvidia drivers.

sinny

Member

Registered: 2014-05-24

Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

well i got here native like perfomance in dota2, with hv-time and 2+ cores, remove that smp option maybe?

http://pastie.org/9639320

i'm using this config for streaming on headless server for last 2-3 months and with 340.82 driver it works without problem.

I copied and edited that config and it seemed to help a bit.  It's still not running anywhere near native performance, but Dota 2 is at least playable.  I'm running a 6-core Bulldozer CPU, so maybe I just need to throw better hardware at it.  The VM is running the 334 Nvidia drivers.

try to disable any sound for vm - just drop any sound-related parameters from xml or commandline
also look at real qemu invocation parameters and look for something that is not needed or simply wrong
check that your memory allocation is normal (hugetlbfs?)
review your cpu allocation scheme (your current config binds to cpus 0-4, but are these cpus isolated from host usage?)

p.s. all this was written too many times all over the thread already, is copy-pasting someone's xml the best of your abilities?

Last edited by sinny (2014-10-12 04:58:03)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lspci -nn

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750 / R7 250E] [1002:683f]
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]
02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750 / R7 250E] [1002:683f]
02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]
04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF119 [GeForce GT 610] [10de:104a] (rev a1)
04:00.1 Audio device [0403]: NVIDIA Corporation GF119 HDMI Audio Controller [10de:0e08] (rev a1)

dmesg: http://pastebin.com/wp56ts9Y

sudo qemu-system-x86_64 \
-boot menu=on \
-enable-kvm \
-M q35 \
-m 4096 \
-cpu host \
-net none \
-rtc base=localtime \
-smp 4,sockets=1,cores=4,threads=1 \
-netdev tap,ifname=tap0,id=tap0 \
-drive file='/mnt/hdd/qemu-vfio-win.img',id=disk,format=raw,cache=none \
-drive file='/mnt/hdd/virtio.iso',id=cdrom,format=raw,readonly=on \
-device ioh3420,addr=02.0,multifunction=on,port=1,chassis=1,id=root.0 \
-device ioh3420,addr=04.0,multifunction=on,port=1,chassis=2,id=root.1 \
-device ioh3420,addr=06.0,multifunction=on,port=1,chassis=3,id=root.2 \
-device virtio-blk-pci,bus=root.2,addr=04.0,drive=disk \
-device ide-cd,unit=0,drive=cdrom \
-device ich9-intel-hda,bus=root.2,addr=00.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-device virtio-net-pci,netdev=tap0,bus=root.2,addr=01.0 \
-device vfio-pci,host=01:00.0,bus=root.0,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.0,addr=00.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=off \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 \
-vga none \
-device qxl,bus=root.2,addr=02.0

BSOD 116 after catalyst install inside windows VM.
Second card doesn't start yet - device code 10.

I have patched host nvidia driver so /dev/vga_arbiter would be

count:3,PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none(0:0)

QEMU 2.1.2

AMD Catalyst 14.9

Also:

options kvm-amd npt=0
blacklist nouveau
options nouveau modeset=0

Full dmesg(quite lenghty): http://pastebin.com/HiJh81Ct

ls /sys/kernel/iommu_groups/*/devices:

/sys/kernel/iommu_groups/1/devices:
0000:00:02.0  0000:01:00.0  0000:01:00.1

/sys/kernel/iommu_groups/2/devices:
0000:00:04.0  0000:02:00.0  0000:02:00.1

Full lspci: http://pastebin.com/0ZED3z7U

Any ideas how to:
- fix BSOD? disabling HDMI audio doesn't help anymore
- fix code 10 on second gpu?

Maybe i should use -M pc and -nodefaults?

UPD:Same dmesg messages appear, but for the different device, the gpu itself, when passing only GPUs without HDMI audio.

Last edited by Duelist (2014-10-12 14:03:53)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

novist

Member

Registered: 2014-03-14

Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I wonder - has anyone tried and succeeded making this work on a laptop? All in all - it should work just fine on laptop with APU and dedicated GPU. What i dont know is how laptop could switch display to dedicated gpu. Anyone know if this is even possible with today's hardware?

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I wonder - has anyone tried and succeeded making this work on a laptop? All in all - it should work just fine on laptop with APU and dedicated GPU. What i dont know is how laptop could switch display to dedicated gpu. Anyone know if this is even possible with today's hardware?

I've built a pair of systems with APU and dual graphics.

It does not switch. It works the other way.

APU has it's own GPU inside, sitting on pci-e bus(thanks to UNB and integrated pci-e controller), which handles the video output.
When running dual graphics(having discrete gpu) - it's simply just crossfire between those cards.
On the desktop part, some applications may not support crossfire(or dual graphics may be turned off) and chosen device for rendering will depend from chosen output - for example, if you have your screen plugged in your motherboard then APU's GPU will render the scene and vice versa.
On the notebook part - some notebooks have video output routed physically to the dedicated GPU(usually those notebooks have non-APU versions, but that's quite rare), some have video output routed to APU.

It is not Nvidia Optimus, these notebooks just don't switch GPUs for energy saving, instead, they have zerocore technology in crossfire which cuts power to the secondary GPU.

BUT! All those shiny new("new" as 2012-new) technologies won't work with that particular technology quite right - i couldn't get fglrx working on host AND passing the second card to VM at the same time. fglrx seem to be messing around and scanning whole PC for related vendor-dev-ids and touching those devices without asking anyone.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Drakulix

Member

Registered: 2014-10-10

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I guess getting ignored means, nobody else has an idea?

sinny

Member

Registered: 2014-05-24

Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I guess getting ignored means, nobody else has an idea?

probably. you got very experimental hardware set on top of experimental softwasre feature (==experimental squared?)

plus, me personnally would not expect someone trying to achieve anything (related to the discussion here) on apple garb devices - you either get these to forget everything "complex" or stick with other hardware choices that allow more flexibility/choices

so i'm afraid you are on your own there or, if you are lucky, there will be very little pre-existing experience for you to lean on

Drakulix

Member

Registered: 2014-10-10

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I guess getting ignored means, nobody else has an idea?

probably. you got very experimental hardware set on top of experimental softwasre feature (==experimental squared?)

plus, me personnally would not expect someone trying to achieve anything (related to the discussion here) on apple garb devices - you either get these to forget everything "complex" or stick with other hardware choices that allow more flexibility/choices

so i'm afraid you are on your own there or, if you are lucky, there will be very little pre-existing experience for you to lean on

Sure it is an experimental setup and sure there are probably not many users, that did try this already.
To be honest, I did not buy this hardware intentionally to do kvm-stuff on it or even install linux. I just recently switched and I am still stunned by the possibilities. Thats why I wanted to give it a try, because surprisingly enough my hardware seemed to meet all requirements. I already got quite far, that is why I want to try on.

And Code 43 seems to be a very common issue, so I did hope, there were some more hints then just "add kvm=off", "remove the hv_* functions" and "put your card in another PCIe-Port" (which would obviously not work on my setup).

I am also not afraid of looking at huge log files or even try to write patches on my own, I just don't know where to start with this weird error. So if anybody has some hints, what I could try, that would be great.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I guess getting ignored means, nobody else has an idea?

Just following your wishes:

Please don't tell me right away, that VGA Passthough will not work on my setup.

Seriously, laptops with dual graphics do strange and bizarre things and Code 43 is a pretty standard "I give up" error code from the Nvidia driver.  We don't have the source, so we can only guess what might be wrong.  Sometimes these laptops require vendor drivers for the discrete GPU to work rather than the standard Nvidia driver.  When that GPU gets extracted into a VM, it may not behave exactly like devices supported by the standard driver.  Especially on an Apple product, it's not clear that expecting the Windows driver to work is valid.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Drakulix

Member

Registered: 2014-10-10

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I guess getting ignored means, nobody else has an idea?

Just following your wishes:

Please don't tell me right away, that VGA Passthough will not work on my setup.

I think, you can guess, that I just wanted to avoid the usual "anti-apple-hate".

Seriously, laptops with dual graphics do strange and bizarre things and Code 43 is a pretty standard "I give up" error code from the Nvidia driver.  We don't have the source, so we can only guess what might be wrong.  Sometimes these laptops require vendor drivers for the discrete GPU to work rather than the standard Nvidia driver.  When that GPU gets extracted into a VM, it may not behave exactly like devices supported by the standard driver.  Especially on an Apple product, it's not clear that expecting the Windows driver to work is valid.

I did always use the unmodified nvidia drivers from their website on a bootcamp installed system without any issue (never installed them either though apple's magical setup program or windows update). Apple isn't even using nvidia optimus, it is just a hardware mux, no weird shared buffers between the two graphic cards. I guess that is as close to desktop graphics, as it can get on a laptop.
Does anybody at least have an idea, how I can get more verbose output from the involved kernel modules and/or qemu? I start reading into the source code if necessary, I am not yet giving up.

Last edited by Drakulix (2014-10-13 09:19:54)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

We don't have the source, so we can only guess what might be wrong.

Even worse.

/* _NVRM_COPYRIGHT_BEGIN_
*
* Copyright 2011 by NVIDIA Corporation.  All rights reserved.  All
* information contained herein is proprietary and confidential to NVIDIA
* Corporation.  Any use, reproduction, or disclosure without the written
* permission of NVIDIA Corporation is prohibited.
*
* _NVRM_COPYRIGHT_END_
*/

Even if we had - there would be legal issues. Because any use is prohibited. And asking nvidia something like "we want to get our KVM vga passthrough working" would be obviously pointless because... you know why.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Chetyre

Member

Registered: 2013-03-27

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Upgraded my kernel to 3.17, but I lost my host graphics.

Actually that happened to in 3.16, but I never mentioned it because I was thinking of messing with ovmf (which failed)

I need the VGA arbiter patch. With it on I can't see anything on host, with it off I get garbled graphics as expected. Back in 3.15 everything just worked. What could be wrong?