Arch Linux

corna

Member

Registered: 2015-05-13

Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

1. What windows do you use? Win 7? Have you followed the instructions from OVMF whitepaper? Win 8? Win 8 won't need QXL at all.
2. ISA debug console, like

-debugcon file:debug.log -global isa-debugcon.iobase=0x402 

should produce debug.log with OVMF logging.
Have you tried installing windows with QXL and then installing the drivers despite seeing code 12? Maybe QXL will fall off(windows will disable it) and the GPU will fire up nicely(like it happened on my setup).

I'm using Windows 7 (look at my previous posts). If I use -M q35 I get an error 12, while if I use -M pc-i440fx-2.2 it reports an error 43 (even if I'm using kvm=off and no hyper-v enlightments)
A simple workaround for all the DRI problems is to use Wayland: in Wayland DRI works and the current status on gnome 3.14 is quite good.
I haven't read the OVMF whitepaper, I'm reading it right now thanks

crashtua

Member

Registered: 2015-05-22

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone.
I have following configuration:

sabertooth fx990 r2.0
fx 8320
gtx 980(vm)
gtx 750ti(host)

kvm modules options:

cat /etc/modprobe.d/kvm*
options kvm-amd npt=0
options kvm ignore_msrs=1

vm command line:

sudo qemu-system-x86_64 \
-name qwerty_nosound \
-machine pc-i440fx-2.0,accel=kvm,usb=off \
-cpu host,kvm=off \
-m 8192 \
-mem-path /dev/hugepages \
-realtime mlock=off \
-smp 8,sockets=1,cores=8,threads=1 \
-uuid d616d9cd-aa9d-564a-7999-704787e5ea6c \
-no-user-config \
-nodefaults \
-rtc base=localtime \
-usb \
-boot menu=off,strict=on \
-msg timestamp=on \
-device vfio-pci,host=06:00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=06:00.1 \
-device virtio-scsi-pci,id=scsi \
-device AC97,id=sound0,bus=pci.0,addr=0x6 \
-drive file=/home/eugene/Downloads/ru_windows_7_professional_with_sp1_vl_build_x64_dvd_u_677774.iso,id=isocd,if=none -device scsi-cd,drive=isocd \
-drive file=/home/dev/windows.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \
-drive file=/home/eugene/Downloads/virtio-win-0.1.96.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \
-drive file=/home/eugene/dota.img,id=disk1,format=raw,if=none -device scsi-hd,drive=disk1 \
-netdev user,id=inet \
-device e1000,netdev=inet \
-netdev tap,id=hoa,script=/usr/qemu-ifup \
-device e1000,netdev=hoa,mac=DE:AD:BE:EF:68:98 \
-nographic \
-monitor stdio

Everything works fine, but every 10-15 seconds during gaming fps drops to 0 for 1 second and then back to normal. Can someone tell how this lags can be removed?

Last edited by crashtua (2015-05-22 12:31:55)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

1. What windows do you use? Win 7? Have you followed the instructions from OVMF whitepaper? Win 8? Win 8 won't need QXL at all.
2. ISA debug console, like

-debugcon file:debug.log -global isa-debugcon.iobase=0x402 

should produce debug.log with OVMF logging.
Have you tried installing windows with QXL and then installing the drivers despite seeing code 12? Maybe QXL will fall off(windows will disable it) and the GPU will fire up nicely(like it happened on my setup).

I'm using Windows 7 (look at my previous posts). If I use -M q35 I get an error 12, while if I use -M pc-i440fx-2.2 it reports an error 43 (even if I'm using kvm=off and no hyper-v enlightments)
A simple workaround for all the DRI problems is to use Wayland: in Wayland DRI works and the current status on gnome 3.14 is quite good.
I haven't read the OVMF whitepaper, I'm reading it right now thanks

Use 440FX because q35 gives you no advantages. Breeze through the OVMF whitepaper, you should be able to find something that you've missed, because i can't see the problem right now. Also, seems like you forgot about the ISA debug console...

Everything works fine, but every 10-15 seconds during gaming fps drops to 0 for 1 second and then back to normal. Can someone tell how this lags can be removed?

Well, first you'll need to clear up your config.

-machine pc-i440fx-2.0,accel=kvm,usb=off \

Are you sure about that line? accel=kvm is soooo "old school", that it's not needed anymore.
Turning off USB looks strange, especially having "-usb \" later on.

-mem-path /dev/hugepages \ 

That may be the problematic place, as hugepages and npt may work not-so-well on AMD platforms, try toggling that and NPT.

-realtime mlock=off \

Now THAT's a strange thing to do. Back in the day, i've forced mlock=on only to find out it's default on.
Are you sure you know why you've turned it off and what that option does?

-device AC97,id=sound0,bus=pci.0,addr=0x6 \

I can see a soundcard device, but no sound output options, do you have sound on your VM?..

-netdev user,id=inet \
-device e1000,netdev=inet \
-netdev tap,id=hoa,script=/usr/qemu-ifup \
-device e1000,netdev=hoa,mac=DE:AD:BE:EF:68:98 \

Since you're obviously playing DotA, do you really need a user-mode networking AND a tap interface simultaneously? User mode networking can be very CPU consuming.

The general advice:
Monitor your CPU load. You're using 8 cores out of 8 available ones, maybe the host system does something strange.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

crashtua

Member

Registered: 2015-05-22

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

That may be the problematic place, as hugepages and npt may work not-so-well on AMD platforms, try toggling that and NPT.

You mean set kvm-amd.npt to 1 and leave -mem-path /dev/hugepages?

Now THAT's a strange thing to do. Back in the day, i've forced mlock=on only to find out it's default on.
Are you sure you know why you've turned it off and what that option does?

No, I am not. Just grabbed some qemu execute string from internet, and it worked for me, but as you can see, not as good as I expected:)
And thanks for reply, because it is really hard to follow up this 200-page topic.

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
Thanks for the suggestion about the vmx setting and looking at the qemu command line. (beginning p. 206, end of p 205)  However, the problem persists.  Unfortunately, I didn't check the functioning of XP Mode every step of the way in going from qemu-script q35 with SeaBios to libvirt 440fx with OVMF and UEFI, so I don't know which part of the transition made it stop working.
This is the command line from /var/log/libvirt/qemu:

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/sbin/qemu-system-x86_64
-name Windows7 -S -machine pc-i440fx-2.2,accel=kvm,usb=off -cpu SandyBridge,+vmx,kvm=off
-drive file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on
-drive file=/var/lib/libvirt/qemu/nvram/Windows7_VARS.fd,if=pflash,format=raw,unit=1
-m 8192 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu -realtime mlock=off -smp 6,sockets=6,cores=1,threads=1
-uuid f63ea610-45ae-4daa-bc14-4e2204XXXXX -nographic -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows7.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control
-rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet
-no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on
-device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5
-drive file=/dev/sr0,if=none,id=drive-ide0-0-1,readonly=on,format=raw
-device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1
-drive file=/home/me/virtio-win-0.1-100.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw
-device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
-drive file=/dev/cryptvg2/win7,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0xa,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:XX:XX:XX:XX,bus=pci.0,addr=0x3
-device intel-hda,id=sound0,bus=pci.0,addr=0x4
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x8
-device vfio-pci,host=03:00.1,id=hostdev1,bus=pci.0,addr=0x9 -device usb-host,hostbus=4,hostaddr=4,id=hostdev2
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7
-msg timestamp=on 

This is the old version:

QEMU_AUDIO_TIMER_PERIOD=0 QEMU_AUDIO_DRV=alsa \
sudo qemu-system-x86_64 -enable-kvm -M q35 -m 8192 \
-cpu host,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 -smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none -nographic \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \
-usbdevice tablet -usbdevice host:0803:3095 -usbdevice host:0461:4d22 \
-usbdevice host:1131:1001 -usbdevice host:2304:023f \
-drive file=/dev/mapper/cryptvg2-win7,id=disk,if=virtio \
-device piix4-ide,bus=pcie.0,id=piix4-ide \
-drive file=/dev/cdrom,id=cddisk -device ide-cd,bus=piix4-ide.1,drive=cddisk \
-net vde,vlan=0 -net nic,vlan=0,macaddr=52:45:4d:XX:XX:XX,model=virtio \
-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-rtc base=localtime \
-monitor unix:/home/me/monitor,server,nowait

I got rid of the hv stuff and upgraded the Nvidia drivers after converting to OVMF.  Obviously the cpu has changed form "host" to "SandyBridge" (there was no host option when I made the new version using VMM), and some of the usb devices have been changed around.  I suspect the main difference is libvirt.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I got rid of the hv stuff and upgraded the Nvidia drivers after converting to OVMF.  Obviously the cpu has changed form "host" to "SandyBridge" (there was no host option when I made the new version using VMM), and some of the usb devices have been changed around.  I suspect the main difference is libvirt.

libvirt is just a mechanism for managing the VM, it's still just using QEMU in the end, so if XP mode can be done via QEMU commandline, it should be possible to do it with libvirt.  To get the "host" CPU option, just manually type "host-passthrough" in the virt-manager drop-down box.  It's not listed as a select-able option, but it works.  You can also manually edit the xml to something like:

  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Everything works fine, but every 10-15 seconds during gaming fps drops to 0 for 1 second and then back to normal. Can someone tell how this lags can be removed?

Perhaps something to do with leaving no cores available for the host?  You have an 8-core host and you've created an 8-core guest.  How is the normal running of the host supposed to not interfere with the VM?  It's gotta get CPU time somehow.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw Thanks, I'll give it a try.  <rant> There's only so much reconfiguration one can do.  I'm concerned that Windows will once again decide that I have changed machines and that I have 3 days to activate. The last two times the automatic version didn't work and I had to call the Microsoft number, still semiautomatic.  Sooner or later I assume that won't work either and they'll insist I buy a new licence.  Rather annoying since I've had only one installation and am merely trying to optimize the VM.  Since that takes time (witness the last 200+ pages), activation is necessary at some point. Oi <rant/>

Re libvirt, surely there's more to it than that.  Libvirt uses virsh to attach devices; I use to just use vfio-bind. Also, I tried passing the qemu command line from /var/log/libvirt to the command line in a script, and get nothing, black screen, no VM.

Last edited by mostlyharmless (2015-05-22 17:07:11)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Re libvirt, surely there's more to it than that.  Libvirt uses virsh to attach devices; I use to just use vfio-bind. Also, I tried passing the qemu command line from /var/log/libvirt to the command line in a script, and get nothing, black screen, no VM.

libvirt passes various file descriptors by number rather than letting QEMU open things on its own, it also configures networks and passes tap devices, so you can't simply cut and paste the command line to a shell prompt and expect it to work.  You can however re-parse it into something that will work by hand, I've done this many times in debugging issues.  Why do you think that virsh nodedev-detach is really all that much different from vfio-bind?  libvirt knows how to manipulate devices and drivers via sysfs too.  If anything, the vfio-bind script is the path to avoid.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah, I removed the network stuff, I assume the rest could run; I'll do more trimming if the cpu change doesn't do it.  I don't know about virsh vs vfio-bind; I only know what I've read here, and switched to virsh accordingly when I switched to libvirt.  But, as you point out, there are subtle differences, else one would not be preferable to the other. 

Before I switched to libvirt, I tried using virsh instead of vfio-bind with my old script and got the dreaded Windows 7B error every time. Switched back to vfio-bind, it disappeared again.  So there's something Windows detects differently between the two.  I have no idea whether it's part of the current XP mode problem.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

That may be the problematic place, as hugepages and npt may work not-so-well on AMD platforms, try toggling that and NPT.

You mean set kvm-amd.npt to 1 and leave -mem-path /dev/hugepages?

Now THAT's a strange thing to do. Back in the day, i've forced mlock=on only to find out it's default on.
Are you sure you know why you've turned it off and what that option does?

No, I am not. Just grabbed some qemu execute string from internet, and it worked for me, but as you can see, not as good as I expected:)
And thanks for reply, because it is really hard to follow up this 200-page topic.

As i've been suggested on IRC: man 3 mlock. Basically, by saying "mlock=off" you're saying "HA! Let that VM memory be swapped!". Which is a very strange thing to do, considering our system being more or less realtime. So chop off that line. Also, using VFIO pins the memory in one place, preventing it from being moved around in the system memory, so, as i understand it, it will never actually be swapped, and that option won't do anything useful.

Test the VM performance with all combinations of npt=1 or npt=0 and hugepages enabled/disabled. That's all. You might eventually get better performance with npt=1 and hugepages enabled, or it may do the opposite thing due to AMD problems, as was noted two years old in the op-post.

And regarding your CPU usage, as aw suggests... I have 4 cores, of which i'm allowing VM to use all 4 cores, and i haven't observed anything alike to your problems when neither host nor guest system is doing something very cpu intensive. When the host loads all four cores with some stuff - the VM struggles and lags, and vice versa.
A generally good idea would be to isolate some(maybe all seven, leaving host with one core) CPU cores from host usage and dedicate them to the VM. Usually this helps a lot when pursuiting latency issues. You could set those cores to NoHZ mode(or something like that, i don't quite remember) or doing cpuset.

And there's a good in-depth manual available from redhat.

P.S.
Я не помню чтобы даже на хабре была эта mlock=off. Откуда?..

Last edited by Duelist (2015-05-22 18:48:30)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

crashtua

Member

Registered: 2015-05-22

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

P.S.
Я не помню чтобы даже на хабре была эта mlock=off. Откуда?..

Честно? Я хз откуда я это взял хДД

Thanks everyone for reply, have good points for optimization. With leaving 2 cores for host and npt=1+hugepages i have constant, but 60 fps in dota(0-110 before changes), so most likely I am on right track.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

P.S.
Я не помню чтобы даже на хабре была эта mlock=off. Откуда?..

Честно? Я хз откуда я это взял хДД

Thanks everyone for reply, have good points for optimization. With leaving 2 cores for host and npt=1+hugepages i have constant, but 60 fps in dota(0-110 before changes), so most likely I am on right track.

60fps sounds suspiciously like an in-game limitter

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

paperinick

Member

Registered: 2015-04-14

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I was too lazy to fix the paths, so i just did

that's good enough, many thanks! I will experiment next week and hopefully see some progress!!

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

P.S.
Я не помню чтобы даже на хабре была эта mlock=off. Откуда?..

Честно? Я хз откуда я это взял хДД

Thanks everyone for reply, have good points for optimization. With leaving 2 cores for host and npt=1+hugepages i have constant, but 60 fps in dota(0-110 before changes), so most likely I am on right track.

60fps sounds suspiciously like an in-game limitter

Assuming it's still a source-based stuffgame, there's a variable called fps_max which, well, sets the limit.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Jodaco

Member

Registered: 2014-07-30

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So in general my config has been working great for me, but I do have random lockups of the guest and host with no information in the logs whatsoever or any pattern to when they happen. I am just using qemu with command line parameters rather than libvirt. I am on an intel platform.   I was wondering if anyone has any suggestions as to what kernel debugging options or other possible flags to pass to qemu for extra information so that I might track this down.

mqddb

Member

Registered: 2015-04-24

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi ALL:
     If CPU Integrated Graphics can passthrough to VM?

crashtua

Member

Registered: 2015-05-22

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

60fps sounds suspiciously like an in-game limitter

Assuming it's still a source-based stuffgame, there's a variable called fps_max which, well, sets the limit.

Doublechecked, no limit and no vsync enabled. But anyway, it is stable fps, even new witcher performs on stable 40, and ocasional freezes cant break normal gameplay. And older games, like tomb rider, works on constant 60(everything max, vsync enabled).

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi ALL:
     If CPU Integrated Graphics can passthrough to VM?

Intel, no, not yet.  AMD, maybe.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

protake

Member

Registered: 2014-09-09

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Finally found some time to set up everything properly a few days ago and it works pretty well. The only thing that's bugging me is the mouse/keyboard. When passing them through, of course they aren't accessible from the host any more. Is there any way to make them accessible to both the guest and host? It'd be nice for the guest to be "escapable".

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Synergy works great for mouse and keyboard.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Finally found some time to set up everything properly a few days ago and it works pretty well. The only thing that's bugging me is the mouse/keyboard. When passing them through, of course they aren't accessible from the host any more. Is there any way to make them accessible to both the guest and host? It'd be nice for the guest to be "escapable".

I use libvirt's graphics window(doing win7+QXL being disabled), which looks like this:

(Due to HDD hardware migration process, i can't run the guest right now).
When the system starts, VM starts to emulate QXL graphics, and that provides me a window where keyboard and mouse input can be grabbed. To escape the guest, i must push ctrl-shift-alt which happens very rarely in normal VM use(not doing emacs stuff, for example).
So far - so good, input latency is low enough to play quake 3 arena competitively.
Before moving to libvirt i did almost the same thing, adding -device qxl to my qemu commandline and running without -nographics to have a small window where input may be grabbed.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

ghormoon

Member

Registered: 2015-05-20

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi, I've got into state where I actually see windows booting (the logo). after that, screen goes black. I can rdp into it and it says device works OK. maybe it remembers something wrong from the previous (cirrus) setup
I'll try to reinstall it, if I figure out how to make it boot from CD without pressing "any key", since usb keyboard doesn't work at that point

another (maybe related, maybe not) problem I have is that when shutting down the domain, I get crash on "kernel tried to execute NX-protected page - exploit attempt?", I can't even halt the host after that ...
dmesg: http://pastebin.com/c3bRQQFe

any ideas on any of those problems?

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi, I've got into state where I actually see windows booting (the logo). after that, screen goes black. I can rdp into it and it says device works OK. maybe it remembers something wrong from the previous (cirrus) setup
I'll try to reinstall it, if I figure out how to make it boot from CD without pressing "any key", since usb keyboard doesn't work at that point

another (maybe related, maybe not) problem I have is that when shutting down the domain, I get crash on "kernel tried to execute NX-protected page - exploit attempt?", I can't even halt the host after that ...
dmesg: http://pastebin.com/c3bRQQFe

any ideas on any of those problems?

You're not using OVMF, right? If so...
What the mess you have with your hardware?
What devices do you have and what are you trying to pass-through?
I see that you have kernel modesetting enabled and radeon module enabled, and that just attracts problems.

 [    9.016962] fb: switching to radeondrmfb from VESA VGA

Framebuffer thingie works via VGA, and you're trying to isolate VGA access and use it from vfio(and the VM) doing x-vga=on...

...

How the hell this thing is supposed to work? That's a notebook, it has one screen only! Are you trying to do secondary pass-through or throw a VNC connection from-guest-to-host to have a small window with windows(sic!) running 3D stuff in it? Or what?

Seems like you unbind the device from radeon and attach it to vfio-pci... but the device isn't properly returned to it's un-enabled state:

 [  218.594123] [drm:rv770_stop_dpm [radeon]] *ERROR* Could not force DPM to low.

and something hints me that you'll have trouble with ROM, assuming 03:00.0 is actually the GPU in question(you haven't showed us lspci output yet), because...
AHAHAHAHA I AM GOING INSANE, THAT'S AN SD CONTROLLER! WHICH IS BROKEN TOO!

 [  227.044242] vfio-pci 0000:03:00.0: Invalid ROM contents

And then even more weird cra shi things happen:

[  227.054680] dmar: DRHD: handling fault status reg 3
[  227.054688] dmar: DMAR:[DMA Read] Request device [00:1d.0] fault addr ef000
               DMAR:[fault reason 06] PTE Read access is not set

THAT looks very bad, what the hell is that 00:1d.0 device? Looks like you're trying to pass-through it too.

OH WELL, just give me the LSPCI output and tell me what are you trying to do, all i see now is a mess of hardware and virtualization everywhere.
Please, list iommu_groups too.

...

[ 1150.557220] mmc0: SDHCI controller on PCI [0000:03:00.0] using DMA
[ 1150.639153] vfio_pci_disable: Failed to reset device 0000:05:00.0 (-11)

Seems like somewhere there you've shutdown your VM, and...
Your system just exploded with errors:

[ 1159.834105] [drm:uvd_v1_0_start [radeon]] *ERROR* UVD not responding, trying to reset the VCPU!!!
eight more lines alike
[ 1169.050536] [drm:uvd_v1_0_start [radeon]] *ERROR* UVD not responding, trying to reset the VCPU!!!
[ 1169.070551] [drm:uvd_v1_0_start [radeon]] *ERROR* UVD not responding, giving up!!!
[ 1169.070572] [drm:evergreen_startup [radeon]] *ERROR* radeon: error initializing UVD (-1).

I have absolutely no clue what that means, but three exclamation marks and "trying to reset the VCPU" looks very serious.
But nevertheless, it tries to continue and...

[ 1170.117249] radeon 0000:01:00.0: fb0: radeondrmfb frame buffer device
[ 1170.117251] radeon 0000:01:00.0: registered panic notifier
[ 1170.135759] [drm] Initialized radeon 2.41.0 20080528 for 0000:01:00.0 on minor 0
[ 1170.136029] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
[ 1170.136065] BUG: unable to handle kernel paging request at ffff880429522c40

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

ghormoon

Member

Registered: 2015-05-20

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lspci:
00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
00:16.3 Serial controller: Intel Corporation 6 Series/C200 Series Chipset Family KT Controller (rev 04)
00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 04)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b4)
00:1c.2 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 3 (rev b4)
00:1c.3 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 4 (rev b4)
00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b4)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)
00:1f.0 ISA bridge: Intel Corporation QM67 Express Chipset Family LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port SATA AHCI Controller (rev 04)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Seymour [Radeon HD 6400M/7400M Series]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series]
03:00.0 System peripheral: JMicron Technology Corp. SD/MMC Host Controller (rev 30)
03:00.2 SD Host controller: JMicron Technology Corp. Standard SD Host Controller (rev 30)
04:00.0 Network controller: Intel Corporation Centrino Advanced-N 6205 [Taylor Peak] (rev 34)
05:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04)

virt manager xml: http://pastebin.com/6PqWB18r (note that x-vga=on is added in the qemu-wrapper)

as for the questions asked:
OVMF: no, it seems to be unsupported on my HW.

I'm passing in 00:1a.0 00:1b.0 00:1d.0 03:00.0 03:00.2 01:00.0 01:00.1 05:00.0 i.e. basically everything, that is not a pci/sata conroller or a network card.

true I've let the radeon to load, I'll try next time without it. linux vm (almost the same config) pickes it ok though

how is it supposed to work - to have vga and all the periphetals passed in, host will be headless.  most of the work I'll do on the vm with VGA passed in with some other VMs  around for testing, services and such.

SD controller is broken by default interesting though that it works in the vm

as for the crash in the end, that happens in the linux VM too, though until shutdown that works ok.