Arch Linux

Flyser

Member

Registered: 2013-12-19

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

20 fps? Ouch, that's bad. gtx 660 is capable of much-much more.

Could be related to the streaming... I don't have the time to test in detail today, but I can test a different game during the week.
While I did notice a drop in performance between native and kvm in Assassins Creed 4, it wasn't huge ...

Kaidax

Member

Registered: 2014-02-09

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

20 fps? Ouch, that's bad. gtx 660 is capable of much-much more.

Could be related to the streaming...

Not if you have >20 fps for any other game at the same resolution... Did you try streaming at 640x480 all graphics options turned off?

I don't have the time to test in detail today, but I can test a different game during the week.
While I did notice a drop in performance between native and kvm in Assassins Creed 4, it wasn't huge ...

Well, I've noticed that I can't get more than 62 fps in Arkham City no matter what... (gameplay, not benchmark mode) Also, Crysis is jerkier and drops frames more than on native...
But Borderlands 2 is the only game I've encountered that COMPLETELY DIES on kvm, so it makes for a much better test.

DanaGoyette

Member

Registered: 2014-01-03

Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've switched back to using libvirt, since EFI+i440fx works reliably with several devices assigned.

However, after a suspend/resume cycle (needed because the bus reset doesn't work),
the VM suddenly complains about one of the devices not being assignable!

I see this in the libvirt log:

error : virPCIDeviceIsAssignable:2260 : internal error: Device 0000:08:00.0 is behind a switch lacking ACS and cannot be assigned

08:00.0 sits behind a PLX PEX8606, which absolutely DOES have ACS.
It seems libvirt adds a second layer of checking, which somehow gets broken by suspend/resume.

Edit: "fixed" by editing /etc/libvirt/qemu.conf to set relaxed_acs_check = 1

EDIT again: What do I need to do to get PCI bus reset to work?  Right now, I still have to suspend and resume the host to get the card reset when it's in a bad state.
At least host suspend/resume finally work, since I removed the Asus sound card.

Last edited by DanaGoyette (2014-02-11 04:16:12)

doubledr

Member

Registered: 2013-12-26

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just curious, does any developer in QEMU notice about poor mem performance in Q35 setting?

Last edited by doubledr (2014-02-11 21:48:21)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just curious, does any developer in QEMU notice about poor mem performance in Q35 setting?

I'd suggest finding a test case that shows the problem, is reproducible, and preferably does not depend on assigned devices and does use a Linux guest (or simple tool like memtest86).  Report it to kvm & qemu-devel mailing lists.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Borg8401

Member

Registered: 2014-02-11

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

First off, I want to say thanks to nbhs and anyone else who has posted useful information or have helped others on this thread.  I've finally got passthrough working on my system and wanted to share my results and see if anyone has some suggestions as to how I could possibly improve my experience.  I'm currently running Arch Linux 2014.02.01, I haven't downloaded the patched kernel and qemu and seabios were installed from pacman.  I'm willing to give them all a shot but as my current setup is working fairly well I thought I'd ask for some input before proceeding.

Hardware

Asus Rampage IV Extreme
Intel i7 3930K @ 4.0GHz (4 cores, 8 threads given to guest)
16GB DDR3 2133MHz Corsair Vengeance (12288MB given to guest)
AMD 4890 - Sapphire (Host - using open source drivers)
Nvidia GTX 780 - Asus OC DirectCU II (Guest)
Kingston HyperX 3K 120GB (Host, 16GB swap, 80GB raw image for guest)
Seagate Barracuda 3TB (Given to guest as slave - games held here)

My starting script is (loosely) based on the one used by Kaidax as it improved my WEI score from 5.9 to 7.6 (based on hard disk performance, other values did not change) as compared to OP's first post.  I added vfi-bind into my starting script as it needs to be ran every time I reboot my host for some reason and this was a simple solution to not forget.   Running "export SDL_VIDEO_X11_DGAMOUSE=0" helped some weird lag/loss of input/stuttering that I'm experiencing with my mouse, however it has not completely solved the issue and I also suffer from something similar at times when using my keyboard; I added it into my script for no good reason really.  On "-machine type=pc,accel=kvm" I still have no idea if there is a difference either way when using this option, so it can stay until I discover otherwise.

/sbin/Win7

vfio-bind 0000:03:00.0 0000:03:00.1
export SDL_VIDEO_X11_DGAMOUSE=0
qemu-system-x86_64 -enable-kvm -machine type=pc,accel=kvm \
-m 12288 -mem-prealloc \
-cpu host -smp 8,sockets=1,cores=4,threads=2 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=pci.0,x-vga=on \
-device vfio-pci,host=03:00.1,bus=pci.0 \
-drive file=/home/borg8401/windows.img,id=hd,format=raw,index=0 \
-drive file=/dev/sdb,id=hd2,format=raw,index=1 \
-usb -usbdevice host:1038:1200 -usbdevice host:06a3:0ccb

Results:

I've uploaded a video that is comprised of one three minute clips each of War Thunder, Borderlands 2, and Battlefield 4 to demonstrate my current performance.  After uploading the video I realized my FPS was only being monitored during War Thunder, but you should still be able to get a general feel for the performance that I'm getting (I will upload a fixed video and edit this post).  http://www.youtube.com/watch?v=oJB1Om6qJsg

War Thunder plays at a (nearly) constant framerate of 60 FPS with vsync on and apart from occasional twitches is very playable.  Borderlands 2 could be considered playable (by some) but is very choppy and the framerate is painfully inconsistent, definitely no where close to native performance.  Battlefield 4 was easily the worst of the three and the lighting effects seemed to destroy any hopes of it being an enjoyable experience; also Punkbuster kicks me when trying to join an online game, stating that I have no ping.

Any suggestions moving forward would be greatly appreciated, if necessary I could go ahead and attempt installing the kernel, qemu and seabios from OP's post; I just don't want to break what I currently have working in the process as I'm still fairly confused on how to go about building things in arch.

Last edited by Borg8401 (2014-02-12 04:48:38)

doubledr

Member

Registered: 2013-12-26

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just curious, does any developer in QEMU notice about poor mem performance in Q35 setting?

I'd suggest finding a test case that shows the problem, is reproducible, and preferably does not depend on assigned devices and does use a Linux guest (or simple tool like memtest86).  Report it to kvm & qemu-devel mailing lists.

It is not that easy, all mem benchmark software crash... I can't find a good benchmark software.

doubledr

Member

Registered: 2013-12-26

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

First off, I want to say thanks to nbhs and anyone else who has posted useful information or have helped others on this thread.  I've finally got passthrough working on my system and wanted to share my results and see if anyone has some suggestions as to how I could possibly improve my experience.  I'm currently running Arch Linux 2014.02.01, I haven't downloaded the patched kernel and qemu and seabios were installed from pacman.  I'm willing to give them all a shot but as my current setup is working fairly well I thought I'd ask for some input before proceeding.

Hardware

Asus Rampage IV Extreme
Intel i7 3930K @ 4.0GHz (4 cores, 8 threads given to guest)
16GB DDR3 2133MHz Corsair Vengeance (12288MB given to guest)
AMD 4890 - Sapphire (Host - using open source drivers)
Nvidia GTX 780 - Asus OC DirectCU II (Guest)
Kingston HyperX 3K 120GB (Host, 16GB swap, 80GB raw image for guest)
Seagate Barracuda 3TB (Given to guest as slave - games held here)

My starting script is (loosely) based on the one used by Kaidax as it improved my WEI score from 5.9 to 7.6 (based on hard disk performance, other values did not change) as compared to OP's first post.  I added vfi-bind into my starting script as it needs to be ran every time I reboot my host for some reason and this was a simple solution to not forget.   Running "export SDL_VIDEO_X11_DGAMOUSE=0" helped some weird lag/loss of input/stuttering that I'm experiencing with my mouse, however it has not completely solved the issue and I also suffer from something similar at times when using my keyboard; I added it into my script for no good reason really.  On "-machine type=pc,accel=kvm" I still have no idea if there is a difference either way when using this option, so it can stay until I discover otherwise.

/sbin/Win7

vfio-bind 0000:03:00.0 0000:03:00.1
export SDL_VIDEO_X11_DGAMOUSE=0
qemu-system-x86_64 -enable-kvm -machine type=pc,accel=kvm \
-m 12288 -mem-prealloc \
-cpu host -smp 8,sockets=1,cores=4,threads=2 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=pci.0,x-vga=on \
-device vfio-pci,host=03:00.1,bus=pci.0 \
-drive file=/home/borg8401/windows.img,id=hd,format=raw,index=0 \
-drive file=/dev/sdb,id=hd2,format=raw,index=1 \
-usb -usbdevice host:1038:1200 -usbdevice host:06a3:0ccb

Results:
http://i.imgur.com/yBdxtB3.png

I've uploaded a video that is comprised of one three minute clips each of War Thunder, Borderlands 2, and Battlefield 4 to demonstrate my current performance.  After uploading the video I realized my FPS was only being monitored during War Thunder, but you should still be able to get a general feel for the performance that I'm getting (I will upload a fixed video and edit this post).  http://www.youtube.com/watch?v=oJB1Om6qJsg

War Thunder plays at a (nearly) constant framerate of 60 FPS with vsync on and apart from occasional twitches is very playable.  Borderlands 2 could be considered playable (by some) but is very choppy and the framerate is painfully inconsistent, definitely no where close to native performance.  Battlefield 4 was easily the worst of the three and the lighting effects seemed to destroy any hopes of it being an enjoyable experience; also Punkbuster kicks me when trying to join an online game, stating that I have no ping.

Any suggestions moving forward would be greatly appreciated, if necessary I could go ahead and attempt installing the kernel, qemu and seabios from OP's post; I just don't want to break what I currently have working in the process as I'm still fairly confused on how to go about building things in arch.

lighting is a serious problem in MS DX11 implementation. The same story happens in Dolphin WII emulator. When I am facing lighting in WII game, my fps in WII will drop from 60 to 20. It turns out that this is a CPU bounded problem. In Dolphin, I have to turn "Skip EFB access from CPU" on to avoid this problem. In general case, I guess pinning CPU to VM will help this situation.

PS: thx for your post, now I know that pcie will work with 440fx, let me try it in my box.

Borg8401

Member

Registered: 2014-02-11

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lighting is a serious problem in MS DX11 implementation. The same story happens in Dolphin WII emulator. When I am facing lighting in WII game, my fps in WII will drop from 60 to 20. It turns out that this is a CPU bounded problem. In Dolphin, I have to turn "Skip EFB access from CPU" on to avoid this problem. In general case, I guess pinning CPU to VM will help this situation.

PS: thx for your post, now I know that pcie will work with 440fx, let me try it in my box.

Attempted googling pinning of the CPU to qemu, not finding much on the subject (just stuff regarding NUMA and libvirt, which I don't use).  Could anyone point me in the right direction?

doubledr

Member

Registered: 2013-12-26

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lighting is a serious problem in MS DX11 implementation. The same story happens in Dolphin WII emulator. When I am facing lighting in WII game, my fps in WII will drop from 60 to 20. It turns out that this is a CPU bounded problem. In Dolphin, I have to turn "Skip EFB access from CPU" on to avoid this problem. In general case, I guess pinning CPU to VM will help this situation.

PS: thx for your post, now I know that pcie will work with 440fx, let me try it in my box.

Attempted googling pinning of the CPU to qemu, not finding much on the subject (just stuff regarding NUMA and libvirt, which I don't use).  Could anyone point me in the right direction?

It is in the top post, "performance improvement" section

Borg8401

Member

Registered: 2014-02-11

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It is in the top post, "performance improvement" section

Holy crap, I even remember staring at that earlier, thanks! lol

DanaGoyette

Member

Registered: 2014-01-03

Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How can I get the VFIO_DEVICE_PCI_BUS_RESET changes into 3.13 and current qemu?  I tried to build the vfio-vga-reset branch of qemu on Ubuntu 13.10, and got:

hw/usb/host-libusb.c:243:5: error: ‘libusb_get_port_path’ is deprecated (declared at /usr/include/libusb-1.0/libusb.h:1358): Use libusb_get_port_numbers instead [-Werror=deprecated-declarations]

My host has been suspending and resuming reasonably reliably since removing the Asus sound card, so long as the guest is not running.  Still, it would be better to have the reset working.

I hope the bus reset won't require having x-vga=on. 
Judging by the lack of glitches, EFI mode in Windows 8.1 doesn't seem to need the VGA registers.

doubledr

Member

Registered: 2013-12-26

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

UPDATE: I can confirm that vfio will work with 440FX. Yes, my VIA VL80X USB 3 adpater is back! All you have to do is to  remove "-M Q35" and change "pcie.0" to "pci.0" in the script.

Borg8401

Member

Registered: 2014-02-11

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

On pinning cores, I wish to pass through 4 physical cores, and 4 virtual cores to achieve my 4 core 8 thread setup from my startup script.  Should I pass through cores 0-3 and 6-9 or cores 0-7?  For clarification, I am unsure if it's setup:

physical-physical-physical-physical-physical-physical-virtual-virtual-virtual-virtual-virtual-virtual
-or-
physical-virtual-physical-virtual-physical-virtual-physical-virtual-physical-virtual-physical-virtual

Also, is there a way to deny access to the physical and virtual cores I select above to my host and limit it only to the remaining cores/threads?

EDIT:
I have tried all of the following combinations with nearly identical levels of success/performance when testing with BF4:

0-3 and 6-9 -cpu host -smp 4,sockets=1,cores=4,threads=2
0-3 and 6-9 -cpu host -smp 8,sockets=1,cores=4,threads=2
0-3 and 6-9 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=2
0-3 and 6-9 -cpu SandyBridge -smp 8,sockets=1,cores=4,threads=2

2-5 and 8-11 -cpu host -smp 4,sockets=1,cores=4,threads=2
2-5 and 8-11 -cpu host -smp 8,sockets=1,cores=4,threads=2
2-5 and 8-11 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=2
2-5 and 8-11 -cpu SandyBridge -smp 8,sockets=1,cores=4,threads=2

0-3 -cpu host -smp 4,sockets=1,cores=4,threads=1
0-3 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=1

2-5 -cpu host -smp 4,sockets=1,cores=4,threads=1
2-5 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=1

0-3 -smp 4,sockets=1,cores=4,threads=1

2-5 -smp 4,sockets=1,cores=4,threads=1

It seems nothing I do to my CPU affects my experience, perhaps my performance issues are coming from somewhere else?  Could it be possible that even though GPU-Z (in Windows guest) reports PCIE is running at x16 3.0 that this may not be the case?  I could see that being a definite bottleneck that may not be noticeable in less demanding games, for instance upon testing Star Wars the Old Republic I had the same performance as if I were on a native Windows install.  Something strange I have noticed however is graphics settings don't have an impact on performance either, I get the same FPS in BF4 on ultra or on low.

Last edited by Borg8401 (2014-02-12 09:49:01)

Kaidax

Member

Registered: 2014-02-09

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

On pinning cores, I wish to pass through 4 physical cores, and 4 virtual cores to achieve my 4 core 8 thread setup from my startup script.  Should I pass through cores 0-3 and 6-9 or cores 0-7?  For clarification, I am unsure if it's setup:

physical-physical-physical-physical-physical-physical-virtual-virtual-virtual-virtual-virtual-virtual
-or-
physical-virtual-physical-virtual-physical-virtual-physical-virtual-physical-virtual-physical-virtual

It seems nothing I do to my CPU affects my experience, perhaps my performance issues are coming from somewhere else?

It's the first way. You can pin cores at runtime in virt-manager and verify that benchmarks won't like it one bit if you assign 0,6,1,7 to a 4-core VM, while 0,1,8,9 is fine. Pinning cores is a last touch to ensure performance, it won't make a huge difference; the scheduler is not as dumb as to assign two saturated vcores to a single core if it can help it, and the gains of reduced context-switching won't manifest in gaming.

Could it be possible that even though GPU-Z (in Windows guest) reports PCIE is running at x16 3.0 that this may not be the case?

That would've impacted your score in benchmarks like 3DMark. I get the same score as on Windows.

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

On pinning cores, I wish to pass through 4 physical cores, and 4 virtual cores to achieve my 4 core 8 thread setup from my startup script.  Should I pass through cores 0-3 and 6-9 or cores 0-7?  For clarification, I am unsure if it's setup:

physical-physical-physical-physical-physical-physical-virtual-virtual-virtual-virtual-virtual-virtual
-or-
physical-virtual-physical-virtual-physical-virtual-physical-virtual-physical-virtual-physical-virtual

Also, is there a way to deny access to the physical and virtual cores I select above to my host and limit it only to the remaining cores/threads?

EDIT:
I have tried all of the following combinations with nearly identical levels of success/performance when testing with BF4:

0-3 and 6-9 -cpu host -smp 4,sockets=1,cores=4,threads=2
0-3 and 6-9 -cpu host -smp 8,sockets=1,cores=4,threads=2
0-3 and 6-9 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=2
0-3 and 6-9 -cpu SandyBridge -smp 8,sockets=1,cores=4,threads=2

2-5 and 8-11 -cpu host -smp 4,sockets=1,cores=4,threads=2
2-5 and 8-11 -cpu host -smp 8,sockets=1,cores=4,threads=2
2-5 and 8-11 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=2
2-5 and 8-11 -cpu SandyBridge -smp 8,sockets=1,cores=4,threads=2

0-3 -cpu host -smp 4,sockets=1,cores=4,threads=1
0-3 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=1

2-5 -cpu host -smp 4,sockets=1,cores=4,threads=1
2-5 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=1

0-3 -smp 4,sockets=1,cores=4,threads=1

2-5 -smp 4,sockets=1,cores=4,threads=1

It seems nothing I do to my CPU affects my experience, perhaps my performance issues are coming from somewhere else?  Could it be possible that even though GPU-Z (in Windows guest) reports PCIE is running at x16 3.0 that this may not be the case?  I could see that being a definite bottleneck that may not be noticeable in less demanding games, for instance upon testing Star Wars the Old Republic I had the same performance as if I were on a native Windows install.  Something strange I have noticed however is graphics settings don't have an impact on performance either, I get the same FPS in BF4 on ultra or on low.

you could try playing arround with kvm-intel options

modinfo kvm-intel

i found out disabling npt on my amd system (modprobe kvm-amd npt=0) made a huge difference in performance, as for vcpu pinning im using cset and then using taskset to pin each vcpu to a cpu core, so on my 8 core amd system, arch gets cores 0-1 and windows 2-7

You can find my cset package here: http://www.fileswap.com/dl/yzZMZTTREg/

Last edited by nbhs (2014-02-12 11:09:27)

Kaidax

Member

Registered: 2014-02-09

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

you could try playing arround with kvm-intel options

modinfo kvm-intel

i found out disabling npt on my amd system (modprobe kvm-amd npt=0) made a huge difference in performance, as for vcpu pinning im using cset and then using taskset to pin each vcpu to a cpu core, so on my 8 core amd system, arch gets cores 0-1 and windows 2-7

You can find my cset package here: http://www.fileswap.com/dl/yzZMZTTREg/

Passthrough doesn't work at all with

 modprobe kvm_intel ept=0 

and gpu-less benchmarks seem to be unaffected.

anickname

Member

Registered: 2014-01-07

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does anyone have some scripts for systemd or libvirt to start / poweroff the virtual machines ?
For libvirt, maybe some minimal templates (xml) ?

Last edited by anickname (2014-02-12 17:15:52)

doubledr

Member

Registered: 2013-12-26

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

On pinning cores, I wish to pass through 4 physical cores, and 4 virtual cores to achieve my 4 core 8 thread setup from my startup script.  Should I pass through cores 0-3 and 6-9 or cores 0-7?  For clarification, I am unsure if it's setup:

physical-physical-physical-physical-physical-physical-virtual-virtual-virtual-virtual-virtual-virtual
-or-
physical-virtual-physical-virtual-physical-virtual-physical-virtual-physical-virtual-physical-virtual

Also, is there a way to deny access to the physical and virtual cores I select above to my host and limit it only to the remaining cores/threads?

EDIT:
I have tried all of the following combinations with nearly identical levels of success/performance when testing with BF4:

0-3 and 6-9 -cpu host -smp 4,sockets=1,cores=4,threads=2
0-3 and 6-9 -cpu host -smp 8,sockets=1,cores=4,threads=2
0-3 and 6-9 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=2
0-3 and 6-9 -cpu SandyBridge -smp 8,sockets=1,cores=4,threads=2

2-5 and 8-11 -cpu host -smp 4,sockets=1,cores=4,threads=2
2-5 and 8-11 -cpu host -smp 8,sockets=1,cores=4,threads=2
2-5 and 8-11 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=2
2-5 and 8-11 -cpu SandyBridge -smp 8,sockets=1,cores=4,threads=2

0-3 -cpu host -smp 4,sockets=1,cores=4,threads=1
0-3 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=1

2-5 -cpu host -smp 4,sockets=1,cores=4,threads=1
2-5 -cpu SandyBridge -smp 4,sockets=1,cores=4,threads=1

0-3 -smp 4,sockets=1,cores=4,threads=1

2-5 -smp 4,sockets=1,cores=4,threads=1

It seems nothing I do to my CPU affects my experience, perhaps my performance issues are coming from somewhere else?  Could it be possible that even though GPU-Z (in Windows guest) reports PCIE is running at x16 3.0 that this may not be the case?  I could see that being a definite bottleneck that may not be noticeable in less demanding games, for instance upon testing Star Wars the Old Republic I had the same performance as if I were on a native Windows install.  Something strange I have noticed however is graphics settings don't have an impact on performance either, I get the same FPS in BF4 on ultra or on low.

Interesting, adjusting graphic setting should disable some CPU bounded visual effect code paths. Unless this issue is in the basic visual effects. Have you tried to limit the number of VCPU? It might be a SMP issue.

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just an addendum to anyone trying to run the amd binary drivers in the host:  there is clearly an IOMMU/vt-d conflict with the fglrx driver, at least in the ASUS RIVE.  Enable vt-d in bios: black screen in X with 99% CPU and no terminal response. Disable vt-d with no other change:  normal 3-D acceleration and function.  I've confirmed this with kernels 3.10.7, 3.12.1 and 3.13.0/2 with the last three beta amd drivers.  Since the radeon drivers don't support 3-D very well on the 7900 yet, I'm going to try switching the cards so that the nvidia is host and the amd is guest.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just an addendum to anyone trying to run the amd binary drivers in the host:  there is clearly an IOMMU/vt-d conflict with the fglrx driver, at least in the ASUS RIVE.  Enable vt-d in bios: black screen in X with 99% CPU and no terminal response. Disable vt-d with no other change:  normal 3-D acceleration and function.  I've confirmed this with kernels 3.10.7, 3.12.1 and 3.13.0/2 with the last three beta amd drivers.  Since the radeon drivers don't support 3-D very well on the 7900 yet, I'm going to try switching the cards so that the nvidia is host and the amd is guest.

iommu=pt to put host devices in passthrough mode may also be an option.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

No, IOMMU=pt or in fact no IOMMU on the kernel line makes no difference.  Just having vt-d activated in BIOS is sufficient to bork fglrx.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

No, IOMMU=pt or in fact no IOMMU on the kernel line makes no difference.  Just having vt-d activated in BIOS is sufficient to bork fglrx.

Perhaps interrupt remapping support is broken, try nointremap.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Blind Tree Frog

Member

Registered: 2013-12-31

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I've still got much of this thread to read through, but before I burn too many cycles trying various things to see if I can get this to work the way I want I have two basic questions that I would like to knock out quickly....

1) Do I need multiple video cards or can this be done with just one?  Ideally I would like to run the host as just a headless host and deal with it through ssh, so I would prefer not to have another card in there.   It looks life vfio-pci would require multiple cards, but pci-assign might?

2) Any source I should look at to set this up from a fresh install?  Reapplying the approach in the first post should work, but I'll take any other compiled info as well if someone wrote something up.

(I will update this post if I find the answers while reading the thread, but putting them out there anyhow)

DanaGoyette

Member

Registered: 2014-01-03

Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm occasionally getting a guest video driver hang, accompanied by an oops on the host.
This also invariably results in a freeze on host shutdown.

[165349.140201] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
[165349.144001] BUG: unable to handle kernel paging request at ffffea0005587340
[165349.147069] IP: [<ffffea0005587340>] 0xffffea0005587340
[165349.149381] PGD 47e5f3067 PUD 47e5f2067 PMD 80000004718001e3 
[165349.152172] Oops: 0011 [#1] SMP 
[165349.153542] Modules linked in: usb_storage(F) xt_multiport(F) xt_conntrack(F) ipt_REJECT(F) sb_edac(F) dm_crypt(F) ip6table_filter(F) ip6_tables(F) xt_CHECKSUM(F) iptable_mangle(F) ipt_MASQUERADE(F) iptable_nat(F) nf_conntrack_ipv4(F) nf_defrag_ipv4(F) nf_nat_ipv4(F) nf_nat(F) nf_conntrack(F) xt_tcpudp(F) bridge(F) stp(F) llc(F) iptable_filter(F) ip_tables(F) x_tables(F) autofs4(F) deflate(F) ctr(F) twofish_generic(F) twofish_avx_x86_64(F) twofish_x86_64_3way(F) twofish_x86_64(F) twofish_common(F) camellia_generic(F) camellia_aesni_avx2(F) camellia_aesni_avx_x86_64(F) camellia_x86_64(F) serpent_avx2(F) serpent_avx_x86_64(F) serpent_sse2_x86_64(F) xts(F) serpent_generic(F) blowfish_generic(F) blowfish_x86_64(F) blowfish_common(F) cast5_avx_x86_64(F) cast5_generic(F) cast_common(F) des_generic(F) cmac(F) xcbc(F) rmd160(F) crypto_null(F) af_key(F) xfrm_algo(F) parport_pc(F) ppdev(F) bnep(F) rfcomm(F) bluetooth(F) joydev(F) iTCO_wdt(F) iTCO_vendor_support(F) dm_multipath(F) scsi_dh(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) kvm_intel(F) snd_usb_audio(F) snd_hda_codec_hdmi(F) kvm(F) snd_usbmidi_lib(F) crct10dif_pclmul(F) hid_logitech_dj(F) crc32_pclmul(F) ghash_clmulni_intel(F) aesni_intel(F) snd_hda_codec_realtek(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) snd_hda_intel(F) i915(F) snd_hda_codec(F) snd_seq_midi(F) shpchp(F) snd_seq_midi_event(F) snd_hwdep(F) snd_rawmidi(F) snd_pcm(F) snd_seq(F) drm_kms_helper(F) snd_page_alloc(F) snd_seq_device(F) snd_timer(F) drm(F) lpc_ich(F) snd(F) i2c_algo_bit(F) soundcore(F) video(F) mac_hid(F) vfio_pci(F) vfio_iommu_type1(F) vfio(F) w83627ehf(F) hwmon_vid(F) coretemp(F) msr(F) cpuid(F) jc42(F) nfsd(F) binfmt_misc(F) auth_rpcgss(F) nfs_acl(F) nfs(F) lp(F) parport(F) lockd(F) sunrpc(F) fscache(F) nls_iso8859_1(F) zfs(POF) zunicode(POF) zavl(POF) zcommon(POF) znvpair(POF) spl(OF) btrfs(F) xor(F) hid_generic(F) raid6_pq(F) libcrc32c(F) usbhid(F) hid(F) firewire_ohci(F) firewire_core(F) crc_itu_t(F) ahci(F) libahci(F) e1000e(F) ptp(F) pps_core(F) microcode(F)
[165349.234609] CPU: 4 PID: 2092 Comm: libvirtd Tainted: PF       W  O 3.13.0-acs #1
[165349.237988] Hardware name: Supermicro X10SAT/X10SAT, BIOS 1.0 08/27/2013
[165349.240918] task: ffff880455471800 ti: ffff880452446000 task.ti: ffff880452446000
[165349.245214] RIP: 0010:[<ffffea0005587340>]  [<ffffea0005587340>] 0xffffea0005587340
[165349.248924] RSP: 0018:ffff880452447d98  EFLAGS: 00010246
[165349.251166] RAX: 0000000000000088 RBX: ffff880467e63098 RCX: 0000000000000000
[165349.253973] RDX: ffff880067ad5748 RSI: 0000000000560054 RDI: ffff880467e63098
[165349.257448] RBP: ffff880452447dc0 R08: 0000000000000246 R09: 0000000000000000
[165349.261025] R10: ffffffff8122f679 R11: ffffea0011496c00 R12: ffff880467e63140
[165349.263848] R13: ffffea0005587340 R14: 0000000000000000 R15: ffffffffffffffed
[165349.267308] FS:  00007f8dcf8de840(0000) GS:ffff88047eb00000(0000) knlGS:0000000000000000
[165349.271130] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[165349.273672] CR2: ffffea0005587340 CR3: 0000000452763000 CR4: 00000000001407e0
[165349.277059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[165349.279770] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[165349.283501] Stack:
[165349.284079]  ffffffff814aabf6 ffffffff817342cc 0000000000000004 ffff880467e63098
[165349.287573]  0000000000000004 ffff880452447df0 ffffffff814ab68d 0000000000000004
[165349.291175]  ffff880467e63098 ffff880467e63140 0000000000000246 ffff880452447e20
[165349.294208] Call Trace:
[165349.295446]  [<ffffffff814aabf6>] ? __rpm_callback+0x36/0xc0
[165349.298879]  [<ffffffff817342cc>] ? notifier_call_chain+0x4c/0x70
[165349.304086]  [<ffffffff814ab68d>] rpm_idle+0x1bd/0x2b0
[165349.307279]  [<ffffffff814ab7dd>] __pm_runtime_idle+0x5d/0x80
[165349.310657]  [<ffffffff8149f085>] __device_release_driver+0x65/0xf0
[165349.314992]  [<ffffffff8149f133>] device_release_driver+0x23/0x30
[165349.318650]  [<ffffffff8149dead>] unbind_store+0xbd/0xe0
[165349.321664]  [<ffffffff8149d2d4>] drv_attr_store+0x24/0x40
[165349.324955]  [<ffffffff8122e878>] sysfs_write_file+0x128/0x1c0
[165349.329196]  [<ffffffff811b7934>] vfs_write+0xb4/0x1f0
[165349.332390]  [<ffffffff811b8369>] SyS_write+0x49/0xa0
[165349.335567]  [<ffffffff817389bf>] tracesys+0xe1/0xe6
[165349.338873] Code: 00 00 00 ff ff ff ff 01 00 00 00 00 01 10 00 00 00 ad de 00 02 20 00 00 00 ad de 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 ff ff 02 00 00 00 00 00 00 00 00 00 00 00 00 00 
[165349.349115] RIP  [<ffffea0005587340>] 0xffffea0005587340
[165349.352128]  RSP <ffff880452447d98>
[165349.354904] CR2: ffffea0005587340
[165349.363300] ---[ end trace d2abebf797ae841e ]---