KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9 (Page 87) / Kernel & Hardware / Arch Linux Forums

You are not logged in.

Topics: Active | Unanswered

Pages: Previous 1 … 85 86 87 88 89 … 234 Next

Topic closed

#2151 2014-06-18 23:03:05

andy123: Member; Registered: 2011-11-04; Posts: 169; Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jajacomp wrote:

I have problem with screen size on monitor. If i set resolution 1080 - then image don't fill all screen, just 80%. How fix if? Thank!

This isn't really specific to this thread, but anyways… It's called overscan and there is an option in the AMD Catalyst Control Center to disable it.

i'm sorry for my poor english wirting skills…

Offline

#2152 2014-06-18 23:47:28

mv0lnicky: Member; Registered: 2014-03-05; Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jajacomp wrote:

I have problem with screen size on monitor. If i set resolution 1080 - then image don't fill all screen, just 80%. How fix if? Thank!

The answer is less than a page away...

https://bbs.archlinux.org/viewtopic.php … 4#p1426184

Offline

#2153 2014-06-19 14:42:57

rabcor: Banned; Registered: 2013-02-09; Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey I ran into some problems with my ACS Override patch (on the official gentoo/arch kernels) turns out the patch breaks vdpau (hardware decoding) for me and all applications that use or rely on it.

I tried to install the linux-mainline kernel from the OP, but now I cannot start X, here is my xorg.log. What did I do wrong?

Making Logitech G510 Work on Linux!

Offline

#2154 2014-06-19 15:01:51

aw: Member; Registered: 2013-10-04; Posts: 921; Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

Hey I ran into some problems with my ACS Override patch (on the official gentoo/arch kernels) turns out the patch breaks vdpau (hardware decoding) for me and all applications that use or rely on it.

This seems very unlikely to me, but if it were to break something it would probably be an indicator that the hardware in fact does not have isolation between downstream ports and the kernel is correct in grouping them together. Is the issue with a VM running? vdpau in the VM?

I tried to install the linux-mainline kernel from the OP, but now I cannot start X, here is my xorg.log. What did I do wrong?

It just seems to be telling us to go look at dmesg

[   114.334] (EE) NVIDIA: Failed to initialize the NVIDIA kernel module. Please see the
[   114.334] (EE) NVIDIA:     system's kernel log for additional error messages and
[   114.334] (EE) NVIDIA:     consult the NVIDIA README for details.

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2155 2014-06-19 15:03:39

rabcor: Banned; Registered: 2013-02-09; Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah I just found out through a bit of coincidence it was not the ACS Override patch that broke my hwdec (I updated the post)

It is the intel_iommu=on(CONFIG_INTEL_IOMMU_DEFAULT_ON=y) setting that causes my issue.

The problem is with VDPAU on the host, not the VM (vm not even running) it also doesn't matter if devices are bound to vfio-pci.

And the following commands find nothing. (They do on the official kernel though)

$dmesg | grep nvidia
$dmesg | grep NVIDIA

The only thing I've tried to solve the problem with the linux-mainline kernel is to re-compile the nvidia drivers.It had no effect.As for the issue with Intel DMA Remapping by default, maybe that should be reported to somewhere?

Last edited by rabcor (2014-06-19 15:38:43)

Making Logitech G510 Work on Linux!

Offline

#2156 2014-06-19 15:22:09

shawly: Member; Registered: 2014-05-24; Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Aight my mainboard died today <3
I won't ever buy MSI products again.. tzz..

But I came up with another question, has anyone got crossfire working with passthrough?
Thinking about buying another HD 6950 and run both with crossfire, would this be possible?

Offline

#2157 2014-06-19 18:27:44

Kaurin: Member; From: Ireland; Registered: 2011-12-11; Posts: 66

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

I've come on board with this and really want to make it work.

I was hitting This bug, and with help of aw, was able to fix it (Still waiting confirmation from aw).

Here is my setup:

Mobo: Gigabyte Q87M-D2H, BIOS F7 01/17/2014 (latest bios)
CPU: Intel Core i7 4770
Ram: 32GB @ 1600Mhz
GPU: Gigabyte Nvidia GTX 770, 4GB slightly overclocked by Gigabyte
OS: Arch

My primary display is set to the Intel integrated graphics, and I'm running a bare Arch + qemu, so no graphical environment.

If I run Linux with a graphical environment, It's going to be a virt, with spice probably, and then use that via Windows, which is also a virt :>

Edit:
Grub:

linux /vmlinuz-3.15.0-rc6-ARCH-dirty root=/dev/mapper/sys-root ro intel_iommu=on vfio_iommu_type1

Before even trying GPU passthrough, this was the command line I used to install/test Windows7:

qemu-system-x86_64 \
-name win7 \
-enable-kvm \
-M q35 \
-net bridge,br=br0 \
-net nic,macaddr=52:54:04:fa:8d:ee,model=virtio \
-nographic \
-spice addr=127.0.0.1,port=5900,disable-ticketing \
-vga qxl \
-cpu host \
-smp cores=2,threads=2 \
-m 8G \
-soundhw hda \
-balloon none \
-rtc base=localtime,clock=vm \
-drive file=/dev/sdb,index=0,if=virtio,discard=on

Win7 works like this, using spice (via ssh tunnel), and everything is dandy.

I've tried testing the GPU passtrhough (as per OP), like so:

Bind the GPU to vfio-pci:

/usr/bin/vfio-bind 0000:01:00.0 0000:01:00.1

This is from journalctl -b0 -ocat

/usr/bin/vfio-bind 0000:01:00.0 0000:01:00.1
VFIO - User Level meta-driver version: 0.3
vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900

Try to boot:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=3,threads=2 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

Running this yields no errors, and I could hear the GPU fans spin down, and back up. However, the display does not turn on (no-signal state).

Then, I tried this:

qemu-system-x86_64 \
-name win7 \
-enable-kvm \
-M q35 \
-net bridge,br=br0 \
-net nic,macaddr=52:54:04:fa:8d:ee,model=virtio \
-nographic \
-spice addr=127.0.0.1,port=5900,disable-ticketing \
-vga qxl \
-cpu host \
-smp cores=2,threads=2 \
-m 8G \
-soundhw hda \
-balloon none \
-rtc base=localtime,clock=vm \
-drive file=/dev/sdb,index=0,if=virtio,discard=on \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

Basically the same command for win7 + GPU test. I logged in via spice, and noticed that I have a "Standard VGA Graphics Adapter" in Device Manager, but the device cannot start (Code 10). Hardware IDs look good: 10DE-1184

The "High Definition Audio Device" with the nvidia HWID seems to work (Can't really test it because I don't have HDMI cables / devices)

Can anyone suggest what to try next in order to get GPU-passtrhough to work?

Before anyone asks, the monitor is connected to the DVI port of the nvidia card since before I began testing GPU passthrough

Update: After a Windows Update/Reboot, the Nvidia GPU is shown as "NVIDIA GeForce GTX 770", and is now code 12 and still not working. (Cannot start b/c resources).

I have not mentioned this before, but I've tried doing the GPU passthrough on XEN last week, and hit the same thing there (device name OK + code 12). After seeing that gpu passthrough with nvidia is something I might not be able to do, i turned to KVM where I see people running virts + nvidia successfully.

Last edited by Kaurin (2014-06-19 19:44:15)

Offline

#2158 2014-06-19 19:43:20

MCMjolnir: Member; Registered: 2014-06-19; Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all.

I'm new here, but I've been reading the thread for a couple of months since I got the urge to build a server for streaming Steam from a workstation that recently came into my possession. Finally got all the parts together, and didn't have much luck with Arch since I'm not really familiar with it, so I'm now running Ubuntu Server 14.04.

Here's my config:
CPUs: 2x Xeon E5-2680
Memory: 64 GB ECC RAM @ 1600MHz
Mobo: Asus Z9PA-D8
GPU (Guest 1): MSI GTX770 4GB
GPU (Guest 2): Asus GTX560Ti 2GB

It's been going fairly well so far. Besides figuring out how to make an upstart equivalent of the systemd service, I haven't had any major issues setting up, to this point. I've been using virtual machine manager from my laptop, and configured a guest image I'm happy with. However, when I try to start the VM with the graphics card added, I get this error:

Error starting domain: internal error: Process exited prior to exec: libvirt:  error : cannot limit locked memory to 9663676416: Operation not permitted


Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 96, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 117, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1162, in startup
    self._backend.create()
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 866, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: internal error: Process exited prior to exec: libvirt:  error : cannot limit locked memory to 9663676416: Operation not permitted

Any ideas on what I'm doing wrong?

Offline

#2159 2014-06-19 19:44:32

sinny: Member; Registered: 2014-05-24; Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Kaurin wrote:

Hi all,
I've come on board with this and really want to make it work.
I was hitting This bug, and with help of aw, was able to fix it (Still waiting confirmation from aw).
Here is my setup:
Mobo: Gigabyte Q87M-D2H, BIOS F7 01/17/2014 (latest bios)
CPU: Intel Core i7 4770
Ram: 32GB @ 1600Mhz
GPU: Gigabyte Nvidia GTX 770, 4GB slightly overclocked by Gigabyte
OS: Arch
My primary display is set to the Intel integrated graphics, and I'm running a bare Arch + qemu, so no graphical environment.
If I run Linux with a graphical environment, It's going to be a virt, with spice probably, and then use that via Windows, which is also a virt :>
Edit:
Grub:
linux /vmlinuz-3.15.0-rc6-ARCH-dirty root=/dev/mapper/sys-root ro intel_iommu=on vfio_iommu_type1
Before even trying GPU passthrough, this was the command line I used to install/test Windows7:
qemu-system-x86_64 \
-name win7 \
-enable-kvm \
-M q35 \
-net bridge,br=br0 \
-net nic,macaddr=52:54:04:fa:8d:ee,model=virtio \
-nographic \
-spice addr=127.0.0.1,port=5900,disable-ticketing \
-vga qxl \
-cpu host \
-smp cores=2,threads=2 \
-m 8G \
-soundhw hda \
-balloon none \
-rtc base=localtime,clock=vm \
-drive file=/dev/sdb,index=0,if=virtio,discard=on
Win7 works like this, using spice (via ssh tunnel), and everything is dandy.
I've tried testing the GPU passtrhough (as per OP), like so:
Bind the GPU to vfio-pci:
/usr/bin/vfio-bind 0000:01:00.0 0000:01:00.1
This is from journalctl -b0 -ocat
/usr/bin/vfio-bind 0000:01:00.0 0000:01:00.1
VFIO - User Level meta-driver version: 0.3
vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
Try to boot:
qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=3,threads=2 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1
Running this yields no errors, and I could hear the GPU fans spin down, and back up. However, the display does not turn on (no-signal state).
Then, I tried this:
qemu-system-x86_64 \
-name win7 \
-enable-kvm \
-M q35 \
-net bridge,br=br0 \
-net nic,macaddr=52:54:04:fa:8d:ee,model=virtio \
-nographic \
-spice addr=127.0.0.1,port=5900,disable-ticketing \
-vga qxl \
-cpu host \
-smp cores=2,threads=2 \
-m 8G \
-soundhw hda \
-balloon none \
-rtc base=localtime,clock=vm \
-drive file=/dev/sdb,index=0,if=virtio,discard=on \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 
Basically the same command for win7 + GPU test. I logged in via spice, and noticed that I have a "Standard VGA Graphics Adapter" in Device Manager, but the device cannot start (Code 10). Hardware IDs look good: 10DE-1184
The "High Definition Audio Device" with the nvidia HWID seems to work (Can't really test it because I don't have HDMI cables / devices)
Can anyone suggest what to try next in order to get GPU-passtrhough to work?
Before anyone asks, the monitor is connected to the DVI port of the nvidia card since before I began testing GPU passthrough :)

do you expect anyone to guess all the tech details needed to even try helping you? this is simply rude, seriously...

anyway, my guess is missing vga arbiter patch in kernel

Offline

#2160 2014-06-19 19:54:50

Kaurin: Member; From: Ireland; Registered: 2011-12-11; Posts: 66

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I'm very sorry if it came off that way. It was definitely not my intention.
I will try to browse the thread again to see what I was missing.

I will look into the vga arbiter patch. Thank you!

Offline

#2161 2014-06-19 21:07:33

rakataka: Member; Registered: 2014-06-19; Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Kaurin wrote:

Hi all,
I've come on board with this and really want to make it work.
I was hitting This bug, and with help of aw, was able to fix it (Still waiting confirmation from aw).
Here is my setup:
Mobo: Gigabyte Q87M-D2H, BIOS F7 01/17/2014 (latest bios)
CPU: Intel Core i7 4770
Ram: 32GB @ 1600Mhz
GPU: Gigabyte Nvidia GTX 770, 4GB slightly overclocked by Gigabyte
OS: Arch
My primary display is set to the Intel integrated graphics, and I'm running a bare Arch + qemu, so no graphical environment.
If I run Linux with a graphical environment, It's going to be a virt, with spice probably, and then use that via Windows, which is also a virt :>
Edit:
Grub:
linux /vmlinuz-3.15.0-rc6-ARCH-dirty root=/dev/mapper/sys-root ro intel_iommu=on vfio_iommu_type1
Before even trying GPU passthrough, this was the command line I used to install/test Windows7:
qemu-system-x86_64 \
-name win7 \
-enable-kvm \
-M q35 \
-net bridge,br=br0 \
-net nic,macaddr=52:54:04:fa:8d:ee,model=virtio \
-nographic \
-spice addr=127.0.0.1,port=5900,disable-ticketing \
-vga qxl \
-cpu host \
-smp cores=2,threads=2 \
-m 8G \
-soundhw hda \
-balloon none \
-rtc base=localtime,clock=vm \
-drive file=/dev/sdb,index=0,if=virtio,discard=on
Win7 works like this, using spice (via ssh tunnel), and everything is dandy.
I've tried testing the GPU passtrhough (as per OP), like so:
Bind the GPU to vfio-pci:
/usr/bin/vfio-bind 0000:01:00.0 0000:01:00.1
This is from journalctl -b0 -ocat
/usr/bin/vfio-bind 0000:01:00.0 0000:01:00.1
VFIO - User Level meta-driver version: 0.3
vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
Try to boot:
qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=3,threads=2 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1
Running this yields no errors, and I could hear the GPU fans spin down, and back up. However, the display does not turn on (no-signal state).
Then, I tried this:
qemu-system-x86_64 \
-name win7 \
-enable-kvm \
-M q35 \
-net bridge,br=br0 \
-net nic,macaddr=52:54:04:fa:8d:ee,model=virtio \
-nographic \
-spice addr=127.0.0.1,port=5900,disable-ticketing \
-vga qxl \
-cpu host \
-smp cores=2,threads=2 \
-m 8G \
-soundhw hda \
-balloon none \
-rtc base=localtime,clock=vm \
-drive file=/dev/sdb,index=0,if=virtio,discard=on \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 
Basically the same command for win7 + GPU test. I logged in via spice, and noticed that I have a "Standard VGA Graphics Adapter" in Device Manager, but the device cannot start (Code 10). Hardware IDs look good: 10DE-1184
The "High Definition Audio Device" with the nvidia HWID seems to work (Can't really test it because I don't have HDMI cables / devices)
Can anyone suggest what to try next in order to get GPU-passtrhough to work?
Before anyone asks, the monitor is connected to the DVI port of the nvidia card since before I began testing GPU passthrough
Update: After a Windows Update/Reboot, the Nvidia GPU is shown as "NVIDIA GeForce GTX 770", and is now code 12 and still not working. (Cannot start b/c resources).
I have not mentioned this before, but I've tried doing the GPU passthrough on XEN last week, and hit the same thing there (device name OK + code 12). After seeing that gpu passthrough with nvidia is something I might not be able to do, i turned to KVM where I see people running virts + nvidia successfully.

As it happens, we have an almost identical PC (I have the same motherboard, CPU, GFX card MSI GTX770 but with 2GB VRAM instead of 4, 32 GB RAM).
I may have been using kvm passthrough for a little bit longer than you. I updated today to linux-mainline 3.15.0 from 3.14.xx (can't remember exact xx). VGA passthrough worked fine on 3.14 with the following kernel line:

linux /boot/vmlinuz-linux-mainline root=/dev/sda1 rw modprobe.blacklist=nouveau,wmi,mxm_wmi,ttm nouveau.modeset=0 intel_iommu=on intel_iommu=igfx_off pcie_acs_override=downstream kvm.ignore_msrs=1 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0 nohz=off

I should note that I have not used the patch you refer to in "This bug", so I get the related error in my boot log.
I am now (with the new kernel version 3.15.0) also getting a blank screen (I have a dedicated monitor connected to the HDMI output of my GTX770 card and I do not have the binary NVIDIA drivers install on the linux host) whereas it worked fine before with Windows 8.1 Pro 64bit and a number of games.
I have two things to note:

I have noticed a new error when running quemu with passthough which I was not getting with the 3.14.xx kernel:

VFIO - User Level meta-driver version: 0.3
Jun 19 22:59:05 laptop kernel: vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
Jun 19 22:59:05 laptop kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
Jun 19 22:59:09 laptop kernel: [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
Jun 19 23:00:00 laptop kernel: [drm:hsw_unclaimed_reg_clear] *ERROR* Unknown unclaimed register before writing to 4510c

I have read in previous thread posts that this is indicative of not having the i935 arbiter patch compiled but I used the package in the 1st post of this thread and just did

makepkg; pacman -U *.pkg.tar.xz

with no changes to the config.

If you take a look at my kernel line (which I have carried over from 3.14.xx) I have used a
```
intel_iommu=on intel_iommu=igfx_off
```
instead of plain
```
intel_iommu=on
```
The extra
```
intel_iommu=igfx_off
```
made the difference between being able to do VGA passthough on 3.14.xx and failing with an error I cannot remember.

I wonder whether it is possible that something has been changed with respect to intel_iommu in 3.15.
If I find anything new I will update.

Offline

#2162 2014-06-19 21:34:32

aw: Member; Registered: 2013-10-04; Posts: 921; Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Seriously people, if you have Intel graphics in your system, you need the i915 kernel patch. If you have radeon host graphics, you need the other vga arbiter patch. If you can't assign the devices you want because of iommu grouping, you need the ACS override patch. If nothing happens on the display connected to the assigned GPU when you start the VM, you're probably missing the i915 or vga arbiter patches. If you get screen corruption on the host, you're definitely missing them. If you get weird intel error messages like above, you're definitely missing the i915 patch.

EDIT: If you think you have the i915 or ACS override patch, but it's not working, then you're not using the right kernel commandline options to enable them.

Last edited by aw (2014-06-19 21:36:04)

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2163 2014-06-19 21:49:12

rakataka: Member; Registered: 2014-06-19; Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Seriously people, if you have Intel graphics in your system, you need the i915 kernel patch. If you have radeon host graphics, you need the other vga arbiter patch. If you can't assign the devices you want because of iommu grouping, you need the ACS override patch. If nothing happens on the display connected to the assigned GPU when you start the VM, you're probably missing the i915 or vga arbiter patches. If you get screen corruption on the host, you're definitely missing them. If you get weird intel error messages like above, you're definitely missing the i915 patch.
EDIT: If you think you have the i915 or ACS override patch, but it's not working, then you're not using the right kernel commandline options to enable them.

You are right, in my case (getting a blank screen but having used the correct kernel: 3.15.0 mainline from 1st post), I just need to add

i915.enable_hd_vgaarb=1

to my kernel command line an it worked fine.

As you can attest, this is neither horribly complex, but it's not simple either with multiple special cases. Also, I normally read change logs (I'm a developer myself) but I did not see one in a reasonably prominent place where I would be informed that I need to add an extra kernel parameter after upgrading to 3.15.

Please do not think that I take all your free work for granted, I am certainly very thankful for it and I try to do the extra work myself in order not to burden those doing the actual work. However, sometimes, one may miss some crucial information due to the lack of dedicated documentation, and searching on a forum is not always helpful (again, not saying that *you* should provide docs, just saying that mistakes may happen due to a lack of them).

Offline

#2164 2014-06-19 21:58:08

aw: Member; Registered: 2013-10-04; Posts: 921; Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rakataka wrote:

aw wrote:
Seriously people, if you have Intel graphics in your system, you need the i915 kernel patch. If you have radeon host graphics, you need the other vga arbiter patch. If you can't assign the devices you want because of iommu grouping, you need the ACS override patch. If nothing happens on the display connected to the assigned GPU when you start the VM, you're probably missing the i915 or vga arbiter patches. If you get screen corruption on the host, you're definitely missing them. If you get weird intel error messages like above, you're definitely missing the i915 patch.
EDIT: If you think you have the i915 or ACS override patch, but it's not working, then you're not using the right kernel commandline options to enable them.
You are right, in my case (getting a blank screen but having used the correct kernel: 3.15.0 mainline from 1st post), I just need to add
i915.enable_hd_vgaarb=1
to my kernel command line an it worked fine.
As you can attest, this is neither horribly complex, but it's not simple either with multiple special cases. Also, I normally read change logs (I'm a developer myself) but I did not see one in a reasonably prominent place where I would be informed that I need to add an extra kernel parameter after upgrading to 3.15.
Please do not think that I take all your free work for granted, I am certainly very thankful for it and I try to do the extra work myself in order not to burden those doing the actual work. However, sometimes, one may miss some crucial information due to the lack of dedicated documentation, and searching on a forum is not always helpful (again, not saying that *you* should provide docs, just saying that mistakes may happen due to a lack of them).

Understood, for some reason the patches in the first post always get the changelog stripped out, the upstream patches do include a proper description:

ACS override
i915 VGA arbiter
VGA arbiter

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#2165 2014-06-19 22:08:44

rakataka: Member; Registered: 2014-06-19; Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeap, in my case (i915 VGA arbiter) I hope that if I had seen:

+module_param_named(enable_hd_vgaarb, i915.enable_hd_vgaarb, bool, 0444);
+MODULE_PARM_DESC(enable_hd_vgaarb,
+	"Enable support for VGA arbitration on Intel HD IGD. (default: false)");

I would have taken the hint that a new parameter was available and I would have checked what's needed.

Thanks for the relevant links.

Offline

#2166 2014-06-20 03:38:38

ahl: Member; Registered: 2014-03-16; Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I have been hit with a possible regression bug. A while ago I reported this:

System 2: AMD FX 8350 + Asus Sabretooth 990FX R2.0 (BIOS 2104)
- OK
- passthrough of the integrated audio device also tested to work with Win 7

But now, passing through the integrated audio device no longer works properly. When playing audio in guest I only get some horrible noise and, at the same time, lots of these messages appear on host:

AMD-Vi: Event logged [IO_PAGE_FAULT device=00:14.2 domain=0x0025 address=0xffffffffffffffc0 flags=0x0010]

These are some of the most obvious changes since working setup:
ASUS BIOS 2104 -> 2501
Changed the host gpu card from HD3870 -> HD5450 (Can't test 3870 right now)
Guest card is still HD6670.
Linux 3.14.0-rc8(approx.) -> 3.14.8-gentoo
Qemu 1.7.0 -> 2.0.0-r1 (Already tried downgrading, It didn't help)

lsgroup output has also changed. I could have sworn first 4 of these were in the same group and I had to bind them all. Now they are separate (Edit: Perhaps it was some other system after all):

### Group 11 ###
    00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42)
### Group 12 ###
    00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40)
### Group 13 ###
    00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
### Group 14 ###
    00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40)
### Group 15 ###
    00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller

Any idea what could introduce such a problem?

UPDATE

After some very thorough testing, I found an explanation but no solution. It may be a BIOS problem. Changing the kernel did nothing.

BIOS SABERTOOTH-990FX-R20-ASUS-2501.CAP (8a19662ee80ff8b75122260baf3de1b760eeb1e6)
BIOS SABERTOOTH-990FX-R20-ASUS-2301.CAP (a593fe9cef0a5b9c5e6eff3ea3f7c5a49bd7743e)
- Problem as described above

BIOS SABERTOOTH-990FX-R20-ASUS-2104.CAP (66bd1583c73ae28ca003b5fd58edc3b998eaf215)
- When playing the sound samples in Win 7, sometimes they play correctly, sometimes they sound very distorted and sometimes they don't play at all
- No error messages appear on host

For my sanity, I'm just going to assume that the latest bioses are actually working correctly. If it does not work properly it better throw some errors rather than being half-broken. Perhaps it never fully worked -- I can't be absolutely sure anymore.

Last edited by ahl (2014-06-20 16:44:58)

Offline

#2167 2014-06-20 08:23:49

Bronek: Member; From: London; Registered: 2014-02-14; Posts: 119

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Understood, for some reason the patches in the first post always get the changelog stripped out, the upstream patches do include a proper description:
ACS override
i915 VGA arbiter
VGA arbiter

Thanks for the links. I understand why ACS override was rejected and there is a discussion under i915 arbiter patch (leading nowhere, because the only opponent cannot be bothered to write alternative solution, as he openly admits), but what happened to the last one? There is no discussion following your post, but it is not present in current http://git.kernel.org/cgit/linux/kernel … a/vgaarb.c either.

Does that mean that VGA pass-through users cannot rely on upstream kernel if they want to use (otherwise well-functioning, but poorly supported by kernel) hardware such as Intel IGP or AMD Radeons ? Given that nVidia seem to be going out of their way to prevent use of consumer cards in such configuration ("Error 43"), this is very discouraging ...

Last edited by Bronek (2014-06-20 11:49:01)

Offline

#2168 2014-06-20 17:22:38

slis: Member; Registered: 2014-06-02; Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

can someone pls help ...

I am hitting strange bug with HDMI sound...

It starts to lag on higher cpu usage and stays distorted only with 660 and 770 cards... 560 works fine...
did vm reinstall, driver reinstall, slots change, acs parameters... i got no clue what is wrong...

i7 2600
gigabyte g1 sniper3
debian

thx

dmesg:

 
0.551184] ------------[ cut here ]------------
[    0.551190] WARNING: CPU: 0 PID: 1 at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x6b/0x80()
[    0.551191] Modules linked in:
[    0.551193] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.15.0-rc7-awilliam+ #1
[    0.551194] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./G1.Sniper 3, BIOS F7 08/23/2012
[    0.551196]  0000000000000009 ffffffff814c0182 0000000000000000 ffffffff8105ccc2
[    0.551198]  ffff88044a0dc098 ffff88044a0dc000 ffff88044a0dc098 0000000000000000
[    0.551200]  0000000000000000 ffffffff812ac7eb ffffffff813b3496 0009880400000000
[    0.551202] Call Trace:
[    0.551207]  [<ffffffff814c0182>] ? dump_stack+0x41/0x51
[    0.551210]  [<ffffffff8105ccc2>] ? warn_slowpath_common+0x72/0x90
[    0.551212]  [<ffffffff812ac7eb>] ? pci_find_upstream_pcie_bridge+0x6b/0x80
[    0.551215]  [<ffffffff813b3496>] ? intel_iommu_add_device+0x36/0x210
[    0.551218]  [<ffffffff813a8a50>] ? bus_set_iommu+0x50/0x50
[    0.551220]  [<ffffffff813a8a70>] ? add_iommu_group+0x20/0x50
[    0.551222]  [<ffffffff8135e683>] ? bus_for_each_dev+0x53/0x90
[    0.551224]  [<ffffffff813a8a3e>] ? bus_set_iommu+0x3e/0x50
[    0.551227]  [<ffffffff81913bf9>] ? intel_iommu_init+0x53f/0x586
[    0.551231]  [<ffffffff818d743e>] ? memblock_find_dma_reserve+0x131/0x131
[    0.551233]  [<ffffffff818d744c>] ? pci_iommu_init+0xe/0x37
[    0.551235]  [<ffffffff81002152>] ? do_one_initcall+0xf2/0x140
[    0.551238]  [<ffffffff8107aa1a>] ? parse_args+0x1fa/0x330
[    0.551240]  [<ffffffff818cf060>] ? kernel_init_freeable+0x15f/0x1e1
[    0.551242]  [<ffffffff818ce8ac>] ? do_early_param+0x81/0x81
[    0.551245]  [<ffffffff814b24c0>] ? rest_init+0x70/0x70
[    0.551247]  [<ffffffff814b24c5>] ? kernel_init+0x5/0xf0
[    0.551249]  [<ffffffff814cd78c>] ? ret_from_fork+0x7c/0xb0
[    0.551251]  [<ffffffff814b24c0>] ? rest_init+0x70/0x70
[    0.551254] ---[ end trace bf54b21f328db5e9 ]---

[ 2404.025436] kvm [3767]: vcpu0 unhandled rdmsr: 0x1a2
[ 2404.025505] kvm [3767]: vcpu1 unhandled rdmsr: 0x1a2
[ 2404.025566] kvm [3767]: vcpu2 unhandled rdmsr: 0x1a2
[ 2404.025623] kvm [3767]: vcpu3 unhandled rdmsr: 0x1a2

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-3.15.0-rc7-awilliam+ root=UUID=dee8e431-f41a-4ba9-89d0-dc053e70ab5e ro elevator=deadline quiet intel_iommu=on pcie_acs_override=id:8086:0101 transparent_hugepage=always

lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation 2nd Generation Core Processor Family DRAM Controller [8086:0100] (rev 09)
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port [8086:0101] (rev 09)
00:02.0 VGA compatible controller [0300]: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller [8086:0102] (rev 09)
00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)
00:16.0 Communication controller [0780]: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 [8086:1e3a] (rev 04)
00:19.0 Ethernet controller [0200]: Intel Corporation 82579V Gigabit Network Connection [8086:1503] (rev 04)
00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)
00:1c.0 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 [8086:1e10] (rev c4)
00:1c.1 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 2 [8086:1e12] (rev c4)
00:1c.5 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev c4)
00:1c.6 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 7 [8086:1e1c] (rev c4)
00:1c.7 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 [8086:1e1e] (rev c4)
00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)
00:1f.0 ISA bridge [0601]: Intel Corporation Z77 Express Chipset LPC Controller [8086:1e44] (rev 04)
00:1f.2 SATA controller [0106]: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] [8086:1e02] (rev 04)
00:1f.3 SMBus [0c05]: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller [8086:1e22] (rev 04)
01:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ba)
02:08.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ba)
02:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ba)
02:10.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ba)
02:11.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ba)
03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1)
03:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] [10de:1200] (rev a1)
04:00.1 Audio device [0403]: NVIDIA Corporation GF114 HDMI Audio Controller [10de:0e0c] (rev a1)
05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK106 [GeForce GTX 660] [10de:11c0] (rev a1)
05:00.1 Audio device [0403]: NVIDIA Corporation GK106 HDMI Audio Controller [10de:0e0b] (rev a1)
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] [10de:1200] (rev a1)
06:00.1 Audio device [0403]: NVIDIA Corporation GF114 HDMI Audio Controller [10de:0e0c] (rev a1)
08:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11)
09:00.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev 41)
0c:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11)

qemu

  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-2.1'>hvm</type>
    <boot dev='hd'/>
    <bootmenu enable='yes'/>
  </os>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Haswell</model>
    <vendor>Intel</vendor>
    <topology sockets='1' cores='4' threads='1'/>
    <feature policy='force' name='rtm'/>
    <feature policy='force' name='invpcid'/>
    <feature policy='force' name='lahf_lm'/>
    <feature policy='force' name='erms'/>
    <feature policy='force' name='bmi2'/>
    <feature policy='force' name='smep'/>
    <feature policy='force' name='avx2'/>
    <feature policy='force' name='hle'/>
    <feature policy='force' name='bmi1'/>
    <feature policy='force' name='fsgsbase'/>
    <feature policy='force' name='movbe'/>
    <feature policy='force' name='fma'/>
  </cpu>
  <clock offset='utc'>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=03:00.1,bus=root.1,addr=00.1,multifunction=on'/>
    <qemu:env name='DISPLAY' value=':0'/>
  </qemu:commandline>

i tried to remove hvtime and other cpu options but it starts to lag sometimes after 1min sometimes after 10mins...

others cards on DVI without sound are working normal at same time...

i tried almost everything i could find... but ("alien",lagging) hdmi sounds are here allways..

i think its something with that nvidia pci bridge PLX, but i don't know what to do...

Last edited by slis (2014-06-20 18:16:59)

Offline

#2169 2014-06-20 20:47:54

abdullah: Member; Registered: 2014-06-10; Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I keep getting this error

error starting domain: internal error: Process exited prior to exec: libvirt:  error : cannot limit locked memory to 7390363648: Operation not permitted

Through virt-manager.
I don't know how to debug this, but here is my specs :
uname -r :

3.15.1-custom

lspci -k :

...
01:00.0 VGA compatible controller: NVIDIA Corporation G86 [GeForce 8500 GT] (rev a1)
	Subsystem: XFX Pine Group Inc. Device 230b
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV770 [Radeon HD 4870]
	Subsystem: PC Partner Limited / Sapphire Technology Device 0851
	Kernel driver in use: pci-stub
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV770 HDMI Audio [Radeon HD 4850/4870]
	Subsystem: PC Partner Limited / Sapphire Technology Radeon HD 4850 512MB GDDR3 PCI-E Dual Slot Fansink
	Kernel driver in use: pci-stub
...

I'm trying to pass the AMD card "02:00.0".

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-3.15.1-custom root=UUID=850616ae-85d0-4248-a5f6-91cc6aae6b7d ro quiet kvm.ignore_msrs=1 nomodeset intel_iommu=on pci-stub.ids=1002:9440,1002:aa30

dmesg | grep pci-stub

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.15.1-custom root=UUID=850616ae-85d0-4248-a5f6-91cc6aae6b7d ro quiet kvm.ignore_msrs=1 nomodeset intel_iommu=on pci-stub.ids=1002:9440,1002:aa30
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.15.1-custom root=UUID=850616ae-85d0-4248-a5f6-91cc6aae6b7d ro quiet kvm.ignore_msrs=1 nomodeset intel_iommu=on pci-stub.ids=1002:9440,1002:aa30
[    4.172224] pci-stub: add 1002:9440 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    4.172229] pci-stub 0000:02:00.0: claimed by stub
[    4.172230] pci-stub: add 1002:AA30 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    4.172233] pci-stub 0000:02:00.1: claimed by stub
[   25.757900] pci-stub 0000:02:00.0: claimed by stub
[   46.332286] pci-stub 0000:02:00.0: claimed by stub
[   46.332539] pci-stub 0000:02:00.1: claimed by stub
[  792.054112] pci-stub 0000:02:00.0: claimed by stub
[  792.054287] pci-stub 0000:02:00.1: claimed by stub
[  794.473400] pci-stub 0000:02:00.0: claimed by stub
[  794.473583] pci-stub 0000:02:00.1: claimed by stub

I did apply the following patches : acs override patch + i915 vga arbiter fixes, I'm using Ubuntu 14.01 LTS.

One more Question, what it is meant by "Systemd service" ? I can't figure out what the equivalent in Debian based distros.

Last edited by abdullah (2014-06-20 20:50:37)

Offline

#2170 2014-06-20 21:26:40

andy123: Member; Registered: 2011-11-04; Posts: 169; Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

abdullah wrote:

Hello,

I keep getting this error

error starting domain: internal error: Process exited prior to exec: libvirt:  error : cannot limit locked memory to 7390363648: Operation not permitted

Through virt-manager.
I don't know how to debug this, but here is my specs :
uname -r :

3.15.1-custom

lspci -k :

...
01:00.0 VGA compatible controller: NVIDIA Corporation G86 [GeForce 8500 GT] (rev a1)
	Subsystem: XFX Pine Group Inc. Device 230b
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV770 [Radeon HD 4870]
	Subsystem: PC Partner Limited / Sapphire Technology Device 0851
	Kernel driver in use: pci-stub
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV770 HDMI Audio [Radeon HD 4850/4870]
	Subsystem: PC Partner Limited / Sapphire Technology Radeon HD 4850 512MB GDDR3 PCI-E Dual Slot Fansink
	Kernel driver in use: pci-stub
...

I'm trying to pass the AMD card "02:00.0".

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-3.15.1-custom root=UUID=850616ae-85d0-4248-a5f6-91cc6aae6b7d ro quiet kvm.ignore_msrs=1 nomodeset intel_iommu=on pci-stub.ids=1002:9440,1002:aa30

dmesg | grep pci-stub

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.15.1-custom root=UUID=850616ae-85d0-4248-a5f6-91cc6aae6b7d ro quiet kvm.ignore_msrs=1 nomodeset intel_iommu=on pci-stub.ids=1002:9440,1002:aa30
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.15.1-custom root=UUID=850616ae-85d0-4248-a5f6-91cc6aae6b7d ro quiet kvm.ignore_msrs=1 nomodeset intel_iommu=on pci-stub.ids=1002:9440,1002:aa30
[    4.172224] pci-stub: add 1002:9440 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    4.172229] pci-stub 0000:02:00.0: claimed by stub
[    4.172230] pci-stub: add 1002:AA30 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    4.172233] pci-stub 0000:02:00.1: claimed by stub
[   25.757900] pci-stub 0000:02:00.0: claimed by stub
[   46.332286] pci-stub 0000:02:00.0: claimed by stub
[   46.332539] pci-stub 0000:02:00.1: claimed by stub
[  792.054112] pci-stub 0000:02:00.0: claimed by stub
[  792.054287] pci-stub 0000:02:00.1: claimed by stub
[  794.473400] pci-stub 0000:02:00.0: claimed by stub
[  794.473583] pci-stub 0000:02:00.1: claimed by stub

I did apply the following patches : acs override patch + i915 vga arbiter fixes, I'm using Ubuntu 14.01 LTS.

One more Question, what it is meant by "Systemd service" ? I can't figure out what the equivalent in Debian based distros.

I don't use an Intel CPU/GPU on my setup, but since you don't use the kernel parameter, please read this post.

The equivalent to a systemd service is an upstart service or an initscript.

And since you're using libvirt, where is you vm's xml file?

Oh, and what do you use the nvidia card for? And which driver do you use it with?

Are the HD4xxx series even supported with this setup?

i'm sorry for my poor english wirting skills…

Offline

#2171 2014-06-20 22:28:28

abdullah: Member; Registered: 2014-06-10; Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

abdullah wrote:

Hello,

I keep getting this error

error starting domain: internal error: Process exited prior to exec: libvirt:  error : cannot limit locked memory to 7390363648: Operation not permitted

Through virt-manager.
I don't know how to debug this, but here is my specs :
uname -r :

3.15.1-custom

lspci -k :

...
01:00.0 VGA compatible controller: NVIDIA Corporation G86 [GeForce 8500 GT] (rev a1)
	Subsystem: XFX Pine Group Inc. Device 230b
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV770 [Radeon HD 4870]
	Subsystem: PC Partner Limited / Sapphire Technology Device 0851
	Kernel driver in use: pci-stub
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV770 HDMI Audio [Radeon HD 4850/4870]
	Subsystem: PC Partner Limited / Sapphire Technology Radeon HD 4850 512MB GDDR3 PCI-E Dual Slot Fansink
	Kernel driver in use: pci-stub
...

I'm trying to pass the AMD card "02:00.0".

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-3.15.1-custom root=UUID=850616ae-85d0-4248-a5f6-91cc6aae6b7d ro quiet kvm.ignore_msrs=1 nomodeset intel_iommu=on pci-stub.ids=1002:9440,1002:aa30

dmesg | grep pci-stub

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.15.1-custom root=UUID=850616ae-85d0-4248-a5f6-91cc6aae6b7d ro quiet kvm.ignore_msrs=1 nomodeset intel_iommu=on pci-stub.ids=1002:9440,1002:aa30
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.15.1-custom root=UUID=850616ae-85d0-4248-a5f6-91cc6aae6b7d ro quiet kvm.ignore_msrs=1 nomodeset intel_iommu=on pci-stub.ids=1002:9440,1002:aa30
[    4.172224] pci-stub: add 1002:9440 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    4.172229] pci-stub 0000:02:00.0: claimed by stub
[    4.172230] pci-stub: add 1002:AA30 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    4.172233] pci-stub 0000:02:00.1: claimed by stub
[   25.757900] pci-stub 0000:02:00.0: claimed by stub
[   46.332286] pci-stub 0000:02:00.0: claimed by stub
[   46.332539] pci-stub 0000:02:00.1: claimed by stub
[  792.054112] pci-stub 0000:02:00.0: claimed by stub
[  792.054287] pci-stub 0000:02:00.1: claimed by stub
[  794.473400] pci-stub 0000:02:00.0: claimed by stub
[  794.473583] pci-stub 0000:02:00.1: claimed by stub

I did apply the following patches : acs override patch + i915 vga arbiter fixes, I'm using Ubuntu 14.01 LTS.

One more Question, what it is meant by "Systemd service" ? I can't figure out what the equivalent in Debian based distros.

I don't use an Intel CPU/GPU on my setup, but since you don't use the kernel parameter, please read this post.

The equivalent to a systemd service is an upstart service or an initscript.

And since you're using libvirt, where is you vm's xml file?

Oh, and what do you use the nvidia card for? And which driver do you use it with?

Are the HD4xxx series even supported with this setup?

The nvidia card is used as the host card (nouva), until I changed it right before posting this, so, no the parameter got nothing to do with this, I'm trying to pass the secondary card.
is this how you debug the VM's ? where I even can find the VM's xml?

I will pass the nvidia card right after I successfully pass the ATI card, or I may get a new one.
I don't know what kind of support you are referring to, but it is working right ?

Offline

#2172 2014-06-21 01:01:16

MCMjolnir: Member; Registered: 2014-06-19; Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think I'm having the same issue, also using virt-manager. Here's the xml file from one of my VMs.

<domain type='kvm'>
  <name>Head1</name>
  <uuid>2f16058f-e2e3-a4a5-5b74-4a5e32623bf8</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>8</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>SandyBridge</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='smx'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='dca'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='monitor'/>
  </cpu>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/Win8Steam.qcow2'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:2e:35:17'/>
      <source network='default'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <video>
      <model type='vga' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
  </devices>
</domain>

You should find the xml files in /etc/libvirt/qemu/ abdullah.

Last edited by MCMjolnir (2014-06-21 01:30:00)

Offline

#2173 2014-06-21 08:38:03

sinny: Member; Registered: 2014-05-24; Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

abdullah wrote:

Hello,

I keep getting this error

error starting domain: internal error: Process exited prior to exec: libvirt:  error : cannot limit locked memory to 7390363648: Operation not permitted

...

wild guess:

[root@sinny ~]# grep root /etc/libvirt/qemu.conf
user = "root"
group = "root"
[root@sinny ~]#

i think i've seen this somewhere already, but can't remember where

Offline

#2174 2014-06-21 08:40:05

Nesousx: Member; Registered: 2012-03-27; Posts: 46; Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

MCMjolnir wrote:

Hi all.
I'm new here, but I've been reading the thread for a couple of months since I got the urge to build a server for streaming Steam from a workstation that recently came into my possession. Finally got all the parts together, and didn't have much luck with Arch since I'm not really familiar with it, so I'm now running Ubuntu Server 14.04.
Here's my config:
CPUs: 2x Xeon E5-2680
Memory: 64 GB ECC RAM @ 1600MHz
Mobo: Asus Z9PA-D8
GPU (Guest 1): MSI GTX770 4GB
GPU (Guest 2): Asus GTX560Ti 2GB
It's been going fairly well so far. Besides figuring out how to make an upstart equivalent of the systemd service, I haven't had any major issues setting up, to this point. I've been using virtual machine manager from my laptop, and configured a guest image I'm happy with. However, when I try to start the VM with the graphics card added, I get this error:
Error starting domain: internal error: Process exited prior to exec: libvirt:  error : cannot limit locked memory to 9663676416: Operation not permitted


Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 96, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 117, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1162, in startup
    self._backend.create()
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 866, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: internal error: Process exited prior to exec: libvirt:  error : cannot limit locked memory to 9663676416: Operation not permitted
Any ideas on what I'm doing wrong?

This looks like an apparmor error, please read here: http://wiki.apparmor.net/index.php/Libvirt.

Basically, you can disable apparmor security for the Windows guest, by editing the the file /etc/apparmor.d/libvirt/libvirt-<uuid>.files and setting security profile to none or something like this. Probably need to restart the libvirt service too.

Now using virt-manager, and going on the General tab of the Windows machine, you can check that the security profile is no active anymore. By the way you find the "uuid" in virt-manager too.

Offline

#2175 2014-06-21 11:48:05

MCMjolnir: Member; Registered: 2014-06-19; Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OK, I tried setting security_driver = "none" in /etc/libvirt/qemu.conf as per the link but it didn't seem to do anything. So instead I ran

sudo /etc/init.d/apparmor stop
sudo update-rc.d -f apparmor remove

rebooted the host, and now the guest boots! (although this may be a bit heavy handed).

However, I'm now facing an issue in the guest where the card is showing up in device manager but it's giving me a code 43. I see this issue has been resolved already, but I'm not sure how I implement it (I'm still very much a fledgling linux user).

Offline

Pages: Previous 1 … 85 86 87 88 89 … 234 Next

Topic closed

Board footer

Atom topic feed