Arch Linux

TripleSpeeder

Member

Registered: 2011-05-02

Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks a lot Duelist! I found the root cause that stopped windows setup from continuing: It is the -cpu parameter.
This is the qemu command-line as provided by libvirt:

-cpu qemu64,hv_time

The moment I remove parameter hv_time, windows setup continues. And this problem only happens with smp=2 or higher. With only one core assigned to the VM it boots also with hv_time.

Any ideas out there what might be the root cause?

Edit: Note that this is still without any device pass-through, just using the default emulated vga. Trying to get setup running completely with passed-through NVidia card is the next step

I experienced crashes during the windows installation in a non-vfio setup as well. The root cause were missing kernel config options for libvirt, but I guess you are using the archlinux kernel, right?

Yes, i tried both current Arch linux kernel and patched kernel from OP.

tritron4

Member

Registered: 2012-04-14

Posts: 153

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does anyone knows how convert xen disk image into kvm . I used dd of lvm to get img of partition. When I try to boot img under kvm I get error: file /grub/i386-pc/normal.mod not found
I tried to reinstall grub but I can get the error to go away

sinny

Member

Registered: 2014-05-24

Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does anyone knows how convert xen disk image into kvm . I used dd of lvm to get img of partition. When I try to boot img under kvm I get error: file /grub/i386-pc/normal.mod not found
I tried to reinstall grub but I can get the error to go away

how the hell is your question related to this thread's topic?

not to mention that judging by error message alone it's obvious that it may be architecture mismatch

Last edited by sinny (2014-12-01 17:04:43)

mastersplinter777

Member

Registered: 2012-10-09

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

https://imageshack.com/i/p8J5tWtEp
https://imageshack.com/i/ipmipYYxp
I finaly succeeded to launch OSX 10.9.5 Mavericks on my Arch + Qemu 2.1.2

Last edited by mastersplinter777 (2014-12-01 21:10:37)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

https://imageshack.com/i/p8J5tWtEp
https://imageshack.com/i/ipmipYYxp
I finaly succeeded to launch OSX 10.9.5 Mavericks on my Arch + Qemu 2.1.2

Cool.  And since running a VM with GPU acceleration in a window always brings the newbies out, let's just point out right now that that's a VNC connection to the VM, guest-based VNC server.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

mastersplinter777

Member

Registered: 2012-10-09

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

https://imageshack.com/i/p8J5tWtEp
https://imageshack.com/i/ipmipYYxp
I finaly succeeded to launch OSX 10.9.5 Mavericks on my Arch + Qemu 2.1.2

Cool.  And since running a VM with GPU acceleration in a window always brings the newbies out, let's just point out right now that that's a VNC connection to the VM, guest-based VNC server.

VNC connection is the only more or less comfotable way to connect graphically to VM if I have one physical display with different inputs... Besides I also use VNC connection with my Win8.1 VM and it is rather fast, so I can even play some 3d games or chat using my webcam in ooVoo

sumpfmensch

Member

Registered: 2014-06-13

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello there,

after trying the whole passthrough journey with ubuntu, I finally gave up and tried to follow the guide in the first post with a new arch installation but I just can't get to pass through my VTX R9 280 3GB to a guest on my arch host. The mobo is the Gigabyte Z97 UD3H BK and the Processor the new 4690k (which supports VT-d).

Passing through an old 8800 GTX seems to work (i.e. gives me the seabios "No boot device found" output) using the same commands and setup. My R9 280 however does not want to output any signal at all (neither via HDMI, DVI nor DVI->VGA).

Since I'm all out of ideas (reading and trying since 3 days almost fulltime now) I'd like to ask for any further advice. The huge amount of posts I skimmed through in this thread usually seem to have get an error message when receiving no signal. I don't see anything problematic in dmesg and no output from qemu at all.

Here is my current setup which seems to work with the Nvidia card (removing the entries for 0000:01:00.1 in the vfio-bind script and the qemu call).

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=eed2a69c-c80f-4820-92bb-2b2afe2b7ad5 rw quiet pci-stub.ids=1002:679a,1002:aaa0 i915.enable_hd_vgaarb=1 intel_iommu=on pcie_acs_override=downstream

Here i added pcie_acs_override=downstream later on although I think it should not be needed having both 0000:01:00.0 and 0000:01:00.0 on iommu group 13 (and no other devices there).

The script for vfio binding: cat vfio-bind

DEVICES="0000:01:00.0 0000:01:00.1"

modprobe vfio-pci
for dev in $DEVICES; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done

ls /sys/bus/pci/drivers/vfio-pci/                 

0000:01:00.0  0000:01:00.1  bind  module  new_id  remove_id  uevent  unbind

The final command I use to start the machine:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
 -bios /usr/share/qemu/bios.bin \
 -vga none -nographic\
 -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/root/ati_r9_280.rom \
 -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

(I tried it with and without the romfile i got from /sys/.../rom - no difference - played around with the pc chipset - or should I continue trying to relocate the devices?)

cat /etc/modprobe.d/blacklist.conf

blacklist radeon
blacklist nouveau 

dmesg

...
[  221.927937] VFIO - User Level meta-driver version: 0.3
[  350.698684] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[  350.722340] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270
[  350.722345] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0
[  353.236170] kvm: zapping shadow pages for mmio generation wraparound

Do you have any Ideas or pointers where I may continue my quest for passthrough? (I can't switch to OVMF since I got two windows 7 licences I want to run on VMs which I do not want to throw away.)

I would just love to see that bios screen - from that point on I already have Ideas how to proceed.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello there,

after trying the whole passthrough journey with ubuntu, I finally gave up and tried to follow the guide in the first post with a new arch installation but I just can't get to pass through my VTX R9 280 3GB to a guest on my arch host. The mobo is the Gigabyte Z97 UD3H BK and the Processor the new 4690k (which supports VT-d).

Passing through an old 8800 GTX seems to work (i.e. gives me the seabios "No boot device found" output) using the same commands and setup. My R9 280 however does not want to output any signal at all (neither via HDMI, DVI nor DVI->VGA).

Since I'm all out of ideas (reading and trying since 3 days almost fulltime now) I'd like to ask for any further advice. The huge amount of posts I skimmed through in this thread usually seem to have get an error message when receiving no signal. I don't see anything problematic in dmesg and no output from qemu at all.

Here is my current setup which seems to work with the Nvidia card (removing the entries for 0000:01:00.1 in the vfio-bind script and the qemu call).

When you switch from the Nvidia card to the Radeon, are you physically swapping them?  If not I'd suspect something with primary VGA or the nvidia driver getting in there and locking the VGA arbiter.  If so, I dunno...

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

sumpfmensch

Member

Registered: 2014-06-13

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When you switch from the Nvidia card to the Radeon, are you physically swapping them?  If not I'd suspect something with primary VGA or the nvidia driver getting in there and locking the VGA arbiter.  If so, I dunno...

Yes, I shut down the machine, pull the card and the power connectors out and insert the other one. My Primary VGA is indeed the Intel card but I hoped I'd have solved the arbiter problems with the kernel provided in post 1.

Maybe the following is interesting, too:

When I boot the vm with an ubuntu live disk, the attached ATI card AND the default std ouput, lspci shows me that the ATI card is present ... While the radeon driver gets loaded assigned to the card, it is not activated by default. It does neither show up in the display settings not output anything. When I boot the linux image on bare metal, (with the Intel card set as primary) it was used for a second screen.

EDIT: I'm using the -M pc currently since I have problems booting the image ("no boot device") with q35
EDIT2: Ok - getting used to the different drive syntax for q35 - however i cannot get the radeon to output anything on the virtualized ubuntu. Already starting to consider switching to xen and starting all over again or even selling the card

Hope somebody can think of anything to try next.

Last edited by sumpfmensch (2014-12-02 00:48:42)

consty

Member

Registered: 2014-12-02

Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

First of all, thank you a lot for the post. It is awesome and really well documented. 10/10.

Even me, with my rudimentary linux skills, I was able to make the passthrough to the KVM, after facing all kind of issues. Yes, I had all of them!.

- My box is a:
intel i7
gigabyte mobo
nvidia gtx780
and the damn:
06:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller (rev 10)

- I am running:
Linux nexus2 3.17.2-1-mainline #1 SMP PREEMPT Mon Nov 24 17:50:40 CET 2014 x86_64 GNU/Linux

And when I activate intel_iommu=on, the raid6 which is partly held by the pcie-SATA marvell controller, stops working. So I have to decide whether I want vfio or my raid storage

I was in the same boat as you with the marvell 9230. I found that booting with intel_iommu=on AND iommu=pt allowed me to passthrough hardware while still having access to my RAID controller.

I don't know if this causes any system instability or if I'm just a special case where this works for only me.

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I  wanna buy a new CPU and motherboard.

Bearing in mind, we get round about 15-20 % less performance in a VM, I'd like to compensate that with stronger, better hardware. My guest OS is Windows 8.1 and I own a MSI GTX 980

I just bought the i7-4790k http://www.newegg.com/Product/Product.a … 6819117369

But I am not 100 % sure I should stay with it (it's still sealed and I haven't decided on the motherboard yet)

There are two other interesting options:

the  I7-5820K http://www.newegg.com/Product/Product.a … 6819117402

and the I7-5930K http://www.newegg.com/Product/Product.a … 6819117403

I just don't know if it's worth going for the more expensive and more power hungry options. What would you guys do if we forget the cost for a moment? It's all about 4k gaming really

Last edited by 4kGamer (2014-12-02 07:37:53)

thearcherblog

Member

Registered: 2014-10-30

Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I  wanna buy a new CPU and motherboard.

Bearing in mind, we get round about 15-20 % less performance in a VM, I'd like to compensate that with stronger, better hardware. My guest OS is Windows 8.1 and I own a MSI GTX 980

I just bought the i7-4790k http://www.newegg.com/Product/Product.a … 6819117369

But I am not 100 % sure I should stay with it (it's still sealed and I haven't decided on the motherboard yet)

There are two other interesting options:

the  I7-5820K http://www.newegg.com/Product/Product.a … 6819117402

and the I7-5930K http://www.newegg.com/Product/Product.a … 6819117403

I just don't know if it's worth going for the more expensive and more power hungry options. What would you guys do if we forget the cost for a moment? It's all about 4k gaming really

Hello,

To be honest I have a 4790k and the performance is really good. No problem at all.

I think the 4790k can be faster (and cheaper) for gaming and the other one probably is better for doing more multitasking (according to cores). Anyway, the clock is faster on 4790k.

About playing games with 4K resolution... is up to your graphic cards really and I'm not expert on that But I guess that you probably need more than one if you really want it in Ultra mode

So, my two cents... 4790K and a Z97 motherboard / X99 (check for vt-d support (I have the Extreme 6 Z97 with good results)) and use the internal IGP for host and two graphic cards for passthrough (two NVIDIA GTX 770 should be enough and I know for sure that works fine ).

I hope it helps

Regards,

TheArcher

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thanks a lot for your input! Really appreciated. Yeah, the i7-4790k is faster than the other ones. I think I will stick with it then. And thank you for suggesting a motherboard. I have seen so many that I don't know which one I should buy. But it's great to hear the ASRock Z97 EXTREME6 is doing a fine job in a VM.

One more question: what CPU cooler do you use? And are you happy with it?

Last edited by 4kGamer (2014-12-02 08:50:26)

thearcherblog

Member

Registered: 2014-10-30

Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thanks a lot for your input! Really appreciated. Yeah, the i7-4790k is faster than the other ones. I think I will stick with it then. And thank you for suggesting a motherboard. I have seen so many that I don't know which one I should buy. But it's great to hear the ASRock Z97 EXTREME6 is doing a fine job in a VM.

One more question: what CPU cooler do you use? And are you happy with it?

A pleasure if I helped you

Is not the proper thread for this but, Noctua is a good brand for normal coolers. If you look for water cooling... H110 is a good one for non super geeks

Anyway, it also depends of your case. In my case I chose a not too big one to have my computer at the living room. I have Noctua fans and I'm happy about the noise but I think can be better results.

My idle temperatures are around 34C (room is near 20) and running the VM is around 37-38C.

When I play dragon age, graphic card goes to 80C and the computer is on 70-72C.

Can be better? yes. My though was to have the H110 but there was not enough room on my case for it.

I didn't notice any difference between running the VM or Windows as a baremetal to be honest.

I hope it helps

Regards,

TheArcher

Last edited by thearcherblog (2014-12-02 10:05:17)

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think I read somewhere about the h110. I will have a look into that. You have been really helpful. Now I can make some decisions

thank you!

thearcherblog

Member

Registered: 2014-10-30

Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think I read somewhere about the h110. I will have a look into that. You have been really helpful. Now I can make some decisions

thank you!

You welcome

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I  wanna buy a new CPU and motherboard.

Sounds like you're already committed to a pricey upgrade, but FWIW I decided to go a more mild route upgrading from the i5-3470T.  I found my motherboard is compatible with a Xeon E3-1245v2, found a reasonable deal for one on ebay, and now I've got 2x the cores and a 500MHz boost.  Including a Cooler Master Hyper 212 Plus, I'm in it for well below the price of an i7-4790k alone, and it seems to have enough juice to run Steam in-home-streaming with the GTX750 (4-cores for the VM, leaving 4 threads for the host).

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yeah, but you know what you're doing I am just following guides without the knowledge Seriously, I am very grateful for your input and I am sure I will need it again once I get my new motherboard and run the new CPU. Hopefully towards the weekend

MatthiasArch

Member

Registered: 2014-12-02

Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi There!
I'm very happy this thread exists as everything is working (more or less) out of the box
I have an Intel i5 4670 on an ASRock B85M-ITX and a Gigabyte Nvidia GTX750TI.
Running Archlinux default

Linux loewenbraeu 3.17.4-1-ARCH #1 SMP PREEMPT Fri Nov 21 21:14:42 CET 2014 x86_64 GNU/Linux

with current arch qemu/kvm/libvirt with Windows 8.1 Pro as OVMF uefi booted guest and Nvidia card passed through. Everything works like a charm UNTIL I shut down the VM!

Then Hell kicks in:

[ 1164.246920] ------------[ cut here ]------------
[ 1164.246935] WARNING: CPU: 3 PID: 303 at drivers/gpu/drm/i915/i915_gem_gtt.c:1492 gen6_ggtt_insert_entries+0x146/0x150 [i915]()
[ 1164.246936] Modules linked in: bluetooth msr cpufreq_stats tun fuse vfio_pci vfio_iommu_type1 vfio xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc nvram led_class joydev mousedev uas usb_storage snd_hda_codec_hdmi cfg80211 rfkill nct6775 hwmon_vid evdev iTCO_wdt mac_hid iTCO_vendor_support coretemp hwmon intel_rapl x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_realtek kvm_intel snd_hda_codec_generic kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i915 aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse pcspkr drm_kms_helper serio_raw i2c_i801 alx drm snd_hda_intel mdio tpm_tis tpm snd_hda_controller snd_hda_codec
[ 1164.246957]  nuvoton_cir snd_hwdep intel_gtt rc_core snd_pcm video i2c_algo_bit battery button snd_timer mei_me i2c_core snd mei soundcore shpchp lpc_ich processor sch_fq_codel nfs lockd sunrpc fscache ext4 crc16 mbcache jbd2 hid_cherry hid_generic usbhid hid sd_mod crc_t10dif crct10dif_common ahci libahci atkbd libps2 ehci_pci xhci_hcd ehci_hcd crc32c_intel libata scsi_mod usbcore usb_common i8042 serio
[ 1164.246975] CPU: 3 PID: 303 Comm: Xorg.bin Tainted: G           O   3.17.4-1-ARCH #1
[ 1164.246976] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B85M-ITX, BIOS P2.20 09/01/2014
[ 1164.246977]  0000000000000000 000000008bc678ec ffff8800a5cffc68 ffffffff81537b0e
[ 1164.246978]  0000000000000000 ffff8800a5cffca0 ffffffff8107078d 00000000fc8c9000
[ 1164.246980]  00000000ffffffff ffffc9000598b424 ffff8800bb417a58 ffffc90005989100
[ 1164.246981] Call Trace:
[ 1164.246985]  [<ffffffff81537b0e>] dump_stack+0x4d/0x6f
[ 1164.246988]  [<ffffffff8107078d>] warn_slowpath_common+0x7d/0xa0
[ 1164.246989]  [<ffffffff810708ba>] warn_slowpath_null+0x1a/0x20
[ 1164.246994]  [<ffffffffa0531786>] gen6_ggtt_insert_entries+0x146/0x150 [i915]
[ 1164.246999]  [<ffffffffa05309b9>] ggtt_bind_vma+0xc9/0xf0 [i915]
[ 1164.247005]  [<ffffffffa053b3cb>] i915_gem_object_set_cache_level+0x29b/0x360 [i915]
[ 1164.247009]  [<ffffffffa053b53d>] i915_gem_set_caching_ioctl+0xad/0xf0 [i915]
[ 1164.247015]  [<ffffffffa03affef>] drm_ioctl+0x1df/0x680 [drm]
[ 1164.247017]  [<ffffffff811da5f0>] do_vfs_ioctl+0x2d0/0x4b0
[ 1164.247019]  [<ffffffff811e4cae>] ? __fget+0x6e/0xb0
[ 1164.247021]  [<ffffffff811da851>] SyS_ioctl+0x81/0xa0
[ 1164.247023]  [<ffffffff8153db29>] system_call_fastpath+0x16/0x1b
[ 1164.247024] ---[ end trace 59b5489690ced9e7 ]---
[ 1164.247025] ------------[ cut here ]------------
[ 1164.247030] WARNING: CPU: 3 PID: 303 at drivers/gpu/drm/i915/intel_uncore.c:47 assert_device_not_suspended.isra.7+0x43/0x50 [i915]()
[ 1164.247031] Device suspended
[ 1164.247032] Modules linked in: bluetooth msr cpufreq_stats tun fuse vfio_pci vfio_iommu_type1 vfio xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc nvram led_class joydev mousedev uas usb_storage snd_hda_codec_hdmi cfg80211 rfkill nct6775 hwmon_vid evdev iTCO_wdt mac_hid iTCO_vendor_support coretemp hwmon intel_rapl x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_realtek kvm_intel snd_hda_codec_generic kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i915 aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse pcspkr drm_kms_helper serio_raw i2c_i801 alx drm snd_hda_intel mdio tpm_tis tpm snd_hda_controller snd_hda_codec
[ 1164.247049]  nuvoton_cir snd_hwdep intel_gtt rc_core snd_pcm video i2c_algo_bit battery button snd_timer mei_me i2c_core snd mei soundcore shpchp lpc_ich processor sch_fq_codel nfs lockd sunrpc fscache ext4 crc16 mbcache jbd2 hid_cherry hid_generic usbhid hid sd_mod crc_t10dif crct10dif_common ahci libahci atkbd libps2 ehci_pci xhci_hcd ehci_hcd crc32c_intel libata scsi_mod usbcore usb_common i8042 serio
[ 1164.247061] CPU: 3 PID: 303 Comm: Xorg.bin Tainted: G        W  O   3.17.4-1-ARCH #1
[ 1164.247062] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B85M-ITX, BIOS P2.20 09/01/2014
[ 1164.247062]  0000000000000000 000000008bc678ec ffff8800a5cffbc0 ffffffff81537b0e
[ 1164.247064]  ffff8800a5cffc08 ffff8800a5cffbf8 ffffffff8107078d ffff8800bb410000
[ 1164.247065]  0000000000101008 ffff8800bb410068 ffff8800bb417a58 0000000000000001
[ 1164.247067] Call Trace:
[ 1164.247068]  [<ffffffff81537b0e>] dump_stack+0x4d/0x6f
[ 1164.247070]  [<ffffffff8107078d>] warn_slowpath_common+0x7d/0xa0
[ 1164.247071]  [<ffffffff8107080c>] warn_slowpath_fmt+0x5c/0x80
[ 1164.247077]  [<ffffffffa0556423>] assert_device_not_suspended.isra.7+0x43/0x50 [i915]
[ 1164.247082]  [<ffffffffa0559de7>] hsw_write32+0x37/0x160 [i915]
[ 1164.247086]  [<ffffffffa0531716>] gen6_ggtt_insert_entries+0xd6/0x150 [i915]
[ 1164.247091]  [<ffffffffa05309b9>] ggtt_bind_vma+0xc9/0xf0 [i915]
[ 1164.247096]  [<ffffffffa053b3cb>] i915_gem_object_set_cache_level+0x29b/0x360 [i915]
[ 1164.247100]  [<ffffffffa053b53d>] i915_gem_set_caching_ioctl+0xad/0xf0 [i915]
[ 1164.247104]  [<ffffffffa03affef>] drm_ioctl+0x1df/0x680 [drm]
[ 1164.247106]  [<ffffffff811da5f0>] do_vfs_ioctl+0x2d0/0x4b0
[ 1164.247108]  [<ffffffff811e4cae>] ? __fget+0x6e/0xb0
[ 1164.247109]  [<ffffffff811da851>] SyS_ioctl+0x81/0xa0
[ 1164.247111]  [<ffffffff8153db29>] system_call_fastpath+0x16/0x1b
[ 1164.247112] ---[ end trace 59b5489690ced9e8 ]---
[ 1164.247114] [drm:hsw_unclaimed_reg_detect] *ERROR* Unclaimed register detected. Please use the i915.mmio_debug=1 to debug this problem.
[ 1164.247115] ------------[ cut here ]------------
[ 1164.247120] WARNING: CPU: 3 PID: 303 at drivers/gpu/drm/i915/intel_uncore.c:47 assert_device_not_suspended.isra.7+0x43/0x50 [i915]()
[ 1164.247120] Device suspended
[ 1164.247121] Modules linked in: bluetooth msr cpufreq_stats tun fuse vfio_pci vfio_iommu_type1 vfio xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc nvram led_class joydev mousedev uas usb_storage snd_hda_codec_hdmi cfg80211 rfkill nct6775 hwmon_vid evdev iTCO_wdt mac_hid iTCO_vendor_support coretemp hwmon intel_rapl x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_realtek kvm_intel snd_hda_codec_generic kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i915 aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse pcspkr drm_kms_helper serio_raw i2c_i801 alx drm snd_hda_intel mdio tpm_tis tpm snd_hda_controller snd_hda_codec
[ 1164.247138]  nuvoton_cir snd_hwdep intel_gtt rc_core snd_pcm video i2c_algo_bit battery button snd_timer mei_me i2c_core snd mei soundcore shpchp lpc_ich processor sch_fq_codel nfs lockd sunrpc fscache ext4 crc16 mbcache jbd2 hid_cherry hid_generic usbhid hid sd_mod crc_t10dif crct10dif_common ahci libahci atkbd libps2 ehci_pci xhci_hcd ehci_hcd crc32c_intel libata scsi_mod usbcore usb_common i8042 serio
[ 1164.247150] CPU: 3 PID: 303 Comm: Xorg.bin Tainted: G        W  O   3.17.4-1-ARCH #1
[ 1164.247151] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B85M-ITX, BIOS P2.20 09/01/2014
[ 1164.247151]  0000000000000000 000000008bc678ec ffff8800a5cffbb8 ffffffff81537b0e
[ 1164.247153]  ffff8800a5cffc00 ffff8800a5cffbf0 ffffffff8107078d ffff8800bb410068
[ 1164.247154]  0000000000101008 ffffc9000598b424 0000000000101008 ffff8800bb410000
[ 1164.247155] Call Trace:
[ 1164.247157]  [<ffffffff81537b0e>] dump_stack+0x4d/0x6f
[ 1164.247158]  [<ffffffff8107078d>] warn_slowpath_common+0x7d/0xa0
[ 1164.247160]  [<ffffffff8107080c>] warn_slowpath_fmt+0x5c/0x80
[ 1164.247165]  [<ffffffffa0556423>] assert_device_not_suspended.isra.7+0x43/0x50 [i915]
[ 1164.247169]  [<ffffffffa05592d5>] gen6_read32+0x35/0x150 [i915]
[ 1164.247174]  [<ffffffffa0531729>] gen6_ggtt_insert_entries+0xe9/0x150 [i915]
[ 1164.247178]  [<ffffffffa05309b9>] ggtt_bind_vma+0xc9/0xf0 [i915]
[ 1164.247183]  [<ffffffffa053b3cb>] i915_gem_object_set_cache_level+0x29b/0x360 [i915]
[ 1164.247188]  [<ffffffffa053b53d>] i915_gem_set_caching_ioctl+0xad/0xf0 [i915]
[ 1164.247191]  [<ffffffffa03affef>] drm_ioctl+0x1df/0x680 [drm]
[ 1164.247193]  [<ffffffff811da5f0>] do_vfs_ioctl+0x2d0/0x4b0
[ 1164.247195]  [<ffffffff811e4cae>] ? __fget+0x6e/0xb0
[ 1164.247196]  [<ffffffff811da851>] SyS_ioctl+0x81/0xa0
[ 1164.247198]  [<ffffffff8153db29>] system_call_fastpath+0x16/0x1b
[ 1164.247199] ---[ end trace 59b5489690ced9e9 ]---
[ 1164.248115] input: Logitech USB Gaming Mouse as /devices/pci0000:00/0000:00:14.0/usb3/3-3/3-3:1.0/0003:046D:C049.0007/input/input31
[ 1164.248166] hid-generic 0003:046D:C049.0007: input,hidraw2: USB HID v1.11 Mouse [Logitech USB Gaming Mouse] on usb-0000:00:14.0-3/input0
[ 1164.249614] hid-generic 0003:046D:C049.0008: hiddev0,hidraw3: USB HID v1.11 Device [Logitech USB Gaming Mouse] on usb-0000:00:14.0-3/input1
[ 1165.942981] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
[ 1165.943032] BUG: unable to handle kernel paging request at ffff880045645000
[ 1165.943074] IP: [<ffff880045645000>] 0xffff880045645000
[ 1165.943108] PGD 2b2c067 PUD 80000000400001e3 
[ 1165.943136] Oops: 0011 [#1] PREEMPT SMP 
[ 1165.943162] Modules linked in: bluetooth msr cpufreq_stats tun fuse vfio_pci vfio_iommu_type1 vfio xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc nvram led_class joydev mousedev uas usb_storage snd_hda_codec_hdmi cfg80211 rfkill nct6775 hwmon_vid evdev iTCO_wdt mac_hid iTCO_vendor_support coretemp hwmon intel_rapl x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_realtek kvm_intel snd_hda_codec_generic kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i915 aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse pcspkr drm_kms_helper serio_raw i2c_i801 alx drm snd_hda_intel mdio tpm_tis tpm snd_hda_controller snd_hda_codec
[ 1165.943620]  nuvoton_cir snd_hwdep intel_gtt rc_core snd_pcm video i2c_algo_bit battery button snd_timer mei_me i2c_core snd mei soundcore shpchp lpc_ich processor sch_fq_codel nfs lockd sunrpc fscache ext4 crc16 mbcache jbd2 hid_cherry hid_generic usbhid hid sd_mod crc_t10dif crct10dif_common ahci libahci atkbd libps2 ehci_pci xhci_hcd ehci_hcd crc32c_intel libata scsi_mod usbcore usb_common i8042 serio
[ 1165.943846] CPU: 0 PID: 464 Comm: libvirtd Tainted: G        W  O   3.17.4-1-ARCH #1
[ 1165.943894] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B85M-ITX, BIOS P2.20 09/01/2014
[ 1165.943952] task: ffff88021acd4670 ti: ffff8802221e4000 task.ti: ffff8802221e4000
[ 1165.943998] RIP: 0010:[<ffff880045645000>]  [<ffff880045645000>] 0xffff880045645000
[ 1165.944046] RSP: 0018:ffff8802221e7d70  EFLAGS: 00010286
[ 1165.944078] RAX: 0000000000000088 RBX: ffff880222b4d098 RCX: 0000000000000000
[ 1165.944121] RDX: ffff8802154dab88 RSI: ffff880222b4d098 RDI: ffff880222b4d098
[ 1165.944165] RBP: ffff8802221e7d90 R08: 0000000000000246 R09: ffffea00029e7c00
[ 1165.944208] R10: ffffffffa06c91b8 R11: 0000000000000000 R12: ffff880222b4d146
[ 1165.944251] R13: ffff880045645000 R14: 0000000000000000 R15: fffffffffffffff2
[ 1165.944295] FS:  00007f6c8235c780(0000) GS:ffff88022e200000(0000) knlGS:0000000000000000
[ 1165.944344] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1165.944379] CR2: ffff880045645000 CR3: 000000021dcb2000 CR4: 00000000001407f0
[ 1165.944422] Stack:
[ 1165.944434]  ffffffff813b5772 ffff880222b4d098 ffff880222b4d098 0000000000000004
[ 1165.944480]  ffff8802221e7dc0 ffffffff813b65c9 ffff880222b4d098 0000000000000004
[ 1165.944526]  ffff880222b4d146 0000000000000246 ffff8802221e7df0 ffffffff813b6703
[ 1165.944573] Call Trace:
[ 1165.944590]  [<ffffffff813b5772>] ? __rpm_callback+0x32/0x70
[ 1165.944626]  [<ffffffff813b65c9>] rpm_idle+0x259/0x340
[ 1165.944658]  [<ffffffff813b6703>] __pm_runtime_idle+0x53/0x70
[ 1165.944695]  [<ffffffff812e6fd4>] pci_device_remove+0x74/0xc0
[ 1165.944731]  [<ffffffff813aca4f>] __device_release_driver+0x7f/0xf0
[ 1165.944770]  [<ffffffff813acae3>] device_release_driver+0x23/0x30
[ 1165.944808]  [<ffffffff813ab82d>] unbind_store+0xed/0x150
[ 1165.944842]  [<ffffffff813aaaa5>] drv_attr_store+0x25/0x40
[ 1165.944877]  [<ffffffff8123ec7a>] sysfs_kf_write+0x3a/0x50
[ 1165.944911]  [<ffffffff8123e1ee>] kernfs_fop_write+0xee/0x180
[ 1165.944946]  [<ffffffff811c7967>] vfs_write+0xb7/0x200
[ 1165.944978]  [<ffffffff811c85d9>] SyS_write+0x59/0xd0
[ 1165.945010]  [<ffffffff8153db29>] system_call_fastpath+0x16/0x1b
[ 1165.945047] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
[ 1165.945190] RIP  [<ffff880045645000>] 0xffff880045645000
[ 1165.945224]  RSP <ffff8802221e7d70>
[ 1165.948044] CR2: ffff880045645000
[ 1165.969330] ---[ end trace 59b5489690ced9ea ]---

This seems a little bit like those VGA arbitration missing issues - except that I thought to avoid that problematic with using UEFI in guest? The host archlinux is also booted using UEFI, CSM is disabled in bios. (Booting using CSM makes no difference, though).
nouveau is blacklisted and the nvidia card set to pci stub. Bootparams:

intel_iommu=on pci-stub.ids=01:00.0,01:00.1

After VM shutdown, lspci hangs forever, libvirt crashed and a system shutdown is not possible as some process hangs...

Any idea where to dig into?

Thanks a lot!

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

intel_iommu=on pci-stub.ids=01:00.0,01:00.1

Nak, pci-stub.ids takes vendor and device IDs, not bus address.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

kidchunks

Member

Registered: 2014-12-03

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been cracking at this for about a week (off and on) and get stuck with the same problem on multiple setups. Before I talk about my problem, I tested the card by installing windows 8.1 on the host and everything worked fine.

Hardware Specs:

Test 1:
Mobo: Supermicro X8DAH+-F
CPU: Xeon X5670
Video Card: Sapphire r9 290x Tri-X OC
OS: Debian Wheezy with latest kernel release from alex william and qemu-kvm from wheezy backports (v2.1)

Test 2:
Same hardware as above
OS: Ubuntu 14.04 LTS (everything stock)

Now, I've done all the necessary changes from post 1 (vfio binding, kernel command, stubbing and etc). I didn't do any patches given I don't think I need them.

Using the following to launch start the VM:

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host -smp 8,sockets=1,cores=8,threads=1 \
-bios /usr/share/seabios/bios.bin -vga none -rtc base=localtime -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=85:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=85:00.1,bus=root.1,addr=00.1 \
-drive file=/root/win7.img,cache=none,if=virtio \
-cdrom /root/win7.iso \
-usb -usbdevice host:046d:c068 \
-no-reboot 

Devices:
85:00.0 <-- video card
85:00.1 <-- audio for card

When I start the VM, I can install the OS no problem through a monitor connected to the video card (no drivers installed). As soon as I install the drivers (any catalyst driver! I tested 13.12 through 14.11.2).

I get the following after the start screen (window logo):

vfio-pci 0000:85:00.0: irq 89 for MSI/MSI-X

the irq number changes each time

Immediately following that message..the VM process is killed. I tested both windows 7 and windows 8.1..same issue.

I'm not sure how to trouble this because dmesg doesn't provide anything other than the irq message. I've also tried searching this thread but seems no one has run into this edge case.

If anyone has any ideas, let me know!

EDIT: Updated error message

Last edited by kidchunks (2014-12-03 14:53:32)

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

try without -device vfio-pci,host=85:00.1,bus=root.1,addr=00.1 \ ?
i don't have amd, but reading this thread i think you need to remove amd audio device

edit: and try 440fx chipset not q35

Last edited by slis (2014-12-03 10:57:11)

dwe11er

Member

Registered: 2014-03-18

Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

try without -device vfio-pci,host=85:00.1,bus=root.1,addr=00.1 \ ?
i don't have amd, but reading this thread i think you need to remove amd audio device

edit: and try fx440 chipset not q35

It actually depends on card (or maybe mobo's ACPI?). My HD7870 works fine with HD Audio assigned as function 0x1 on 440FX.

Last edited by dwe11er (2014-12-03 09:29:11)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I  wanna buy a new CPU and motherboard.

Sounds like you're already committed to a pricey upgrade, but FWIW I decided to go a more mild route upgrading from the i5-3470T.  I found my motherboard is compatible with a Xeon E3-1245v2, found a reasonable deal for one on ebay, and now I've got 2x the cores and a 500MHz boost.  Including a Cooler Master Hyper 212 Plus, I'm in it for well below the price of an i7-4790k alone, and it seems to have enough juice to run Steam in-home-streaming with the GTX750 (4-cores for the VM, leaving 4 threads for the host).

You do know that xeon lacks integrated video core.
He may be needing it to get host output. However, sometimes buying a cheap-as-cardboard GT610 can be more profitable than buying i7 for integrated video.

Whoa, just noted that this post is #3499.

Last edited by Duelist (2014-12-03 12:46:49)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You do know that xeon lacks integrated video core.
He may be needing it to get host output. However, sometimes buying a cheap-as-cardboard GT610 can be more profitable than buying i7 for integrated video.

Whoa, just noted that this post is #3499.

Check ark, E3-1245v2 has HD P4000 video.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users