You are not logged in.

Pages: 1 … 73 74 75 76 77 … 234

Topic closed

#1851 2014-05-27 07:27:55

thelamer
Member
Registered: 2014-05-25
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I thought I would write down what I needed to do to get my gaming VM up and running. I am not really in a position to answer any hardcore technical questions as this mostly came from googling and this thread.

I am happy to report I got to get some Wolfenstein in on my linux desktop even if it was only for 2 hours out of the entire weekend sad .

GameVM

I am running relatively bleeding edge hardware. It is a i7-4790 on a Z87X-UD4H and a XFX R9 270x I bought at Best Buy ( it was the weekend and I was desperate to test)

First problem I had is with IOMMU enabled is my marvell 88SE9172 SATA controller was broken on boot(running the current stable kernel 3.14) which dropped 2 disks out of my 16 driver raid array. Two things I can say is, praise science for raid 6 and bitmap array rebuilds, otherwise I would have said nope right there and just made sure my data was safe.

Second problem was that my motherboard has a i915 VGA adapter and is effected by an outstanding arbitration bug.

To resolve this I compiled Alex Williamson's fork of the kernel mainline and applied a i915 patch he put together ( big shout out to Alex he is the man and extrememly helpful on all the forums he communicates on. As far as I know is the current king of VFIO help it seems like).

I am running Debian Jessie on my server to checkout and compile Alex's kernel I needed to: 

git clone git://github.com/awilliam/linux-vfio.git linux-vfio
cd linux-vfio
git checkout dma-alias-v4

created a file called i915patch:

http://pastebin.com/dZqD3k5a

Now patched it in: 

patch -i i915patch

Now to compile: 

cp /boot/config-`uname -r` ./.config
make menuconfig

At this point all the VFIO stuff will be built as modules given my current kernel config, but I ended up setting them anyway.

edit the .config file and change/add the following lines: 

CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_VGA=y

Now compiled kernel and output debs: 

fakeroot make-kpkg -j8 --initrd --append-to-version=-awilliam kernel_image kernel_headers

Before I install them updated grub options:

in /etc/default/grub: 

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 i915.enable_hd_vgaarb=1"

Now install them: 

dpkg -i linux-image-3.15.0-rc6-awilliam-i915-2+_3.15.0-rc6-awilliam-10.00.Custom_amd64.deb
dpkg -i linux-headers-3.15.0-rc6-awilliam-i915-2+_3.15.0-rc6-awilliam-10.00.Custom_amd64.deb

Finally blacklist radeon: 

echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf

Reboot!

Ok so now I am booted and have a kernel that finally uses lube. Through troubleshooting I found out I needed the latest qemu and seabios from the current git tree. The ones bundled in jessie even in experimental did not cut it.

So lets build qemu: 

git clone http://git.qemu.org/git/qemu.git
cd qemu
./configure --enable-libusb
make
make install

Resulting binaries will be in /usr/local/bin/ so I just use full path to ensure I am using the latest.

Now build seabios: 

git clone git://git.seabios.org/seabios.git seabios
cd seabios/
make menuconfig (save and exit) 
make

Resulting bios.bin in the out/ subfolder.

Ok so I got all the tools I need to build a gaming VM, not gonna lie at this point I was 3 days in and was getting pretty excited to play the new Wolfenstein on my linux Desktop from my server using steam's new streaming service.

Get networking setup:

/etc/network/interfaces: 

 # This file describes the network interfaces available on your system
 # and how to activate them. For more information, see interfaces(5).
  
 # The loopback network interface
 auto lo br0 
 iface lo inet loopback

 # Set up interfaces manually, avoiding conflicts with, e.g., network manager
 iface eth0 inet manual

 # Bridge setup
iface br0 inet static 
   pre-up ip tuntap add dev tap0 mode tap user root
   pre-up ip link set tap0 up
   bridge_ports all tap0
   address 192.168.1.150
   netmask 255.255.255.0
   gateway 192.168.1.1
   bridge_stp off
   bridge_maxwait 0
   bridge_fd      0
   post-down ip link set tap0 down
   post-down ip tuntap del dev tap0 mode tap

I only have one NIC and I plan on bridging it.

Setup qemu.conf: (needed for the card to work properly)

/etc/qemu/qemu.conf 

group = "root"
user = "root"
clear_emulator_capabilities = 0

First installed windows 7 (windows 8 and 8.1 did not work for me something is broken in the catalyst driver for now) 


/usr/local/bin/qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host -smp 8,sockets=1,cores=4,threads=8 \
-bios /storage/OSs/seabios/out/bios.bin \
-boot menu=on \
-vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=00:1b.0,bus=pcie.0 \
-device piix4-ide,bus=pcie.0,id=piix4-ide \
-device ide-hd,bus=piix4-ide.0,drive=disk -drive file=/storage/OSs/vm/windows7.img,id=disk,format=raw \
-device ide-cd,bus=piix4-ide.1,drive=isocd -drive file=/storage/OSs/vm/windows7.iso,id=isocd \
-usb -usbdevice host:046d:c051 -usbdevice host:046d:c315 \
-net nic -net tap,ifname=tap0,script=no,downscript=no \
-vnc :1

Explanation, 01:00.0 is my 270x, 00:1b.0 is my MB audio the hdmi audio on the card did not work for me, initially to get windows to install I use the piix4 controller as it does not need drivers at install time virtio or q35 at install did not work for me, the usb devices are keyboard and mouse.

Got monitor output Hooray!

Install windows like normal and shutdown.

Now I installed the virtio drivers found here: http://alt.fedoraproject.org/pub/alt/vi … st/images/

/usr/local/bin/qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host -smp 8,sockets=1,cores=4,threads=8 \
-bios /storage/OSs/seabios/out/bios.bin \
-boot menu=on \
-vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=00:1b.0,bus=pcie.0 \
-device piix4-ide,bus=pcie.0,id=piix4-ide \
-device virtio-scsi-pci,id=storage -drive file=/storage/OSs/vm/storage.img,id=storage,format=raw \
-device ide-hd,bus=piix4-ide.0,drive=disk -drive file=/storage/OSs/vm/windows7.img,id=disk,format=raw \
-device ide-cd,bus=piix4-ide.1,drive=isocd -drive file=/storage/OSs/vm/virtio-win-0.1-74.iso,id=isocd \
-usb -usbdevice host:046d:c051 -usbdevice host:046d:c315 \
-net nic,model=virtio -net tap,ifname=tap0,script=no,downscript=no \
-vnc :1

Here I added a disk image and changed my nic to virtio so I could easily install the drivers in windows 7 and poweroff.

Now that you have virtio installed boot the machine make sure it works work: 

/usr/local/bin/qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host -smp 8,sockets=1,cores=4,threads=8 \
-bios /storage/OSs/seabios/out/bios.bin \
-boot menu=on \
-vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=00:1b.0,bus=pcie.0 \
-device virtio-scsi-pci,id=scsi \
-device scsi-hd,drive=disk -drive file=/storage/OSs/vm/windows7.img,id=disk,format=raw \
-device scsi-hd,drive=data -drive file=/storage/OSs/vm/storage.img,id=data,format=raw \
-usb -usbdevice host:046d:c051 -usbdevice host:046d:c315 \
-net nic,model=virtio -net tap,ifname=tap0,script=no,downscript=no \
-vnc :1

Ok driver time. For me I spent a whole day installing and re-installing windows and fighting BSODs only to find out I needed to reboot the host after a driver upgrade. Maybe catalyst writes a new rom to the card or something during the install not sure.

So with your machine still open installed the latest beta drivers on amd's website for my card. Then powered off (not reboot only got BSOD with reboot)

Rebooted the host.

Fire back up: 

/usr/local/bin/qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host -smp 8,sockets=1,cores=4,threads=8 \
-bios /storage/OSs/seabios/out/bios.bin \
-boot menu=on \
-vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=00:1b.0,bus=pcie.0 \
-device virtio-scsi-pci,id=scsi \
-device scsi-hd,drive=disk -drive file=/storage/OSs/vm/windows7.img,id=disk,format=raw \
-device scsi-hd,drive=data -drive file=/storage/OSs/vm/storage.img,id=data,format=raw \
-usb -usbdevice host:046d:c051 -usbdevice host:046d:c315 \
-net nic,model=virtio -net tap,ifname=tap0,script=no,downscript=no \
-vnc :1

From here I installed tightvnc and left my monitor and usb devices in the dust. Planning on getting one of these:

http://www.ebay.com/itm/Headless-server … 1087766664

With steam running I just enabled in game streaming: https://support.steampowered.com/kb_art … -RIAV-1617

Now I need to get steamOS as a network boot option for all my HTPCs so I can play games on the couch.

Last edited by thelamer (2014-05-27 07:28:45)

Offline

#1852 2014-05-27 11:18:21

zzz3000
Member
Registered: 2014-02-16
Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Namelles_One wrote:
zzz3000 wrote:

I have the same problem with HD7750.
The problem with windows radeon driver dissapeared only after I had installed  fedora rawhide, so I am waiting fedora 21.

Thank you! "Rawhide" - it is a keyword ^^

If you don't want to go bleeding edge for the whole distro, you can use the Fedora virt-preview repo: http://fedoraproject.org/wiki/Virtualiz … Repository

It effectively gives you "rawhide" virt bits while keeping the rest on the latest stable version.

Thanks. This is realy help me.

Offline

#1853 2014-05-27 15:52:16

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the report and details!

thelamer wrote:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 i915.enable_hd_vgaarb=1"

You shouldn not need vfio_iommu_type1.allow_unsafe_interrupts=1 on your hardware

Finally blacklist radeon: 

echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf

Some distros require rebuilding the initramfs after this or else the driver is loaded before we get to the rootfs

Now build seabios: 

git clone git://git.seabios.org/seabios.git seabios
cd seabios/
make menuconfig (save and exit) 
make

Resulting bios.bin in the out/ subfolder.

Is this really needed?  I don't know of anything wrong with the BIOS supplied by upstream QEMU.

Setup qemu.conf: (needed for the card to work properly)

/etc/qemu/qemu.conf 

group = "root"
user = "root"
clear_emulator_capabilities = 0

This is only needed for libvirt support, which you don't seem to be using, at least not yet.

Now I installed the virtio drivers found here: http://alt.fedoraproject.org/pub/alt/vi … st/images/

/usr/local/bin/qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host -smp 8,sockets=1,cores=4,threads=8 \
-bios /storage/OSs/seabios/out/bios.bin \
-boot menu=on \
-vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=00:1b.0,bus=pcie.0 \
-device piix4-ide,bus=pcie.0,id=piix4-ide \
-device virtio-scsi-pci,id=storage -drive file=/storage/OSs/vm/storage.img,id=storage,format=raw \
-device ide-hd,bus=piix4-ide.0,drive=disk -drive file=/storage/OSs/vm/windows7.img,id=disk,format=raw \
-device ide-cd,bus=piix4-ide.1,drive=isocd -drive file=/storage/OSs/vm/virtio-win-0.1-74.iso,id=isocd \
-usb -usbdevice host:046d:c051 -usbdevice host:046d:c315 \
-net nic,model=virtio -net tap,ifname=tap0,script=no,downscript=no \
-vnc :1

Here I added a disk image and changed my nic to virtio so I could easily install the drivers in windows 7 and poweroff.

You can also pass the virtio ISO as a 2nd CD during install, load both the network and disk drivers during install, then you don't need to do this 2-step process.

From here I installed tightvnc and left my monitor and usb devices in the dust. Planning on getting one of these:

http://www.ebay.com/itm/Headless-server … 1087766664

Heh, neat.  Thanks

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1854 2014-05-27 21:57:01

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alex, so anymore ideas to my lcpsi output at post #1849 and #1850?

Offline

#1855 2014-05-27 22:36:23

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:

Alex, so anymore ideas to my lcpsi output at post #1849 and #1850?

None yet.  I was going to suggest that we should save/restore ATS support as part of bus reset path since that's the only notable difference in #1849, but then I found that we already do that, so I don't know why it's showing up disabled.  For #1850, does a host suspend/resume make it work or is that just another failing case?  The differences in the correctable error status registers don't seem like they should make a difference.

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1856 2014-05-27 22:47:35

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

None yet.  I was going to suggest that we should save/restore ATS support as part of bus reset path since that's the only notable difference in #1849, but then I found that we already do that, so I don't know why it's showing up disabled.  For #1850, does a host suspend/resume make it work or is that just another failing case?  The differences in the correctable error status registers don't seem like they should make a difference.

In #1850 it is another working case. However I'm not sure if radeon driver maintainer might help as they may know some magic of these devices.

Offline

#1857 2014-05-28 06:53:53

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I want to passthrought my ATI radeon 38xx card to a KVM guest (windows), host is using i7's onboard video card.

Blacklisted radeon driver, and successfully bind these two driver to vfio-pci. (via the script in first post)

root@robin:~# lspci | grep ATI
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV670 [Radeon HD 3690/3850]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV670/680 HDMI Audio [Radeon HD 3690/3800 Series]

Getting this error:

Error starting domain: internal error: early end of file from monitor: possible problem:
qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev2,bus=pci.0,addr=0x6: vfio: error, group 6 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev2,bus=pci.0,addr=0x6: vfio: failed to get group 6
qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev2,bus=pci.0,addr=0x6: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev2,bus=pci.0,addr=0x6: Device 'vfio-pci' could not be initialized

All of devices are listed in same iommu_group, including my secondary network and onboard network card and graphics.

root@robin:/dev# ls /sys/bus/pci/devices/0000\:01\:00.0/iommu_group/devices/
0000:00:1c.0  0000:00:1c.5  0000:01:00.0  0000:02:00.0  0000:04:00.0
0000:00:1c.4  0000:00:1c.6  0000:01:00.1  0000:03:00.0
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04)
00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4)
00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4)
00:1c.5 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 6 (rev c4)
00:1c.6 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4)
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)
00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04)
00:1f.2 IDE interface: Intel Corporation 7 Series/C210 Series Chipset Family 4-port SATA Controller [IDE mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
00:1f.5 IDE interface: Intel Corporation 7 Series/C210 Series Chipset Family 2-port SATA Controller [IDE mode] (rev 04)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV670 [Radeon HD 3690/3850]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV670/680 HDMI Audio [Radeon HD 3690/3800 Series]
02:00.0 Ethernet controller: Qualcomm Atheros AR8161 Gigabit Ethernet (rev 10)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
04:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 41)

My kernel cmdline

root=UUID=94c11fcb-ec91-4b49-b21d-d2cb08d5929e ro quiet rootfstype=xfs add_efi_memmap intel_iommu=on initrd=\EFI\debian\initrd.img
Linux robin 3.14-1-amd64 #1 SMP Debian 3.14.4-1 (2014-05-13) x86_64 GNU/Linux

How i can only pass the ATI card to the guest? Not all the devices in iommu group.

Thanks

Offline

#1858 2014-05-28 10:32:15

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

buraktamturk wrote:

How i can only pass the ATI card to the guest? Not all the devices in iommu group.

You need the ACS override patch for your hardware. It is not in mainline Linux kernel (neither in 3.14 nor in upcoming 3.15). You can get it from https://lkml.org/lkml/2013/5/30/513 or in the linux-mainline archive from #1 post in this thread. You need to patch and rebuild your kernel for it.

Offline

#1859 2014-05-28 12:15:20

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:
buraktamturk wrote:

How i can only pass the ATI card to the guest? Not all the devices in iommu group.

You need the ACS override patch for your hardware. It is not in mainline Linux kernel (neither in 3.14 nor in upcoming 3.15). You can get it from https://lkml.org/lkml/2013/5/30/513 or in the linux-mainline archive from #1 post in this thread. You need to patch and rebuild your kernel for it.

Thank you, applied patches and rebuilded the kernel, it worked. My new cmdline is

root=UUID=94c11fcb-ec91-4b49-b21d-d2cb08d5929e ro quiet rootfstype=xfs add_efi_memmap intel_iommu=on initrd=\EFI\debian\initrd.img vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 pci-stub.ids=1002:9505,1002:aa18 pcie_acs_override=downstream

Guest see the graphics card but just used the emulated one, but i see there is two Standard VGA Adapter, i think guest excepting install the drivers to show up the monitor (or excepting to me remove regular one).

Now when i tried to stop the virtual machine i get following error (dmesg output of error) and kvm, libvirt is complately hanged until i reboot the computer. Is there anything i miss?

[  639.297949] BUG: unable to handle kernel paging request at 000000fd000000fc
[  639.298021] IP: [<000000fd000000fc>] 0xfd000000fc
[  639.298066] PGD 0 
[  639.298083] Oops: 0010 [#1] SMP 
[  639.298111] Modules linked in: tun nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc bridge 8021q garp stp mrp llc snd_hda_codec_via snd_hda_codec_generic snd_hda_codec_hdmi nls_utf8 nls_cp437 vfat fat x86_pkg_temp_thermal intel_powerclamp iTCO_wdt iTCO_vendor_support ppdev intel_rapl coretemp crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel joydev snd_hda_intel aes_x86_64 lrw snd_hda_codec gf128mul snd_hwdep glue_helper ablk_helper cryptd snd_pcm efi_pstore evdev psmouse pcspkr serio_raw snd_timer snd soundcore i2c_i801 efivars lpc_ich i915 mfd_core drm_kms_helper drm i2c_algo_bit mei_me i2c_core mei parport_pc tpm_infineon battery parport tpm_tis tpm video button processor ip6table_filter ip6_tables ipt_REJECT xt_tcpudp xt_limit nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables fuse autofs4 xfs crc32c libcrc32c dm_mod sd_mod crc_t10dif crct10dif_generic hid_generic ata_generic usbhid hid crct10dif_pclmul crct10dif_common ata_piix libata scsi_mod ehci_pci xhci_hcd ehci_hcd r8169 alx mii mdio usbcore thermal fan usb_common thermal_sys
[  639.298909] CPU: 0 PID: 2361 Comm: libvirtd Tainted: G        W    3.14.4-rt5 #3
[  639.298951] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z77M-D3H, BIOS F15a 12/31/2013
[  639.299005] task: ffff88041dcaac80 ti: ffff88041ce52000 task.ti: ffff88041ce52000
[  639.299046] RIP: 0010:[<000000fd000000fc>]  [<000000fd000000fc>] 0xfd000000fc
[  639.299091] RSP: 0018:ffff88041ce53de0  EFLAGS: 00010202
[  639.299121] RAX: 0000000000000008 RBX: ffff88041e520098 RCX: 0000000000000000
[  639.299160] RDX: 000000fd000000fc RSI: ffff88041e520098 RDI: ffff88041e520098
[  639.299200] RBP: ffff88041e520140 R08: 0000000000000000 R09: 0000000000000008
[  639.299239] R10: 0000000000000074 R11: ffff88041ce53ba6 R12: 0000000000000004
[  639.299278] R13: 0000000000000000 R14: 000000000000000c R15: ffff88041c01e810
[  639.299318] FS:  00007f3da4c98700(0000) GS:ffff88042f200000(0000) knlGS:0000000000000000
[  639.299362] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  639.299394] CR2: 000000fd000000fc CR3: 000000041b183000 CR4: 00000000001407e0
[  639.299433] Stack:
[  639.299448]  ffffffff813ab9d7 0000000000000004 ffff88041e520098 0000000000000004
[  639.299497]  ffffffff813ac3b8 0000000000000004 ffff88041e520098 ffff88041e520140
[  639.299546]  0000000000000246 ffffffff813ac504 ffff88041e520098 ffffffff81885b60
[  639.299594] Call Trace:
[  639.299617]  [<ffffffff813ab9d7>] ? __rpm_callback+0x27/0x70
[  639.299653]  [<ffffffff813ac3b8>] ? rpm_idle+0x198/0x290
[  639.299686]  [<ffffffff813ac504>] ? __pm_runtime_idle+0x54/0x80
[  639.299721]  [<ffffffff812ee2e7>] ? pci_device_remove+0x67/0xa0
[  639.299757]  [<ffffffff813a1e45>] ? __device_release_driver+0x75/0xf0
[  639.299794]  [<ffffffff813a1ed9>] ? device_release_driver+0x19/0x30
[  639.299831]  [<ffffffff813a0e35>] ? unbind_store+0xb5/0xe0
[  639.299865]  [<ffffffff81233c2e>] ? kernfs_fop_write+0xbe/0x120
[  639.299901]  [<ffffffff811c5080>] ? vfs_write+0xb0/0x1e0
[  639.299932]  [<ffffffff811c5a2d>] ? SyS_write+0x3d/0xa0
[  639.299965]  [<ffffffff81508e39>] ? system_call_fastpath+0x16/0x1b
[  639.299999] Code:  Bad RIP value.
[  639.300024] RIP  [<000000fd000000fc>] 0xfd000000fc
[  639.300058]  RSP <ffff88041ce53de0>
[  639.300079] CR2: 000000fd000000fc
[  639.313543] ---[ end trace 1613cc056b6b661d ]---

Offline

#1860 2014-05-28 12:35:51

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Anyone knows how to add ",multifunction=on,x-vga=on" to pci config on virsh xml configuration file?

My configuration file is follows

<domain type='kvm'>                                                                                                                                                                                                                                                                                      
  <name>Windows7</name>
  <uuid>42716343-35d0-e786-5a86-d6d66dc6ced2</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.0'>hvm</type>
    <loader>/usr/share/ovmf/OVMF.fd</loader>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/storage/share/programlar/ISO/en_windows_8.1_professional_vl_with_update_x64_dvd_4065194.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/buraktamturk/Downloads/virtio-win-0.1-74.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/storage/vm/disk/Windows7_C.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:76:17:e2'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </interface>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x045e'/>
        <product id='0x0752'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x045e'/>
        <product id='0x00cb'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Thanks

Offline

#1861 2014-05-28 12:58:26

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

buraktamturk wrote:

Anyone knows how to add ",multifunction=on,x-vga=on" to pci config on virsh xml configuration file?

My configuration file is follows

<domain type='kvm'>                                                                                                                                                                                                                                                                                      
  <name>Windows7</name>
  <uuid>42716343-35d0-e786-5a86-d6d66dc6ced2</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.0'>hvm</type>
    <loader>/usr/share/ovmf/OVMF.fd</loader>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/storage/share/programlar/ISO/en_windows_8.1_professional_vl_with_update_x64_dvd_4065194.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/buraktamturk/Downloads/virtio-win-0.1-74.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/storage/vm/disk/Windows7_C.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:76:17:e2'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </interface>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x045e'/>
        <product id='0x0752'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x045e'/>
        <product id='0x00cb'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Thanks

Devices you want to pass have to be defined using qemu:commandlie. Here is my config: http://pastebin.com/ZVwvv9Hc

Because of this, you have to bind them to vfio by hand. You can use my libvirt's hook script to bind these devices to vfio (or bind HD Audio to vfio and rebind it to host's driver after vm is down; should work with every pci device): http://pastebin.com/fznjWcZ4 (put it as /etc/libvirt/hooks/qemu and change device id's).

Offline

#1862 2014-05-28 14:25:52

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

Devices you want to pass have to be defined using qemu:commandlie. Here is my config: http://pastebin.com/ZVwvv9Hc

Because of this, you have to bind them to vfio by hand. You can use my libvirt's hook script to bind these devices to vfio (or bind HD Audio to vfio and rebind it to host's driver after vm is down; should work with every pci device): http://pastebin.com/fznjWcZ4 (put it as /etc/libvirt/hooks/qemu and change device id's).

Very thanks.

I feel very stupid. I can't find the bus that graphics card attached.

  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
  </qemu:commandline>

I get "Bus 'root.1' not found", i can't find the bus with lspci. (tried several ones including root.0 to 5 and not giving bus number but didn't do the trick)

How i can find it? The device i want to attach is Advanced Micro Devices, Inc. [AMD/ATI] RV670 [Radeon HD 3690/3850]

Thank you again,

lspci output

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09)
        Subsystem: Gigabyte Technology Co., Ltd Device 5000
        Flags: bus master, fast devsel, latency 0
        Capabilities: [e0] Vendor Specific Information: Len=0c <?>

00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09) (prog-if 00 [VGA controller])
        Subsystem: Gigabyte Technology Co., Ltd Device d000
        Flags: bus master, fast devsel, latency 0, IRQ 45
        Memory at f7800000 (64-bit, non-prefetchable) [size=4M]
        Memory at d0000000 (64-bit, prefetchable) [size=256M]
        I/O ports at f000 [size=64]
        Expansion ROM at <unassigned> [disabled]
        Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [d0] Power Management version 2
        Capabilities: [a4] PCI Advanced Features
        Kernel driver in use: i915

00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI])
        Subsystem: Gigabyte Technology Co., Ltd Device 5007
        Flags: bus master, medium devsel, latency 0, IRQ 42
        Memory at f7f00000 (64-bit, non-prefetchable) [size=64K]
        Capabilities: [70] Power Management version 2
        Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
        Kernel driver in use: xhci_hcd

00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
        Subsystem: Gigabyte Technology Co., Ltd Device 1c3a
        Flags: bus master, fast devsel, latency 0, IRQ 44
        Memory at f7f19000 (64-bit, non-prefetchable) [size=16]
        Capabilities: [50] Power Management version 3
        Capabilities: [8c] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Kernel driver in use: mei_me

00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) (prog-if 20 [EHCI])
        Subsystem: Gigabyte Technology Co., Ltd Device 5006
        Flags: bus master, medium devsel, latency 0, IRQ 16
        Memory at f7f17000 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [50] Power Management version 2
        Capabilities: [58] Debug port: BAR=1 offset=00a0
        Capabilities: [98] PCI Advanced Features
        Kernel driver in use: ehci-pci

00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
        Subsystem: Gigabyte Technology Co., Ltd Device a014
        Flags: bus master, fast devsel, latency 0, IRQ 46
        Memory at f7f10000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [50] Power Management version 2
        Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [130] Root Complex Link
        Kernel driver in use: snd_hda_intel

00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 0000e000-0000efff
        Memory behind bridge: f7e00000-f7efffff
        Prefetchable memory behind bridge: 00000000e0000000-00000000efffffff
        Capabilities: [40] Express Root Port (Slot+), MSI 00
        Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
        Capabilities: [90] Subsystem: Gigabyte Technology Co., Ltd Device 5001
        Capabilities: [a0] Power Management version 2
        Kernel driver in use: pcieport

00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
        I/O behind bridge: 0000d000-0000dfff
        Memory behind bridge: f7d00000-f7dfffff
        Capabilities: [40] Express Root Port (Slot+), MSI 00
        Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
        Capabilities: [90] Subsystem: Gigabyte Technology Co., Ltd Device 5001
        Capabilities: [a0] Power Management version 2
        Kernel driver in use: pcieport

00:1c.5 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 6 (rev c4) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
        I/O behind bridge: 0000c000-0000cfff
        Memory behind bridge: f7c00000-f7cfffff
        Prefetchable memory behind bridge: 00000000f0000000-00000000f00fffff
        Capabilities: [40] Express Root Port (Slot+), MSI 00
        Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
        Capabilities: [90] Subsystem: Gigabyte Technology Co., Ltd Device 5001
        Capabilities: [a0] Power Management version 2
        Kernel driver in use: pcieport

00:1c.6 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4) (prog-if 01 [Subtractive decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=04, subordinate=05, sec-latency=0
        Capabilities: [40] Express Root Port (Slot+), MSI 00
        Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
        Capabilities: [90] Subsystem: Gigabyte Technology Co., Ltd Device 5001
        Capabilities: [a0] Power Management version 2

00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) (prog-if 20 [EHCI])
        Subsystem: Gigabyte Technology Co., Ltd Device 5006
        Flags: bus master, medium devsel, latency 0, IRQ 23
        Memory at f7f16000 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [50] Power Management version 2
        Capabilities: [58] Debug port: BAR=1 offset=00a0
        Capabilities: [98] PCI Advanced Features
        Kernel driver in use: ehci-pci

00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04)
        Subsystem: Gigabyte Technology Co., Ltd Device 5001
        Flags: bus master, medium devsel, latency 0
        Capabilities: [e0] Vendor Specific Information: Len=0c <?>
        Kernel driver in use: lpc_ich

00:1f.2 IDE interface: Intel Corporation 7 Series/C210 Series Chipset Family 4-port SATA Controller [IDE mode] (rev 04) (prog-if 8f [Master SecP SecO PriP PriO])
        Subsystem: Gigabyte Technology Co., Ltd Device b005
        Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 19
        I/O ports at f110 [size=8]
        I/O ports at f100 [size=4]
        I/O ports at f0f0 [size=8]
        I/O ports at f0e0 [size=4]
        I/O ports at f0d0 [size=16]
        I/O ports at f0c0 [size=16]
        Capabilities: [70] Power Management version 3
        Capabilities: [b0] PCI Advanced Features
        Kernel driver in use: ata_piix

00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
        Subsystem: Gigabyte Technology Co., Ltd Device 5001
        Flags: medium devsel, IRQ 18
        Memory at f7f15000 (64-bit, non-prefetchable) [size=256]
        I/O ports at f040 [size=32]

00:1f.5 IDE interface: Intel Corporation 7 Series/C210 Series Chipset Family 2-port SATA Controller [IDE mode] (rev 04) (prog-if 85 [Master SecO PriO])
        Subsystem: Gigabyte Technology Co., Ltd Device b002
        Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 19
        I/O ports at f0b0 [size=8]
        I/O ports at f0a0 [size=4]
        I/O ports at f090 [size=8]
        I/O ports at f080 [size=4]
        I/O ports at f070 [size=16]
        I/O ports at f060 [size=16]
        Capabilities: [70] Power Management version 3
        Capabilities: [b0] PCI Advanced Features
        Kernel driver in use: ata_piix

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV670 [Radeon HD 3690/3850] (prog-if 00 [VGA controller])
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 2542
        Flags: fast devsel, IRQ 11
        Memory at e0000000 (64-bit, prefetchable) [disabled] [size=256M]
        Memory at f7e20000 (64-bit, non-prefetchable) [disabled] [size=64K]
        I/O ports at e000 [disabled] [size=256]
        Expansion ROM at f7e00000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Kernel driver in use: vfio-pci

01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV670/680 HDMI Audio [Radeon HD 3690/3800 Series]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] RV670/680 HDMI Audio [Radeon HD 3690/3800 Series]
        Flags: fast devsel, IRQ 17
        Memory at f7e30000 (64-bit, non-prefetchable) [disabled] [size=16K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Kernel driver in use: vfio-pci

02:00.0 Ethernet controller: Qualcomm Atheros AR8161 Gigabit Ethernet (rev 10)
        Subsystem: Gigabyte Technology Co., Ltd Device e000
        Flags: bus master, fast devsel, latency 0, IRQ 48
        Memory at f7d00000 (64-bit, non-prefetchable) [size=256K]
        I/O ports at d000 [size=128]
        Capabilities: [40] Power Management version 3
        Capabilities: [58] Express Endpoint, MSI 00
        Capabilities: [c0] MSI: Enable+ Count=1/16 Maskable+ 64bit+
        Capabilities: [d8] MSI-X: Enable- Count=16 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [180] Device Serial Number ff-5e-25-1c-90-2b-34-ff
        Kernel driver in use: alx

03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
        Subsystem: Realtek Semiconductor Co., Ltd. Device 0123
        Flags: bus master, fast devsel, latency 0, IRQ 43
        I/O ports at c000 [size=256]
        Memory at f7c00000 (64-bit, non-prefetchable) [size=4K]
        Memory at f0000000 (64-bit, prefetchable) [size=16K]
        Expansion ROM at f0020000 [disabled] [size=128K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [160] Device Serial Number 71-02-00-00-68-4c-e0-00
        Kernel driver in use: r8169

04:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 41) (prog-if 01 [Subtractive decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=04, secondary=05, subordinate=05, sec-latency=32
        Capabilities: [90] Power Management version 2
        Capabilities: [a0] Subsystem: Gigabyte Technology Co., Ltd Device 8892

lspci -t

-[0000:00]-+-00.0
           +-02.0
           +-14.0
           +-16.0
           +-1a.0
           +-1b.0
           +-1c.0-[01]--+-00.0
           |            \-00.1
           +-1c.4-[02]----00.0
           +-1c.5-[03]----00.0
           +-1c.6-[04-05]----00.0-[05]--
           +-1d.0
           +-1f.0
           +-1f.2
           +-1f.3
           \-1f.5

Offline

#1863 2014-05-28 17:56:13

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

<qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/var/lib/libvirt/firmware/vbios_GF106.rom'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>

First device is PCH bridge emulated by qemu to which you want connect gpu you're passing, second and third are gpu itself and gpu's hda. TBH you can just copy-paste it wink

EDIT:
Is it only me or new NVIDIA 337.88 driver still has issues with virtualized Windows? I'm getting code 43, 335.23 doesn't have this issue.

Last edited by dwe11er (2014-05-28 18:12:54)

Offline

#1864 2014-05-28 18:30:23

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:
<qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/var/lib/libvirt/firmware/vbios_GF106.rom'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>

First device is PCH bridge emulated by qemu to which you want connect gpu you're passing, second and third are gpu itself and gpu's hda. TBH you can just copy-paste it wink

EDIT:
Is it only me or new NVIDIA 337.88 driver still has issues with virtualized Windows? I'm getting code 43, 335.23 doesn't have this issue.

Thank You! I just see the windows installer on the monitor that connected to the ati card. I was giving up!

Now i understant what "bus" means here. Just had to change "ioh3420,bus=pcie.0" to "ioh3420,bus=pci.0". Everything seems working! except OVMF UEFI bios firmware (Black screen at all, UEFI doesn't important for me at this point).

What does "romfile=/var/lib/libvirt/firmware/vbios_GF106.rom" means? I do not have such file.

Thank you again!!

Offline

#1865 2014-05-28 19:24:02

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now another problem at windows installation's final step :'(. Is there really something that i miss?

svrhqhD.png

Offline

#1866 2014-05-28 19:37:17

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

buraktamturk wrote:

Now another problem at windows installation's final step :'(. Is there really something that i miss?

Did you rebooted your Windows during driver installation? Could you try to make a clean host reboot and then start VM with installed driver.

Offline

#1867 2014-05-28 19:53:00

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

Is it only me or new NVIDIA 337.88 driver still has issues with virtualized Windows? I'm getting code 43, 335.23 doesn't have this issue.

It's not only you. Same here for my gtx660

EDIT: Downgrading worked

Last edited by Flyser (2014-05-28 19:59:28)

Offline

#1868 2014-05-28 20:08:55

buraktamturk
Member
Registered: 2014-05-28
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mbroemme wrote:
buraktamturk wrote:

Now another problem at windows installation's final step :'(. Is there really something that i miss?

Did you rebooted your Windows during driver installation? Could you try to make a clean host reboot and then start VM with installed driver.

Setup automatically rebooted guest after completing step 1. installing windows.

When installation is in 2. Step ("Completing Installation"), screen flickered (i think it was installing the graphics driver), then BSOD show up. When i start the guest again, (still standard vga driver), installer said something like "the system was unexceptedly restarted and setup cannot contiune"

Now tried with Windows 8.1 Update 1 iso, the os is installed without any graphics driver, and after setup, i manually installed latest graphics of ati 3800. voilla, worked!, resolution is fine and ATI control center is working.
So i have to block the default bundled ati driver installation of unmodified Windows 7 (i belive radeon5400 and older cards have microsoft driver by default), i will figure out somehow. I will edit this post when i can successfully boot windows 7 os.

Thank you again.

Offline

#1869 2014-05-28 22:12:33

thelamer
Member
Registered: 2014-05-25
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

buraktamturk wrote:
mbroemme wrote:
buraktamturk wrote:

Now another problem at windows installation's final step :'(. Is there really something that i miss?

Did you rebooted your Windows during driver installation? Could you try to make a clean host reboot and then start VM with installed driver.

Setup automatically rebooted guest after completing step 1. installing windows.

When installation is in 2. Step ("Completing Installation"), screen flickered (i think it was installing the graphics driver), then BSOD show up. When i start the guest again, (still standard vga driver), installer said something like "the system was unexceptedly restarted and setup cannot contiune"

Now tried with Windows 8.1 Update 1 iso, the os is installed without any graphics driver, and after setup, i manually installed latest graphics of ati 3800. voilla, worked!, resolution is fine and ATI control center is working.
So i have to block the default bundled ati driver installation of unmodified Windows 7 (i belive radeon5400 and older cards have microsoft driver by default), i will figure out somehow. I will edit this post when i can successfully boot windows 7 os.

Thank you again.

For me to get windows 7 working I needed to reboot the host (not guest) after the ati driver installation. (just selected poweroff after the driver install)

I was getting the same atikpmag BSOD you were.

It is worth a try.

Offline

#1870 2014-05-28 23:08:46

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:
dwe11er wrote:

Is it only me or new NVIDIA 337.88 driver still has issues with virtualized Windows? I'm getting code 43, 335.23 doesn't have this issue.

It's not only you. Same here for my gtx660

EDIT: Downgrading worked

Same here *sigh* time to start comparing traces :-\

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1871 2014-05-29 03:23:38

gutleib
Member
Registered: 2014-05-29
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi!
I am working on passthrough my gpu to windows VM by this guide, but I can't resolve one bug:

me@host:~$sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on  -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=06:00.0,bus=root.1,addr=01.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom -device vfio-pci,host=06:00.1,bus=root.1,addr=01.1 \
-vga cirrus -cdrom /home/me/Win_8.1.iso

gives

qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on: vfio: Device does not support requested feature x-vga
qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on: vfio: failed to get device 0000:05:00.0
qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on: Device 'vfio-pci' could not be initialized

System is Ubuntu 14.04, 

me@host:~# cat /boot/config-3.13.0-27-generic| grep VFIO
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y
CONFIG_KVM_VFIO=y
me@host:~# lsmod |grep vfio
vfio_pci               36474  0 
vfio_iommu_type1       17636  0 
vfio                   24873  2 vfio_iommu_type1,vfio_pci
me@host:~# lspci -s 05:00.0 -v
05:00.0 3D controller: NVIDIA Corporation GF110 [GeForce GTX 590] (rev a1)
	Subsystem: NVIDIA Corporation Device 0868
	Flags: bus master, fast devsel, latency 0, IRQ 16
	Memory at ec000000 (32-bit, non-prefetchable) [size=16M]
	Memory at e0000000 (64-bit, prefetchable) [size=128M]
	Memory at e8000000 (64-bit, prefetchable) [size=32M]
	Expansion ROM at ed080000 [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [78] Express Endpoint, MSI 00
	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
	Capabilities: [100] Virtual Channel
	Capabilities: [128] Power Budgeting <?>
	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Kernel driver in use: vfio-pci

Ummm... Does anyone have an idea? At least how to debug?

Last edited by gutleib (2014-05-29 03:24:41)

Offline

#1872 2014-05-29 03:29:48

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gutleib wrote:
qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on: vfio: Device does not support requested feature x-vga
05:00.0 3D controller: NVIDIA Corporation GF110 [GeForce GTX 590] (rev a1)

Ummm... Does anyone have an idea? At least how to debug?

The class code for you device is "3D controller", not "VGA compatible controller", therefore x-vga doesn't work for it because it doesn't support VGA.

EDIT: Are you using SLI and picked the slave card as your VGA device by mistake?

Last edited by aw (2014-05-29 03:35:00)

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1873 2014-05-29 03:39:40

gutleib
Member
Registered: 2014-05-29
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ah! thanks
search doesn't show much on how to fix this, do you have a suggestion?

edit: i see 06.00.0 is vga. going to try use it

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom  \
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=06:00.0,bus=root.1,addr=01.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on \
-device vfio-pci,host=06:00.1,bus=root.1,addr=01.1

hmmm.
No errors.
Nothing in vnc.
Nothing on screen, though.

Last edited by gutleib (2014-05-29 03:54:46)

Offline

#1874 2014-05-29 04:08:31

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gutleib wrote:

Ah! thanks
search doesn't show much on how to fix this, do you have a suggestion?

edit: i see 06.00.0 is vga. going to try use it

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom  \
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=06:00.0,bus=root.1,addr=01.0,multifunction=on,romfile=/root/Gigabyte.GTX590.1536.110223.rom,x-vga=on \
-device vfio-pci,host=06:00.1,bus=root.1,addr=01.1

hmmm.
No errors.
Nothing in vnc.
Nothing on screen, though.

Maybe you should start with one card and build from there.  You haven't identified what patches are in your kernel, your host graphics device, the version of qemu used, etc.  Besides, you've made a non-standard PCIe configuration in the guest by using PCI slots 0 and 1 behind your root port.  Typically only slot 0 is scanned by the guest.  Once you get your VGA card working, add another ioh3420 and put the other card behind it.

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1875 2014-05-29 04:47:05

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

buraktamturk wrote:

So i have to block the default bundled ati driver installation of unmodified Windows 7 (i belive radeon5400 and older cards have microsoft driver by default), i will figure out somehow. I will edit this post when i can successfully boot windows 7 os.

in fact, you dont have to block or figure anything.
just dont use passthrough during installation/inital setup (==use only emulated qemu vga, not host cards)
after you are done, you can add passthrough and install needed graphics drivers

in my case (HD 6450) it served one more purpose - i was getting BSOD (stop 0x0000007b i think?) every time i tried to poweroff guest (and at least once after some idle time - i think win7 tried to power down monitor) with passthrough. the following (before installing amd drivers) resolved the issue: i disabled any power saving related stuff i could find - HDD/monitor sleep-on-idle, etc. plus, after installing drivers, just in case, i disabled anything power saving in driver's settings.

gutleib wrote:

hmmm.
No errors.
Nothing in vnc.
Nothing on screen, though.

you sure you don't use intel's IGD for host and miss vga arbiter patch?

Offline

Pages: 1 … 73 74 75 76 77 … 234

Topic closed

Board footer

Powered by FluxBB