KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9 (Page 166) / Kernel & Hardware / Arch Linux Forums

You are not logged in.

Topics: Active | Unanswered

Pages: Previous 1 … 164 165 166 167 168 … 234 Next

Topic closed

#4126 2015-02-09 11:42:37

evilsephiroth: Member; Registered: 2014-09-16; Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:

NVIDIA GRID K2 passthrough successful, but with code 43 after install nvidia driver.
driver info: 347.25-quadro-grid-desktop-notebook-win8-win7-64bit-international-whql
here is my qemu commandline:
#!/bin/sh
qemu-system-x86_64 -cpu host,kvm=off -smp 4,sockets=2,cores=2,threads=1 \
-m 8192 -M q35 -enable-kvm \
-rtc base=localtime,clock=host \
-acpitable file=/var/lib/libvirt/images/LENOVO-TC-90-MSFT-2.1.BIN \
-device virtio-scsi-pci,id=scsi \
-drive file=/var/lib/libvirt/images/win7_nvidia_k2_clean.img,cache=writeback,\
if=none,format=qcow2,aio=native,id=virtio-scsi-disk0 \
-device scsi-hd,drive=virtio-scsi-disk0 \
-net nic,model=virtio,macaddr=52:54:00:1a:2b:3c \
-net tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
-vga none -nographic \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-monitor stdio

According to this thread, no solution yet for drivers > 340.52. I'm stuck at that version too.(gtx 560Ti on Windows 7 machine)

Offline

#4127 2015-02-09 11:57:28

slis: Member; Registered: 2014-06-02; Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You need bios from GT 630 and use it with romfile= option, for working reboot.

Offline

#4128 2015-02-09 12:41:52

evilsephiroth: Member; Registered: 2014-09-16; Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

You need bios from GT 630 and use it with romfile= option, for working reboot.

is it a problem only with gt630?

I'm doing a openelec vm with a gt610...

My vm windows 7 with gtx 560 TI doesn't suffer this problem...

Offline

#4129 2015-02-09 12:42:37

slis: Member; Registered: 2014-06-02; Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hmm, for me it was must with geforce fermi

Offline

#4130 2015-02-09 12:50:01

evilsephiroth: Member; Registered: 2014-09-16; Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

edit double post

Last edited by evilsephiroth (2015-02-09 12:50:46)

Offline

#4131 2015-02-09 12:50:44

slis: Member; Registered: 2014-06-02; Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

from that link

4 The GeForce GT 620 (OEM) card is a rebranded GeForce GT 520.
5 The GeForce GT 620 card is a rebranded GeForce GT 530.
6 This revision of GeForce GT 630 (DDR3) card is a rebranded GeForce GT 440 (DDR3).

Offline

#4132 2015-02-09 13:39:54

bpbastos: Member; Registered: 2014-11-21; Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

evilsephiroth wrote:

slis wrote:
You need bios from GT 630 and use it with romfile= option, for working reboot.
is it a problem only with gt630?
I'm doing a openelec vm with a gt610...
My vm windows 7 with gtx 560 TI doesn't suffer this problem...

I'm already using the romfile option but seems like my bios is corrupt or something because I could boot and use the card, except shutdown and reboot, with seabios but if try ovmf I get a lot of artifacts.
Do you have the bios for ZOTAC GeForce GT 630 Synergy Edition 2GB? I haven't found any.

Offline

#4133 2015-02-09 13:41:19

aw: Member; Registered: 2013-10-04; Posts: 921; Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:

NVIDIA GRID K2 passthrough successful, but with code 43 after install nvidia driver.
driver info: 347.25-quadro-grid-desktop-notebook-win8-win7-64bit-international-whql

Nvidia K-series GRID, Tesla, and Quadro (2000+) are supported by Nvidia in non-vga passthrough mode. Configure the device as secondary graphics for the VM and do not use the x-vga option.

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4134 2015-02-09 13:43:44

aw: Member; Registered: 2013-10-04; Posts: 921; Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

evilsephiroth wrote:

According to this thread, no solution yet for drivers > 340.52. I'm stuck at that version too.(gtx 560Ti on Windows 7 machine)

What are you talking about?! I'm running 347.25 here with GTX750. KVM and Hyper-V extensions need to be hidden/disabled to avoid Code 43 on GeForce cards.

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4135 2015-02-09 13:56:17

flack: Member; Registered: 2014-06-12; Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

bpbastos wrote:

evilsephiroth wrote:
slis wrote:
You need bios from GT 630 and use it with romfile= option, for working reboot.
is it a problem only with gt630?
I'm doing a openelec vm with a gt610...
My vm windows 7 with gtx 560 TI doesn't suffer this problem...
I'm already using the romfile option but seems like my bios is corrupt or something because I could boot and use the card, except shutdown and reboot, with seabios but if try ovmf I get a lot of artifacts.
Do you have the bios for ZOTAC GeForce GT 630 Synergy Edition 2GB? I haven't found any.

Are you try this one? At techpowerup site

Offline

#4136 2015-02-09 14:03:33

bpbastos: Member; Registered: 2014-11-21; Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

flack wrote:

bpbastos wrote:
evilsephiroth wrote:
is it a problem only with gt630?
I'm doing a openelec vm with a gt610...
My vm windows 7 with gtx 560 TI doesn't suffer this problem...
I'm already using the romfile option but seems like my bios is corrupt or something because I could boot and use the card, except shutdown and reboot, with seabios but if try ovmf I get a lot of artifacts.
Do you have the bios for ZOTAC GeForce GT 630 Synergy Edition 2GB? I haven't found any.

Are you try this one? At techpowerup site

Thank you @flack.

EDIT:
The bios from techpowerup doesn't support EFI.

[root@home-server rom-parser]# ./rom-parser Zotac.GT630.2048.120511.rom
Valid ROM signature found @0h, PCIR offset 188h
        PCIR: type 0, vendor: 10de, device: 0f00, class: 030000
        PCIR: revision 0, vendor revision: 1
        Last image

Last edited by bpbastos (2015-02-09 14:11:08)

Offline

#4137 2015-02-09 14:56:58

hotfunction: Member; Registered: 2015-02-05; Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hotfunction wrote:

Hi everyone, i've been reading a lot of posts here, and i wanted to try the passthrough by myself, but seemed that i got some error here, and firstly according to the steps of the guide, i have been doing step by steps, and here's my build :
Processor : Xeon E3-1200 (VT-d Enabled in BIOS)
ubuntu display GPU : Radeon HD4870
Passthrough GPU : GeForce GTX 560 Ti Hawx
So steps i've been done so far :
1. Ubuntu Server 12.04, then do-release-upgrade to 14.04
2. Downloaded Kernel 3.18-5 from Linux homepage
3. Downloaded 3.18-5 linux-mainline from nhbs's first post
4. Patched ACS and i915_317 patch from linux-mainline from nbhs's post
5. Qemu 2.2.0, also Seabios standard and no patches
6. lspci | grep NVIDIA :

05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] [10de:1200] (rev a1)
05:00.1 Audio device [0403]: NVIDIA Corporation GF114 HDMI Audio Controller [10de:0e0c] (rev a1)
7. edited /etc/default/grub :

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c"
and our /proc/cmdline :

BOOT_IMAGE=/vmlinuz-3.18.5 root=/dev/mapper/SSLABVFIO--vg-root ro quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c vt.handoff=7
8. also blacklisted radeon on /etc/modprobe.d/blacklist.conf
9. dmesg pci-stub :
[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.18.5 root=/dev/mapper/SSLABVFIO--vg-root ro quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c vt.handoff=7
[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.18.5 root=/dev/mapper/SSLABVFIO--vg-root ro quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c vt.handoff=7
[ 2.787575] pci-stub: add 10DE:1200 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 2.787587] pci-stub 0000:05:00.0: claimed by stub
[ 2.787593] pci-stub: add 10DE:0E0C sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 2.787597] pci-stub 0000:05:00.1: claimed by stub
10. i checked lspci -vnn for the graphics cards in the device :
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] R700 [Radeon HD 4870 X2] [1002:9441] (prog-if 00 [VGA controller])
Subsystem: ASUSTeK Computer Inc. Device [1043:0284]
Flags: bus master, fast devsel, latency 0, IRQ 11
Memory at d0000000 (64-bit, prefetchable) [size=256M]
Memory at f6220000 (64-bit, non-prefetchable) [size=64K]
I/O ports at b000 [size=256]
Expansion ROM at f6200000 [disabled] [size=128K]
Capabilities: [50] Power Management version 3
Capabilities: [58] Express Legacy Endpoint, MSI 00
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] RV770 HDMI Audio [Radeon HD 4850/4870] [1002:aa30]
Subsystem: ASUSTeK Computer Inc. Device [1043:aa30]
Flags: bus master, fast devsel, latency 0, IRQ 36
Memory at f6230000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [50] Power Management version 3
Capabilities: [58] Express Legacy Endpoint, MSI 00
Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Kernel driver in use: snd_hda_intel
04:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] R700 [Radeon HD 4870 X2] [1002:9441]
Subsystem: ASUSTeK Computer Inc. Device [1043:0284]
Flags: bus master, fast devsel, latency 0, IRQ 11
Memory at c0000000 (64-bit, prefetchable) [size=256M]
Memory at f6120000 (64-bit, non-prefetchable) [size=64K]
I/O ports at a000 [size=256]
Expansion ROM at f6100000 [disabled] [size=128K]
Capabilities: [50] Power Management version 3
Capabilities: [58] Express Legacy Endpoint, MSI 00
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] [10de:1200] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:2601]
Flags: fast devsel, IRQ 11
Memory at f4000000 (32-bit, non-prefetchable) [disabled] [size=32M]
Memory at e0000000 (64-bit, prefetchable) [disabled] [size=128M]
Memory at e8000000 (64-bit, prefetchable) [disabled] [size=64M]
I/O ports at e000 [disabled] [size=128]
Expansion ROM at f6000000 [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [b4] Vendor Specific Information: Len=14 <?>
Capabilities: [100] Virtual Channel
Capabilities: [128] Power Budgeting <?>
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Kernel driver in use: pci-stub
05:00.1 Audio device [0403]: NVIDIA Corporation GF114 HDMI Audio Controller [10de:0e0c] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:2601]
Flags: bus master, fast devsel, latency 0, IRQ 10
Memory at f6080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Kernel driver in use: pci-stub
11. after all went good, i create vfio-pci1.cfg in /etc/vfio-pci1.cfg :
0000:05:00.0
0000:05:00.1
12. Booting VM's Script :
#!/bin/bash
configfile=/etc/vfio-pci1.cfg
vfiobind() {
dev="$1"
vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
device=$(cat /sys/bus/pci/devices/$dev/device)
if [ -e /sys/bus/pci/devices/$dev/driver ]; then
echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
fi
echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
}
modprobe vfio-pci
cat $configfile | while read line;do
echo $line | grep ^# >/dev/null 2>&1 && continue
vfiobind $line
done
sudo qemu-system-x86_64 -vga none -M q35 -hda /home/sslab719/VMimages/VM.img -enable-kvm -m 2048 -cpu host,kvm=off \
-smp 2,sockets=1,cores=2,threads=1 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-drive file=/home/sslab719/VMimages/VM.img,id=disk,format=qcow2 \
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/sslab719/MSI.GTX560Ti.1024.110825.rom \
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \
-net user,hostfwd=tcp::10022-:22 -net nic
#-boot menu=on
exit 0
Here at this point, i am always having black QEMU Monitor with "compat_monitor0 console"
Qemu 2.2.0 monitor - type help for more information -- when i'm using -vga none right over here.
If i remove -vga none there, i can boot to the VMs normally, and i can see the lspci went in correctly, but it does seem like it doesn't have the real GPU power... the passthrough went there and it shows up in lspci but i am guessing that theres no GPU clock, shader, and memory power.. I tried to install heaven engine benchmark in linux, and i tried to run it on, and it fails, and here i am guessing that perhaps the problem might be the GPU doesn't really passthrough successfully, but instead its just a name showup on the VMs, and i can still see 00:01.0 VGA compatible Controller: Device 1234:1111, and the others are our passthrough'd GPU 01:00.0..
Here by reading i know that we should add -vga none in order to passthrough it normally, but it seemed that -vga none didn't bring a help for me, and instead its just gave me a weird black qemu monitor which i can type some commands of qemu itself.
i tried -vga std and others, but those didn't worked too..
so i checked my dmesg when turning my VM to see whether vfio works correctly, here are some important points at my dmesg:
[ 0.210689] vgaarb: setting as boot device: PCI:0000:03:00.0
[ 0.210691] vgaarb: device added: PCI:0000:03:00.0,decodes=io+mem,owns=io+mem,locks=none
[ 0.210695] vgaarb: device added: PCI:0000:05:00.0,decodes=io+mem,owns=none,locks=none
[ 0.210696] vgaarb: loaded
[ 0.210697] vgaarb: bridge control possible 0000:05:00.0
[ 0.210697] vgaarb: bridge control possible 0000:03:00.0
[ 0.210846] SCSI subsystem initialized
[ 0.210873] libata version 3.00 loaded.
[ 0.210889] ACPI: bus type USB registered
[ 0.210902] usbcore: registered new interface driver usbfs
[ 0.210908] usbcore: registered new interface driver hub
[ 0.210919] usbcore: registered new device driver usb
[ 0.211004] PCI: Using ACPI for IRQ routing
[ 0.212320] PCI: pci_cache_line_size set to 64 bytes
[ 0.212378] e820: reserve RAM buffer [mem 0x0009ec00-0x0009ffff]
[ 0.212380] e820: reserve RAM buffer [mem 0xbdf9c000-0xbfffffff]
[ 0.212381] e820: reserve RAM buffer [mem 0xbdfa3000-0xbfffffff]
[ 0.212382] e820: reserve RAM buffer [mem 0xbec81000-0xbfffffff]
[ 0.212383] e820: reserve RAM buffer [mem 0xbf000000-0xbfffffff]
[ 0.212384] e820: reserve RAM buffer [mem 0x83e000000-0x83fffffff]
[ 0.266687] pci 0000:03:00.0: Video device with shadowed ROM
[ 0.266730] PCI: CLS 64 bytes, default 64
[ 0.266768] Trying to unpack rootfs image as initramfs...
[ 2.132383] Freeing initrd memory: 156660K (ffff880024df6000 - ffff88002e6f3000)
[ 2.132421] DMAR: No ATSR found
[ 2.132438] IOMMU 0 0xfed90000: using Queued invalidation
[ 2.132439] IOMMU: Setting RMRR:
[ 2.132450] IOMMU: Setting identity map for device 0000:00:14.0 [0xbe51f000 - 0xbe53bfff]
[ 2.132468] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbe51f000 - 0xbe53bfff]
[ 2.132481] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbe51f000 - 0xbe53bfff]
[ 2.132489] IOMMU: Prepare 0-16MiB unity mapping for LPC
[ 2.132494] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[ 2.132571] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
[ 2.787575] pci-stub: add 10DE:1200 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 2.787587] pci-stub 0000:05:00.0: claimed by stub
[ 2.787593] pci-stub: add 10DE:0E0C sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 2.787597] pci-stub 0000:05:00.1: claimed by stub
[ 3.495732] VFIO - User Level meta-driver version: 0.3
[ 3.529871] FS-Cache: Loaded
[ 3.692679] RPC: Registered named UNIX socket transport module.
[ 3.692682] RPC: Registered udp transport module.
[ 3.692683] RPC: Registered tcp transport module.
[ 3.692684] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 3.722255] ppdev: user-space parallel port driver
[ 3.731469] parport_pc 00:05: reported by Plug and Play ACPI
[ 3.731516] parport0: PC-style at 0x378, irq 5 [PCSPP]
[ 3.816825] lp0: using parport0 (interrupt-driven).
[ 3.821640] wmi: Mapper loaded
[ 3.866631] init: avahi-cups-reload main process (565) terminated with status 1
[ 3.878995] ACPI Warning: SystemIO range 0x0000000000000428-0x000000000000042f conflicts with OpRegion 0x0000000000000400-0x000000000000047f (\PMIO) (20140926/utaddress-258)
[ 3.879000] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[ 3.879003] ACPI Warning: SystemIO range 0x0000000000000540-0x000000000000054f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20140926/utaddress-258)
[ 3.879005] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[ 3.879006] ACPI Warning: SystemIO range 0x0000000000000530-0x000000000000053f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20140926/utaddress-258)
[ 3.879007] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[ 3.879008] ACPI Warning: SystemIO range 0x0000000000000500-0x000000000000052f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20140926/utaddress-258)
[ 3.879010] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[ 3.879010] lpc_ich: Resource conflict(s) found affecting gpio_ich
[ 3.962690] device-mapper: multipath: version 1.7.0 loaded
[ 3.962999] mei_me 0000:00:16.0: irq 34 for MSI/MSI-X
[ 3.977288] AVX version of gcm_enc/dec engaged.
[ 3.977290] AES CTR mode by8 optimization enabled
[ 4.182963] snd_hda_intel 0000:00:1b.0: irq 35 for MSI/MSI-X
[ 4.183050] snd_hda_intel 0000:03:00.1: Handle VGA-switcheroo audio client
[ 4.183071] snd_hda_intel 0000:03:00.1: irq 36 for MSI/MSI-X
[ 4.202764] input: HDA ATI HDMI HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:03:00.1/sound/card1/input5
[ 4.221910] sound hdaudioC0D0: autoconfig: line_outs=1 (0x14/0x0/0x0/0x0/0x0) type:line
[ 4.221914] sound hdaudioC0D0: speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[ 4.221915] sound hdaudioC0D0: hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)
[ 4.221916] sound hdaudioC0D0: mono: mono_out=0x0
[ 4.221917] sound hdaudioC0D0: dig-out=0x11/0x0
[ 4.221918] sound hdaudioC0D0: inputs:
[ 4.221919] sound hdaudioC0D0: Front Mic=0x19
[ 4.221921] sound hdaudioC0D0: Rear Mic=0x18
[ 4.221922] sound hdaudioC0D0: Line=0x1a
[ 4.234307] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input6
[ 4.234879] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input7
[ 4.234930] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1b.0/sound/card0/input8
[ 4.235658] input: HDA Intel PCH Line Out as /devices/pci0000:00/0000:00:1b.0/sound/card0/input9
[ 4.236186] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card0/input10
[ 4.237295] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 4.258359] init: Failed to obtain startpar-bridge instance: Unknown parameter: INSTANCE
[ 4.291101] r8169 0000:06:00.0 eth0: link down
[ 4.291120] r8169 0000:06:00.0 eth0: link down
[ 4.291144] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 6.646476] r8169 0000:06:00.0 eth0: link up
[ 6.646484] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 9.469643] init: failsafe main process (690) killed by TERM signal
[ 9.511040] audit_printk_skb: 21 callbacks suppressed
[ 10.109377] init: plymouth-upstart-bridge main process ended, respawning
[ 10.278910] cgroup: systemd-logind (574) created nested cgroup for controller "memory" which has incomplete hierarchy support. Nested cgroups may change behavior in the future.
[ 10.278913] cgroup: "memory" requires setting use_hierarchy to 1 on the root
[ 10.676638] random: nonblocking pool is initialized
[ 162.674652] vfio-pci 0000:05:00.0: enabling device (0000 -> 0003)
[ 166.777854] kvm: zapping shadow pages for mmio generation wraparound
i saw some kvm zapping shadow pages here, and these dmesg show up when its on -vga none..
and when i removed -vga none, and i can enter the VM heres the dmesg :
[ 162.674652] vfio-pci 0000:05:00.0: enabling device (0000 -> 0003)
[ 166.777854] kvm: zapping shadow pages for mmio generation wraparound
[ 262.768505] kvm: zapping shadow pages for mmio generation wraparound
[ 269.342192] kvm [2329]: vcpu0 ignored rdmsr: 0x345
[ 269.342216] kvm [2329]: vcpu0 ignored wrmsr: 0x680 data 0
[ 269.342218] kvm [2329]: vcpu0 ignored wrmsr: 0x681 data 0
[ 269.342219] kvm [2329]: vcpu0 ignored wrmsr: 0x682 data 0
[ 269.342220] kvm [2329]: vcpu0 ignored wrmsr: 0x683 data 0
[ 269.342222] kvm [2329]: vcpu0 ignored wrmsr: 0x684 data 0
[ 269.342223] kvm [2329]: vcpu0 ignored wrmsr: 0x685 data 0
[ 269.342224] kvm [2329]: vcpu0 ignored wrmsr: 0x686 data 0
[ 269.342225] kvm [2329]: vcpu0 ignored wrmsr: 0x687 data 0
[ 269.342227] kvm [2329]: vcpu0 ignored wrmsr: 0x688 data 0
[ 269.342228] kvm [2329]: vcpu0 ignored wrmsr: 0x689 data 0
Right over here i am stuck in order to try a GPU passthrough.. did i miss anything? or did my dmesg shows any incorrect steps that i've been through? And also, why would the -vga none goes black instead of going to the VMs directly..
So would someone help about this case? i've been stuck for two weeks and nothing came up with any good..
Thanks before !

Can someone help me on this please...

Offline

#4138 2015-02-09 15:37:08

aw: Member; Registered: 2013-10-04; Posts: 921; Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hotfunction wrote:

Hi everyone, i've been reading a lot of posts here, and i wanted to try the passthrough by myself, but seemed that i got some error here, and firstly according to the steps of the guide, i have been doing step by steps, and here's my build :
Processor : Xeon E3-1200 (VT-d Enabled in BIOS)
ubuntu display GPU : Radeon HD4870
Passthrough GPU : GeForce GTX 560 Ti Hawx

Step 1, please learn how to use code boxes rather than quote boxes on the list.

So steps i've been done so far :
1. Ubuntu Server 12.04, then do-release-upgrade to 14.04
2. Downloaded Kernel 3.18-5 from Linux homepage
3. Downloaded 3.18-5 linux-mainline from nhbs's first post
4. Patched ACS and i915_317 patch from linux-mainline from nbhs's post
5. Qemu 2.2.0, also Seabios standard and no patches
6. lspci | grep NVIDIA :

05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] [10de:1200] (rev a1)
05:00.1 Audio device [0403]: NVIDIA Corporation GF114 HDMI Audio Controller [10de:0e0c] (rev a1)
7. edited /etc/default/grub :
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c"
and our /proc/cmdline :
BOOT_IMAGE=/vmlinuz-3.18.5 root=/dev/mapper/SSLABVFIO--vg-root ro quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c vt.handoff=7

If you don't have Intel integrated graphics, you don't need the i915 patch. You're not even using the ACS override patch, so you may be able to use a stock kernel.

8. also blacklisted radeon on /etc/modprobe.d/blacklist.conf

Why? Radeon is your host device. How are you going to have host graphics without it?

9. dmesg pci-stub :

[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.18.5 root=/dev/mapper/SSLABVFIO--vg-root ro quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c vt.handoff=7
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.18.5 root=/dev/mapper/SSLABVFIO--vg-root ro quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1200,10de:0e0c vt.handoff=7
[    2.787575] pci-stub: add 10DE:1200 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.787587] pci-stub 0000:05:00.0: claimed by stub
[    2.787593] pci-stub: add 10DE:0E0C sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.787597] pci-stub 0000:05:00.1: claimed by stub

10. i checked lspci -vnn for the graphics cards in the device :

03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] R700 [Radeon HD 4870 X2] [1002:9441] (prog-if 00 [VGA controller])
        Subsystem: ASUSTeK Computer Inc. Device [1043:0284]
        Flags: bus master, fast devsel, latency 0, IRQ 11
        Memory at d0000000 (64-bit, prefetchable) [size=256M]
        Memory at f6220000 (64-bit, non-prefetchable) [size=64K]
        I/O ports at b000 [size=256]
        Expansion ROM at f6200000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>

03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] RV770 HDMI Audio [Radeon HD 4850/4870] [1002:aa30]
        Subsystem: ASUSTeK Computer Inc. Device [1043:aa30]
        Flags: bus master, fast devsel, latency 0, IRQ 36
        Memory at f6230000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Kernel driver in use: snd_hda_intel

04:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] R700 [Radeon HD 4870 X2] [1002:9441]
        Subsystem: ASUSTeK Computer Inc. Device [1043:0284]
        Flags: bus master, fast devsel, latency 0, IRQ 11
        Memory at c0000000 (64-bit, prefetchable) [size=256M]
        Memory at f6120000 (64-bit, non-prefetchable) [size=64K]
        I/O ports at a000 [size=256]
        Expansion ROM at f6100000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>

05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] [10de:1200] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:2601]
        Flags: fast devsel, IRQ 11
        Memory at f4000000 (32-bit, non-prefetchable) [disabled] [size=32M]
        Memory at e0000000 (64-bit, prefetchable) [disabled] [size=128M]
        Memory at e8000000 (64-bit, prefetchable) [disabled] [size=64M]
        I/O ports at e000 [disabled] [size=128]
        Expansion ROM at f6000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [b4] Vendor Specific Information: Len=14 <?>
        Capabilities: [100] Virtual Channel
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Kernel driver in use: pci-stub

05:00.1 Audio device [0403]: NVIDIA Corporation GF114 HDMI Audio Controller [10de:0e0c] (rev a1)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:2601]
        Flags: bus master, fast devsel, latency 0, IRQ 10
        Memory at f6080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Kernel driver in use: pci-stub

See how much nicer that is in a code box?

11. after all went good, i create vfio-pci1.cfg in /etc/vfio-pci1.cfg :
0000:05:00.0
0000:05:00.1
12. Booting VM's Script :
#!/bin/bash

configfile=/etc/vfio-pci1.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id

}

modprobe vfio-pci

cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
        vfiobind $line
done

sudo qemu-system-x86_64 -vga none -M q35 -hda /home/sslab719/VMimages/VM.img -enable-kvm -m 2048 -cpu host,kvm=off \
-smp 2,sockets=1,cores=2,threads=1 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-drive file=/home/sslab719/VMimages/VM.img,id=disk,format=qcow2 \
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/sslab719/MSI.GTX560Ti.1024.110825.rom \
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \
-net user,hostfwd=tcp::10022-:22 -net nic
#-boot menu=on

exit 0
Here at this point, i am always having black QEMU Monitor with "compat_monitor0 console"
Qemu 2.2.0 monitor - type help for more information -- when i'm using -vga none right over here.
If i remove -vga none there, i can boot to the VMs normally, and i can see the lspci went in correctly, but it does seem like it doesn't have the real GPU power... the passthrough went there and it shows up in lspci but i am guessing that theres no GPU clock, shader, and memory power.. I tried to install heaven engine benchmark in linux, and i tried to run it on, and it fails, and here i am guessing that perhaps the problem might be the GPU doesn't really passthrough successfully, but instead its just a name showup on the VMs, and i can still see 00:01.0 VGA compatible Controller: Device 1234:1111, and the others are our passthrough'd GPU 01:00.0..
Here by reading i know that we should add -vga none in order to passthrough it normally, but it seemed that -vga none didn't bring a help for me, and instead its just gave me a weird black qemu monitor which i can type some commands of qemu itself.
i tried -vga std and others, but those didn't worked too..

That black window is the sdl graphics head for the VM which appears because you haven't used the -nographic option. The GPU output should be on a monitor connected to the assigned GPU. I'd also recommend, like I almost always do, using 440fx rather than q35 for the VM. It looks like you're using a Linux guest, but afaik, the nvidia driver does not have an issue with PCIe topology in Linux (until you get to lots of GPUs and that may already be fixed too).

so i checked my dmesg when turning my VM to see whether vfio works correctly, here are some important points at my dmesg:

[    0.210689] vgaarb: setting as boot device: PCI:0000:03:00.0
[    0.210691] vgaarb: device added: PCI:0000:03:00.0,decodes=io+mem,owns=io+mem,locks=none
[    0.210695] vgaarb: device added: PCI:0000:05:00.0,decodes=io+mem,owns=none,locks=none
[    0.210696] vgaarb: loaded
[    0.210697] vgaarb: bridge control possible 0000:05:00.0
[    0.210697] vgaarb: bridge control possible 0000:03:00.0
[    0.210846] SCSI subsystem initialized
[    0.210873] libata version 3.00 loaded.
[    0.210889] ACPI: bus type USB registered
[    0.210902] usbcore: registered new interface driver usbfs
[    0.210908] usbcore: registered new interface driver hub
[    0.210919] usbcore: registered new device driver usb
[    0.211004] PCI: Using ACPI for IRQ routing
[    0.212320] PCI: pci_cache_line_size set to 64 bytes
[    0.212378] e820: reserve RAM buffer [mem 0x0009ec00-0x0009ffff]
[    0.212380] e820: reserve RAM buffer [mem 0xbdf9c000-0xbfffffff]
[    0.212381] e820: reserve RAM buffer [mem 0xbdfa3000-0xbfffffff]
[    0.212382] e820: reserve RAM buffer [mem 0xbec81000-0xbfffffff]
[    0.212383] e820: reserve RAM buffer [mem 0xbf000000-0xbfffffff]
[    0.212384] e820: reserve RAM buffer [mem 0x83e000000-0x83fffffff]
[    0.266687] pci 0000:03:00.0: Video device with shadowed ROM
[    0.266730] PCI: CLS 64 bytes, default 64
[    0.266768] Trying to unpack rootfs image as initramfs...
[    2.132383] Freeing initrd memory: 156660K (ffff880024df6000 - ffff88002e6f3000)
[    2.132421] DMAR: No ATSR found
[    2.132438] IOMMU 0 0xfed90000: using Queued invalidation
[    2.132439] IOMMU: Setting RMRR:
[    2.132450] IOMMU: Setting identity map for device 0000:00:14.0 [0xbe51f000 - 0xbe53bfff]
[    2.132468] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbe51f000 - 0xbe53bfff]
[    2.132481] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbe51f000 - 0xbe53bfff]
[    2.132489] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    2.132494] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    2.132571] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
[    2.787575] pci-stub: add 10DE:1200 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.787587] pci-stub 0000:05:00.0: claimed by stub
[    2.787593] pci-stub: add 10DE:0E0C sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.787597] pci-stub 0000:05:00.1: claimed by stub
[    3.495732] VFIO - User Level meta-driver version: 0.3
[    3.529871] FS-Cache: Loaded
[    3.692679] RPC: Registered named UNIX socket transport module.
[    3.692682] RPC: Registered udp transport module.
[    3.692683] RPC: Registered tcp transport module.
[    3.692684] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    3.722255] ppdev: user-space parallel port driver
[    3.731469] parport_pc 00:05: reported by Plug and Play ACPI
[    3.731516] parport0: PC-style at 0x378, irq 5 [PCSPP]
[    3.816825] lp0: using parport0 (interrupt-driven).
[    3.821640] wmi: Mapper loaded
[    3.866631] init: avahi-cups-reload main process (565) terminated with status 1
[    3.878995] ACPI Warning: SystemIO range 0x0000000000000428-0x000000000000042f conflicts with OpRegion 0x0000000000000400-0x000000000000047f (\PMIO) (20140926/utaddress-258)
[    3.879000] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[    3.879003] ACPI Warning: SystemIO range 0x0000000000000540-0x000000000000054f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20140926/utaddress-258)
[    3.879005] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[    3.879006] ACPI Warning: SystemIO range 0x0000000000000530-0x000000000000053f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20140926/utaddress-258)
[    3.879007] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[    3.879008] ACPI Warning: SystemIO range 0x0000000000000500-0x000000000000052f conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20140926/utaddress-258)
[    3.879010] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[    3.879010] lpc_ich: Resource conflict(s) found affecting gpio_ich
[    3.962690] device-mapper: multipath: version 1.7.0 loaded
[    3.962999] mei_me 0000:00:16.0: irq 34 for MSI/MSI-X
[    3.977288] AVX version of gcm_enc/dec engaged.
[    3.977290] AES CTR mode by8 optimization enabled
[    4.182963] snd_hda_intel 0000:00:1b.0: irq 35 for MSI/MSI-X
[    4.183050] snd_hda_intel 0000:03:00.1: Handle VGA-switcheroo audio client
[    4.183071] snd_hda_intel 0000:03:00.1: irq 36 for MSI/MSI-X
[    4.202764] input: HDA ATI HDMI HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:04.0/0000:03:00.1/sound/card1/input5
[    4.221910] sound hdaudioC0D0: autoconfig: line_outs=1 (0x14/0x0/0x0/0x0/0x0) type:line
[    4.221914] sound hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[    4.221915] sound hdaudioC0D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)
[    4.221916] sound hdaudioC0D0:    mono: mono_out=0x0
[    4.221917] sound hdaudioC0D0:    dig-out=0x11/0x0
[    4.221918] sound hdaudioC0D0:    inputs:
[    4.221919] sound hdaudioC0D0:      Front Mic=0x19
[    4.221921] sound hdaudioC0D0:      Rear Mic=0x18
[    4.221922] sound hdaudioC0D0:      Line=0x1a
[    4.234307] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input6
[    4.234879] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1b.0/sound/card0/input7
[    4.234930] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:1b.0/sound/card0/input8
[    4.235658] input: HDA Intel PCH Line Out as /devices/pci0000:00/0000:00:1b.0/sound/card0/input9
[    4.236186] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card0/input10
[    4.237295] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    4.258359] init: Failed to obtain startpar-bridge instance: Unknown parameter: INSTANCE
[    4.291101] r8169 0000:06:00.0 eth0: link down
[    4.291120] r8169 0000:06:00.0 eth0: link down
[    4.291144] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    6.646476] r8169 0000:06:00.0 eth0: link up
[    6.646484] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    9.469643] init: failsafe main process (690) killed by TERM signal
[    9.511040] audit_printk_skb: 21 callbacks suppressed
[   10.109377] init: plymouth-upstart-bridge main process ended, respawning
[   10.278910] cgroup: systemd-logind (574) created nested cgroup for controller "memory" which has incomplete hierarchy support. Nested cgroups may change behavior in the future.
[   10.278913] cgroup: "memory" requires setting use_hierarchy to 1 on the root
[   10.676638] random: nonblocking pool is initialized
[  162.674652] vfio-pci 0000:05:00.0: enabling device (0000 -> 0003)
[  166.777854] kvm: zapping shadow pages for mmio generation wraparound

i saw some kvm zapping shadow pages here, and these dmesg show up when its on -vga none..

and when i removed -vga none, and i can enter the VM heres the dmesg :

[  162.674652] vfio-pci 0000:05:00.0: enabling device (0000 -> 0003)
[  166.777854] kvm: zapping shadow pages for mmio generation wraparound
[  262.768505] kvm: zapping shadow pages for mmio generation wraparound
[  269.342192] kvm [2329]: vcpu0 ignored rdmsr: 0x345
[  269.342216] kvm [2329]: vcpu0 ignored wrmsr: 0x680 data 0
[  269.342218] kvm [2329]: vcpu0 ignored wrmsr: 0x681 data 0
[  269.342219] kvm [2329]: vcpu0 ignored wrmsr: 0x682 data 0
[  269.342220] kvm [2329]: vcpu0 ignored wrmsr: 0x683 data 0
[  269.342222] kvm [2329]: vcpu0 ignored wrmsr: 0x684 data 0
[  269.342223] kvm [2329]: vcpu0 ignored wrmsr: 0x685 data 0
[  269.342224] kvm [2329]: vcpu0 ignored wrmsr: 0x686 data 0
[  269.342225] kvm [2329]: vcpu0 ignored wrmsr: 0x687 data 0
[  269.342227] kvm [2329]: vcpu0 ignored wrmsr: 0x688 data 0
[  269.342228] kvm [2329]: vcpu0 ignored wrmsr: 0x689 data 0

Nope, nothing particularly important in either of those

Right over here i am stuck in order to try a GPU passthrough.. did i miss anything? or did my dmesg shows any incorrect steps that i've been through? And also, why would the -vga none goes black instead of going to the VMs directly..
So would someone help about this case? i've been stuck for two weeks and nothing came up with any good..
Thanks before !

If 440fx and -nographic don't help and you're looking at the right output from the GPU, I'd next be suspicious of the ROM file. It looks like one from techpowerup. Does it actually match your card? Can you dump the ROM from your card instead? Do you even need a dump of the ROM?

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4139 2015-02-09 16:02:01

The_Moves: Member; Registered: 2015-01-06; Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

If 440fx and -nographic don't help and you're looking at the right output from the GPU, I'd next be suspicious of the ROM file. It looks like one from techpowerup. Does it actually match your card? Can you dump the ROM from your card instead? Do you even need a dump of the ROM?

What is the benefit of adding the "romfile/rom" attribute? Under what conditions should it be provided?

Offline

#4140 2015-02-09 16:16:48

aw: Member; Registered: 2013-10-04; Posts: 921; Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The_Moves wrote:

aw wrote:
If 440fx and -nographic don't help and you're looking at the right output from the GPU, I'd next be suspicious of the ROM file. It looks like one from techpowerup. Does it actually match your card? Can you dump the ROM from your card instead? Do you even need a dump of the ROM?
What is the benefit of adding the "romfile/rom" attribute? Under what conditions should it be provided?

The romfile= option makes QEMU read the ROM from a file rather than the device itself. It should only be used if you have trouble reading the ROM, such as seeing the invalid ROM contents message in dmesg, or if you're replacing the ROM with one that has features the on-device ROM does not, ex. you've hacked on it yourself or you're providing an updated ROM, maybe with UEFI support. The ROM is read from the device once and cached in QEMU, so there shouldn't be any issues with re-reading the physical ROM at VM reset. Also, I feel like I shouldn't need to say this, but if you attempt to update any assigned device ROM from inside the VM, you do so at your own risk.

The rombar= option is used to hide the device ROM if you don't want to use it, for instance if you assign a NIC with a PXE ROM but you don't want to boot from the device.

We've seen some instance of people here using rombar=0,romfile=$FILE. Unless you know exactly what you're doing, this is almost certainly wrong. This says to hide the device ROM BAR, but still provide a ROM, which gets pushed into the legacy ROMs address space, disassociating it from the device. Don't do this.

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4141 2015-02-09 16:26:10

Duelist: Member; Registered: 2014-09-22; Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

The romfile= option makes QEMU read the ROM from a file rather than the device itself.

..a quick distraction..
Is it 100% "accurate"? So if i'll actually flash that ROM on the device, nothing will change for the VM?

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Offline

#4142 2015-02-09 16:32:34

aw: Member; Registered: 2013-10-04; Posts: 921; Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist wrote:

aw wrote:
The romfile= option makes QEMU read the ROM from a file rather than the device itself.
..a quick distraction..
Is it 100% "accurate"? So if i'll actually flash that ROM on the device, nothing will change for the VM?

So long as we can read the ROM from the device, that should be true. QEMU also takes a lazy approach to reading the ROM, not attempting to do so until the guest tries to read it. Theoretically this makes sure that the device has been reset and is in a proper state to access the ROM. I rely on the on-card ROM for most of my devices.

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4143 2015-02-09 19:50:23

apex8: Member; Registered: 2014-03-29; Posts: 60

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apex8 wrote:

aw wrote:
Nothing has changed here, the user always needs to be able to lock enough pages for the VM. You're not going to get very far with a locked memory limit of 40MB. If you use libvirt (and do not hide the vfio device in <qemu:arg>!) this happens automatically. If you run qemu by hand, the easiest solution is to run as root or use sudo.
Ah I remember that. I think I adjusted this via
cat /etc/security/limits.conf | grep @users
@users           soft    memlock         4301000
@users           hard    memlock         4301000
This was fine to run qemu as non root until now..

OK I found and fixed the issue...

cat /etc/security/limits.conf
@users           soft    memlock         4301000
@users           hard    memlock         4301000

*               -       rtprio          0
*               -       nice            0
@audio          -       rtprio          65
@audio          -       nice           -10
@audio          -       memlock         4301000

The memlock for @audio was set to 40MB. I haven't touched this file for month and obviously the last value which was set was taken into account. Still wondering why this broke suddenly...
Thanks for the hints!

Offline

#4144 2015-02-09 20:02:06

DelusionalLogic: Member; Registered: 2013-05-07; Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've gotten it to work, but for some reason Visual studio wont install. Have anyone had any success installing it? It's some error about wpfgfx.dll, which makes me afraid it has something to do with the graphics card.

Offline

#4145 2015-02-09 20:06:51

dwe11er: Member; Registered: 2014-03-18; Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

DelusionalLogic wrote:

I've gotten it to work, but for some reason Visual studio wont install. Have anyone had any success installing it? It's some error about wpfgfx.dll, which makes me afraid it has something to do with the graphics card.

Works for me.

Offline

#4146 2015-02-09 22:18:15

valnare: Member; Registered: 2015-02-09; Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I managed to get my AMD Radeon 290X to work with VGA passtrough. TRIM also works on the virtual disk controller for the SSD after changing the controller type to IDE and AHCI. I wanted to pass a real disk to the VM, so I don't loose the ability to natively boot into Windows.

For playback I am using the HDMI output of my GPU. Unfortunately the audio there stutters while playing sound. The same happens when I try to use the virtualized AC97 output. Did someone else also had problems with the HDMI sound output? Everything else is working perfectly as far as I can tell. Thank you for the great Guide!

Here is the script I am using to start the VM:

export QEMU_PA_SAMPLES=1024
export QEMU_PA_SERVER="127.0.0.1"
export QEMU_AUDIO_DRV=pa
export VMPATH="/home/felix"
export DISK="/dev/disk/by-id/ata-SanDisk_SDSSDHII960G_144340400518"

qemu-system-x86_64 -enable-kvm -m 8192 -cpu Haswell -smp 4,sockets=1,cores=4,threads=1 -vga none \
-mem-path /dev/hugepages -monitor stdio \
-device vfio-pci,host=01:00.0,x-vga=on \
-device vfio-pci,host=01:00.1 \
-net nic -net bridge,br=br0 \
-usb -usbdevice host:045e:0719 -usbdevice host:046d:c07d -usbdevice host:04d9:2013 -usbdevice host:1852:50d0 \
-soundhw ac97 \
-drive id=disk,file=$DISK,if=none,discard=on,detect-zeroes=on -device ahci,id=ahci -device ide-drive,drive=disk,bus=ahci.0

Offline

#4147 2015-02-10 03:00:18

erganzi: Member; Registered: 2014-07-09; Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thanks a lot.

aw wrote:

erganzi wrote:
NVIDIA GRID K2 passthrough successful, but with code 43 after install nvidia driver.
driver info: 347.25-quadro-grid-desktop-notebook-win8-win7-64bit-international-whql
Nvidia K-series GRID, Tesla, and Quadro (2000+) are supported by Nvidia in non-vga passthrough mode. Configure the device as secondary graphics for the VM and do not use the x-vga option.

when I configure the device as secondary graphics for the VM and do not use the x-vga option, the driver report code 12.
like "This device cannot find enough free resources that it can use. If you want to use this device, you will need to disable one of the other devices on this system. (Code 12)"

but if I disable the first device, and reboot it reported code 12 too.

here is my qemu commandline:

#!/bin/sh

qemu-system-x86_64 -cpu host,kvm=off -smp 1,sockets=1,cores=1,threads=1 \
-m 8192 -M q35 -enable-kvm \
-rtc base=localtime,clock=host \
-acpitable file=/var/lib/libvirt/images/LENOVO-TC-90-MSFT-2.1.BIN \
-device virtio-scsi-pci,id=scsi \
-drive file=/var/lib/libvirt/images/win7_nvidia_k2_clean.img,cache=writeback,\
if=none,format=qcow2,aio=native,id=virtio-scsi-disk0 \
-device scsi-hd,drive=virtio-scsi-disk0 \
-net nic,model=virtio,macaddr=52:54:00:1a:2b:3c \
-net tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
-vga std -vnc :3 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on \
-usb -device usb-tablet \
-monitor stdio

Offline

#4148 2015-02-10 03:01:45

aw: Member; Registered: 2013-10-04; Posts: 921; Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

erganzi wrote:

thanks a lot.
aw wrote:
erganzi wrote:
NVIDIA GRID K2 passthrough successful, but with code 43 after install nvidia driver.
driver info: 347.25-quadro-grid-desktop-notebook-win8-win7-64bit-international-whql
Nvidia K-series GRID, Tesla, and Quadro (2000+) are supported by Nvidia in non-vga passthrough mode. Configure the device as secondary graphics for the VM and do not use the x-vga option.
when I configure the device as secondary graphics for the VM and do not use the x-vga option, the driver report code 12.

Use 440fx, not q35

EDIT: You might just be able to move the GPU to the root complex, but behind a root port is known to cause the problem right now.

Last edited by aw (2015-02-10 03:12:46)

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#4149 2015-02-10 05:22:37

hotfunction: Member; Registered: 2015-02-05; Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi and thank you for your reply sir aw !

If you don't have Intel integrated graphics, you don't need the i915 patch. You're not even using the ACS override patch, so you may be able to use a stock kernel.

yes, i checked on my lspci and i think that my graphics card doesn't have IGP on it, so i changed the boot parameter and in cmdline changes into this :

BOOT_IMAGE=/vmlinuz-3.18.5 root=/dev/mapper/SSLABVFIO--vg-root ro quiet splash pcie_acs_override=downstream intel_iommu=on pci-stub.ids=10de:1200,10de:0e0c vt.handoff=7

next, i also cleared the blacklisted radeon on /etc/modprobe.d/blacklist.conf, and i added nvidia, for passthrough instead.

That black window is the sdl graphics head for the VM which appears because you haven't used the -nographic option. The GPU output should be on a monitor connected to the assigned GPU. I'd also recommend, like I almost always do, using 440fx rather than q35 for the VM.

i tried adding -nographic and putting back -vga none on the script, but the session is stuck there, and sometimes it made the whole machine stuck until i had to hard-reboot it. i also tried adding -nographic without -vga none, and gives the same result too.. so i checked using top while having other stuck'd session, and qemu shows up that they own 99.8% of CPU usage.. so i tried to kill the process, and that stucked session went back, so that means actually that -nographic option still boots up the VM, but i can't see any output.. any idea on this?

also as for the 440fx version, i configured my script, and it shows no good, still only black monitor, and if i add -vga none, same black_compat0 monitor still show up like before.. here is my script :

#!/bin/bash

configfile=/etc/vfio-pci1.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id

}

modprobe vfio-pci

cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
        vfiobind $line
done

sudo qemu-system-x86_64 -vga none -hda /home/sslab719/VMimages/VM.img -enable-kvm -m 2048 -cpu host,kvm=off \
-smp 2,sockets=1,cores=2,threads=1 \
-device vfio-pci,host=05:00.0,x-vga=on,romfile=/home/sslab719/MSI.GTX560Ti.1024.110825.rom \
-device vfio-pci,host=05:00.1 \
-net user,hostfwd=tcp::10022-:22 -net nic
#-boot menu=on

exit 0

I'd next be suspicious of the ROM file. It looks like one from techpowerup. Does it actually match your card? Can you dump the ROM from your card instead? Do you even need a dump of the ROM?

yes i am downloading ones from the techpowerup.. i am not sure whether its going to match perfectly, since 560 ti has a lot of variants, im using MSI GTX560 Ti twin frozr 2 version, and i am searching for better accurate rom for the graphics card.
so is that possible that because of the ROM that it brokes the whole process? i dont know why but when the nographic and -vga none added it will crash the pc instead..

Additional :
i tried to keep using q35 version of my script, and i tried to download some benchmark, so i downloaded GpuTest, and once i open it, the whole PC crashes and i had to hard reboot it, is it because of CPU usage that made the PC overloads until it had crash, or is it a stuck because of GPU operation fails?

Last edited by hotfunction (2015-02-10 15:41:13)

Offline

#4150 2015-02-10 07:13:43

erganzi: Member; Registered: 2014-07-09; Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

erganzi wrote:
thanks a lot.
aw wrote:
Nvidia K-series GRID, Tesla, and Quadro (2000+) are supported by Nvidia in non-vga passthrough mode. Configure the device as secondary graphics for the VM and do not use the x-vga option.
when I configure the device as secondary graphics for the VM and do not use the x-vga option, the driver report code 12.
Use 440fx, not q35
EDIT: You might just be able to move the GPU to the root complex, but behind a root port is known to cause the problem right now.

when I changed to i440fx, the driver report code 43.

my qemu commandline:

qemu-system-x86_64 -cpu SandyBridge,kvm=off -smp 1,sockets=1,cores=1,threads=1 \
-m 8192 -enable-kvm \
-rtc base=localtime,clock=host \
-acpitable file=/var/lib/libvirt/images/LENOVO-TC-90-MSFT-2.1.BIN \
-device virtio-scsi-pci,id=scsi \
-drive file=/var/lib/libvirt/images/win7_nvidia_k2_clean.img,cache=writeback,\
if=none,format=qcow2,aio=native,id=virtio-scsi-disk0 \
-device scsi-hd,drive=virtio-scsi-disk0 \
-net nic,model=virtio,macaddr=52:54:00:1a:2b:3c \
-net tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
-vga std -vnc :3 \
-device vfio-pci,host=04:00.0,addr=05.0,multifunction=on \
-usb -device usb-tablet \
-monitor stdio

Offline

Pages: Previous 1 … 164 165 166 167 168 … 234 Next

Topic closed

Board footer

Atom topic feed