You are not logged in.
aw wrote:It's possible you need the "other" VGA arbiter patch listed in question 4 of the FAQ. That bug in arbitration was found with radeon, but if nouveau behaves similarly, the arbiter might be getting stuck with VGA routing enabled to the wrong device.
Thanks. Will try that, but the same thing happens when the host is running the nvidia drivers.
With the nvidia driver the problem is likely different, see FAQ question 9 and the link in the comment. The nvidia driver takes the vga lock and never releases it, so the monitor stays blank in that case because the guest is stalled waiting for VGA access.
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
@ansa89 you could try to remove that bios-256k option and u don't need gpu romfile option... and if u edit that script when adding gpu, care for spaces / new lines... that could be reason for not booting
Now I'm using this script:
# Bind VGA
DEVICES="0000:01:00.0 0000:01:00.1"
for dev in $DEVICES ; do
vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
device=$(cat /sys/bus/pci/devices/$dev/device)
if [ -e /sys/bus/pci/devices/$dev/driver ]; then
echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
fi
echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done
# Start QEMU
qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,kvm=off \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-rtc base=localtime \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-drive file=/dev/sdc,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
-boot menu=on \
-netdev tap,ifname=qemu0,id=qemu_tap -device e1000,netdev=qemu_tap,mac=00:16:3E:12:34:56 \
-usb -usbdevice tabletThe script is ok, since if I change "-vga none" to "-vga std" everything works (so the problem is due to GTX 650 being the primary VGA device).
Moreover you said "not booting", but this isn't correct: the VM boots, but the monitor stays black and I can't connect to VNC server inside VM (I'm enough sure about this because I can arping the VM ip).
Last edited by Ansa89 (2014-09-01 09:42:04)
Offline
Cenm wrote:aw wrote:It's possible you need the "other" VGA arbiter patch listed in question 4 of the FAQ. That bug in arbitration was found with radeon, but if nouveau behaves similarly, the arbiter might be getting stuck with VGA routing enabled to the wrong device.
Thanks. Will try that, but the same thing happens when the host is running the nvidia drivers.
With the nvidia driver the problem is likely different, see FAQ question 9 and the link in the comment. The nvidia driver takes the vga lock and never releases it, so the monitor stays blank in that case because the guest is stalled waiting for VGA access.
Thanks again.
Which host driver would you suggest? I see some reports of instability with nouveau, is that still the case?
Offline
aw wrote:Cenm wrote:Thanks. Will try that, but the same thing happens when the host is running the nvidia drivers.
With the nvidia driver the problem is likely different, see FAQ question 9 and the link in the comment. The nvidia driver takes the vga lock and never releases it, so the monitor stays blank in that case because the guest is stalled waiting for VGA access.
Thanks again.
Which host driver would you suggest? I see some reports of instability with nouveau, is that still the case?
Since I have no idea what you use the host graphics for, use whatever meets your needs. I prefer to use open source drivers where I can, but with graphics cards you may only get full performance and functionality with proprietary drivers.
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
So Ive been experiencing video stuttering when testing games and tried disabling nested page tables to see if that helped, but my system has an intel processor so this didnt seem to do anything.
I found the option for the intel equivelant (extended page tables) which seems to be kvm-intel.ept=0
I and added that to my /etc/modprobe.d/intel-kvm.conf.
With extended page tables disabled my windows guest crashes during boot.
Is that not how I would do the equivalent of disabling nested page tables on an intel processor?
Has anyone done this with an intel processor in an attempt to decrease stutter?
Any ideas where to go from here?
Last edited by sitonapanotis (2014-09-03 02:59:16)
Offline
So Ive been experiencing video stuttering when testing games and tried disabling nested page tables to see if that helped, but my system has an intel processor so this didnt seem to do anything.
I found the option for the intel equivelant (extended page tables) which seems to be kvm-intel.ept=0
I and added that to my /etc/modprobe.d/intel-kvm.conf.
With extended page tables disabled my windows guest crashes during boot.
Is that not how I would do the equivalent of disabling nested page tables on an intel processor?Has anyone done this with an intel processor in an atempt to decrease stutter?
Any ideas where to go from here?
Nested paging typically provides a significant performance improvement, so I'm not sure why you'd want to disable it. Some people do claim that disabling nested guest support improves performance, but (a) I haven't seen substantiated evidence of that and (b) nested guests != nested paging.
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
sitonapanotis wrote:So Ive been experiencing video stuttering when testing games and tried disabling nested page tables to see if that helped, but my system has an intel processor so this didnt seem to do anything.
I found the option for the intel equivelant (extended page tables) which seems to be kvm-intel.ept=0
I and added that to my /etc/modprobe.d/intel-kvm.conf.
With extended page tables disabled my windows guest crashes during boot.
Is that not how I would do the equivalent of disabling nested page tables on an intel processor?Has anyone done this with an intel processor in an atempt to decrease stutter?
Any ideas where to go from here?Nested paging typically provides a significant performance improvement, so I'm not sure why you'd want to disable it. Some people do claim that disabling nested guest support improves performance, but (a) I haven't seen substantiated evidence of that and (b) nested guests != nested paging.
Ah I was only trying it because OP suggested it, but what I have read about it does seem to make what you are saying make more sense.
Any other ideas for troubleshooting latency/stuttering issues?
edit: in windows LatencyMon is reporting some pretty high latency http://i.imgur.com/G299o61.jpg
Last edited by sitonapanotis (2014-09-03 03:23:55)
Offline
Hi,
I've been using vfio-vga passthrough for quite some time now with my Radaeon R9 290 and everything has been working really great.
However after seeing aw's blog, I wanted to try to get libvirt working and then move to an OVMF setup but I'm having a few issues which I can't seem to resolve.
I have successfully setup 1 ubuntu guest using libvirt and virsh but I can't get my Windows8 guest, which has a Radeon R9 290 passed through to it to start. I'm getting the following:
[root@i7-4770s ~]# virsh start windows8
error: Failed to start domain windows8
error: internal error: early end of file from monitor: possible problem:
2014-09-03T13:07:19.859907Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f256 220c0c0, 0x0, 0x8be00000, 0x7f24b4200000) = -12 (Cannot allocate memory)
2014-09-03T13:07:19.859970Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listene r initialization failed for container
2014-09-03T13:07:19.859982Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setu p container for group 1
2014-09-03T13:07:19.860017Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1
2014-09-03T13:07:19.860031Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initializatio n failed.
2014-09-03T13:07:19.860045Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,
I see some other posts on here about the same issue, but that seems to be when they wish to run start the guest as another user other than root. Regardless I have tried changing my ulimits but with no effect:
[root@i7-4770s ~]# ulimit -l
8650752
I have tried changing the RAM allocated to the Guest to 2GB from 6GB to see if there is any difference but no luck also.
Any suggestions please?
I'm running Arch Linux with the Kernel from the OP (3.15.0-1-mainline)
Thanks
Graham
Last edited by gneville (2014-09-03 13:15:00)
Offline
Just spotted the following in dmseg, need to investigate further:
[ 49.781160] qemu-system-x86: sending ioctl 5326 to a partition!
[ 49.781168] qemu-system-x86: sending ioctl 80200204 to a partition!
[ 49.781572] qemu-system-x86: sending ioctl 5326 to a partition!
[ 49.781577] qemu-system-x86: sending ioctl 80200204 to a partition!
[ 49.940979] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
[ 49.940993] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
Hi,
I've been using vfio-vga passthrough for quite some time now with my Radaeon R9 290 and everything has been working really great.
However after seeing aw's blog, I wanted to try to get libvirt working and then move to an OVMF setup but I'm having a few issues which I can't seem to resolve.
I have successfully setup 1 ubuntu guest using libvirt and virsh but I can't get my Windows8 guest, which has a Radeon R9 290 passed through to it to start. I'm getting the following:
[root@i7-4770s ~]# virsh start windows8
error: Failed to start domain windows8
error: internal error: early end of file from monitor: possible problem:
2014-09-03T13:07:19.859907Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f256 220c0c0, 0x0, 0x8be00000, 0x7f24b4200000) = -12 (Cannot allocate memory)
2014-09-03T13:07:19.859970Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listene r initialization failed for container
2014-09-03T13:07:19.859982Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setu p container for group 1
2014-09-03T13:07:19.860017Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1
2014-09-03T13:07:19.860031Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initializatio n failed.
2014-09-03T13:07:19.860045Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,I see some other posts on here about the same issue, but that seems to be when they wish to run start the guest as another user other than root. Regardless I have tried changing my ulimits but with no effect:
[root@i7-4770s ~]# ulimit -l
8650752I have tried changing the RAM allocated to the Guest to 2GB from 6GB to see if there is any difference but no luck also.
Any suggestions please?
I'm running Arch Linux with the Kernel from the OP (3.15.0-1-mainline)
Thanks
Graham
Offline
aw wrote:sitonapanotis wrote:So Ive been experiencing video stuttering when testing games and tried disabling nested page tables to see if that helped, but my system has an intel processor so this didnt seem to do anything.
I found the option for the intel equivelant (extended page tables) which seems to be kvm-intel.ept=0
I and added that to my /etc/modprobe.d/intel-kvm.conf.
With extended page tables disabled my windows guest crashes during boot.
Is that not how I would do the equivalent of disabling nested page tables on an intel processor?Has anyone done this with an intel processor in an atempt to decrease stutter?
Any ideas where to go from here?Nested paging typically provides a significant performance improvement, so I'm not sure why you'd want to disable it. Some people do claim that disabling nested guest support improves performance, but (a) I haven't seen substantiated evidence of that and (b) nested guests != nested paging.
Ah I was only trying it because OP suggested it, but what I have read about it does seem to make what you are saying make more sense.
Any other ideas for troubleshooting latency/stuttering issues?edit: in windows LatencyMon is reporting some pretty high latency http://i.imgur.com/G299o61.jpg
main idea: explore and investigate yourself. the functionality in question is experimental after all. plus, it's the first time i hear about video stutter at all (maybe last few months i was not paying enough attention for all the reading i did, but still) so it may be not that common of a problem for you to wait for some outside help.
just try everything you can think of, including weird (==seemingly unrelated) stuff like playing around with audio settings (this, along with switch to hugetlbfs for memory, resolved my fps problems) or cpu isolation (run vm on cores NOT used by host at all)
Offline
Any other ideas for troubleshooting latency/stuttering issues?
edit: in windows LatencyMon is reporting some pretty high latency http://i.imgur.com/G299o61.jpg
Have a look at this post https://bbs.archlinux.org/viewtopic.php … 0#p1285910. If using Windows 8.x turning off dynticks could help, also verify whether your host kernel is preemptible (should have CONFIG_PREEMPT set).
Offline
I'm using this for two guests and one guest (Linux guest with nVidia card) brings down the host on guest reboot.
Asrock Z97 Extreme6 mainboard
i4790K
i915 for host
AMD HD7950 for Win8.1 guest
nVidia GT 630 for Linux guest
on kernel 3.16 and qemu 2.1.0 with i915 and ACS override patches applied.
Boot parameters are
BOOT_IMAGE=/vmlinuz-3.16.1-vfio-i915-acs-vol-7.g90bc0f1-desktop root=/dev/mapper/vg1-root resume=/dev/vg1/swap splash=silent quiet showopts intel_iommu=on i915.enable_hd_vgaarb=1 pcie_acs_override=downstream pci-stub.ids=1002:679a,1002:aaa0,10de:0f00,10de:0bea,8086:8ca0 modeset.nouveau=0 modeset.radeon=0The devices I'm passing through are
00:1b.0 Audio device [0403]: Intel Corporation Device [8086:8ca0]
Subsystem: ASRock Incorporation Device [1849:1151]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti PRO [Radeon HD 7950] [1002:679a]
Subsystem: Gigabyte Technology Co., Ltd Device [1458:254c]
Kernel driver in use: vfio-pci
Kernel modules: radeon
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] [1002:aaa0]
Subsystem: Gigabyte Technology Co., Ltd Device [1458:aaa0]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF108 [GeForce GT 630] [10de:0f00] (rev a1)
Subsystem: ASUSTeK Computer Inc. Device [1043:8498]
Kernel driver in use: vfio-pci
Kernel modules: nouveau
02:00.1 Audio device [0403]: NVIDIA Corporation GF108 High Definition Audio Controller [10de:0bea] (rev a1)
Subsystem: ASUSTeK Computer Inc. Device [1043:8498]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intelor the two graphics cards and the on-board sound (for the Win8.1 guest).
The Win8.1 guest with the AMD 7950 works perfectly. I was using an AMD 5450 for the second guest running Linux as well but was having the host crash on reboot with the radeon open source drivers, as well as low, stuttering desktop performance when using Cinnamon as DE (no issues with XFCE or LXDE), both with open radeon and fglrx drivers.
The AMD 5450 was swapped out for a nVidia 630 and this resolved the Cinnamon desktop performance issue, however when rebooting the guest this will again lock up the host (hard reset needed).
QEMU is invoked as
/usr/bin/qemu-system-x86_64 -name vm-linux -enable-kvm -rtc base=utc \
-M q35 -m 4096 -mem-path /dev/hugepages \
-smp sockets=1,cores=4 -cpu host \
-vga none -nographic -monitor stdio -serial none -parallel none -nodefconfig -boot menu=off \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=0000:02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=0000:02:00.1,bus=root.1,addr=00.1 -device virtio-scsi-pci,id=scsi \
-device ahci,bus=pcie.0,id=ahci -drive file=/dev/vg2/linux,id=disk,format=raw,cache=none,aio=threads,discard=unmap \
-device scsi-hd,drive=disk -drive file=/home/maarten/Downloads/openSUSE-13.1-DVD-x86_64.iso,id=isocd
-device ide-cd,bus=ahci.1,drive=isocd
-net nic,model=virtio,macaddr=52:54:00:6b:04:1d -net bridge,br=br0I don't have this issue with the Windows guest at any point. I don't know if this is an issue resolved with the "other" VGA arbiter patch on https://lkml.org/lkml/2014/5/25/94, I do have the gfx card output on screen but I don't have complete understanding of what's going on here so I'm not certain. Thanks.
edit: I can shutdown the Linux guest without having the host hang but the guest cannot be started again:
(qemu) qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:02:00.0
Device option ROM contents are probably invalid (check dmesg).
Skip option ROM probe with rombar=0, or load from file with romfile=Last edited by siddharta (2014-09-03 18:52:42)
Offline
Hi,
I've been following this thread for a while now, and a few months ago tried to passthrough my 2 NVIDIA Graphics cards, with the same results I am getting now, but this time I am determined to get it to work.
My specs are.
Asus X79-Deluxe
Intel 4930k
64 GB Ram
2GB NVIDIA GeForce GTX 770 (10de:1184) (Primary 16x Slot)
1GB NVIDIA GeForce GTX550Ti (10de:1244) (Secondary 16x Slot)
I am running the latest linux-mainline kernel from the #1 post with all patches, along with qemu-git, as of about 3hours ago. As I have no built in GPU with the processor, I am finding it very hard to release the primary GPU after booting. Here is my info and then I will get to what it is and isn't doing.
lspci
01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
02:00.0 VGA compatible controller: NVIDIA Corporation GF116 [GeForce GTX 550 Ti] (rev a1)
02:00.1 Audio device: NVIDIA Corporation GF116 High Definition Audio Controller (rev a1)
lspci -n
01:00.0 0300: 10de:1184 (rev a1)
01:00.1 0403: 10de:0e0a (rev a1)
02:00.0 0300: 10de:1244 (rev a1)
02:00.1 0403: 10de:0bee (rev a1) Kernel command line: initrd=\initramfs-linux-mainline.img root=/dev/sda2 rw video=efifb:off intel_iommu=on pci-stub.ids=10de:1184,10de:0e0a,10de:1244,10de:0bee
[ 0.950144] pci-stub: add 10DE:1184 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 0.950164] pci-stub 0000:01:00.0: claimed by stub
[ 0.950182] pci-stub: add 10DE:0E0A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 0.950194] pci-stub 0000:01:00.1: claimed by stub
[ 0.950203] pci-stub: add 10DE:1244 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 0.950214] pci-stub 0000:02:00.0: claimed by stub
[ 0.950226] pci-stub: add 10DE:0BEE sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 0.950237] pci-stub 0000:02:00.1: claimed by stubKernel Arguments (efifb:off to turn framebuffer off)
root=/dev/sda2 rw video=efifb:off intel_iommu=on pci-stub.ids=10de:1184,10de:0e0a,10de:1244,10de:0bee
vfio-bind
0000:01:00.0 0000:01:00.1 0000:02:00.0 0000:02.1
qemu (executed over ssh)
qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/root/GK104.rom \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -nographic
qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 -nographic
blacklist nouveauCurrently I can use device 02:00.0 absolutely fine, but if I try and use 01:00.0, I don't get any errors, just the monitor goes to sleep and this is in dmesg
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900I can still exit qemu fine, no lockups etc...
I have tried changing the cards around, and moving them to slower PCI slots, the GPU started by POST is the one that refuses to pass through. Both cards will pass through individually, providing they are not the first card. and I only get an Invalid ROM message when the GTX770 is in Slot 1, not in any other slot, I have verified my BIOS Dump with rom-parser, everything checks out.
When passing through the second card (in any PCI Slot, Either of the cards) I get the following in DMESG
vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)
vfio-pci 0000:02:00.1: enabling device (0000 -> 0002)But whenever I try and passthrough the first card, I get this
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)The VGA part of the first card never gets enabled.
All todays testing was done on a fresh install with only the minimum for this installed, no xorg or any sort of graphics drivers.
I believe something is grabbing the primary card, but I have exhausted most of the search results related to passing through your only graphics cards without leaving one for the host.
Last edited by friedcpu (2014-09-04 08:10:42)
Offline
aw wrote:sitonapanotis wrote:So Ive been experiencing video stuttering when testing games and tried disabling nested page tables to see if that helped, but my system has an intel processor so this didnt seem to do anything.
I found the option for the intel equivelant (extended page tables) which seems to be kvm-intel.ept=0
I and added that to my /etc/modprobe.d/intel-kvm.conf.
With extended page tables disabled my windows guest crashes during boot.
Is that not how I would do the equivalent of disabling nested page tables on an intel processor?Has anyone done this with an intel processor in an atempt to decrease stutter?
Any ideas where to go from here?Nested paging typically provides a significant performance improvement, so I'm not sure why you'd want to disable it. Some people do claim that disabling nested guest support improves performance, but (a) I haven't seen substantiated evidence of that and (b) nested guests != nested paging.
Ah I was only trying it because OP suggested it, but what I have read about it does seem to make what you are saying make more sense.
Any other ideas for troubleshooting latency/stuttering issues?edit: in windows LatencyMon is reporting some pretty high latency http://i.imgur.com/G299o61.jpg
- huge pages
- vCPU pinning
- don't over commit CPUs
- try to get your assigned devices using MSI
- ???
Please share your findings.
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
Just spotted the following in dmseg, need to investigate further:
[ 49.781160] qemu-system-x86: sending ioctl 5326 to a partition!
[ 49.781168] qemu-system-x86: sending ioctl 80200204 to a partition!
[ 49.781572] qemu-system-x86: sending ioctl 5326 to a partition!
[ 49.781577] qemu-system-x86: sending ioctl 80200204 to a partition!
[ 49.940979] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
[ 49.940993] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
What version of libvirt are you using? Seems like it doesn't fully support vfio--based device assignment. libvirt should be setting the locked memory limit for the vm process to a sufficient value to pin all of guest memory.
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
Thanks for the reply aw, this was totally my fault! I didn't add "clear_emulator_capabilities = 0" to /etc/libvirt/qemu.conf.
I am running libvirt 1.2.9.
I have successfully booted my Windows8 guest with the Radeon R9 290 passed through using libvirt now using a Q35 machine.
However when I try to define the OVMF Guest I get the following message:
error: Failed to define domain from /root/windows-ovmf-virsh-xml.xml
error: internal error: PCI bus is not compatible with the device at 0000:00:01.0. Device requires a standard PCI slot, which is not provided by bus 0000:00
My XML contains:
<type arch='i686' machine='pc'>hvm</type>
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
</controller><qemu:commandline>
<qemu:arg value='-no-hpet'/>
<qemu:arg value='-drive'/>
<qemu:arg value='if=pflash,format=raw,readonly,file=/home/xen/vfio/OVMF.fd'/>
<qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
<qemu:arg value='-device'/>
<qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
<qemu:arg value='-device'/>
<qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/>
<qemu:arg value='-device'/>
<qemu:arg value='ahci,bus=pcie.0,id=ahci'/>
</qemu:commandline>
I've tried different varients of configuration but not having much luck, how does your xml look?
gneville wrote:Just spotted the following in dmseg, need to investigate further:
[ 49.781160] qemu-system-x86: sending ioctl 5326 to a partition!
[ 49.781168] qemu-system-x86: sending ioctl 80200204 to a partition!
[ 49.781572] qemu-system-x86: sending ioctl 5326 to a partition!
[ 49.781577] qemu-system-x86: sending ioctl 80200204 to a partition!
[ 49.940979] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
[ 49.940993] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceededWhat version of libvirt are you using? Seems like it doesn't fully support vfio--based device assignment. libvirt should be setting the locked memory limit for the vm process to a sufficient value to pin all of guest memory.
Thanks
Graham
Last edited by gneville (2014-09-04 15:13:46)
Offline
Thanks for the reply aw, this was totally my fault! I didn't add "clear_emulator_capabilities = 0" to /etc/libvirt/qemu.conf.
I am running libvirt 1.2.9.
I have successfully booted my Windows8 guest with the Radeon R9 290 passed through using libvirt now using a Q35 machine.
However when I try to define the OVMF Guest I get the following message:
error: Failed to define domain from /root/windows-ovmf-virsh-xml.xml
error: internal error: PCI bus is not compatible with the device at 0000:00:01.0. Device requires a standard PCI slot, which is not provided by bus 0000:00My XML contains:
<type arch='i686' machine='pc'>hvm</type>
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
</controller><qemu:commandline>
<qemu:arg value='-no-hpet'/>
<qemu:arg value='-drive'/>
<qemu:arg value='if=pflash,format=raw,readonly,file=/home/xen/vfio/OVMF.fd'/>
<qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
<qemu:arg value='-device'/>
<qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
<qemu:arg value='-device'/>
<qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/>
<qemu:arg value='-device'/>
<qemu:arg value='ahci,bus=pcie.0,id=ahci'/>
</qemu:commandline>I've tried different varients of configuration but not having much luck, how does your xml look?
Please re-read the blog post, especially the bit about OVMF not supporting Q35. If you assign the GPU with a hostdev definition (which you can do when you don't need the x-vga=on option, which you don't need when using OVMF), then you don't need any special libvirt configurations (like retaining capabilities, running as root, or defining ACLs for /dev/vfio/foo).
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
Thanks again, aw. I'll re-read it.
I did change the PC to:
<type arch='i686' machine='pc'>hvm</type>
I'm guessing the config will be something like this then, I'll give it a go when I get home:
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</hostdev>
Please re-read the blog post, especially the bit about OVMF not supporting Q35. If you assign the GPU with a hostdev definition (which you can do when you don't need the x-vga=on option, which you don't need when using OVMF), then you don't need any special libvirt configurations (like retaining capabilities, running as root, or defining ACLs for /dev/vfio/foo).
Offline
Thanks again, aw. I'll re-read it.
I did change the PC to:
<type arch='i686' machine='pc'>hvm</type>
I'm guessing the config will be something like this then, I'll give it a go when I get home:
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</hostdev>
Yes, exactly. My full xml looks like this:
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>Windows8-OVMF</name>
<uuid>98dd8584-aa6a-49c3-b19e-8cbb942d5261</uuid>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static'>2</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='2'/>
<vcpupin vcpu='1' cpuset='3'/>
</cputune>
<os>
<type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
</hyperv>
<kvm>
<hidden state='on'/>
</kvm>
</features>
<cpu mode='custom' match='exact'>
<model fallback='allow'>SandyBridge</model>
<topology sockets='1' cores='2' threads='1'/>
</cpu>
<clock offset='localtime'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/home/alwillia/local/bin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/mnt/store/vm/win8.ovmf'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</disk>
<controller type='pci' index='0' model='pci-root'/>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
</controller>
<interface type='bridge'>
<mac address='02:12:34:56:78:92'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<input type='tablet' bus='usb'/>
<sound model='ich6'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</sound>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</hostdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</memballoon>
</devices>
<qemu:commandline>
<qemu:arg value='-drive'/>
<qemu:arg value='if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd'/>
</qemu:commandline>
</domain>http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
edit: Rebooting the Linux guest will lock up the host. I can shutdown the Linux guest without having the host hang but the guest cannot then be started again:
(qemu) qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:02:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile=
In trying to resolve this I've been going through this thread again as I had diagnonally read most posts involving nVidia cards (didn't own one until now). I dumped the ROM of the GT 630 from within the guest with the nVidia gpu using
echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom
cat /sys/bus/pci/devices/0000:01:00.0/rom > asus-gt630-dcsl-2gd3-v2-vbios.rom
echo 0 > /sys/bus/pci/devices/0000:01:00.0/romand added this to the QEMU invocation as
-device vfio-pci,host=$GPU,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/user/asus-gt630-dcsl-2gd3-v2-vbios.rombut no success - host still hangs on guest reboot, same as without romfile specified.
The other suggestion in the error message, adding rombar=0, doesn't work either. - there is no display output when specifying this. I didn't try both together as a previous post indicated this doesn't make sense. I don't find another way forward in this thread unless I missed something. Any help would be very much appreciated, this is really bugging me.
A second issue I'm seeing is when passing through a USB controller
### Group 25 ###
0b:00.0 USB controller: ASMedia Technology Inc. Device 1142that is bound to pci-stub at boot
0b:00.0 USB controller [0c03]: ASMedia Technology Inc. Device [1b21:1142]
Subsystem: ASRock Incorporation Device [1849:1142]
Kernel driver in use: pci-stub
Kernel modules: xhci_hcd. Using
-device vfio-pci,host=$USB,bus=pcie.0in the QEMU command results in dmesg
[2014-09-04T17:45:38.943613+01:00 kvmhost-3 kernel: [ 1553.980492] dmar: DRHD: handling fault status reg 2
2014-09-04T17:45:38.943638+01:00 kvmhost-3 kernel: [ 1553.980504] dmar: DMAR:[DMA Read] Request device [0b:00.0] fault addr ee000
2014-09-04T17:45:38.943642+01:00 kvmhost-3 kernel: [ 1553.980504] DMAR:[fault reason 06] PTE Read access is not setand nothing beyond that. The guest doesn't see the USB controller.
I'm attempting to passthrough the whole controller because passing a single usb device (regardless of the device type and port the device is attached to) gives a stream of errors as
2014-09-04T10:09:31.794293+01:00 kvmhost-3 kernel: [ 153.883612] xhci_hcd 0000:00:14.0: Command completion event does not match command
2014-09-04T10:09:31.794311+01:00 kvmhost-3 kernel: [ 153.883633] xhci_hcd 0000:00:14.0: Timeout while waiting for setup device command
2014-09-04T10:09:37.010279+01:00 kvmhost-3 kernel: [ 159.099819] xhci_hcd 0000:00:14.0: Timeout while waiting for setup device command
2014-09-04T10:09:37.211259+01:00 kvmhost-3 kernel: [ 159.300820] usb 3-9.1: device not accepting address 4, error -62
2014-09-04T10:09:37.291256+01:00 kvmhost-3 kernel: [ 159.380909] usb 3-9.1: reset full-speed USB device number 4 using xhci_hcd
2014-09-04T10:09:37.291260+01:00 kvmhost-3 kernel: [ 159.381010] usb 3-9.1: Device not responding to setup address.
2014-09-04T10:09:37.492268+01:00 kvmhost-3 kernel: [ 159.581889] usb 3-9.1: Device not responding to setup address.
2014-09-04T10:09:37.693274+01:00 kvmhost-3 kernel: [ 159.782852] usb 3-9.1: device not accepting address 4, error -71
2014-09-04T10:09:37.693287+01:00 kvmhost-3 kernel: [ 159.783255] usb 3-9.1: USB disconnect, device number 4
2014-09-04T10:09:37.693288+01:00 kvmhost-3 kernel: [ 159.783347] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff880444cff940
2014-09-04T10:09:37.774261+01:00 kvmhost-3 kernel: [ 159.863962] usb 3-9.1: new full-speed USB device number 6 using xhci_hcd
2014-09-04T10:09:37.855287+01:00 kvmhost-3 kernel: [ 159.944987] usb 3-9.1: device descriptor read/64, error -32
2014-09-04T10:09:37.956285+01:00 kvmhost-3 kernel: [ 160.045876] xhci_hcd 0000:00:14.0: Setup ERROR: setup context command for slot 6.
2014-09-04T10:09:37.956292+01:00 kvmhost-3 kernel: [ 160.045878] usb 3-9.1: hub failed to enable device, error -22
2014-09-04T10:09:38.037284+01:00 kvmhost-3 kernel: [ 160.126992] usb 3-9.1: new full-speed USB device number 7 using xhci_hcd
2014-09-04T10:09:38.118289+01:00 kvmhost-3 kernel: [ 160.208006] usb 3-9.1: device descriptor read/64, error -32
2014-09-04T10:09:38.219271+01:00 kvmhost-3 kernel: [ 160.308883] xhci_hcd 0000:00:14.0: Setup ERROR: setup context command for slot 7.
2014-09-04T10:09:38.219278+01:00 kvmhost-3 kernel: [ 160.308886] usb 3-9.1: hub failed to enable device, error -22
2014-09-04T10:09:38.299285+01:00 kvmhost-3 kernel: [ 160.389022] usb 3-9.1: new full-speed USB device number 8 using xhci_hcd
2014-09-04T10:09:38.299291+01:00 kvmhost-3 kernel: [ 160.389127] usb 3-9.1: Device not responding to setup address.
2014-09-04T10:09:38.500284+01:00 kvmhost-3 kernel: [ 160.589984] usb 3-9.1: Device not responding to setup address.
2014-09-04T10:09:38.701266+01:00 kvmhost-3 kernel: [ 160.790900] usb 3-9.1: device not accepting address 8, error -71
2014-09-04T10:09:38.781261+01:00 kvmhost-3 kernel: [ 160.870989] usb 3-9.1: new full-speed USB device number 9 using xhci_hcd
2014-09-04T10:09:38.781268+01:00 kvmhost-3 kernel: [ 160.871083] usb 3-9.1: Device not responding to setup address.
2014-09-04T10:09:38.982281+01:00 kvmhost-3 kernel: [ 161.071992] usb 3-9.1: Device not responding to setup address.
2014-09-04T10:09:39.183287+01:00 kvmhost-3 kernel: [ 161.272923] usb 3-9.1: device not accepting address 9, error -71
2014-09-04T10:09:39.183294+01:00 kvmhost-3 kernel: [ 161.273058] usb 3-9-port1: unable to enumerate USB deviceand brings down the host eventually. Any insight into this is also appreciated. I've googled what I can and it may be a problem with libusb that I'm seeing but I'm not certain how to pinpoint the root cause definitely.
Offline
siddharta wrote:edit: Rebooting the Linux guest will lock up the host. I can shutdown the Linux guest without having the host hang but the guest cannot then be started again:
(qemu) qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:02:00.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile=In trying to resolve this I've been going through this thread again as I had diagnonally read most posts involving nVidia cards (didn't own one until now). I dumped the ROM of the GT 630 from within the guest with the nVidia gpu using
echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom cat /sys/bus/pci/devices/0000:01:00.0/rom > asus-gt630-dcsl-2gd3-v2-vbios.rom echo 0 > /sys/bus/pci/devices/0000:01:00.0/romand added this to the QEMU invocation as
-device vfio-pci,host=$GPU,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/user/asus-gt630-dcsl-2gd3-v2-vbios.rombut no success - host still hangs on guest reboot, same as without romfile specified.
The other suggestion in the error message, adding rombar=0, doesn't work either. - there is no display output when specifying this. I didn't try both together as a previous post indicated this doesn't make sense. I don't find another way forward in this thread unless I missed something. Any help would be very much appreciated, this is really bugging me.
Host hangs can be pretty challenging to debug, especially without at least a serial console. I'd probably start with the shutdown case you list and compare lspci -vvv in the host before and after a guest boot + shutdown. Also, don't forget that you're applying and enabling ACS override in your kernel, which assumes isolation where it may not exist. I'm not sure if that's contributing or not, but you may want to try the card in another slot rather than ignoring the advertised isolation of devices. Things like ASPM and AER may play a role. You can find options to change or disable these in kernel-parameters.txt (https://git.kernel.org/cgit/linux/kerne … meters.txt).
A second issue I'm seeing is when passing through a USB controller
### Group 25 ### 0b:00.0 USB controller: ASMedia Technology Inc. Device 1142that is bound to pci-stub at boot
0b:00.0 USB controller [0c03]: ASMedia Technology Inc. Device [1b21:1142] Subsystem: ASRock Incorporation Device [1849:1142] Kernel driver in use: pci-stub Kernel modules: xhci_hcd. Using
-device vfio-pci,host=$USB,bus=pcie.0in the QEMU command results in dmesg
[2014-09-04T17:45:38.943613+01:00 kvmhost-3 kernel: [ 1553.980492] dmar: DRHD: handling fault status reg 2 2014-09-04T17:45:38.943638+01:00 kvmhost-3 kernel: [ 1553.980504] dmar: DMAR:[DMA Read] Request device [0b:00.0] fault addr ee000 2014-09-04T17:45:38.943642+01:00 kvmhost-3 kernel: [ 1553.980504] DMAR:[fault reason 06] PTE Read access is not setand nothing beyond that. The guest doesn't see the USB controller.
Another quality ASMedia product...
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
Host hangs can be pretty challenging to debug, especially without at least a serial console. I'd probably start with the shutdown case you list and compare lspci -vvv in the host before and after a guest boot + shutdown. Also, don't forget that you're applying and enabling ACS override in your kernel, which assumes isolation where it may not exist. I'm not sure if that's contributing or not, but you may want to try the card in another slot rather than ignoring the advertised isolation of devices. Things like ASPM and AER may play a role. You can find options to change or disable these in kernel-parameters.txt (https://git.kernel.org/cgit/linux/kerne … meters.txt).
Thanks for your feedback. Interesting.
lspci -vvv -s 02:00.gives
02:00.0 VGA compatible controller: NVIDIA Corporation GF108 [GeForce GT 630] (rev a1) (prog-if 00 [VGA controller])
Subsystem: ASUSTeK Computer Inc. Device 8498
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 10
Region 0: Memory at ee000000 (32-bit, non-prefetchable) [disabled] [size=16M]
Region 1: Memory at b0000000 (64-bit, prefetchable) [disabled] [size=256M]
Region 3: Memory at c0000000 (64-bit, prefetchable) [disabled] [size=32M]
Region 5: I/O ports at d000 [disabled] [size=128]
Expansion ROM at ef000000 [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [78] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #1, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: [b4] Vendor Specific Information: Len=14 <?>
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01
Status: NegoPending- InProgress-
Capabilities: [128 v1] Power Budgeting <?>
Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Kernel driver in use: pci-stub
Kernel modules: nouveauwhereas after boot and shutdown of guest this becomes (with diff before after)
3,9c3,9
< Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
< Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
< Interrupt: pin A routed to IRQ 10
< Region 0: Memory at ee000000 (32-bit, non-prefetchable) [disabled] [size=16M]
< Region 1: Memory at b0000000 (64-bit, prefetchable) [disabled] [size=256M]
< Region 3: Memory at c0000000 (64-bit, prefetchable) [disabled] [size=32M]
< Region 5: I/O ports at d000 [disabled] [size=128]
---
> Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
> Interrupt: pin A routed to IRQ 17
> Region 0: Memory at ee000000 (32-bit, non-prefetchable) [size=16M]
> Region 1: Memory at b0000000 (64-bit, prefetchable) [size=256M]
> Region 3: Memory at c0000000 (64-bit, prefetchable) [size=32M]
> Region 5: I/O ports at d000 [size=128]
15c15
< Address: 0000000000000000 Data: 0000
---
> Address: 00000000fee004b8 Data: 0000
17c17
< DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
---
> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64usbut I'm not certain how to relate this correctly and whether this would be related to the issues I'm seeing (host hang on guest reboot and host crash + QEMU stating "invalid ROM contents" on guest shutdown+boot again).
Another quality ASMedia product...
I wouldn't touch it with a ten foot pole if I would have USB device passthrough working properly, which unfortunately is not the case. edit: interestingly enough I find the same USB passthrough errors in dmesg on an Asrock Z77 mainboard but these don't seem to bring down the host there. Thanks.
Last edited by siddharta (2014-09-04 19:50:00)
Offline
Getting there slowly, but still having issues.
It looks like I've managed to successfully pass through my Radeon R9 290, although I can only tell his if I enable cirrus graphics and VNC and when at the UEFI Shell performing "pci" to see the Vendor and Device IDs that match the same as when performing "lsusb -n"
However, I can't get any display to appear when starting the guest without cirrus and VNC and can't get it to boot in to Windows either.
My XML now looks like so
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0' />
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</hostdev>
<qemu:commandline>
<qemu:arg value='-no-hpet'/>
<qemu:arg value='-drive'/>
<qemu:arg value='if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_x64.bin'/>
</qemu:commandline>I have just built the latest TianoCore OVMF image using "yaourt -S ovmf-svn" svn version was 16059.
If I VNC in to the device, I see :
Boot Failed EFI Floppy
Boot Failed EFI Floppy 1
Boot Failed Misc Device
Boot Failed Misc Device 1The next screen shows me a mapping table where by I can see what looks like 3 Hard Drives with MBRs on, but I get no further.
Yes, exactly. My full xml looks like this:
Offline
Getting there slowly, but still having issues.
It looks like I've managed to successfully pass through my Radeon R9 290, although I can only tell his if I enable cirrus graphics and VNC and when at the UEFI Shell performing "pci" to see the Vendor and Device IDs that match the same as when performing "lsusb -n"
However, I can't get any display to appear when starting the guest without cirrus and VNC and can't get it to boot in to Windows either.
My XML now looks like so
<hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0' /> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <qemu:commandline> <qemu:arg value='-no-hpet'/> <qemu:arg value='-drive'/> <qemu:arg value='if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_x64.bin'/> </qemu:commandline>I have just built the latest TianoCore OVMF image using "yaourt -S ovmf-svn" svn version was 16059.
If I VNC in to the device, I see :
Boot Failed EFI Floppy Boot Failed EFI Floppy 1 Boot Failed Misc Device Boot Failed Misc Device 1The next screen shows me a mapping table where by I can see what looks like 3 Hard Drives with MBRs on, but I get no further.
Did you test the ROM for your card to see if it includes EFI support? When I have emulated graphics and an assigned graphics, OVMF initializes both. You will need to reinstall your guest, you can't simply move the disk from a legacy BIOS to a UEFI system and expect it to work, on bare metal or virtual machine.
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
I've just ran the test from your website and I don't see the type 3, an EFI ROM. I'm very surprised as to why not as this is a brand new card, UEFI is written on the box and stated on the website.
Thanks for the heads up with the re-install.
Edit: Just found there is a BIOS selection switch physically on the card! Will dismantle and switch to the other BIOS and test again.
http://www.sapphiretech.com/presentatio … &psn&lid=1
[root@i7-4770s rom-parser]# ./rom-parser /root/r9-290.rom
Valid ROM signature found @0h, PCIR offset 238h
PCIR: type 0, vendor: 1002, device: 67b1, class: 030000
PCIR: revision 0, vendor revision: f2a
Last imageDid you test the ROM for your card to see if it includes EFI support? When I have emulated graphics and an assigned graphics, OVMF initializes both. You will need to reinstall your guest, you can't simply move the disk from a legacy BIOS to a UEFI system and expect it to work, on bare metal or virtual machine.
Last edited by gneville (2014-09-04 20:31:45)
Offline