Arch Linux

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For these days test, I got a strange problem.

First, I have HD5450 and double GTX480 on my MB, and set both GTX480 into pci-stub, lspci -k shows that both of GTX480 has bind into pci-stub.

Second, I run 2 VMs with these GTX480 passthrough by VFIO. And lspci -k shows all of them bind into VFIO-PCI.

1st VM can work normal, it can entered Windows 7. But 2nd VM has no screen when I plug DVI cable. And terminal didn't show any error.

I tried shut down the 1st VM, And only run 2nd VM, no use. Even I tried 1st VM's storage file, still give me no screen.

I know X79 only serves 2 PCI-E x16 and 1 PCI-E x8, so 2nd GTX480 is PCI-E x8,

I'm using Rampage IV Gene. It can only plug 2 video card at the same time, but I use PCI-E cable so I can plug 3 video cards at the same time.

Now I'm thinking that maybe this MB can't do this at real. But when I switch back to Windows 7, it can catch all of 3 cards.

Is anyone have an idea?

Last edited by AKSN74 (2014-03-22 11:51:45)

Jesse2004

Member

Registered: 2012-06-25

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I cant for the life of me sort out my audio for my VM.

I use this:

-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 

I have also tried with -soundhw all and -soundhw ac97.

This is what I get when I fire up the computer:

pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave
alsa: Could not initialize DAC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave
alsa: Could not initialize DAC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
audio: Failed to create voice `dac'
ALSA lib pcm_dsnoop.c:618:(snd_pcm_dsnoop_open) unable to open slave
alsa: Could not initialize ADC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
ALSA lib pcm_dsnoop.c:618:(snd_pcm_dsnoop_open) unable to open slave
alsa: Could not initialize ADC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
audio: Failed to create voice `adc'

In the first post its stated that I may have to use the QEMU_AUDIO_DRV variable. I have tried this (export QEMU_AUDIO_DRV=alsa and QEMU_AUDIO_DRV=pa as root) without success. Anyone in the same boat as me, and/or found a solution?(except passing a USB headset? )

Update: I managed to get audio somewhat working:

I use sudo -E export QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa
and
sudo -E qemu-system-x86_64 ....
The -E option tries to preserve the environment even though you are running with sudo.

However the audio is crackling and sounding really disorted...

I experience the exact same behaviour on my machine. Its working now with ALSA but the quality isnt good. Has anyone fixed this issue or got it working with Pulse Audio (which I would prefer)

Similar problem here. From my reading, the pulseaudio "Connection refused" errors above might be due to the fact that normally pulseaudio is configured in single user mode, which means only a specific session of some user can use pulseaudio in a specific time(unless you are root). If I start a pulseaudio instance as root and then run qemu as root, these errors are gone. But yes, the sound is really harsh and distorted.

Configure pulseaudio in system mode may help eliminate the errors. However I have no idea regarding the ultra-low sound quality. Hope someone can shed light on this issue.

Last edited by Jesse2004 (2014-03-22 12:41:08)

Blind Tree Frog

Member

Registered: 2013-12-31

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I've at least made some progress...

I think part of my earlier problem was that pci-stub wasn't grabbing the USB devices correctly.  Thursday I built another kernel and made sure that pci-stub was built in.  Today I finally installed it.

Booting a Fedora liveCD from iso and it comes up (as it did prior to the new kernel) but mouse and keyboard work.  It still complains about little things and it locked up on me once, but it works and sound and video work.

I then switched back to the crunchbang liveCD.  It booted and got to the menu,  but the keyboard never got power and the system wouldn't respond to the inputs.  So I'm closer.

Still getting the reset errors:

qemu-system-x86_64: vfio: Cannot reset device 0000:00:16.0, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:13.0, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:12.0, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.5, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.4, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.3, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.0, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:16.0, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:13.0, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:12.0, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.5, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.4, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.3, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.0, no available reset mechanism.

And still seeing page faults in dmesg

[  311.887722] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050]
[  311.887725] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050]
[  311.887728] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050]
[  311.887731] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050]
[  311.887734] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050]
[  311.887737] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050]
[  311.887740] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050]
[  311.887743] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050]
[  311.887746] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050]
[  311.887749] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050]
[  311.887754] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050]
[  311.887757] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050]
[  311.887760] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050]
[  311.887763] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050]
[  311.887766] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:13.0 domain=0x0000 address=0x00000000ffffffc0 flags=0x0050]

Though that is a different device this time (compared to last)

I'm not sure if anyone sees anything that I should take note of in the last few dmesg messages, but just in case:

[  357.076479] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x19@0x270
[  357.312658] vfio_cap_init: 0000:00:12.2 hiding cap 0xa
[  357.372555] vfio_cap_init: 0000:00:13.2 hiding cap 0xa
[  357.425797] vfio_cap_init: 0000:00:16.2 hiding cap 0xa
[  485.323872] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x19@0x270
[  485.553417] vfio_cap_init: 0000:00:12.2 hiding cap 0xa
[  485.606661] vfio_cap_init: 0000:00:13.2 hiding cap 0xa
[  485.659901] vfio_cap_init: 0000:00:16.2 hiding cap 0xa
[  517.644634] kvm [893]: vcpu0 unhandled rdmsr: 0xc0010112
[  517.644646] kvm [893]: vcpu0 unhandled rdmsr: 0xc0010048
[  517.766789] kvm [893]: vcpu0 unhandled rdmsr: 0xc0010001
[  517.778451] kvm [893]: vcpu1 unhandled rdmsr: 0xc0010048
[  517.789778] kvm [893]: vcpu2 unhandled rdmsr: 0xc0010048
[  517.801080] kvm [893]: vcpu3 unhandled rdmsr: 0xc0010048
[  517.812380] kvm [893]: vcpu4 unhandled rdmsr: 0xc0010048
[  517.823688] kvm [893]: vcpu5 unhandled rdmsr: 0xc0010048
[  517.909455] kvm: zapping shadow pages for mmio generation wraparound
[  519.701768] vfio-pci 0000:07:00.0: irq 76 for MSI/MSI-X
[  519.741639] vfio-pci 0000:07:00.0: irq 76 for MSI/MSI-X
[  519.741647] vfio-pci 0000:07:00.0: irq 77 for MSI/MSI-X
[  519.748301] vfio-pci 0000:07:00.0: irq 76 for MSI/MSI-X
[  519.748309] vfio-pci 0000:07:00.0: irq 77 for MSI/MSI-X
[  519.748316] vfio-pci 0000:07:00.0: irq 78 for MSI/MSI-X
[  519.768267] vfio-pci 0000:07:00.0: irq 76 for MSI/MSI-X
[  519.768276] vfio-pci 0000:07:00.0: irq 77 for MSI/MSI-X
[  519.768283] vfio-pci 0000:07:00.0: irq 78 for MSI/MSI-X
[  519.768289] vfio-pci 0000:07:00.0: irq 79 for MSI/MSI-X
[  519.778929] vfio-pci 0000:07:00.0: irq 76 for MSI/MSI-X
[  519.778938] vfio-pci 0000:07:00.0: irq 77 for MSI/MSI-X
[  519.778945] vfio-pci 0000:07:00.0: irq 78 for MSI/MSI-X
[  519.778951] vfio-pci 0000:07:00.0: irq 79 for MSI/MSI-X
[  519.778957] vfio-pci 0000:07:00.0: irq 80 for MSI/MSI-X
[  519.808216] vfio-pci 0000:07:00.0: irq 76 for MSI/MSI-X
[  519.808223] vfio-pci 0000:07:00.0: irq 77 for MSI/MSI-X
[  519.808231] vfio-pci 0000:07:00.0: irq 78 for MSI/MSI-X
[  519.808237] vfio-pci 0000:07:00.0: irq 79 for MSI/MSI-X
[  519.808243] vfio-pci 0000:07:00.0: irq 80 for MSI/MSI-X
[  519.808250] vfio-pci 0000:07:00.0: irq 81 for MSI/MSI-X
[  519.824860] vfio-pci 0000:07:00.0: irq 76 for MSI/MSI-X
[  519.824868] vfio-pci 0000:07:00.0: irq 77 for MSI/MSI-X
[  519.824874] vfio-pci 0000:07:00.0: irq 78 for MSI/MSI-X
[  519.824880] vfio-pci 0000:07:00.0: irq 79 for MSI/MSI-X
[  519.824887] vfio-pci 0000:07:00.0: irq 80 for MSI/MSI-X
[  519.824893] vfio-pci 0000:07:00.0: irq 81 for MSI/MSI-X
[  519.824899] vfio-pci 0000:07:00.0: irq 82 for MSI/MSI-X
[  523.395384] vfio-pci 0000:03:00.1: irq 100 for MSI/MSI-X

So 'yay!!' for getting closer, but still not quite there.

tuoni

Member

Registered: 2013-12-19

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

<SNIP>

Update: I managed to get audio somewhat working:

I use sudo -E export QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa
and
sudo -E qemu-system-x86_64 ....
The -E option tries to preserve the environment even though you are running with sudo.

However the audio is crackling and sounding really disorted...

I experience the exact same behaviour on my machine. Its working now with ALSA but the quality isnt good. Has anyone fixed this issue or got it working with Pulse Audio (which I would prefer)

Similar problem here. From my reading, the pulseaudio "Connection refused" errors above might be due to the fact that normally pulseaudio is configured in single user mode, which means only a specific session of some user can use pulseaudio in a specific time(unless you are root). If I start a pulseaudio instance as root and then run qemu as root, these errors are gone. But yes, the sound is really harsh and distorted.

Configure pulseaudio in system mode may help eliminate the errors. However I have no idea regarding the ultra-low sound quality. Hope someone can shed light on this issue.

I use the following invocation for a Win7 x64 guest:

sudo QEMU_AUDIO_DRV=alsa
QEMU_AUDIO_TIMER_PERIOD=0
QEMU_AUDIO_DAC_FIXED_FREQ=48000
QEMU_AUDIO_ADC_FIXED_FREQ=48000
QEMU_ALSA_DAC_BUFFER_SIZE=16384
qemu-system-x86_64 -enable-kvm -M q35
<snip>
-soundhw hda,ac97

In my guest, I have disabled the high-definition audio and have installed the x64 AC'97 drivers from Realtek's website.

My audio is now clear, though there is the tiniest bit of lag occasionally.  Your mileage may vary and I'm sure there is a better way to do it, but I stopped fiddling once I'd gotten it running

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I still can't bootup 2nd VM with another GTX480. Now I focus on some setting problem.

Strange is, I can't start with just 2nd VM. And just start 1st VM is OK.

If I start 1st VM then start 2nd VM, 1st VM worked, 2nd still can't.

But if I converse the steps (start 2nd VM then start 1st VM), 1st VM will got KVM error.

KVM internal error. Suberror: 1
emulation failure
EAX=ffff2b56 EBX=00000000 ECX=00000000 EDX=00000300
ESI=00006d0a EDI=00006d25 EBP=000041f7 ESP=00000014
EIP=00032dcf EFL=00010082 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =2b2b 0002b2b0 ffffffff 00809300
CS =6d23 0006d230 ffffffff 00809b00
SS =0000 00000000 ffffffff 00809300
DS =c000 000c0000 ffffffff 00809300
FS =0000 00000000 ffffffff 00809300
GS =0000 00000000 ffffffff 00809300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     000f68c0 00000037
IDT=     00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

And here is my dmesg when I start 2nd VM only.

[  259.375335] VFIO - User Level meta-driver version: 0.3
[  259.392054] ehci-pci 0000:00:1d.0: remove, state 1
[  259.392062] usb usb2: USB disconnect, device number 1
[  259.392063] usb 2-1: USB disconnect, device number 2
[  259.392065] usb 2-1.1: USB disconnect, device number 3
[  259.399330] hid-generic 0003:04D9:2011.0005: can't resubmit intr, 0000:00:1d.0-1.2/input0, status -19
[  259.414950] usb 2-1.2: USB disconnect, device number 4
[  259.486485] usb 2-1.5: USB disconnect, device number 5
[  259.486487] usb 2-1.5.1: USB disconnect, device number 7
[  259.488913] usb 2-1.6: USB disconnect, device number 6
[  259.495437] ehci-pci 0000:00:1d.0: USB bus 2 deregistered
[  259.598585] device tap1 entered promiscuous mode
[  259.599065] br0: port 3(tap1) entered forwarding state
[  259.599070] br0: port 3(tap1) entered forwarding state
[  259.898723] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)
[  260.030186] vfio_cap_init: 0000:00:1d.0 hiding cap 0xa
[  262.975682] dmar: DRHD: handling fault status reg 2
[  262.975689] dmar: DMAR:[DMA Read] Request device [00:1d.0] fault addr ee000 
[  262.975689] DMAR:[fault reason 06] PTE Read access is not set
[  274.634766] br0: port 3(tap1) entered forwarding state
[  322.927939] audit: type=1400 audit(1395624287.279:64): apparmor="DENIED" operation="capable" profile="/usr/sbin/cupsd" pid=1705 comm="cupsd" capability=36  capname="block_suspend"

Also, I don't know why it got this message in dmesg every boot, and it's a lot of then.

[   12.903046] vgaarb: this pci device is not a vga device

And here is my QEMU start command, I thought that maybe it's anothe problem, too.

(1st VM code)
#!/bin/bash

vfio-bind 0000:02:00.0 0000:02:00.1

qemu-system-x86_64 -enable-kvm -M q35 -m 2048 -cpu host \
-smp 2,sockets=1,cores=1,threads=2 \
-bios /usr/share/qemu/bios.bin -vga none -nographic -localtime \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/home/ak/test.img,format=raw,if=virtio \
-net nic,model=virtio,vlan=0,macaddr=52-54-00-f5-44-04 -net tap,vlan=0,ifname=tap0,script=no

(2nd VM code)
#!/bin/bash

vfio-bind 0000:03:00.0 0000:03:00.1 0000:00:1d.0

qemu-system-x86_64 -enable-kvm -M q35 -m 2048 -cpu host \
-smp 2,sockets=1,cores=1,threads=2 \
-bios /usr/share/qemu/bios.bin -vga none -nographic -localtime \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:1d.0,bus=pcie.0 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/home/ak/test2.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk \
-drive file=/home/ak/ubuntu.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \
-net nic,model=virtio,vlan=0,macaddr=52-54-00-f5-44-08 -net tap,vlan=0,ifname=tap1,script=no

Still, I'm apologize my poor English.

Many thanks

Last edited by AKSN74 (2014-03-24 01:41:44)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

FYI, if you're trying Passmark Performance Test on Intel, avoid -cpu host or any of the models above Penryn, they seem to result in a BSOD.  I've reported the problem to Passmark, so hopefully they'll resolve it soon.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

trimm

Member

Registered: 2014-03-06

Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have the weirdest issue and I dont know how to troubleshoot it.

I can for example play Rocksmith 2014(perfect audio yai!) and Battlefield 4(at Ultra settings) without any issues. The performance is for me just as good as playing on a "real" Windows machine.
However as soon as I quit(!) the game it will freeze the whole machine up. I cant ping the guest from the host so it is totally frozen. The screen is showing whatever was on it the second it froze.
I do not receive any error messages in journalctl, and the Windows guest does not log anything to the event viewer.

The only way to unfreeze my Windows screen is to reboot the host, killing the qemu processes is not possible.

Playing Guild Wars 2 at Ultra Settings does work without any issues though.
(Got a Nvidia GTX 670, i7 CPU(6 dedicated to the guest), 32GB RAM(16GB to the guest). I am not running processor pinning or huge pages, but I do have cache=none set to my hard drives.

Anyone having experience to something similar like this?

ahl

Member

Registered: 2014-03-16

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have finally found the real source of those random hardware failures I was experiencing on host: The HD 7660D IGP of AMD A10-5700. Use of that GPU on host makes the host system very unreliable. Further, there appears to be nothing wrong with the cheap PCI-E NICs I mentioned earlier. It was a simple coincidence that their removal made my system slightly more stable.

Here's a summary of all the hardware I have tested with VGA vfio-pci:

System 1: AMD A10-5700 APU + Asus F2A85-M PRO (BIOS 6308)
- OK (must disable integrated graphics)
- passthrough of the integrated audio device also tested to work with Win 7

System 2: AMD FX 8350 + Asus Sabretooth 990FX R2.0 (BIOS 2104)
- OK
- passthrough of the integrated audio device also tested to work with Win 7

GPU 1: Radeon HD 7660D IGP of AMD A10-5700 APU
- If used on host, crashes the host horribly
- If used on guest, does not crash the host, but does not work either (no picture)
- If booted on headless Linux guest, kernel complains about invalid rom contents (or booting with romfile= extracted earlier stucks the VM in busy loop)
==> USELESS

GPU 2: Asus Radeon HD 3870
- On host: OK
- On Linux guest: OK (glxgears tested)
- On Win 7 guest: Device Manager reports Code 43 (or BSoD STOP 0x3B after forced device reinstall)
- On WinXP guest: Might work, but I have trouble installing drivers
==> For host or Linux guest only

GPU 3: Sapphire Ultimate Radeon HD 6670 (11192-06-20G)
- On host: OK
- On Linux guest: OK (glxgears tested)
- On Win 7 guest: OK (Graphics/Gaming score 6.9)
- On WinXP guest: Might work, but I have trouble installing drivers
==> Very good, well supported GPU

GPU 4: GeForce 9600 GT
GPU 5: Radeon X300
GPU 6: Radeon X1600
- None of these older cards appear to work as these refuse to initialize together with other PCI-E GPUs (lspci on host shows up to 1 card only)
==> Useless???

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GPU 4: GeForce 9600 GT
GPU 5: Radeon X300
GPU 6: Radeon X1600
- None of these older cards appear to work as these refuse to initialize together with other PCI-E GPUs (lspci on host shows up to 1 card only)
==> Useless???

The Radeon X-series are useless.  I have an X500 that I spent some time reverse engineering quirks for, but I never got it working and decided I didn't want to set a precedence for maintaining code for devices that really aren't useful as a modern, high-performance card.  The 9600GT should work.  I think the Nvidia quirks we have work down to the 8-series, maybe even the 7-series, but obviously if the platform won't initialize the slot it's a non-starter.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

ahl

Member

Registered: 2014-03-16

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The 9600GT should work.  I think the Nvidia quirks we have work down to the 8-series, maybe even the 7-series, but obviously if the platform won't initialize the slot it's a non-starter.

Thanks for the info. I did some more testing with my 9600GT. No luck with vfio-pci. All I get is blank picture even if it's the only GPU in the system.

And now it froze the host system (Sabretooth) completely...! I can't even SSH.

Here's how I tried to use it from console:

GRUB: pci-stub.ids=10de:0622
vfio-bind 0000:01:00.0

qemu-system-x86_64 -machine accel=kvm -M q35 -nographic -vga none
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on

It would have been nice, had this worked, but after a host crash I don't think I'll keep trying unless there is some known trick.

Has anyone tested a brand new Radeon R7 260? It might be my next purchace.

ahl

Member

Registered: 2014-03-16

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a project, which is to have 2 windows VM (with each of them having one dedicated graphic card) running on the same archlinux host for my bf and I. From this thread I'm sure that it works with one, however, before buying all the stuff I need I would have liked some validation on some level that it could maybe work (if it definitely can't there's no point in spending money ). From the looks of it I would say it could work but I may be missing something as I never took a look at the actual code.

Certainly an interesting project! How do you plan to assign peripherals like keyboard, mouse, etc. to the hosts? You might want to look for a motherboard that has at least 3 distinct USB controllers, so you can pass through one chip to each VM and have one left for the host.

An addon PCI-E usb controller could also do the trick.
But is it really necessary to passthrough USB controllers? I'm asking this because so far I have experienced absolutely no reason to do that. Not even for light gaming use.

I only use "-usbdevice tablet". It takes care of all input problems. I'm finding the ability to control both host and all guests, with just one keyboard and mouse, an extremely useful feature.

Jesse2004

Member

Registered: 2012-06-25

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I use the following invocation for a Win7 x64 guest:

sudo QEMU_AUDIO_DRV=alsa
QEMU_AUDIO_TIMER_PERIOD=0
QEMU_AUDIO_DAC_FIXED_FREQ=48000
QEMU_AUDIO_ADC_FIXED_FREQ=48000
QEMU_ALSA_DAC_BUFFER_SIZE=16384
qemu-system-x86_64 -enable-kvm -M q35
<snip>
-soundhw hda,ac97

Thanks for sharing!

Jesse2004

Member

Registered: 2012-06-25

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have the weirdest issue and I dont know how to troubleshoot it.

I can for example play Rocksmith 2014(perfect audio yai!) and Battlefield 4(at Ultra settings) without any issues. The performance is for me just as good as playing on a "real" Windows machine.

May I ask your audio configuration?

novist

Member

Registered: 2014-03-14

Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahl or someone - please comment on AMD setup performance. ahl, my setup is similar to your AMD FX 8350 + Asus Sabretooth 990FX R2.0. Does that rig run fine? im not sure why but i have terrible performance. to get a perspective - arma3 playable is at normal settings, better yet low. i run git qemu and kernel is 3.14-rc6 with patches from first post.

One more thing - anyone happen to test 3.14-rc7 kernel? Any stability issues? Because i have some that pop up quite fast after boot. Wondering if im only one unlucky,

ahl

Member

Registered: 2014-03-16

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahl or someone - please comment on AMD setup performance. ahl, my setup is similar to your AMD FX 8350 + Asus Sabretooth 990FX R2.0. Does that rig run fine? im not sure why but i have terrible performance. to get a perspective - arma3 playable is at normal settings, better yet low. i run git qemu and kernel is 3.14-rc6 with patches from first post.

One more thing - anyone happen to test 3.14-rc7 kernel? Any stability issues? Because i have some that pop up quite fast after boot. Wondering if im only one unlucky,

I can't really comment on performance just yet, because I'm not even trying to get top speeds. I only need something that works.
I use a kernel a bit newer than 3.14-rc7. And looks like -rc8 is already out so I suggest trying that.

Edit: Upgraded to 3.14-rc8 + fix_memleak.patch

Last edited by ahl (2014-03-25 12:15:30)

Flyser

Member

Registered: 2013-12-19

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have the weirdest issue and I dont know how to troubleshoot it.

I can for example play Rocksmith 2014(perfect audio yai!) and Battlefield 4(at Ultra settings) without any issues. The performance is for me just as good as playing on a "real" Windows machine.
However as soon as I quit(!) the game it will freeze the whole machine up. I cant ping the guest from the host so it is totally frozen. The screen is showing whatever was on it the second it froze.
I do not receive any error messages in journalctl, and the Windows guest does not log anything to the event viewer.

The only way to unfreeze my Windows screen is to reboot the host, killing the qemu processes is not possible.
Anyone having experience to something similar like this?

I had a similar issue once and I think it was due to a qemu bug that has already been fixed. I am not 100% sure, but iirc just rebuilding the latest  qemu-git fixed it.

novist

Member

Registered: 2014-03-14

Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahl or someone - please comment on AMD setup performance. ahl, my setup is similar to your AMD FX 8350 + Asus Sabretooth 990FX R2.0. Does that rig run fine? im not sure why but i have terrible performance. to get a perspective - arma3 playable is at normal settings, better yet low. i run git qemu and kernel is 3.14-rc6 with patches from first post.

One more thing - anyone happen to test 3.14-rc7 kernel? Any stability issues? Because i have some that pop up quite fast after boot. Wondering if im only one unlucky,

I can't really comment on performance just yet, because I'm not even trying to get top speeds. I only need something that works.
I use a kernel a bit newer than 3.14-rc7. And looks like -rc8 is already out so I suggest trying that.

thanks for heads-up! i tried new kernel. so far system is running for over an hour. keeping fingers crossed it stays that way. there is like 30% increase in fps (from 10 to 15) from rc6 kernel in directx9 complex passmark test. has little play-session, its still laggy. this time i kept gpu-z on checking some stats. apparently gpu most of the time runs at 20% load. that is really wrong... fan speed is reported at 33%. temperature at ~60 degrees under max load. no idea what to make of it.

alexis_evo

Member

Registered: 2013-07-04

Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a project, which is to have 2 windows VM (with each of them having one dedicated graphic card) running on the same archlinux host for my bf and I. From this thread I'm sure that it works with one, however, before buying all the stuff I need I would have liked some validation on some level that it could maybe work (if it definitely can't there's no point in spending money ). From the looks of it I would say it could work but I may be missing something as I never took a look at the actual code.

Certainly an interesting project! How do you plan to assign peripherals like keyboard, mouse, etc. to the hosts? You might want to look for a motherboard that has at least 3 distinct USB controllers, so you can pass through one chip to each VM and have one left for the host.

An addon PCI-E usb controller could also do the trick.
But is it really necessary to passthrough USB controllers? I'm asking this because so far I have experienced absolutely no reason to do that. Not even for light gaming use.

I only use "-usbdevice tablet". It takes care of all input problems. I'm finding the ability to control both host and all guests, with just one keyboard and mouse, an extremely useful feature.

How does this work? Currently I pass through the USB devices (not the entire controller, just specific devices based on vendorid/productid) which means I can only use keyboard/mouse on the guest or the host. I can switch between them, but it requires using qemu console to detach/reattach. If there is an easier method for keyboard sharing, to let you use both on host/guest, I'd love to hear how it works.

ahl

Member

Registered: 2014-03-16

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I only use "-usbdevice tablet". It takes care of all input problems. I'm finding the ability to control both host and all guests, with just one keyboard and mouse, an extremely useful feature.

How does this work? Currently I pass through the USB devices (not the entire controller, just specific devices based on vendorid/productid) which means I can only use keyboard/mouse on the guest or the host. I can switch between them, but it requires using qemu console to detach/reattach. If there is an easier method for keyboard sharing, to let you use both on host/guest, I'd love to hear how it works.

Qemu should open a blank window by default. You can use both mouse and keyboard on that. Any input is automatically sent to the guest.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I only use "-usbdevice tablet". It takes care of all input problems. I'm finding the ability to control both host and all guests, with just one keyboard and mouse, an extremely useful feature.

How does this work? Currently I pass through the USB devices (not the entire controller, just specific devices based on vendorid/productid) which means I can only use keyboard/mouse on the guest or the host. I can switch between them, but it requires using qemu console to detach/reattach. If there is an easier method for keyboard sharing, to let you use both on host/guest, I'd love to hear how it works.

Qemu should open a blank window by default. You can use both mouse and keyboard on that. Any input is automatically sent to the guest.

I use this for quick testing as well, but with the -vnc option to the VM.  It doesn't work well for gaming though, Borderlands2 at least is completely unusable with this mouse input.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

tuoni

Member

Registered: 2013-12-19

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If there is an easier method for keyboard sharing, to let you use both on host/guest, I'd love to hear how it works.

http://synergy-foss.org

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Edit: Upgraded to 3.14-rc8 + fix_memleak.patch

erm ... you mean memleak fix didnt make it to 3.14 yet ?? ouch.

trimm

Member

Registered: 2014-03-06

Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have the weirdest issue and I dont know how to troubleshoot it.

I can for example play Rocksmith 2014(perfect audio yai!) and Battlefield 4(at Ultra settings) without any issues. The performance is for me just as good as playing on a "real" Windows machine.

May I ask your audio configuration?

I am passthroughing my onboard audio card.
-device vfio-pci,host=00:1b.0

And the USB rocksmith cable:
-usb -usbdevice host:12ba:00ff

I had a similar issue once and I think it was due to a qemu bug that has already been fixed. I am not 100% sure, but iirc just rebuilding the latest  qemu-git fixed it.

Thank you so much Flyser, I will definately try to use/reinstall the latest qemu-git!

Edit: is there a reason for why we are not using the AUR package of qemu-git? What is so special with the one OP has posted?
Edit2: Still got the same issue, the following is the only thing in my journalctl while the VM froze:

Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe590 fff97410 00000000 0d00000c 07058000
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe5a0 fff97420 00000000 0d00000c 07058000
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe5b0 fff97430 00000000 0d00000c 07058000
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe5c0 fff97440 00000000 0d00000c 07058000
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe5d0 fff97450 00000000 0d00000c 07058000
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe5e0 fff97460 00000000 0d00000c 07058000
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe5f0 fff97470 00000000 0d00000c 07058000
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe600 fff97480 00000000 0d00000c 07058000
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe610 fff97490 00000000 0d00000a 07058000
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe620 fff974a0 00000000 0d00000c 07058000
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe630 fff974b0 00000000 0d00000c 07058000
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe640 fff974c0 00000000 0d00000c 07058000
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: ERROR Transfer event for disabled endpoint or incorrect stream ring
Mar 25 22:53:42 ultimaarch kernel: xhci_hcd 0000:00:14.0: @00000000ffffe650 fff974d0 00000000 0d00000c 07058000

Last edited by trimm (2014-03-25 20:56:42)

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

anyone tried the new QEMU 2.0 rc0 ?

As soon as i startup my win7 guest with it instead of good old qemu 1.7.0 - i get a BSOD with STOP 0x00..01e
No Config Changes whatsover - i just replaced the qemu binary...

Perhaps the Memory layout Changes in Qemu 2.0 (see http://wiki.qemu.org/ChangeLog/2.0 ) require a Guest-Reinstall from Scratch? i dont hope so, but...

Thanks for your thouhts on that

Greetings
K.

pereczes

Member

Registered: 2014-03-08

Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

FYI: All my adventure with Powercolor 6950 ends here... I think I had a faulty card from the beginning.
It was crashing linux with the original driver when I was using opencl, and now with pass-trough.
The GPU-Z was showing only half of the shaders in win7 guest. Once connecting in guest the DVI, system was
freezing. Was reading in different forums that Powercolor is just crap... better spend few more $$$ and go for better
brand. Managed though to reflash the original Bios, though is showing strange characters in bios and in text mode.
Removed now the card, power dropped from 120W to 80W in iddle mod... Was using 40W for nothing...