Arch Linux

josipva

Member

Registered: 2014-07-21

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have an 4790K with Win 8.1 using 4 vcpu and a 7950, if you'd like I can run a benchmark if you tell me which one. Windows performance index is 8.1 for CPU, memory and graphics.

Could you try cinebench with 4vcpu and 3 vcpu so I can compare it to native 4690 results ?

P.S. Thanks "siddharta" and "aw" for info so far.

Last edited by josipva (2014-08-27 14:34:04)

siddharta

Member

Registered: 2014-05-03

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have an 4790K with Win 8.1 using 4 vcpu and a 7950, if you'd like I can run a benchmark if you tell me which one. Windows performance index is 8.1 for CPU, memory and graphics.

Could you try cinebench with 4vcpu and 3 vcpu so I can compare it to native 4690 results ?

P.S. Thanks "siddharta" and "aw" for info so far.

488cb for 3 vcpu, 590cb for 4 vcpu, 790 for 6 vcpu (for completeness). This could possibly be improved still, this is a clean install and not much tweaking was done.

http://imgur.com/touYvyQ,EqM4eGw#1 (forgot to screencap the 6 vcpu, sorry)

josipva

Member

Registered: 2014-07-21

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Huge thanks. Helps a lot to estimate performance.

EDIT:

Interesting stuff 4vcpu performance puts it in line with 4690.

Last edited by josipva (2014-08-27 19:20:54)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Huge thanks. Helps a lot to estimate performance.

EDIT:

Interesting stuff 4vcpu performance puts it in line with 4690.

Sure, if you only expect one VM to be under load at any given point, your i5-4690 is a good comparison.  If you want the worst case performance where both VMs are under load, I think you'll be closer to the i3 processor.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

bonob

Member

Registered: 2014-02-02

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's not directly related with the topic at hand, but maybe some people around here have the same config as I have and will be able to confirm whether I'm understanding my issue correctly (and possibly confirm if they suffer the same fate.)

I send the audio from my (Windows) VM to my (Arch) host using NetJack2 (mostly following this guide.) It's been working pretty well for the past few months.
Since I rebooted yesterday, it's not working anymore. I noticed that jack2-dbus has just been updated to 1.9.10, so I'm wondering if that's the cause of my trouble.

From what I can see, everything is working as usual, jack can play audio on the host, the virtual cable is plugged on the guest, the qjackctls on either side are not reporting anything unusual.
It seems that the problem is that the jack slave and the master cannot find each other, whereas they used to. So I'm thinking that maybe the auto-discovery feature of NetJack2 is somehow broken?

Last edited by bonob (2014-08-28 15:35:03)

Wimma77

Member

Registered: 2014-08-12

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I don't expect everyone is following my new blog (yet), but I wrote this up yesterday:
...

I've just successfully got my HD7770 passed through to my Win7 VM (trick was to not pass the sound with it - good tip redger) and proved capability.
Still have to work out networking and sound, but progress is always encouraging.
I will have to consider Win8 based on your UEFI comments, but will wait for my GTX750ti first and give that a crack.

Just in addition to this, happy with the performance and can stream a steam game across the network fine (HD7770 anyway, haven't worked out how to get the GTX750 working yet..), but high level question about sound if anyone could help please.
I see the sound config details on p1, and will use for some testing, but going forward I want to use alsa on the host for some distributed home audio stuff (with a couple pcie sound cards).
Does that mean I can really only use pulseaudio for the VM? Will I run into conflicts later if I use alsa on both?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I don't expect everyone is following my new blog (yet), but I wrote this up yesterday:
...

I've just successfully got my HD7770 passed through to my Win7 VM (trick was to not pass the sound with it - good tip redger) and proved capability.
Still have to work out networking and sound, but progress is always encouraging.
I will have to consider Win8 based on your UEFI comments, but will wait for my GTX750ti first and give that a crack.

Just in addition to this, happy with the performance and can stream a steam game across the network fine (HD7770 anyway, haven't worked out how to get the GTX750 working yet..), but high level question about sound if anyone could help please.
I see the sound config details on p1, and will use for some testing, but going forward I want to use alsa on the host for some distributed home audio stuff (with a couple pcie sound cards).
Does that mean I can really only use pulseaudio for the VM? Will I run into conflicts later if I use alsa on both?

Does audio only work with steam in-home-streaming when routed through the GPU audio?  This was the conclusion I came to when I attempted to do streaming on a bare metal system without a monitor connected to the GPU, but I didn't try too hard.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Wimma77

Member

Registered: 2014-08-12

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does audio only work with steam in-home-streaming when routed through the GPU audio?  This was the conclusion I came to when I attempted to do streaming on a bare metal system without a monitor connected to the GPU, but I didn't try too hard.

Yeah, this is my question essentially. I can't add in the audio device to my VM, as it won't boot with it.
Just the video card device itself is fine though, and at least gives me accelerated video capability.
Will try some audio lines to test, but get the feeling I'll still need to keep a second box or dual boot
Be a bummer, as the main VM use for me would be steam streaming.
Not sure if I'll have more luck with my nvidia card, but a quick test yesterday just got me emulator error, like others have had in earlier posts. Still have to sort that out. More testing ahead...

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

steaming doesn't need sound card on guest, if u don't want sound (some games require sound card), but it can use any sound card... i am using   

    <qemu:arg value='-device'/>
    <qemu:arg value='ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0'/>

since i don't need any audio on host / audio-output  on guest, and hdmi audio causes problems after 30mins or so (in my case)....

Last edited by slis (2014-08-29 05:08:24)

Cenm

Member

Registered: 2014-08-29

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've managed to get everything working for the first boot of the VM but it fails for every subsequent boot with the error 'vfio-pci 0000:03:00.0: Invalid ROM contents'

If I specify romfile=/path/to/rom I don't get the error but nothing appears on the screen.

I have also noticed when I kill the VM whatever was on the screen remains there until I start the VM again, at which point the screen turns off.

Any advice would be great.

Wimma77

Member

Registered: 2014-08-12

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

steaming doesn't need sound card on guest, if u don't want sound (some games require sound card), but it can use any sound card... i am using   

    <qemu:arg value='-device'/>
    <qemu:arg value='ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0'/>

since i don't need any audio on host / audio-output  on guest, and hdmi audio causes problems after 30mins or so (in my case)....

Awesome,thanks. Will try this. I do need alsa audio on the host, and I want audio for streaming games, so will see if anything can work on the guest.

Still having issues with nvidia card though.
Is the kvm=off option still required for nvidia?
ref "If you're a GeForce user, libirt just got support for kvm=off and QEMU will now automatically enable the primary device quirk for Nvidia GPUs independent of VGA" posted a while back.

I cannot get anything on screen booting with a guest GTX750ti (HD5450 host), though had video passthrough fine with guest HD7770.
I get the QEMU window for a minute, then
KVM internal error. Suberror: 1
emulation failure
EAX=ffff0 ... etc
I tried cpu kvm=off option, but that fails also with
qemu-system-x86_64: Property '.kvm' not found

Nothing else was changed from the HD7770 setup (except for stub IDs for new card)... ?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

steaming doesn't need sound card on guest, if u don't want sound (some games require sound card), but it can use any sound card... i am using   

    <qemu:arg value='-device'/>
    <qemu:arg value='ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0'/>

since i don't need any audio on host / audio-output  on guest, and hdmi audio causes problems after 30mins or so (in my case)....

Awesome,thanks. Will try this. I do need alsa audio on the host, and I want audio for streaming games, so will see if anything can work on the guest.

Still having issues with nvidia card though.
Is the kvm=off option still required for nvidia?
ref "If you're a GeForce user, libirt just got support for kvm=off and QEMU will now automatically enable the primary device quirk for Nvidia GPUs independent of VGA" posted a while back.

These are two separate things.  kvm=off will always be required as long as the guest Nvidia driver decides to bail when it finds KVM.  The QEMU change allows Nvidia to work with OVMF (ie. without x-vga=on).

I cannot get anything on screen booting with a guest GTX750ti (HD5450 host), though had video passthrough fine with guest HD7770.
I get the QEMU window for a minute, then
KVM internal error. Suberror: 1
emulation failure
EAX=ffff0 ... etc
I tried cpu kvm=off option, but that fails also with
qemu-system-x86_64: Property '.kvm' not found

This means you are not running a new enough QEMU.  You need at least 2.1 for the kvm=off property.

Nothing else was changed from the HD7770 setup (except for stub IDs for new card)... ?

What is your primary graphics for this test?  Does FAQ question 4 apply?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Wimma77

Member

Registered: 2014-08-12

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

These are two separate things.  kvm=off will always be required as long as the guest Nvidia driver decides to bail when it finds KVM.  The QEMU change allows Nvidia to work with OVMF (ie. without x-vga=on).
This means you are not running a new enough QEMU.  You need at least 2.1 for the kvm=off property.
What is your primary graphics for this test?  Does FAQ question 4 apply?

Ok, wasn't sure I'd need kvm=off. Looking at https://bbs.archlinux.org/viewtopic.php … 7#p1313007 shows ubuntu success using older version, and similar host ATI card (I have HD5450) and guest nvidia. Will try newer QEMU this weekend anyway, no harm.
No errors from memory in dmesg like FAQ#3, but as I have a headless server NFI if it had any video issues. VNC to my xfce4 desktop was fine though! lol
Also just noticed card has the hybrid BIOS switch thing, so may set to legacy and try also, as using Win7.
Thanks again.

abdullah

Member

Registered: 2014-06-10

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I don't expect everyone is following my new blog (yet), but I wrote this up yesterday:

http://vfio.blogspot.com/2014/08/primar … t-vga.html

I'm not the first to try this, former active user on this thread DanaGoyette attempted this, but never fully debugged the issues.  Using OVMF (UEFI) for the guest we can avoid all of the issues with the VGA aspects of GPU assignment.  This means that we don't care about how broken the i915 driver is and we don't need the x-vga option for the assigned device, making it just another assigned devices as far as libvirt is concerned.

Also:

http://vfio.blogspot.com/2014/08/upstre … -2014.html

If you're a GeForce user, libirt just got support for kvm=off and QEMU will now automatically enable the primary device quirk for Nvidia GPUs independent of VGA.

Nicely done!, I was thinking about the same thing, I may submit a post or 2, maybe do some translation as well, but let me find the time for it.

Now everything works perfect exept one thing, when ever the proccess of qemu is killed, the guest OS becomes corrupted, is that the matter with the format of the container (using raw data here) ?, what should be done to avoid this problem, what if the powerd is lost suddenly for example.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nicely done!, I was thinking about the same thing, I may submit a post or 2, maybe do some translation as well, but let me find the time for it.

That would be great!

Now everything works perfect exept one thing, when ever the proccess of qemu is killed, the guest OS becomes corrupted, is that the matter with the format of the container (using raw data here) ?, what should be done to avoid this problem, what if the powerd is lost suddenly for example.

Are you specifying any cache option for the guest block device?  There is a cache=unsafe option which uses the host buffer cache, but as the name implies, it's not safe.  I typically use raw images or lvm volumes and I don't have much issue with killing the guest.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

josipva

Member

Registered: 2014-07-21

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sure, if you only expect one VM to be under load at any given point, your i5-4690 is a good comparison.  If you want the worst case performance where both VMs are under load, I think you'll be closer to the i3 processor.

That makes sense. What do you think about newly released i7-5820k (it's only a bit more expensive than 4790k)? Could I clock it a bit, give 5 vcpu to each VM and leave 2vcpu for host while getting equivalent of 4690 performance for each VM (running simultaneously ofc)?

Last edited by josipva (2014-08-29 15:04:25)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sure, if you only expect one VM to be under load at any given point, your i5-4690 is a good comparison.  If you want the worst case performance where both VMs are under load, I think you'll be closer to the i3 processor.

That makes sense. What do you think about newly released i7-5820k (it's only a bit more expensive than 4790k)? Could I clock it a bit, give 5 vcpu to each VM and leave 2vcpu for host while getting equivalent of 4690 performance for each VM (running simultaneously ofc)?

-ENOCPUINARK  But I see elsewhere that it's a hex-core+threads and it's got a lower clock speed (3.3GHz) and since it's not in ARK yet, unknown VT-d support.  With this CPU you've gained 2 more cores, but you've lost too much clock to be comparable to the i5-4690.  You could reasonable have 2 concurrent sockets=1,cores=3,threads=2 VMs but it's difficult to compare 3cores+threads @3.3GHz to 4cores @3.5GHz.  Seems like they could be pretty close aggregate, but your single thread performance is going to take a hit.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

josipva

Member

Registered: 2014-07-21

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's a cpu with unlocked multiplier and i plan to overclock it to atleast 4.0Ghz (early leaks indicate it's a pretty good chip for oc). Seems to me like a good target for dual VM without going for full on expensive 1k$ chip, thats why im asking for opinions

Last edited by josipva (2014-08-29 15:20:32)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's a cpu with unlocked multiplier and i plan to overclock it to atleast 4.0Ghz (early leaks indicate it's a pretty good chip for oc). Seems to me like a good target for dual VM without going for full on expensive 1k$ chip, thats why im asking for opinions

If you can get there and have it stable then it could be a good option (/me makes note to ignore any spectacularly weird bugs from josipva).  Just be aware that there are many K-series processors that do not support VT-d.  For a while they seemed to be mutually exclusive.  Hopefully it will appear in ark soon.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

josipva

Member

Registered: 2014-07-21

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well they added Vt-d support for both i5 and i7 in Devil's canyon release so i doubt it won't be supported in these new Haswell-E cpus. Anyway 4.0Ghz in Haswell is easy to get stable.

(/me makes note to ignore any spectacularly weird bugs from josipva)

Last edited by josipva (2014-08-29 15:30:42)

Ansa89

Member

Registered: 2014-08-30

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all, first post here .
I'm getting some problems setting up a VM with Windows 7 and a GeForce GTX 650 passed through.

Software setup:
- Debian 7 (headless pc, managed through ssh)
- kernel 3.16.1 + ACS patch + i915 arbitrer patch + VGA arbitrer patch
- boot parameters "i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:0fc6,10de:0e1b"
- qemu 2.1
- seabios 1.7.5

Hardware setup:
- MSI Z77A-G43
- Intel i5-3470
- IGD as primary VGA (for host)
- nVidia GeForce GTX 650 as secondary VGA (for guest)

Steps I followed:
- enabled VT-d in motherboard
- installed Windows 7 through VNC using "-vga std" (keeping GTX 650 as secondary graphics card in guest)
- installed tightVNC inside Windows 7
- installed nVidia graphics driver (version 335.23)
- booted Windows 7 with "-vga none"
- black screen and cannot connect to VNC server inside Windows 7 (without any error from qemu)

Script used to start VM:

# Bind VGA
DEVICES="0000:01:00.0 0000:01:00.1"

for dev in $DEVICES ; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done

# Start QEMU
qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host,kvm=off \
        -smp 2,sockets=1,cores=2,threads=1 \
        -bios /usr/share/qemu/bios-256k.bin -vga none \
        -rtc base=localtime \
        -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
        -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=/root/gtx650_bios.rom \
        -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
        -drive file=/dev/sdc,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
        -boot menu=on \
        -netdev tap,ifname=qemu0,id=qemu_tap -device e1000,netdev=qemu_tap,mac=00:16:3E:12:34:56 \
        -usb -usbdevice tablet

If you need more info, just ask.
Thanks in advance.

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

u don't need rombar and romfile...

Ansa89

Member

Registered: 2014-08-30

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I need them as without these two options I was getting some errors about VGA initialization (or something like that).
Adding "rombar" and "romfile" the errors disappeared.

PS: obviously the file passed to "romfile" is the actual rom dumped from my nVidia card.

Wimma77

Member

Registered: 2014-08-12

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This means you are not running a new enough QEMU.  You need at least 2.1 for the kvm=off property.

Ok, so got from git qemu 2.1.50, but when I then try to run my vm I just get a 'QEMU 2.1.50 monitor' prompt?
Even added full path /usr/local/bin/qemu... to make sure was using.
And then I still get the same KVM internal error. Suberror: 1 emulation failure after a short time :\
May have to put the card in another machine to confirm all is working ok for a start (used card I just picked up). Or just stick with ATI cards..

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I need them as without these two options I was getting some errors about VGA initialization (or something like that).
Adding "rombar" and "romfile" the errors disappeared.

Those errors are what we need to debug.  rombar=0 tells QEMU to not expose a PCI option ROM for the device.  Following that with romfile= makes QEMU put that rom in the genroms section, which is sort of like having it builtin to the BIOS.  The problem with that is that it is no longer associated with the device, so unless you know ROM has the ability to scan through all the devices to find the hardware, you've probably just broken the option ROM execution.

Remove the rombar option and report the error, if there is one.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users