Arch Linux

The_Moves

Member

Registered: 2015-01-06

Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Without knowing the host cpu or guest cpu topology, this is pretty meaningless

My processor is an X5660, and I believe I have it setup correct with 2 Cores and 2 Thread. My question is, are the core id the same between 'isolcpus' and my 'vm config'? Is 2 in one the same 2 in the other?

Here is the core thread layout for my Xeon X5660:

[root@kvmhost1 ~]# egrep "^processor|^core id" /proc/cpuinfo
processor       : 0
core id         : 0
processor       : 1
core id         : 1
processor       : 2
core id         : 2
processor       : 3
core id         : 8
processor       : 4
core id         : 9
processor       : 5
core id         : 10
processor       : 6
core id         : 0
processor       : 7
core id         : 1
processor       : 8
core id         : 2
processor       : 9
core id         : 8
processor       : 10
core id         : 9
processor       : 11
core id         : 10
[root@kvmhost1 ~]#

You're messing with the cpu affinity of the qemu process, not the vCPUs.  There's an <emulatorpin> control for this https://libvirt.org/formatdomain.html#elementsCPUTuning

I will look into this.

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek :

I'm using raw disk images (.img) for my VMs . No block devices or zvols . I'm just looking for a way to copy those images to a backup location but qemu always crashes along with the host .

@jasonwryan :

How is my question about qemu VMs disk images unrelated to this qemu thread ?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How is my question about qemu VMs disk images unrelated to this qemu thread ?

How is a question about snapshotting a running VM disk image unrelated to graphics assignment?  Virtualization is a very broad topic, not every question about running or managing a QEMU VM is relevant to this particular thread.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well, there's positive news about my project and negative.
The positive news are:
Three GPUs actually WORK! Separately.
So i can mine my whatever-coins in a virtual machine. A strange thing to do, but combined with openstack that could be a base for some virtual private miner(VPS, but for mining) service.. ...althrough it'd be as pointless as /dev/null as a service.
But also i can crack some passwords! oclHashCat is able to utilize all three GPUs at their full power, so cracking NTLM hashes from filthy wind0wz is now 33% faster, heh.
I can attach a whooping number of screens to the VM: Nine. There was a version of HD7750 with 6 mini-dp ports, but it's cost was nowhere near my setup.
That 9 screens would work with AMD EyeFinity. I've actually tried doing three screens(one fullhd 24" and two square 1280x1024 17") on two GPUs, and i've ran out of video memory quickly, 2gb cards would be much nicer. AFAIK, the video memory does stack when using XDMA crossfire.
But i've distracted from the topic, back to qemu again.
Crossfire works with two cards only even if there's only one card running using vfio-pci and the others are one pci-assign from xen, which is considered legacy.
I've managed to find a way of determining which cards are crossfire-bound - the unbound card still has it's video outputs working, the paired one shares the display connectors with the master.
Also, i am really able to run two or three VMs in parallel, so it's three "seats" per one case. Well, almost, as the third card does not fit inside.
http://i.imgur.com/PN6XNaq.jpg
Again, sorry for the big image.
Negative news:
Crossfire with 3 cards totally doesn't work in my setup.
Maybe the culprit is pci-e bandwidth, as pci-e 2.0 x1 is not enough to overwhelm that physical crossfire bridge.
Or maybe the AMD support was right, and crossfire requires an even number of cards to work. I hope i'll find it out someday, but most likely not.

Now the third card goes to the sponsor(he paid for half of the card) and i continue to run crossfire with two cards, hoping to sell them one day and buy a GTX 750 KalmX. Or...
Hmm, what if i'll find a PLX-based switch which does ACS and stuff and presents three cards as if they were all connected to the proccesor's root port?..
Not only there would be a somewhat bandwidth increase(maybe), but it would enable me to use vfio-pci for all three cards eliminating the reset problems.
Sad thing that pci-e switches are rare an expensive, especially in my location. If i could only borrow it for some time...
The alternate way of proceeding on my way to tri-way crossfire in a VM would be using something on A85X/88X with three pci-e 16x slots(they are hopefully routed as processor-root-ports).
Or running windows bare-metal! That's a thing to try.

-- mod edit: read the Forum Etiquette and only post thumbnails http://wiki.archlinux.org/index.php/For … s_and_Code [jwr] --

Last edited by Duelist (2015-06-30 22:33:09)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Dang , I forgot the thread's title completely  : (

@Duelist :

I guess you can run three AMD gpus on crossfire . I think they call it TriFire but don't quote me on that . I've never purchased one since the ATI days .
PCI-E expansion chasis are really expensive . Magma sells a 3 slot chassis for over $2000 . And they even have a 10 slots chassis for an unknown price , probably would be over $7000 . You're better off building more virtualization hosts with regular hardware .

Last edited by Denso (2015-07-01 00:10:36)

devianceluka

Member

Registered: 2014-05-19

Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

My ASMedia USB 3.0 is working flawlessly (from 1 page before) with OVMF-pure-efi, but not with OVMF-with-csm and QEMU. Can I do something about it anywhere in OVMF-with-csm (because the GPU used with ASMedia is not GOP capable) to load EFI for USB controllers?

EDIT:
On QEMU I get it to work almost flawlessly with Legacy USB 3.0 disabled in BIOS + pcie_aspm=off. After 10-30 reboots of the OS/VM I get DMAR error on host and only the reboot of the host solves it. It looks like the USB controller gets into a power saving mode or whatever and it doesnt unstick/reset anymore and only a host reboot solves it. Can it be fixed with any command/patch? To reset manually?

Last edited by devianceluka (2015-07-01 12:18:04)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Crossfire with 3 cards on my setup doesn't work even on bare metal.
But i've been able to record some benchmarks:
Unigine Heaven Benchmark 4.0: D3D11, quality medium, resolution 1920x1080 fullscreen(basic preset with full resolution).
Bare metal: 654 single card(100%), 1244 crossfire(190% performance with crossfire)
VM:630 single card(96%), 10131087 crossfire(160%173% crossfire from single VM card, 81%87% from bare metal performance).
I ran only one run, so i think some CPU activity on host did interfere with crossfire VM run.
There's also some stuttering occuring in the crossfire VM, i think about turning iommu=pt to iommu=on and maybe running PCAP or something to capture that and try to align it with host CPU and/or disk usage.
Sadly, toggling iommu=pt to iommu=on didn't help much. There's still stuttering present when unigine shows a new area or scene. Especially this is seen in the end of scene #19, where the camera pans to the city, and in the beginning of scene #22, where there's a fade in in the city.
If anyone is able to reproduce that lags, please tell.

Now i'll try to setup a second VM with the third card only.

Last edited by Duelist (2015-07-01 12:57:53)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

The_Moves

Member

Registered: 2015-01-06

Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sadly, toggling iommu=pt to iommu=on didn't help much. There's still stuttering present when unigine shows a new area or scene. Especially this is seen in the end of scene #19, where the camera pans to the city, and in the beginning of scene #22, where there's a fade in in the city.
If anyone is able to reproduce that lags, please tell.

Now i'll try to setup a second VM with the third card only.

This same issue happens to me. For instance, in World of Tanks, gliding over different tanks with the cursor causes the screen to freeze for a split second then jump ahead. In GTA5, some textures take a while to load when traveling at a high rate of speed, but this may be normal. I also noticed the issue on previous Windows 10 builds when swapping between the ICONS. As I play different games, i might be able to reproduce the issue.

Another issue I noticed was with GTA4, when I would load it, it would just crash the host to crash.

I think it might be good to look at different performance recording statistics from the host end, like iostat and vmstat.

With Windows 10, when opening up Task Manager, I also noticed that Windows knows it is a VM. I thought 'kvm=off' was supposed to fix that, but maybe it is just with NVIDIA Drivers?

devianceluka

Member

Registered: 2014-05-19

Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Why nobody wants to help?

The_Moves

Member

Registered: 2015-01-06

Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Why nobody wants to help?

Why are you using 'OVMF-with-csm'?

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Why nobody wants to help?

If you insist:
reset issues are tough. And they require quirks for particular devices.
BUT. Why do you p-t a USB controller, if there's a device passthrough, which gives almost 100% performance and is trouble-free?

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

devianceluka

Member

Registered: 2014-05-19

Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sorry for taking so long, took a nap after nerd rage... I'm already trying for 4 days straight, 12h minimum each day, trying ALOT of combinations of BIOS and linux stuff and rebooting.

Why nobody wants to help?

Why are you using 'OVMF-with-csm'?

Because the GPU used with ASMedia does not support GOP (R7 240).

OVMF-pure-efi - works flawlessly (okay in 100 reboots, it didnt recognize it 1 time - its no problem)
SeaBIOS - works half the time
OVMF-with-csm - doesnt work, before booting Windows logo and after OVMF logo there's " _ " for 1-2 minutes and then it boots and doesnt recognize it in windows. If I disconnect the keyboard before starting VM and connecting back when windows boots, it works "flawlessly"

The real question is why wouldnt it boot with csm and only with pure-efi? Tried to remove all uhci/xhci/usb devices in XML, thought it might be in a dilemma which to choose and it doesnt fix it.

The only explanation from me would be that because SeaBIOS sometimes works sometimes doesnt (legacy ROM), OVMF-with-csm is using legacy ROM from USB controller also then... Where and how to change it to use EFI ROM for everything except GPU?

Why nobody wants to help?

If you insist:
reset issues are tough. And they require quirks for particular devices.
BUT. Why do you p-t a USB controller, if there's a device passthrough, which gives almost 100% performance and is trouble-free?

It is not 100% identical. It is bad for gaming. It is software and CPU accelerated. Would you flash your Android with an emulated USB controller? The question is only time when a trouble will arise and something will not be compatible... I have 4 GPU and 4 USB, so...

----

Also one observation made by me... Sometimes SeaBIOS when it doesnt work, it in my opinion"locks" the controller and even after host reboot it doesnt work. Only solution is pcie_aspm=off to pcie_aspm=force and it works after next host reboot (policy powersave ; performance & default do not work ever).

----

Before someone says it is a BIOS/firmware bug... It MAY be, but it is FIXABLE (by me), but needs a human hand. There 100% must be a software solution/fix. As I'm saying I get like 3-4 lines on host: "dmar dma write request device fault addr". Also if EFI is working, there should atleast be a setting where one can force (like in real MB UEFI BIOS) to force EFI on devices (CSM) if the devices have working EFI ROMs atleast.

----

Host:
ASUS P9X79 PRO
i7-3820
16GB DDR3
256GB 850 PRO + 128GB 850 EVO
2x 2TB WD Black
2x R9 290
2x R7 240
4.0.6-300.fc22.x86_64 (virt-preview - everything latest!)
GRUB: intel_iommu=on pci-stub.ids=1b4b:9130,1b21:0612,1002:6613,1002:aab0,1002:6798,1002:aaa0,8086:1d26,8086:1d2d kvm-intel.nested=1 iommu=pt pcie_aspm=off vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1

----

Almost forgot. If I remove ASMedia USB 3.0 controller from pci-stub, it increases the chances of working in SeaBIOS and surviving reboots by ALOT....

Last edited by devianceluka (2015-07-01 18:22:41)

zir_blazer

Member

Registered: 2013-12-12

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ aw

Did you found any reelevant info for the VT-d support in Chipset discussion? I was waiting for it. With the huge amount of contradicting info going around from both Intel and manufacturers, I still want a final word on that, and you are close enough to the Intel guys to get it (And by final word, I mean someone that can also explain why the documentation incosistency).
https://bbs.archlinux.org/viewtopic.php … 9#p1523059

I'm still pondering how reelevant it is to get a Chipset that Intel claims that has VT-d support, in case I have to make or recommend a new build for this usage. Yes, I know that you can get this working on Desktop, yet I want to know if Server parts have a definite advantage which is not Firmware related.

Also, I'm aware that LGA 1155/1150 parts does not support ACS for the PCIe Controller of the Processor, so all PCIe slots that are connected to it should fall into the same IOMMU Group. However, assuming there is a Motherboard that adds a PLX chip (Typically you see it in some high end Motherboards that claims SLI/Crossfire in 16x/16x mode instead of 8x/8x), can the PLX chip itself isolate the PCIe Cards below it even if there is no isolation between Processor PCIe Controller and the PLX chip?
What about the new trend of PCIe SSDs supporting NVMe? Aren't they PCI devices? Could I do something like, do PCI Passthrough of a PCIe SSD to a VM, then using the OVMF Firmware and QEMU NVMe support to boot from it? I'm also curious on how SATA Express and M.2 devices looks like from this point of view (Do they appear in lspci?), since they use PCIe Lanes and also SATA, which could come from two different places (SATA from Chipset, but PCIe Lanes can come from either the same Chipset or from the Processor itself). I expect these two to be rather hard to work with.

I've heard some news about XenGT(or KVMGT or whatever) in linux 4.1, but lkml doesn't hint me on details.
Can anyone give me a brief info about it's state? Does anybody knows about it?

According to Phoronix there was something related to XenGT that was included in 4.1:
http://www.phoronix.com/scan.php?page=n … -DRM-First

It talks about "client side" and "guest". I believe than it means that the Linux 4.1 Intel i915 Driver is XenGT/KVMGT ready from inside a VM, but it still doesn't mention anything about host side (Since it needs support from both the Linux kernel, QEMU, SeaBIOS, and for Xen, Xen itself. Not sure if the KVM component also needs a hand or is included in the Linux kernel part). So basically, I take that you don't need a special build of the Drivers to use the virtualized GPU from inside a VM, but you would still need to set up the host.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@devianceluka:
Things to try also:
Flashing the motherboard's firmware. Maybe there are updates of that asmedia(i suppose it's soldered to the board just like mine) controller's firmware too.
Digging into the GPU in question - I have R7 250E and i've got my UEFI support working by doing weird and strange things which i completely understand now. Maybe you'll be able to pull that trick too?
Modifying the firmware itself would be almost impossible, sadly. But, i think, you can feed the firmware with any EFI driver for a device just by putting them on a right filesystem(fat32 of some sort). And it will load. Sadly, USB controllers have nothing to do with pci option roms.

P.S.

Would you flash your Android

I'd flash it with a flashbang grenade. Considering all this mess we have on x86 platforms, i don't even want to know anything about android and vendors and hardware and driver blobs and stuff. I don't want to loose my last remains of sanity. So i'm stuck with a buttoney half chinese flip-phone and a dead sony err.. with some soldering iron modifications. I don't have much social life to rely on all those funky whatsapp-viber-facebook-twitter stuff, i don't even get enough e-mails.
If you need 100% insurance that everything will work - boot the windows baremetal if your operation is not available on linux host. If you have security concerns about that, unplug everything unneeded from the machine beforehand. If you set it up right, that is an easy task to do.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

devianceluka

Member

Registered: 2014-05-19

Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What are you saying about UEFI GPU trickery? Dont understand. You hacked it or what? Got a link?
Where to search for asmedia fw update? The MB BIOS is latest...

iH8c0ff33

Member

Registered: 2015-07-02

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone,
I'm trying to passtrough an Nvidia GTX 770.
I used the AUR package linux-vfio and booting with "i915.enable_hd_vgaarb=1 intel_iommu=on" but i keep getting this error:
----------------------------------------------------------------------------------------------------------------------------------------------------------------
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio: error opening /dev/vfio/1: No such file or directory
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio: failed to get group 1                                               
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device initialization failed                                               
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device 'vfio-pci' could not be initialized
----------------------------------------------------------------------------------------------------------------------------------------------------------------
so i searched google for this error and found another boot parameter for ACS override but the error hasn't changed so much
----------------------------------------------------------------------------------------------------------------------------------------------------------------
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio: error opening /dev/vfio/13: No such file or directory
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio: failed to get group 13
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device initialization failed
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device 'vfio-pci' could not be initialized
----------------------------------------------------------------------------------------------------------------------------------------------------------------
I don't know what to do please help me
Thanks

devianceluka

Member

Registered: 2014-05-19

Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@devianceluka:
Things to try also:
Flashing the motherboard's firmware. Maybe there are updates of that asmedia(i suppose it's soldered to the board just like mine) controller's firmware too.
Digging into the GPU in question - I have R7 250E and i've got my UEFI support working by doing weird and strange things which i completely understand now. Maybe you'll be able to pull that trick too?
Modifying the firmware itself would be almost impossible, sadly. But, i think, you can feed the firmware with any EFI driver for a device just by putting them on a right filesystem(fat32 of some sort). And it will load. Sadly, USB controllers have nothing to do with pci option roms.

P.S.

Would you flash your Android

I'd flash it with a flashbang grenade. Considering all this mess we have on x86 platforms, i don't even want to know anything about android and vendors and hardware and driver blobs and stuff. I don't want to loose my last remains of sanity. So i'm stuck with a buttoney half chinese flip-phone and a dead sony err.. with some soldering iron modifications. I don't have much social life to rely on all those funky whatsapp-viber-facebook-twitter stuff, i don't even get enough e-mails.
If you need 100% insurance that everything will work - boot the windows baremetal if your operation is not available on linux host. If you have security concerns about that, unplug everything unneeded from the machine beforehand. If you set it up right, that is an easy task to do.

Are you going to let me in the dust waiting forever with your half completed answer? Can you elaborate more on GPU GOP trickery?

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Are you going to let me in the dust waiting forever with your half completed answer? Can you elaborate more on GPU GOP trickery?

Patience, my friend. I understand your passion, as i have adopted the same feeling some time ago. But sometimes i have to allocate some time on my RL problems. You'd be pretty shocked to know the details, so i won't share them.

I don't know anything about asmedia, but i suggest you digging in the other direction.
(I'd give you a link to my earlier messages about that, but i don't remember when i wrote them so..)
So. The idea is simple. My vendor(ASUS) supplied an update of GPU FW which enabled GOP and that image contained an .efi device driver.
BUT IT WAS BROKEN! So i went and googled for a supplement.
And i've found two things:
1. In the early days of UEFI and GOP(2008-2010, i think) AMD gave a presentation about GOP support being universal for all their 6 and 7 series GPUs. 8 and R5-7-9 stuff didn't exist that day.
2. Some mac fanatics compiled a ROM file with a efi driver from some 6XXX card. So that hinted me that i can take my broken firmware and replace the GOP part with the working one, taken from some 6-series(i think it was called barton or barts..) card.

That i did. I don't remember how to do that properly, so i'm not able to give you step-by-step instructions.
What you need to do: take my UEFI-capable ROM which i made for myself, use EDK2 tool called... called...
damn i forgot. Anyway, try using that to extract the .efi that really works and then you need to compress it back into your own "legacy-only" ROM made for your card exclusively. Append that romfile to your device in a VM(so no need to flash anything!) and hope it will work.

There are tools in EDK2 for that tasks, i just forgot how they were named and where are they.

I'll try lurking through my old messages to find the details, but considering your inpatience..

https://bbs.archlinux.org/viewtopic.php … 1#p1513281
Here. The message with the details and links and stuff. Modifying GPU ROM is easier that USB controllers'.
...just found out that that message contains a link to my previous message about that. LOL.

Last edited by Duelist (2015-07-02 18:41:58)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

devianceluka

Member

Registered: 2014-05-19

Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Are you going to let me in the dust waiting forever with your half completed answer? Can you elaborate more on GPU GOP trickery?

Patience, my friend. I understand your passion, as i have adopted the same feeling some time ago. But sometimes i have to allocate some time on my RL problems. You'd be pretty shocked to know the details, so i won't share them.

I don't know anything about asmedia, but i suggest you digging in the other direction.
(I'd give you a link to my earlier messages about that, but i don't remember when i wrote them so..)
So. The idea is simple. My vendor(ASUS) supplied an update of GPU FW which enabled GOP and that image contained an .efi device driver.
BUT IT WAS BROKEN! So i went and googled for a supplement.
And i've found two things:
1. In the early days of UEFI and GOP(2008-2010, i think) AMD gave a presentation about GOP support being universal for all their 6 and 7 series GPUs. 8 and R5-7-9 stuff didn't exist that day.
2. Some mac fanatics compiled a ROM file with a efi driver from some 6XXX card. So that hinted me that i can take my broken firmware and replace the GOP part with the working one, taken from some 6-series(i think it was called barton or barts..) card.

That i did. I don't remember how to do that properly, so i'm not able to give you step-by-step instructions.
What you need to do: take my UEFI-capable ROM which i made for myself, use EDK2 tool called... called...
damn i forgot. Anyway, try using that to extract the .efi that really works and then you need to compress it back into your own "legacy-only" ROM made for your card exclusively. Append that romfile to your device in a VM(so no need to flash anything!) and hope it will work.

There are tools in EDK2 for that tasks, i just forgot how they were named and where are they.

I'll try lurking through my old messages to find the details, but considering your inpatience..

https://bbs.archlinux.org/viewtopic.php … 1#p1513281
Here. The message with the details and links and stuff. Modifying GPU ROM is easier that USB controllers'.

Thanks man! I will look into it as that's my only option...

iH8c0ff33

Member

Registered: 2015-07-02

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone,
I'm trying to passtrough an Nvidia GTX 770.
I used the AUR package linux-vfio and booting with "i915.enable_hd_vgaarb=1 intel_iommu=on" but i keep getting this error:
----------------------------------------------------------------------------------------------------------------------------------------------------------------
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio: error opening /dev/vfio/1: No such file or directory
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio: failed to get group 1                                               
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device initialization failed                                               
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device 'vfio-pci' could not be initialized
----------------------------------------------------------------------------------------------------------------------------------------------------------------
so i searched google for this error and found another boot parameter for ACS override but the error hasn't changed so much
----------------------------------------------------------------------------------------------------------------------------------------------------------------
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio: error opening /dev/vfio/13: No such file or directory
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio: failed to get group 13
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device initialization failed
qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: Device 'vfio-pci' could not be initialized
----------------------------------------------------------------------------------------------------------------------------------------------------------------
I don't know what to do please help me
Thanks

Ok I think i solved the probled. I tried not to use the Systemd service and running vfio-bind many times then it worked O.o
Thanks anyway

sunmast

Member

Registered: 2014-12-16

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I'm trying to pass through a EVGA GTX 960 card with OVMF method (because I need to use Intel IGP on the host), but I'm hitting the VM unable to reboot problem (unable to reboot VM unless reboot host first, otherwise host freezes). It seems the card is not reset correctly between reboots.

Searching the thread it seems one workaround is to "eject" the card before Windows shutdown, but does this work for OVMF method? In Windows I don't see the card is removable. I've tried deveject and devcon (from WDK), they both don't work. I've also tried to disable the card before shutdown, it doesn't solve the problem.

I'm wondering if there is any command from host side can help? Like reset the card each time when qemu exits?

Thanks in advance!

maelask3

Member

Registered: 2015-07-03

Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Using Libvirt 1.2.17 and QEMU 2.3, Windows 8.1 setup BSODs (SYSTEM_SERVICE_EXCEPTION) no matter if the machine is i440fx or q35, or if the graphics card is passed through or not when it gets to Copying files.
Windows 10 setup doesn't even boot.
Edit : Worth mentioning : using OVMF with a Radeon R9 270X (Gigabyte OC)

Last edited by maelask3 (2015-07-03 12:31:09)

Zeorymer

Member

Registered: 2015-07-03

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AMD-Vi: device: 00:00.2 cap: 0040 seg: 0 flags: 3e info 1300
AMD-Vi:        mmio-addr: 00000000feb20000
AMD-Vi:   DEV_SELECT_RANGE_START         devid: 00:00.0 flags: 00
AMD-Vi:   DEV_RANGE_END                 devid: 00:00.2
AMD-Vi:   DEV_SELECT                         devid: 00:02.0 flags: 00
AMD-Vi:   DEV_SELECT_RANGE_START         devid: 01:00.0 flags: 00
AMD-Vi:   DEV_RANGE_END                 devid: 01:00.1
AMD-Vi:   DEV_SELECT                         devid: 00:04.0 flags: 00
AMD-Vi:   DEV_SELECT                         devid: 02:00.0 flags: 00
AMD-Vi:   DEV_SELECT                         devid: 00:05.0 flags: 00
AMD-Vi:   DEV_SELECT                         devid: 03:00.0 flags: 00
AMD-Vi:   DEV_SELECT                         devid: 00:06.0 flags: 00
AMD-Vi:   DEV_SELECT                         devid: 04:00.0 flags: 00
AMD-Vi:   DEV_SELECT                         devid: 00:07.0 flags: 00
AMD-Vi:   DEV_SELECT                         devid: 05:00.0 flags: 00
AMD-Vi:   DEV_SELECT                         devid: 00:11.0 flags: 00
AMD-Vi:   DEV_SELECT_RANGE_START         devid: 00:12.0 flags: 00
AMD-Vi:   DEV_RANGE_END                 devid: 00:12.2
AMD-Vi:   DEV_SELECT_RANGE_START         devid: 00:13.0 flags: 00
AMD-Vi:   DEV_RANGE_END                 devid: 00:13.2
AMD-Vi:   DEV_SELECT                         devid: 00:14.0 flags: d7
AMD-Vi:   DEV_SELECT                         devid: 00:14.3 flags: 00
AMD-Vi:   DEV_SELECT                         devid: 00:14.4 flags: 00
AMD-Vi:   DEV_ALIAS_RANGE                 devid: 06:00.0 flags: 00 devid_to: 00:14.4
AMD-Vi:   DEV_RANGE_END                 devid: 06:1f.7
AMD-Vi:   DEV_SELECT                         devid: 00:14.5 flags: 00
AMD-Vi:   DEV_SELECT                         devid: 00:15.0 flags: 00
AMD-Vi:   DEV_SELECT_RANGE_START         devid: 07:00.0 flags: 00
AMD-Vi:   DEV_RANGE_END                 devid: 07:00.1
AMD-Vi:   DEV_SELECT_RANGE_START         devid: 00:16.0 flags: 00
AMD-Vi:   DEV_RANGE_END                 devid: 00:16.2
AMD-Vi:   DEV_SPECIAL(IOAPIC[0])                devid: 00:14.0
AMD-Vi:   DEV_SPECIAL(HPET[0])                devid: 00:14.0
AMD-Vi:   DEV_SPECIAL(IOAPIC[255])                devid: 00:00.1
[Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table
[Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table
[Firmware Bug]: AMD-Vi: No southbridge IOAPIC found
AMD-Vi: Disabling interrupt remapping

I'm assuming my table is broken?

I'm not exactly sure how to interpert the iommu dump data.

EDIT:  May help

00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx0 port B) [1002:5a14] (rev 02)
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU) [1002:5a23]
00:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port B) [1002:5a16]
00:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port D) [1002:5a18]
00:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port E) [1002:5a19]
00:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port F) [1002:5a1a]
00:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port G) [1002:5a1b]
00:11.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] [1002:4391] (rev 40)
00:12.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
00:12.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
00:13.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
00:13.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller [1002:4385] (rev 42)
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller [1002:439d] (rev 40)
00:14.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge [1002:4384] (rev 40)
00:14.5 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller [1002:4399]
00:15.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) [1002:43a0]
00:16.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
00:16.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0 [1022:1600]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1 [1022:1601]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2 [1022:1602]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3 [1022:1603]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4 [1022:1604]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5 [1022:1605]
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] RV630 XT [Radeon HD 2600 XT] [1002:9588]
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] RV630 HDMI Audio [Radeon HD 2600 Series] [1002:aa08]
02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 06)
03:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller [1b21:1042]
04:00.0 SATA controller [0106]: JMicron Technology Corp. JMB362 SATA Controller [197b:2362] (rev 10)
05:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller [1b21:1042]
06:06.0 Multimedia audio controller [0401]: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] [13f6:8788]
07:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti PRO [Radeon HD 7950/8950 OEM / R9 280] [1002:679a]
07:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] [1002:aaa0]

Last edited by Zeorymer (2015-07-03 15:58:11)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm assuming my table is broken?

Try

ivrs_ioapic[7]=00:14.0 ivrs_ioapic[8]=00:00.1

mapping ioapic directly. Adjust accordingly, if i recall correctly - that should work for 990FX...

BTW, what versions of everything do you use? I observe fixes for this being added somewhere in 2012. AW may correct me.

Last edited by Duelist (2015-07-03 17:13:45)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Zeorymer

Member

Registered: 2015-07-03

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

http://i.imgur.com/Mvd6M1B.png
I'm assuming my table is broken?

Try

ivrs_ioapic[7]=00:14.0 ivrs_ioapic[8]=00:00.1

mapping ioapic directly. Adjust accordingly, if i recall correctly - that should work for 990FX...

BTW, what versions of everything do you use? I observe fixes for this being added somewhere in 2012. AW may correct me.

Even though it complains about ioapic[9] and ioapic[10]?  Then again, iommu is not my expertise.

Kernel is -ck with required options enabled for passthrough.

Everything else was in Arch repos, though I haven't really gotten that far thanks to Asus not knowing how to make a bios correctly.