Arch Linux

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Code status update:

Linux v3.12 includes support for bus reset and the VFIO interfaces to use it, so you can update to the latest rc release and hopefully throw away any kernel patches you've been using.

On the QEMU front, you'll need to get this series to match the above:

http://lists.nongnu.org/archive/html/qe … 00494.html

Apply to latest qemu.git or pull my tag.

In addition to the bus reset, I've reworked how and when we read the device option ROM, so if you previously needed to supply a romfile= to make things work, try without it.

Also, this patch should hopefully solve the Code 43 issues with Nvidia on Intel for many of you:

http://lists.nongnu.org/archive/html/qe … 00597.html

Thanks,
Alex

Awsome news! thanks a lot Alex!

SpacePirate

Member

Registered: 2013-09-16

Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Code status update:

Linux v3.12 includes support for bus reset and the VFIO interfaces to use it, so you can update to the latest rc release and hopefully throw away any kernel patches you've been using.

On the QEMU front, you'll need to get this series to match the above:

http://lists.nongnu.org/archive/html/qe … 00494.html

Apply to latest qemu.git or pull my tag.

In addition to the bus reset, I've reworked how and when we read the device option ROM, so if you previously needed to supply a romfile= to make things work, try without it.

Also, this patch should hopefully solve the Code 43 issues with Nvidia on Intel for many of you:

http://lists.nongnu.org/archive/html/qe … 00597.html

Thanks,
Alex

So with this new progress i will need a linux 3.12-rc1 or later and a pull from alex` repository (https://github.com/awilliam/qemu-vfio/r … 20131003.0)?
It looks like seabios was merged into qemu so i wont need to install it?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Code status update:

Linux v3.12 includes support for bus reset and the VFIO interfaces to use it, so you can update to the latest rc release and hopefully throw away any kernel patches you've been using.

On the QEMU front, you'll need to get this series to match the above:

http://lists.nongnu.org/archive/html/qe … 00494.html

Apply to latest qemu.git or pull my tag.

In addition to the bus reset, I've reworked how and when we read the device option ROM, so if you previously needed to supply a romfile= to make things work, try without it.

Also, this patch should hopefully solve the Code 43 issues with Nvidia on Intel for many of you:

http://lists.nongnu.org/archive/html/qe … 00597.html

Thanks,
Alex

So with this new progress i will need a linux 3.12-rc1 or later and a pull from alex` repository (https://github.com/awilliam/qemu-vfio/r … 20131003.0)?
It looks like seabios was merged into qemu so i wont need to install it?

Right, you can use seabios as-is in qemu.  If you use my tag note that it does not include the patch in the 2nd link.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

SpacePirate

Member

Registered: 2013-09-16

Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

...

So with this new progress i will need a linux 3.12-rc1 or later and a pull from alex` repository (https://github.com/awilliam/qemu-vfio/r … 20131003.0)?
It looks like seabios was merged into qemu so i wont need to install it?

Right, you can use seabios as-is in qemu.  If you use my tag note that it does not include the patch in the 2nd link.

Alright, I will try to get it running with this. Since i use AMD i dont think i need that patch.

andy123

Member

Registered: 2011-11-04

Posts: 169

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, so I tried 3.12-rc3 and qemu from https://github.com/awilliam/qemu-vfio.
It resulted in a lot of error messages in dmesg, but it worked.
Afterwards, if I start qemu, it says

file_ram_alloc: can't mmap RAM pages: Cannot allocate memory

and does't seem to use my 4 1gb hugepages…
I'll reboot later and try again. My current kernel is compiled with "# CONFIG_DEBUG_KERNEL is not set" for some reason, is a recompile with "CONFIG_DEBUG_KERNEL=y" likely to give useful information?

While I'm at it, has there been any news on the broken hardware front (http://lkml.indiana.edu/hypermail/linux … 00079.html)?

---

i'm sorry for my poor english wirting skills…

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, so I tried 3.12-rc3 and qemu from https://github.com/awilliam/qemu-vfio.
It resulted in a lot of error messages in dmesg, but it worked.
Afterwards, if I start qemu, it says

file_ram_alloc: can't mmap RAM pages: Cannot allocate memory

and does't seem to use my 4 1gb hugepages…
I'll reboot later and try again. My current kernel is compiled with "# CONFIG_DEBUG_KERNEL is not set" for some reason, is a recompile with "CONFIG_DEBUG_KERNEL=y" likely to give useful information?

While I'm at it, has there been any news on the broken hardware front (http://lkml.indiana.edu/hypermail/linux … 00079.html)?

dmesg errors are all from kvm, not vfio.  Do these happen regardless of an assigned device?  What if you don't use hugepages?  What if you boot the guest with a single CPU?  WRT broken hardware, no time lately.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

andy123

Member

Registered: 2011-11-04

Posts: 169

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

…

dmesg errors are all from kvm, not vfio.  Do these happen regardless of an assigned device?  What if you don't use hugepages?  What if you boot the guest with a single CPU?  WRT broken hardware, no time lately.

Yeah, right. To clarify, the vm starts despite the mmap error and without "-mem-path" the error goes away. Not assinging devices or using only one cpu doesn't change anything.
I did some testing and the first vm start after a reboot results in no error from qemu, but the dmesg errors and everything afterwards returns "file_ram_alloc: can't mmap RAM pages: Cannot allocate memory" (but dmesg stays the same).
I tried something simple as a first start to try to reduce error sources:

/usr/bin/qemu-system-x86_64 -enable-kvm -m 4096 -mem-path /dev/hugepages -drive file=/files/vm/arch.img,id=drive0,if=virtio,media=disk -monitor stdio

In case I have the same problem with upstream stable qemu, any idea where to report this?

---

i'm sorry for my poor english wirting skills…

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dmesg errors are all from kvm, not vfio.  Do these happen regardless of an assigned device?  What if you don't use hugepages?  What if you boot the guest with a single CPU?  WRT broken hardware, no time lately.

Yeah, right. To clarify, the vm starts despite the mmap error and without "-mem-path" the error goes away. Not assinging devices or using only one cpu doesn't change anything.

What happens with 2M hugepages?

I did some testing and the first vm start after a reboot results in no error from qemu, but the dmesg errors and everything afterwards returns "file_ram_alloc: can't mmap RAM pages: Cannot allocate memory" (but dmesg stays the same).
I tried something simple as a first start to try to reduce error sources:

/usr/bin/qemu-system-x86_64 -enable-kvm -m 4096 -mem-path /dev/hugepages -drive file=/files/vm/arch.img,id=drive0,if=virtio,media=disk -monitor stdio

In case I have the same problem with upstream stable qemu, any idea where to report this?

My qemu tree only modifies vfio, so if it happens without an assigned device, it should happen on qemu.git.  If you can narrow it down (or even if you can't), please report it to:

kvm@vger.kernel.org, cc qemu-devel@nongnu.org

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

andy123

Member

Registered: 2011-11-04

Posts: 169

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What happens with 2M hugepages?

My qemu tree only modifies vfio, so if it happens without an assigned device, it should happen on qemu.git.  If you can narrow it down (or even if you can't), please report it to:

kvm@vger.kernel.org, cc qemu-devel@nongnu.org

There are no problems with 2M hugepages.
Thanks for those adresses, I did some futher testing and I noticed, that the hugepages seem to "fill up". With 4x1G hugepages, I can start and stop 2 virtual machines with 2gb ram with dmesg errors, but without qemu telling me, that it can't mmap, 4 virtualmachines if they use 1gb each.

---

i'm sorry for my poor english wirting skills…

tarcisioe

Member

Registered: 2013-10-06

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I am trying this with the following setup:

CPU: i7 2600 (supports VT-d)
Motherboard: Gigabyte GA-P67A-UD3-B3 (Current BIOS F4a, VT-d is enabled on BIOS setup)
Video cards:
- GeForce GT 620 (for Linux host)
- GeForce GTX 650 Ti (for Windows)

I can't seem to turn Intel_IOMMU on, though. When I try, Arch boots to weird messages about "can't find DMAR" for various devices and finally hangs, unable to find the hard drive.

The kernel line I am (trying to, actually) using is:
linux    /vmlinuz-linux root=UUID=89c6d568-9a2f-4de8-ad98-5f210c6a7163 rw  quiet iommu=1 intel_iommu=on pci-stub.ids=10de:11c6,10de:0b0e

I can't seem to get past this. Without intel_iommu=on, I get
ĺs: cannot access  /sys/bus/pci/devices/0000:01.00.0/iommu_group/devices: No such file or directory
ĺs: cannot access  /sys/bus/pci/devices/0000:01.00.1/iommu_group/devices: No such file or directory

when trying to run vfio-bind.

Any help will be much appreciated, so thanks in advance.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I am trying this with the following setup:

CPU: i7 2600 (supports VT-d)
Motherboard: Gigabyte GA-P67A-UD3-B3 (Current BIOS F4a, VT-d is enabled on BIOS setup)
Video cards:
- GeForce GT 620 (for Linux host)
- GeForce GTX 650 Ti (for Windows)

I can't seem to turn Intel_IOMMU on, though. When I try, Arch boots to weird messages about "can't find DMAR" for various devices and finally hangs, unable to find the hard drive.

The kernel line I am (trying to, actually) using is:
linux    /vmlinuz-linux root=UUID=89c6d568-9a2f-4de8-ad98-5f210c6a7163 rw  quiet iommu=1 intel_iommu=on pci-stub.ids=10de:11c6,10de:0b0e

I can't seem to get past this. Without intel_iommu=on, I get
ĺs: cannot access  /sys/bus/pci/devices/0000:01.00.0/iommu_group/devices: No such file or directory
ĺs: cannot access  /sys/bus/pci/devices/0000:01.00.1/iommu_group/devices: No such file or directory

when trying to run vfio-bind.

Any help will be much appreciated, so thanks in advance.

IOMMU support is required for VFIO.  Make sure VT-d support is enabled in your BIOS, update to a newer BIOS, and possibly contact your motherboard vendor about fixing their VT-d support.  Perhaps others with this board can share their experience.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

tarcisioe

Member

Registered: 2013-10-06

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

IOMMU support is required for VFIO.  Make sure VT-d support is enabled in your BIOS, update to a newer BIOS, and possibly contact your motherboard vendor about fixing their VT-d support.  Perhaps others with this board can share their experience.

Indeed, updating to the newest BIOS (F9) made me able to boot with intel_iommu=on. For anyone who may care, the option "M.I.T. Intelligent Tweaker > Miscellaneous Settings > Virtualization technology" apparently enables VT-x as well as VT-d on the newer Gigabyte BIOS for GA-P67A-UD3-B3. There is no option called "VT-d" as it was called on F4a.

Now dmesg tells me that Intel IOMMU is on, but I can't seem to get the iommu_group directories under /sys/bus/pci/devices/0000:01:00.1 yet. I can't load vfio_iommu_type1 directly, but I'll try to compile it built-in just in case.

I'm not with my PC right now but I can post any other needed information as soon as I get home.

Thanks

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

IOMMU support is required for VFIO.  Make sure VT-d support is enabled in your BIOS, update to a newer BIOS, and possibly contact your motherboard vendor about fixing their VT-d support.  Perhaps others with this board can share their experience.

Indeed, updating to the newest BIOS (F9) made me able to boot with intel_iommu=on. For anyone who may care, the option "M.I.T. Intelligent Tweaker > Miscellaneous Settings > Virtualization technology" apparently enables VT-x as well as VT-d on the newer Gigabyte BIOS for GA-P67A-UD3-B3. There is no option called "VT-d" as it was called on F4a.

Now dmesg tells me that Intel IOMMU is on, but I can't seem to get the iommu_group directories under /sys/bus/pci/devices/0000:01:00.1 yet. I can't load vfio_iommu_type1 directly, but I'll try to compile it built-in just in case.

I'm not with my PC right now but I can post any other needed information as soon as I get home.

Thanks

If there's no iommu_group link or /sys/kernel/iommu_groups/ is not populated, that suggests that intel-iommu isn't really enabled.  Those are provided by the iommu driver, not vfio.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

ttouch

Member

From: /dev/null

Registered: 2012-05-27

Posts: 130

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sorry to bother again,

I get output only when I omit -vga none.
Is the card used? Or I use qemu's graphics emulation?

---

GitHub | Git Server | Blog

SpacePirate

Member

Registered: 2013-09-16

Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

A somewhat off-topic question: how can i pass isos/images/partitions to the guest? using -cdrom or -drive doesnt work for me. For the isos i have found a way by adding an ahci controller with an cd-drive attached to it, but that looks more like a workaround to me.

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sorry to bother again,

I get output only when I omit -vga none.
Is the card used? Or I use qemu's graphics emulation?

Have you tried the lastest kernel rc + alex qemu tree?

Code status update:

Linux v3.12 includes support for bus reset and the VFIO interfaces to use it, so you can update to the latest rc release and hopefully throw away any kernel patches you've been using.

On the QEMU front, you'll need to get this series to match the above:

http://lists.nongnu.org/archive/html/qe … 00494.html

Apply to latest qemu.git or pull my tag.

In addition to the bus reset, I've reworked how and when we read the device option ROM, so if you previously needed to supply a romfile= to make things work, try without it.

Also, this patch should hopefully solve the Code 43 issues with Nvidia on Intel for many of you:

http://lists.nongnu.org/archive/html/qe … 00597.html

Thanks,
Alex

Last edited by nbhs (2013-10-09 19:19:08)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

A somewhat off-topic question: how can i pass isos/images/partitions to the guest? using -cdrom or -drive doesnt work for me. For the isos i have found a way by adding an ahci controller with an cd-drive attached to it, but that looks more like a workaround to me.

Its on the main guide, you must add a sata/ide/virtio controller then attach a disk to it

SpacePirate

Member

Registered: 2013-09-16

Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

A somewhat off-topic question: how can i pass isos/images/partitions to the guest? using -cdrom or -drive doesnt work for me. For the isos i have found a way by adding an ahci controller with an cd-drive attached to it, but that looks more like a workaround to me.

Its on the main guide, you must add a sata/ide/virtio controller then attach a disk to it

Ah, now i see it, thanks!

EDIT: I`m afraid I can`t get Windows to install. It stops at "Windows data is being expanded (0%)..." i waited for a while but I cant see any progress... Any ideas?

Last edited by SpacePirate (2013-10-09 20:31:16)

tarcisioe

Member

Registered: 2013-10-06

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If there's no iommu_group link or /sys/kernel/iommu_groups/ is not populated, that suggests that intel-iommu isn't really enabled.  Those are provided by the iommu driver, not vfio.

Oh, I didn't check /sys/kernel/iommu_groups. It is indeed not populated. How can intel-iommu not be really enabled, though? I even got a bunch of errors when trying to use it with my old BIOS, and they were fixed by updating it to the latest one.

Don't want this second graphics card to be a waste of money, I am positive that my CPU supports IOMMU and was informed by Gigabyte's technical support that my motherboard indeed supports VT-d.

Any suggestions?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

A somewhat off-topic question: how can i pass isos/images/partitions to the guest? using -cdrom or -drive doesnt work for me. For the isos i have found a way by adding an ahci controller with an cd-drive attached to it, but that looks more like a workaround to me.

Its on the main guide, you must add a sata/ide/virtio controller then attach a disk to it

Ah, now i see it, thanks!

EDIT: I`m afraid I can`t get Windows to install. It stops at "Windows data is being expanded (0%)..." i waited for a while but I cant see any progress... Any ideas?

AHCI is currently extremely slow in QEMU, much slower than IDE on 440FX.  You can use pidstat to see if there's actually I/O to the QEMU process.  Suggest waiting it out and switch to virtio once installed (or try to attach IDE or switch to 440FX model).

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If there's no iommu_group link or /sys/kernel/iommu_groups/ is not populated, that suggests that intel-iommu isn't really enabled.  Those are provided by the iommu driver, not vfio.

Oh, I didn't check /sys/kernel/iommu_groups. It is indeed not populated. How can intel-iommu not be really enabled, though? I even got a bunch of errors when trying to use it with my old BIOS, and they were fixed by updating it to the latest one.

Don't want this second graphics card to be a waste of money, I am positive that my CPU supports IOMMU and was informed by Gigabyte's technical support that my motherboard indeed supports VT-d.

Any suggestions?

dmesg or it didn't happen.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

tarcisioe

Member

Registered: 2013-10-06

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dmesg or it didn't happen.

LOL ok.

[root@jack tarcisioe]# dmesg | grep -e IOMMU
[    0.000000] Intel-IOMMU: enabled

Anything else I should grep for?

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've uploaded new builds for kernel 3.12-rc4, qemu-git + alex patches, and the lastest seabios, all working fine on my pc, ive also cleaned up the guide a bit to reflect the lastest changes

SpacePirate

Member

Registered: 2013-09-16

Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Maybe you could include a note that the AHCI controller will be slow in the first post. using the IDE controller it works fine for me now.

Also i think the best practice to create qemu images is using qemu-img instead of dd, isnt it? I dont know if there are any differences but so far i have only seen the use of qemu-img in other tutorials/wikis/the documentation.

I now got an Windows installation running but cant install the driver for the 6450 on the guest. When I use the installer I get a BSOD and if i try to use Windows` driver update functionality it detects the GPU but says "Failed to read from device"...

Last edited by SpacePirate (2013-10-10 11:12:57)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dmesg or it didn't happen.

LOL ok.

[root@jack tarcisioe]# dmesg | grep -e IOMMU
[    0.000000] Intel-IOMMU: enabled

Anything else I should grep for?

You should have a lot more output than that, for example:

$ dmesg | grep -e IOMMU
[    0.000000] Intel-IOMMU: enabled
[    0.020973] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.020977] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.021052] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.414246] IOMMU 0 0xfed90000: using Queued invalidation
[    0.414248] IOMMU 1 0xfed91000: using Queued invalidation
[    0.414251] IOMMU: Setting RMRR:
[    0.414276] IOMMU: Setting identity map for device 0000:00:02.0 [0xdb800000 - 0xdf9fffff]
[    0.415007] IOMMU: Setting identity map for device 0000:00:1d.0 [0xda3ba000 - 0xda3d0fff]
[    0.415044] IOMMU: Setting identity map for device 0000:00:1a.0 [0xda3ba000 - 0xda3d0fff]
[    0.415077] IOMMU: Setting identity map for device 0000:00:14.0 [0xda3ba000 - 0xda3d0fff]
[    0.415095] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.415108] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users