Arch Linux

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Mouse response is normal , but the key strokes are delayed by ~5 seconds .

@Duelist :

"I would like to be able to provide access to the PCIe devices to the VMs while simultaneously allowing the host OS to still access the PCIe devices."

Last edited by Denso (2015-07-13 02:13:54)

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
I won't be uploading that xml, I'm trying now to run it on Win7, I'll be uploading this XML.
I tried OVMF, thinking it may make a difference, but Windows won't install at all using UEFI. With a Win8.1 ISO, the UEFI shell doesn't recognize the image, and with Win7, the screen goes black when the file copy begins, and the CPU maxes out. I'll update this post with whether the passthrough works with non-UEFI Win7.

Update:
Still doesn't work, OVMF was no success, and Win7 also has a code 43. If any more info is needed for a diagnosis, let me know.

Last edited by morat (2015-07-13 05:29:20)

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Duelist :

"I would like to be able to provide access to the PCIe devices to the VMs while simultaneously allowing the host OS to still access the PCIe devices."

As far as I know, that isn't possible, and even if it were, it would be unwise; data corruption etc. GPUs, network cards etc. are built to handle multiple outputs, but there's a reason a PCIe card can only fit in one slot.

Nazfellun

Member

Registered: 2012-10-22

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I currently have a working setup using Q35 + SeaBIOS, but am trying to change over to using i440fx + OVMF. The VBIOS for the card I'm attempting to pass-through does not appear to have an EFI section according to rom-parser, but the manufacturer has provided an updated version that does.

I've gotten a working VM without the card passed through, but adding the card and specifying the updated VBIOS with a <rom file="..." /> segment appears to just hang after boot -- there's no output on the physical monitor attached to the card, the VM consumes an absolutely constant amount of CPU time, and it never comes up on the network. Remove the card from the VM, and it boots just fine and I can reach it over VNC.

Any idea on how to troubleshoot further? Do I need to try flashing the updated vbios to the card? If so, how do I do this under Linux? I've tried atiflash, but it just returns 'Adapter not found' when I ask it to list current adapters...

Nazfellun

Member

Registered: 2012-10-22

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
I won't be uploading that xml, I'm trying now to run it on Win7, I'll be uploading this XML.
I tried OVMF, thinking it may make a difference, but Windows won't install at all using UEFI. With a Win8.1 ISO, the UEFI shell doesn't recognize the image, and with Win7, the screen goes black when the file copy begins, and the CPU maxes out. I'll update this post with whether the passthrough works with non-UEFI Win7.

Update:
Still doesn't work, OVMF was no success, and Win7 also has a code 43. If any more info is needed for a diagnosis, let me know.

The UEFI shell didn't recognize your Windows 8.1 ISO because it's in UDF format, not iso9660 -- I had this issue initially myself.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I currently have a working setup using Q35 + SeaBIOS, but am trying to change over to using i440fx + OVMF. The VBIOS for the card I'm attempting to pass-through does not appear to have an EFI section according to rom-parser, but the manufacturer has provided an updated version that does.

I've gotten a working VM without the card passed through, but adding the card and specifying the updated VBIOS with a <rom file="..." /> segment appears to just hang after boot -- there's no output on the physical monitor attached to the card, the VM consumes an absolutely constant amount of CPU time, and it never comes up on the network. Remove the card from the VM, and it boots just fine and I can reach it over VNC.

Any idea on how to troubleshoot further? Do I need to try flashing the updated vbios to the card? If so, how do I do this under Linux? I've tried atiflash, but it just returns 'Adapter not found' when I ask it to list current adapters...

I've had that problem and solved it.
If you have an ASUS HD7750 Silent - i can give you a working ROM.
You don't need to flash anything, just append as romfile option.

Hmmm.. finally a thing to put in my signature.

Last edited by Duelist (2015-07-13 08:17:54)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
I won't be uploading that xml, I'm trying now to run it on Win7, I'll be uploading this XML.
I tried OVMF, thinking it may make a difference, but Windows won't install at all using UEFI. With a Win8.1 ISO, the UEFI shell doesn't recognize the image, and with Win7, the screen goes black when the file copy begins, and the CPU maxes out. I'll update this post with whether the passthrough works with non-UEFI Win7.

Update:
Still doesn't work, OVMF was no success, and Win7 also has a code 43. If any more info is needed for a diagnosis, let me know.

The UEFI shell didn't recognize your Windows 8.1 ISO because it's in UDF format, not iso9660 -- I had this issue initially myself.

I got hopeful for a sec, but supposedly not:

[lain@navi .vm]$ file Windows7.iso 
Windows7.iso: ISO 9660 CD-ROM filesystem data 'GSP1RMCPRXFREO_EN_DVD' (bootable)
[lain@navi .vm]$ file Windows81.iso 
Windows81.iso: ISO 9660 CD-ROM filesystem data 'IRM_CCSA_X64FRE_EN-US_DV5' (bootable)

Nazfellun

Member

Registered: 2012-10-22

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've had that problem and solved it.
If you have an ASUS HD7750 Silent - i can give you a working ROM.
You don't need to flash anything, just append as romfile option.

Hmmm.. finally a thing to put in my signature.

That was a fairly confusing series of links, but I ended up following the stuff on the insanelymac link you gave and was able to create a version of my current VBIOS with the EFI GOP added, and rom-parser reports it contains a valid EFI ROM signature... I still get the same issues when trying to boot, however .

Nazfellun

Member

Registered: 2012-10-22

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
I won't be uploading that xml, I'm trying now to run it on Win7, I'll be uploading this XML.
I tried OVMF, thinking it may make a difference, but Windows won't install at all using UEFI. With a Win8.1 ISO, the UEFI shell doesn't recognize the image, and with Win7, the screen goes black when the file copy begins, and the CPU maxes out. I'll update this post with whether the passthrough works with non-UEFI Win7.

Update:
Still doesn't work, OVMF was no success, and Win7 also has a code 43. If any more info is needed for a diagnosis, let me know.

The UEFI shell didn't recognize your Windows 8.1 ISO because it's in UDF format, not iso9660 -- I had this issue initially myself.

I got hopeful for a sec, but supposedly not:

[lain@navi .vm]$ file Windows7.iso 
Windows7.iso: ISO 9660 CD-ROM filesystem data 'GSP1RMCPRXFREO_EN_DVD' (bootable)
[lain@navi .vm]$ file Windows81.iso 
Windows81.iso: ISO 9660 CD-ROM filesystem data 'IRM_CCSA_X64FRE_EN-US_DV5' (bootable)

Get the 'disktype' utility and run it on the images, confirm that they have an El Torito boot record entry that points to a FAT file system, relevant section of the output will look something like this if it does:

  El Torito boot record, catalog at 22
    Bootable non-emulated image, starts at 2828, preloads 4 KiB
      Platform 0x00 (x86), System Type 0x00 (Empty)
    Bootable non-emulated image, starts at 2830, preloads 512 bytes
      Platform 0xEF (EFI), System Type 0x00 (Empty)
      Windows / MS-DOS boot loader
      FAT12 file system (hints score 5 of 5)
        Volume size 1.390 MiB (1457664 bytes, 2847 clusters of 512 bytes)
        Volume name "EFISECTOR"

If it doesn't... get another image, I guess?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
I won't be uploading that xml, I'm trying now to run it on Win7, I'll be uploading this XML.
I tried OVMF, thinking it may make a difference, but Windows won't install at all using UEFI. With a Win8.1 ISO, the UEFI shell doesn't recognize the image, and with Win7, the screen goes black when the file copy begins, and the CPU maxes out. I'll update this post with whether the passthrough works with non-UEFI Win7.

Update:
Still doesn't work, OVMF was no success, and Win7 also has a code 43. If any more info is needed for a diagnosis, let me know.

You didn't hide kvm, this needs to appear within the <features> secions:

    <kvm>
      <hidden state='on'/>
    </kvm>

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Nazfellun

Member

Registered: 2012-10-22

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've had that problem and solved it.
If you have an ASUS HD7750 Silent - i can give you a working ROM.
You don't need to flash anything, just append as romfile option.

Hmmm.. finally a thing to put in my signature.

That was a fairly confusing series of links, but I ended up following the stuff on the insanelymac link you gave and was able to create a version of my current VBIOS with the EFI GOP added, and rom-parser reports it contains a valid EFI ROM signature... I still get the same issues when trying to boot, however .

Scratch that, I built a git copy of ovmf and the VBIOS I built is working using that. Installing drivers now... will report back on whether or not this works out . Thanks for your help either way, Duelist.

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
I won't be uploading that xml, I'm trying now to run it on Win7, I'll be uploading this XML.
I tried OVMF, thinking it may make a difference, but Windows won't install at all using UEFI. With a Win8.1 ISO, the UEFI shell doesn't recognize the image, and with Win7, the screen goes black when the file copy begins, and the CPU maxes out. I'll update this post with whether the passthrough works with non-UEFI Win7.

Update:
Still doesn't work, OVMF was no success, and Win7 also has a code 43. If any more info is needed for a diagnosis, let me know.

You didn't hide kvm, this needs to appear within the <features> secions:

    <kvm>
      <hidden state='on'/>
    </kvm>

Fixed.
Still code 43.

Nazfellun

Member

Registered: 2012-10-22

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Strangely, although I am getting output on the physical card it isn't listed in device manager. Despite this, the AMD Catalyst installer detects the graphics hardware and tries (unsuccessfully) to install it. It gets up to installing the actual driver, the screen flickers once, then the physical display loses signal, the VM goes offline and ramps up to a constant 100% CPU usage with no disk activity. Happens using both the modified copy of my existing VBIOS and the vendor-provided updated one.

My hardware:
Motherboard: Gigabyte 990FXA-UD3
CPU: FX-8350
Host GPU: Gigabyte GTX 750 Ti OC
Guest GPU: Gigabyte R9 290 (initial reference cooler release)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
I won't be uploading that xml, I'm trying now to run it on Win7, I'll be uploading this XML.
I tried OVMF, thinking it may make a difference, but Windows won't install at all using UEFI. With a Win8.1 ISO, the UEFI shell doesn't recognize the image, and with Win7, the screen goes black when the file copy begins, and the CPU maxes out. I'll update this post with whether the passthrough works with non-UEFI Win7.

Update:
Still doesn't work, OVMF was no success, and Win7 also has a code 43. If any more info is needed for a diagnosis, let me know.

You didn't hide kvm, this needs to appear within the <features> secions:

    <kvm>
      <hidden state='on'/>
    </kvm>

Fixed.
Still code 43.

You've also still got the emulated graphics attached.  Perhaps you could take a more thorough read of part4 of my guide?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You didn't hide kvm, this needs to appear within the <features> secions:

    <kvm>
      <hidden state='on'/>
    </kvm>

Fixed.
Still code 43.

You've also still got the emulated graphics attached.  Perhaps you could take a more thorough read of part4 of my guide?

When I detatch the displays from the VM, starting it puts the CPU at exactly 25%, and the memory at 32MiB, with 0 I/O or other activity. Does not respond to VNC or pings. As soon as I re-attach the virtual displays, works a charm. This has got me absolutely stumped, thanks for not just giving a RTFM response.

Edit:
@aw, is any of this likely to be caused by running Win 7 / SeaBIOS? My Win 8 image lacks a FAT boot for UEFI, meaning OVMF was a no go; do I need to source a valid one, or find a way to fix this image?

Last edited by morat (2015-07-13 15:00:20)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Fixed.
Still code 43.

You've also still got the emulated graphics attached.  Perhaps you could take a more thorough read of part4 of my guide?

When I detatch the displays from the VM, starting it puts the CPU at exactly 25%, and the memory at 32MiB, with 0 I/O or other activity. Does not respond to VNC or pings. As soon as I re-attach the virtual displays, works a charm. This has got me absolutely stumped, thanks for not just giving a RTFM response.

I don't see a <loader> line in your xml, so you're using Seabios with Intel host graphics.  That means you need to patch your host kernel or else any VGA region access done by the VGA ROM is going to go to the Intel GPU instead of the assigned GPU.  Have you patched the host kernel and enabled i915 VGA arbitration?  The OVMF approach doesn't require this.

Last edited by aw (2015-07-13 14:58:31)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You've also still got the emulated graphics attached.  Perhaps you could take a more thorough read of part4 of my guide?

When I detatch the displays from the VM, starting it puts the CPU at exactly 25%, and the memory at 32MiB, with 0 I/O or other activity. Does not respond to VNC or pings. As soon as I re-attach the virtual displays, works a charm. This has got me absolutely stumped, thanks for not just giving a RTFM response.

I don't see a <loader> line in your xml, so you're using Seabios with Intel host graphics.  That means you need to patch your host kernel or else any VGA region access done by the VGA ROM is going to go to the Intel GPU instead of the assigned GPU.  Have you patched the host kernel and enabled i915 VGA arbitration?  The OVMF approach doesn't require this.

The instructions in part 5 of your guide arent working for me, so I'll just find a Win8.1 image that works with UEFI; tomorrow.
FYI I get

error: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/libexec/qemu-kvm.vga -help) unexpected exit status 126: libvirt:  error : cannot execute binary /usr/libexec/qemu-kvm.vga: Exec format error

on virsh-ing the vga=on script. Apparently my kernel doesnt have SELinux enforced, or enabled, so that shouldn't be the issue.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When I detatch the displays from the VM, starting it puts the CPU at exactly 25%, and the memory at 32MiB, with 0 I/O or other activity. Does not respond to VNC or pings. As soon as I re-attach the virtual displays, works a charm. This has got me absolutely stumped, thanks for not just giving a RTFM response.

I don't see a <loader> line in your xml, so you're using Seabios with Intel host graphics.  That means you need to patch your host kernel or else any VGA region access done by the VGA ROM is going to go to the Intel GPU instead of the assigned GPU.  Have you patched the host kernel and enabled i915 VGA arbitration?  The OVMF approach doesn't require this.

The instructions in part 5 of your guide arent working for me, so I'll just find a Win8.1 image that works with UEFI; tomorrow.
FYI I get

error: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/libexec/qemu-kvm.vga -help) unexpected exit status 126: libvirt:  error : cannot execute binary /usr/libexec/qemu-kvm.vga: Exec format error

on virsh-ing the vga=on script. Apparently my kernel doesnt have SELinux enforced, or enabled, so that shouldn't be the issue.

As always, no data help.  I can't fix your script without seeing it.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I don't see a <loader> line in your xml, so you're using Seabios with Intel host graphics.  That means you need to patch your host kernel or else any VGA region access done by the VGA ROM is going to go to the Intel GPU instead of the assigned GPU.  Have you patched the host kernel and enabled i915 VGA arbitration?  The OVMF approach doesn't require this.

The instructions in part 5 of your guide arent working for me, so I'll just find a Win8.1 image that works with UEFI; tomorrow.
FYI I get

error: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/libexec/qemu-kvm.vga -help) unexpected exit status 126: libvirt:  error : cannot execute binary /usr/libexec/qemu-kvm.vga: Exec format error

on virsh-ing the vga=on script. Apparently my kernel doesnt have SELinux enforced, or enabled, so that shouldn't be the issue.

As always, no data help.  I can't fix your script without seeing it.

It's your script, instructions followed to the letter.
EDIT:
I've removed the other gpu option, and changed qemu-kvm to qemu-system-x86_64:

 #!/bin/sh
 exec qemu-system-x86_64 \
 `echo "\$@" | \
 sed 's|01:00.0|01:00.0,x-vga=on|g'`

Last edited by morat (2015-07-13 15:37:15)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The instructions in part 5 of your guide arent working for me, so I'll just find a Win8.1 image that works with UEFI; tomorrow.
FYI I get

error: internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin /usr/libexec/qemu-kvm.vga -help) unexpected exit status 126: libvirt:  error : cannot execute binary /usr/libexec/qemu-kvm.vga: Exec format error

on virsh-ing the vga=on script. Apparently my kernel doesnt have SELinux enforced, or enabled, so that shouldn't be the issue.

As always, no data help.  I can't fix your script without seeing it.

It's your script, instructions followed to the letter.

Shall we talk about last time you followed the instructions "to the letter" and failed to include kvm state hidden?  If you can't be bothered to show your work, then I can't be bothered to help.  Sorry.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Always seconds between. If it isn't too much trouble, could you be bothered to read my work?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@morat

Did you: "Be sure to chmod 755 the file before proceeding."

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

# ls -lvm.vga                     
-rwxr-xr-x 1 root root 94 Jul 14 01:35 /usr/libexec/qemu-kvm.vga

Yes, without permissions it returns a permissions error.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Make the first like

#!/bin/sh -x

And run it by hand as:

/usr/libexec/qemu-kvm.vga -help

What does it output?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Make the first like

#!/bin/sh -x

And run it by hand as:

/usr/libexec/qemu-kvm.vga -help

What does it output?

Outputs the full qemu-system-x86_64 help, but virsh still fails.