Stuck setting up pci passthrough

jonnyv · 2017-11-11 15:19:22

Hi all,

Recently windows gave up the ghost on my computer and I decided to go from a dual boot setup to a fully arch one. It shouldn't be too bad since I use windows for only specific apps and some games.

I've been trying for the past few weeks to get PCI passthrough to work however I keep hitting the same wall. I've been following the main entry in the wiki page (PCI passthrough via OVMF) as well as a few variants but the results are the same. As soon as I add the vfio entries in the modprode.d folder and in mkinitcpio.conf files, make a new ramdisk and reboot, I get stuck at loading ramdisk. I wait there for a while with no sign of progress and then have to go back to fallback.

Things I know work:
- VT-x & VT-d enabled in bios
- Internal graphics on the motherboard have been enabled and the motherboard's hdmi port has been connected to the second output of one of my monitors.
- IOMMU is enabled and I can see it in dmesg|grep -e DMAR -e IOMMU.
- Video Card (GTX 970) is in its own group with its HDMI Audio device. Funny thing, it is even the same device ID values as the vfio-pci example in the arch wiki page.
- QEMU/KVM/LibVirt are all working. I've created a windows 10 VM which runs smoothly.

Specs:
- i5 6600
- 32GB of Ram
- Asus Z170-P motherboard
- Asus GTX970
- Two Asus VG247H monitors (each have hdmi + dvi inputs)
- 4.13.11-1-ARCH Kernel
- Nvidia 387.22-4 drivers installed
- Using Gnome on X.Org

Other things I've tried:
- Using the linux-vfio kernel instead both with and without pcie_acs_override=downstream. Hangs at the same place.
- Some posts suggested adding the vfio device IDs right in the grub entry. Still no luck (reference: https://davidyat.es/2016/09/08/gpu-passthrough/)

At this point, I'm pretty sure it is going to be one or two things small that are blocking things but I just can't figure out what. Most of the googling gives out the same info so I'm at wits end.

Thanks ahead for any help.

Lone_Wolf · 2017-11-11 15:47:43

You do have intel microcode updates configured ?
Are you sure the integrated card is set as primary card in the firmware ?

The fact that fallback mkinitcpio works suggests you may be missing something in your mkinitcpio file, please post the mkinitcpio.conf file that doesn't boot.

jonnyv · 2017-11-11 16:46:36

Thanks Lone_wolf, you pointed me in the right direction.

The microcode was updated and I added the same values in mkinitcpio as before (same modules as wiki entry) however what I changed was the setting the internal card as primary in the bios. Once this was done, instead of displaying off the nvidia card, it started displaying off the internal one. I had a small hiccup where my original xorg.conf file prevented X to start but that was only because it was configured with my nvidia cards. Once I renamed the conf file, X started beautifully.

Now, the real test, add the GPU to my VM. Since I was using virt-manager, I added the gpu and audio device using: https://access.redhat.com/documentation … _host.html. The VM booted and the device manager recognized the Geforce 970. I am now downloading the windows geforce drivers and will set them up. What I'd like to know is how do I get the VM to display on my monitor (or is that possible)? Either that or get full use of my monitors somehow. Right now it's running in virtmanager on the (2nd) monitor powered by the internal GPU. I've shown the wiring setup below to help clarify.

GPU/Monitor connections:
Geforce (HDMI): Monitor 1
Geforce (DVI): Monitor 2

Internal (HDMI): Monitor 2

Thanks again

Lone_Wolf · 2017-11-12 13:04:44

The guest should be able to use monitors attached to the geforce card, but you may have forgotten to remove some devices that messes stuff up.

Re-read https://wiki.archlinux.org/index.php/PC … CI_devices

jonnyv · 2017-11-16 03:08:45

Pardon the delay in the reply, I was offline and had to study for an exam.

So I connected the second output of my internal card so that I can maintain two monitors when my VM isn't on.
I managed to have my geforce working on the VM and the output directly to the monitors.
I has to resolve the Error 43 issue which I did by adding a cpu vendor id as per the main wiki.

I loaded up a game (Heroes of the storm) and it played very well.

I did get some random instability in the host OS now and again when restarting the VM or opening another VM so I'll have to look into that. I also compared the game to running it in Wine and both setups have their pros and cons. Wine doesn't require setting up a whole windows VM and takes less space however it is far less stable and looks worse. Setting up wine to get the game to work was a whole other pain in the butt.

I had one last question that I haven't been able to figure out; when you are not running a VM with the external graphics card, is there a way to dynamically reassign them to the host that doesn't require a full reboot? I can setup different ramdisk and grub entries but it is rather unpleasant to go that route.

Thanks

Omar007 · 2017-11-16 11:35:06

You should be able to do that with the nouveau drivers (I haven't had nor seen success with the nvidia binary drivers at all) if you ensure you close down Xorg first when detaching from or reattaching to the host.
Detaching can't be done while the device is in use and Xorg tends to freeze/lock up completely if you do not terminate it before reattaching.
I've not personally tried any of this with Wayland yet but I'd assume the same applies for at least detaching.

Detaching the current driver (should be either vfio-pci or nouveau depending on whether you are going to reattach it to or detach it from the host):

# $dev is the address (e.g. 0000:01:00.0)
echo $dev > /sys/bus/pci/devices/$dev/driver/unbind

Assign vfio-pci driver for passthrough to VM (unbind first):

# $vendor = vendor id, $device = device id (e.g. 10de and 13c2 respectively)
echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id

Assign default driver to device (for reattaching to host, unbind first):

# $dev is the address (e.g. 0000:01:00.0)
echo $dev > /sys/bus/pci/drivers_probe

Last edited by Omar007 (2017-11-16 11:37:44)

jonnyv · 2017-11-17 00:42:00

Cool beans! I'll give that a try. Stopping xorg is about the same as rebooting if I have to shut down all my running apps. Still I had to ask.

Right now I'm not using Wayland but rather Gnome on Xorg just because certain apps were misbehaving otherwise.

Thanks again for the info

Arch Linux

#1 2017-11-11 15:19:22

Stuck setting up pci passthrough

#2 2017-11-11 15:47:43

Re: Stuck setting up pci passthrough

#3 2017-11-11 16:46:36

Re: Stuck setting up pci passthrough

#4 2017-11-12 13:04:44

Re: Stuck setting up pci passthrough

#5 2017-11-16 03:08:45

Re: Stuck setting up pci passthrough

#6 2017-11-16 11:35:06

Re: Stuck setting up pci passthrough

#7 2017-11-17 00:42:00

Re: Stuck setting up pci passthrough

Board footer