qemu windows 10 kvm fails to start up (single gpu passthrough)

randomtheseventh · 2024-01-20 00:26:33

i've been trying to set up a windows 10 vm with single gpu passthrough, but it fails to boot up after i power it on in virt-manager.
i mostly followed this guide, and i used this other guide to set up the qemu hook scripts.
i also replaced mkinitcpio with dracut because that was what the youtuber in the first guide used.

these are my computer specs:

OS: Arch Linux (last updated 2024-01-19 at around 4 pm CST)
Motherboard: ASRock B550M-ITX/ac
Kernel: 6.7.0-arch3-1 (64-bit)
DE: KDE Plasma 5.27.10
Graphics platform: Wayland
CPU: AMD Ryzen 7 5800X
GPU: AMD Radeon RX 7800XT
Memory: 31.3 GiB of RAM

what exactly happens when running the vm

after powering on the vm in virt-manager, my screen turns black for a few seconds (probably because of the hook scripts being executed), and then i'm sent to the sddm login screen.
it started happening after i added my gpu to the hardware list in virt-manager, and i don't know what has been causing the issue; this sudden shutdown supposedly happens if the vm is passing through hardware that is not plugged into the host (based on what i researched prior to posting this), but the gpu is the only device i'm passing through for now.

running sudo systemctl status libvirtd in the terminal returns this:

● libvirtd.service - libvirt legacy monolithic daemon
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; preset: disabled)
     Active: active (running) since Fri 2024-01-19 16:54:15 CST; 52s ago
TriggeredBy: ● libvirtd-ro.socket
             ● libvirtd.socket
             ● libvirtd-admin.socket
       Docs: man:libvirtd(8)
             https://libvirt.org/
   Main PID: 14214 (libvirtd)
      Tasks: 20 (limit: 32768)
     Memory: 68.0M (peak: 70.3M)
        CPU: 1.982s
     CGroup: /system.slice/libvirtd.service
             └─14214 /usr/bin/libvirtd --timeout 120

Jan 19 16:54:15 hostname systemd[1]: Starting libvirt legacy monolithic daemon...
Jan 19 16:54:15 hostname systemd[1]: Started libvirt legacy monolithic daemon.
Jan 19 16:54:33 hostname libvirtd[14214]: libvirt version: 10.0.0
Jan 19 16:54:33 hostname libvirtd[14214]: hostname: hostname
Jan 19 16:54:33 hostname libvirtd[14214]: End of file while reading data: Input/output error

and running sudo cat /var/log/libvirt/qemu/win10-vfio.log returns the following: (the time in this one is messed up for some reason)

2024-01-18 03:10:09.545+0000: shutting down, reason=failed
2024-01-18 03:18:32.751+0000: shutting down, reason=failed
2024-01-18 04:02:27.678+0000: shutting down, reason=failed
2024-01-18 04:10:30.241+0000: shutting down, reason=failed
2024-01-18 04:14:21.409+0000: shutting down, reason=failed
2024-01-18 04:20:13.078+0000: shutting down, reason=failed
2024-01-19 02:21:52.280+0000: shutting down, reason=failed
2024-01-19 21:10:56.309+0000: shutting down, reason=failed
2024-01-19 21:34:28.340+0000: shutting down, reason=failed
2024-01-19 22:34:24.674+0000: shutting down, reason=failed
2024-01-19 22:54:39.211+0000: shutting down, reason=failed

my xml file

<domain type="kvm">
  <name>win10-vfio</name>
  <uuid>i probably shouldn't be showing this</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">16777216</memory>
  <currentMemory unit="KiB">16777216</currentMemory>
  <vcpu placement="static">14</vcpu>
  <os firmware="efi">
    <type arch="x86_64" machine="pc-q35-8.2">hvm</type>
    <firmware>
      <feature enabled="no" name="enrolled-keys"/>
      <feature enabled="yes" name="secure-boot"/>
    </firmware>
    <loader readonly="yes" secure="yes" type="pflash">/usr/share/edk2/x64/OVMF_CODE.secboot.4m.fd</loader>
    <nvram template="/usr/share/edk2/x64/OVMF_VARS.4m.fd">/var/lib/libvirt/qemu/nvram/win10-vfio_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode="custom">
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
    </hyperv>
    <vmport state="off"/>
    <smm state="on"/>
  </features>
  <cpu mode="host-passthrough" check="none" migratable="on">
    <topology sockets="1" dies="1" cores="7" threads="2"/>
    <feature policy="require" name="topoext"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2"/>
      <source file="/run/media/[myusername]/Data/VM Installation Media/win10.qcow2"/>
      <target dev="sda" bus="virtio"/>
      <boot order="1"/>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/[myusername]/Downloads/virtio-win-0.1.240.iso"/>
      <target dev="sdb" bus="sata"/>
      <readonly/>
      <boot order="2"/>
      <address type="drive" controller="0" bus="0" target="0" unit="1"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x16"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0x17"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0x18"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="10" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="10" port="0x19"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
    </controller>
    <controller type="pci" index="11" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="11" port="0x1a"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
    </controller>
    <controller type="pci" index="12" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="12" port="0x1b"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
    </controller>
    <controller type="pci" index="13" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="13" port="0x1c"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
    </controller>
    <controller type="pci" index="14" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="14" port="0x1d"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <interface type="network">
      <mac address="52:54:00:5f:80:1a"/>
      <source network="default"/>
      <model type="e1000e"/>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </interface>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial"/>
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
    </console>
    <input type="tablet" bus="usb">
      <address type="usb" bus="0" port="1"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <graphics type="vnc" port="5901" autoport="no" listen="0.0.0.0">
      <listen type="address" address="0.0.0.0"/>
    </graphics>
    <sound model="ich9">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
    </sound>
    <audio id="1" type="none"/>
    <video>
      <model type="qxl" ram="65536" vram="65536" vgamem="16384" heads="1" primary="yes"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
    </video>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
      </source>
      <rom bar="on"/>
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x08" slot="0x00" function="0x1"/>
      </source>
      <rom bar="on"/>
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
    </hostdev>
    <watchdog model="itco" action="reset"/>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
</domain>

qemu hook scripts

i copy-pasted these from the second guide i mentioned at the start of this thread.

here's the start script:

#!/bin/bash
set -x

# Stop display manager
systemctl stop display-manager.service
systemctl --user -M [my-username]@ stop plasma*

# Unbind VTconsoles: might not be needed
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind

# Unbind EFI Framebuffer
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

# Unload NVIDIA kernel modules
# modprobe -r nvidia_drm nvidia_modeset nvidia_uvm nvidia

# Unload AMD kernel module
modprobe -r amdgpu

# Detach GPU devices from host
# Use your GPU and HDMI Audio PCI host device
virsh nodedev-detach pci_0000_08_00_0
virsh nodedev-detach pci_0000_08_00_1

# Load vfio module
modprobe vfio-pci

and here's my stop script:

#!/bin/bash
set -x

# Attach GPU devices to host
# Use your GPU and HDMI Audio PCI host device
virsh nodedev-reattach pci_0000_08_00_0
virsh nodedev-reattach pci_0000_08_00_1

# Unload vfio module
modprobe -r vfio-pci

# Load AMD kernel module
modprobe amdgpu

# Rebind framebuffer to host
echo "efi-framebuffer.0" > /sys/bus/platform/drivers/efi-framebuffer/bind

# Load NVIDIA kernel modules
#modprobe nvidia_drm
#modprobe nvidia_modeset
#modprobe nvidia_uvm
#modprobe nvidia

# Bind VTconsoles: might not be needed
echo 1 > /sys/class/vtconsole/vtcon0/bind
echo 1 > /sys/class/vtconsole/vtcon1/bind

# Restart Display Manager
systemctl start display-manager

help is greatly appreciated.
thanks in advance.

cgb_spender · 2024-01-20 21:06:09

I would try virtualbox instead, more mature software, it has never failed me, on Ubuntu, Debian, or Arch. Be sure to install the extension pack, if you decide to go this route.

ewaller · 2024-01-20 21:43:49

cgb_spender wrote:

I would try virtualbox instead, more mature software, it has never failed me, on Ubuntu, Debian, or Arch. Be sure to install the extension pack, if you decide to go this route.

I am somewhat surprised that you are suggesting a product by Oracle based upon your history of concerns over big brother back doors.

That said, beware of the licence requirements of VirtualBox if this not purely for personal use. Especially if it is at a business that might be subject to BSA audits.

randomtheseventh · 2024-01-20 22:50:40

cgb_spender wrote:

I would try virtualbox instead, more mature software, it has never failed me, on Ubuntu, Debian, or Arch. Be sure to install the extension pack, if you decide to go this route.

virtualbox isn't really an option for me because i'm mainly going to use the vm for gaming.

cgb_spender · 2024-01-21 10:28:30

ewaller wrote:

cgb_spender wrote:
I would try virtualbox instead, more mature software, it has never failed me, on Ubuntu, Debian, or Arch. Be sure to install the extension pack, if you decide to go this route.
I am somewhat surprised that you are suggesting a product by Oracle based upon your history of concerns over big brother back doors.

Good point :-) but I do not have the vboxdrv module always running on my system, I always load it before use and then unload it. Besides, I am assuming that if a software package is in the extra repository, it must be safe. Am I wrong?

Head_on_a_Stick · 2024-01-21 11:18:07

cgb_spender wrote:

I am assuming that if a software package is in the extra repository, it must be safe. Am I wrong?

Yes.

@OP: have you tried following https://wiki.archlinux.org/title/PCI_pa … h_via_OVMF instead of those other guides? Specifically: https://wiki.archlinux.org/title/PCI_pa … ng_the_GPU

Disclaimer: I haven't actually tried this myself and I use qemu-system-x86_64 directly rather than virt-manager so I haven't checked your configuration.

randomtheseventh · 2024-01-21 18:42:58

Head_on_a_Stick wrote:

@OP: have you tried following https://wiki.archlinux.org/title/PCI_pa … h_via_OVMF instead of those other guides? Specifically: https://wiki.archlinux.org/title/PCI_pa … ng_the_GPU

last week i tried using gpu-passthrough-manager (to do steps 1-3 with a gui), and it made my system unbootable (it got stuck at Loading initial ramdisk...) for a little while after loading the vfio drivers because my gpu used the vfio drivers when booting into linux instead of the default ones. after that, i just assumed that guide was meant for people with more than 1 gpu and didn't try doing it manually because i was afraid that the same thing would happen again.

Head_on_a_Stick · 2024-01-21 19:06:38

Oops, sorry, I presumed that CPU had an iGPU but it does not.

Arch Linux

#1 2024-01-20 00:26:33