You are not logged in.
Hello. I have a windows 11 VM that I am running and I have my discrete GPU passed through to it but every once in a while the VM will "crash" and I am unable to restart it until I reboot my host machine. I say "crash" because when I still see this:
$ sudo virsh list
Id Name State
-----------------------
1 win11 runningBut I am not getting any A/V output from the VM. Doing virsh shutdown also does not shut the machine down (it's been about 20min now) and doing virsh destroy just hangs for a while and then gives this (below) but it seems to destroy it anyways. After it is destoryed, I am able to start it again but still no A/V output.
error: Failed to destroy domain 'win11'
error: Failed to terminate process 1894 with SIGKILL: Device or resource busyHere are my host machine's details:
Arch Linux x86_64
Linux 6.12.60-1-lts
13th Gen Intel(R) Core(TM) i7-13620H (16) @ 4.90 GHz
NVIDIA GeForce RTX 4060 Max-Q / Mobile [Discrete]
Intel UHD Graphics @ 1.50 GHz [Integrated]
16GB DDR5Here is some more specific info on my GPUs that I have in my machine:
$ lspci -nnk | grep -A4 -E "VGA|3D|Display|Audio"
libkmod: ERROR: kmod_config_parse: /etc/modprobe.d/vfio.conf line 2: ignoring bad line starting with 'disable_idle_d3=1'
0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Raptor Lake-P [UHD Graphics] [8086:a7a8] (rev 04)
DeviceName: Onboard - Video
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:13c0]
Kernel driver in use: i915
Kernel modules: i915, xe
--
0000:01:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD107M [GeForce RTX 4060 Max-Q / Mobile] [10de:28a0] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:13c0]
Kernel driver in use: vfio-pci
Kernel modules: nouveau, nvidia_drm, nvidia
0000:01:00.1 Audio device [0403]: NVIDIA Corporation AD107 High Definition Audio Controller [10de:22be] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:13c0]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
10000:e0:06.0 PCI bridge [0604]: Intel Corporation Raptor Lake PCI Express 4.0 Graphics Port [8086:a74d]I am just now seeing the error there due to the line I added in my vfio.conf file. I was looking around other forum posts online about others experiencing this issue and came across a suggestion here to add that in my config. I plan on removing that now since it doesn't work (or I set it incorrectly). Here is that file:
options vfio-pci ids=10de:28a0,10de:22be
disable_idle_d3=1This is my VM log:
2025-12-04 17:20:53.265+0000: Starting external device: TPM Emulator
/usr/bin/swtpm socket --ctrl type=unixio,path=/run/libvirt/qemu/swtpm/1-win11-swtpm.sock,mode=0600 --tpmstate dir=/var/lib/libvirt/swtpm/2bfe159d-401b-474d-b172-dad71720e362/tpm2,mode=0600,lock --log file=/var/log/swtpm/libvirt/qemu/win11-swtpm.log --terminate --tpm2
2025-12-04 17:20:53.315+0000: starting up libvirt version: 11.10.0, qemu version: 10.1.2, kernel: 6.12.60-1-lts, hostname: archlinux
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin \
USER=root \
HOME=/var/lib/libvirt/qemu/domain-1-win11 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-win11/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-win11/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-win11/.config \
/usr/bin/qemu-system-x86_64 \
-name guest=win11,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-win11/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/edk2/x64/OVMF_CODE.secboot.4m.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/win11_VARS.fd","node-name":"libvirt-pflash1-storage","read-only":false}' \
-machine pc-q35-10.1,usb=off,vmport=off,smm=on,dump-guest-core=off,memory-backend=pc.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-storage,hpet=off,acpi=on \
-accel kvm \
-cpu host,migratable=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vpindex=on,hv-runtime=on,hv-synic=on,hv-stimer=on,hv-frequencies=on,hv-tlbflush=on,hv-ipi=on,hv-evmcs=on,hv-avic=on \
-global driver=cfi.pflash01,property=secure,value=on \
-m size=8388608k \
-object '{"qom-type":"memory-backend-memfd","id":"pc.ram","share":true,"x-use-canonical-path-for-ramblock-id":false,"size":8589934592}' \
-overcommit mem-lock=off \
-smp 12,sockets=1,dies=1,clusters=1,cores=6,threads=2 \
-object '{"qom-type":"iothread","id":"iothread1"}' \
-uuid 2bfe159d-401b-474d-b172-dad71720e362 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=32,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x5"}' \
-device '{"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"}' \
-device '{"driver":"pcie-root-port","port":23,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x2.0x7"}' \
-device '{"driver":"pcie-root-port","port":24,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x3"}' \
-device '{"driver":"pcie-root-port","port":25,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x3.0x1"}' \
-device '{"driver":"pcie-root-port","port":26,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x3.0x2"}' \
-device '{"driver":"pcie-root-port","port":27,"chassis":12,"id":"pci.12","bus":"pcie.0","addr":"0x3.0x3"}' \
-device '{"driver":"pcie-root-port","port":28,"chassis":13,"id":"pci.13","bus":"pcie.0","addr":"0x3.0x4"}' \
-device '{"driver":"pcie-root-port","port":29,"chassis":14,"id":"pci.14","bus":"pcie.0","addr":"0x3.0x5"}' \
-device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.2","addr":"0x0"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.3","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/win11.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"discard":"unmap","driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device '{"driver":"ide-hd","bus":"ide.0","drive":"libvirt-1-format","id":"sata0-0-0","bootindex":1}' \
-netdev '{"type":"tap","fd":"33","id":"hostnet0"}' \
-device '{"driver":"e1000e","netdev":"hostnet0","id":"net0","mac":"52:54:00:25:b3:f5","bus":"pci.1","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/1-win11-swtpm.sock \
-tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \
-device '{"driver":"tpm-crb","tpmdev":"tpm-tpm0","id":"tpm0"}' \
-object '{"qom-type":"input-linux","id":"input0","evdev":"/dev/input/by-id/usb-Logitech_USB_Receiver-if02-event-mouse"}' \
-object '{"qom-type":"input-linux","id":"input1","evdev":"/dev/input/by-id/usb-Dell_Dell_USB_Keyboard-event-kbd","repeat":true,"grab_all":true,"grab-toggle":"ctrl-ctrl"}' \
-audiodev '{"id":"audio1","driver":"pa","server":"/run/user/1000/pulse/native"}' \
-device '{"driver":"ich9-intel-hda","id":"sound0","bus":"pcie.0","addr":"0x1b"}' \
-device '{"driver":"hda-micro","id":"sound0-codec0","bus":"sound0.0","cad":0,"audiodev":"audio1"}' \
-global ICH9-LPC.noreboot=off \
-watchdog-action reset \
-device '{"driver":"vfio-pci","host":"0000:01:00.0","id":"hostdev0","bus":"pci.5","addr":"0x0"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.4","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/2 (label charserial0)
ERROR cluster 58699 refcount=0 reference=1
... (there are quite a number of similar lines)
ERROR cluster 187205 refcount=0 reference=1
Rebuilding refcount structure
Repairing cluster 1165 refcount=1 reference=0
... (there are quite a number of similar lines)
Repairing cluster 1343488 refcount=1 reference=0
2025-12-04T18:19:56.179427Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3I did notice though that my GPU is not in D3 as far as my host can tell but the audio is:
$ cat /sys/bus/pci/devices/0000:01:00.0/power_state
D0
$ cat /sys/bus/pci/devices/0000:01:00.1/power_state
D3coldThis is my VM config xml file:
$ sudo cat /etc/libvirt/qemu/win11.xml
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
virsh edit win11
or other application using the libvirt API.
-->
<domain type='kvm'>
<name>win11</name>
<uuid>2bfe159d-401b-474d-b172-dad71720e362</uuid>
<title>Windows 11 VM</title>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/11"/>
</libosinfo:libosinfo>
</metadata>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<memoryBacking>
<source type='memfd'/>
<access mode='shared'/>
</memoryBacking>
<vcpu placement='static'>12</vcpu>
<iothreads>1</iothreads>
<cputune>
<vcpupin vcpu='0' cpuset='2'/>
<vcpupin vcpu='1' cpuset='3'/>
<vcpupin vcpu='2' cpuset='4'/>
<vcpupin vcpu='3' cpuset='5'/>
<vcpupin vcpu='4' cpuset='6'/>
<vcpupin vcpu='5' cpuset='7'/>
<vcpupin vcpu='6' cpuset='8'/>
<vcpupin vcpu='7' cpuset='9'/>
<vcpupin vcpu='8' cpuset='10'/>
<vcpupin vcpu='9' cpuset='11'/>
<vcpupin vcpu='10' cpuset='2'/>
<vcpupin vcpu='11' cpuset='3'/>
<emulatorpin cpuset='0-1,12-15'/>
<iothreadpin iothread='1' cpuset='0-1,12-15'/>
</cputune>
<os firmware='efi'>
<type arch='x86_64' machine='pc-q35-10.1'>hvm</type>
<firmware>
<feature enabled='no' name='enrolled-keys'/>
<feature enabled='yes' name='secure-boot'/>
</firmware>
<loader readonly='yes' secure='yes' type='pflash' format='raw'>/usr/share/edk2/x64/OVMF_CODE.secboot.4m.fd</loader>
<nvram template='/usr/share/edk2/x64/OVMF_VARS.4m.fd' templateFormat='raw' format='raw'>/var/lib/libvirt/qemu/nvram/win11_VARS.fd</nvram>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<hyperv mode='custom'>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
<vpindex state='on'/>
<runtime state='on'/>
<synic state='on'/>
<stimer state='on'/>
<frequencies state='on'/>
<tlbflush state='on'/>
<ipi state='on'/>
<evmcs state='on'/>
<avic state='on'/>
</hyperv>
<vmport state='off'/>
<smm state='on'/>
</features>
<cpu mode='host-passthrough' check='none' migratable='on'>
<topology sockets='1' dies='1' clusters='1' cores='6' threads='2'/>
</cpu>
<clock offset='localtime'>
<timer name='rtc' present='no' tickpolicy='catchup'/>
<timer name='pit' present='no' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' discard='unmap'/>
<source file='/var/lib/libvirt/images/win11.qcow2'/>
<target dev='sda' bus='sata'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='1' port='0x10'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='2' port='0x11'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='3' port='0x12'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='4' port='0x13'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='5' port='0x14'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
</controller>
<controller type='pci' index='6' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='6' port='0x15'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
</controller>
<controller type='pci' index='7' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='7' port='0x16'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
</controller>
<controller type='pci' index='8' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='8' port='0x17'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
</controller>
<controller type='pci' index='9' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='9' port='0x18'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='10' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='10' port='0x19'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
</controller>
<controller type='pci' index='11' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='11' port='0x1a'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
</controller>
<controller type='pci' index='12' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='12' port='0x1b'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
</controller>
<controller type='pci' index='13' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='13' port='0x1c'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/>
</controller>
<controller type='pci' index='14' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='14' port='0x1d'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/>
</controller>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</controller>
<interface type='network'>
<mac address='52:54:00:25:b3:f5'/>
<source network='default'/>
<model type='e1000e'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
<serial type='pty'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<input type='evdev'>
<source dev='/dev/input/by-id/usb-Logitech_USB_Receiver-if02-event-mouse'/>
</input>
<input type='evdev'>
<source dev='/dev/input/by-id/usb-Dell_Dell_USB_Keyboard-event-kbd' grab='all' grabToggle='ctrl-ctrl' repeat='on'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<tpm model='tpm-crb'>
<backend type='emulator' version='2.0'>
<profile name='default-v1'/>
</backend>
</tpm>
<sound model='ich9'>
<codec type='micro'/>
<audio id='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
</sound>
<audio id='1' type='pulseaudio' serverName='/run/user/1000/pulse/native'/>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</hostdev>
<watchdog model='itco' action='reset'/>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</memballoon>
</devices>
</domain>I tried looking through the Arch wiki for anything that might be related and I think that maybe this section is but I have NVIDIA not AMD. I am thinking this is maybe related because I get this:
$ journalctl -u libvirtd | grep -i pci
Nov 26 10:35:53 archlinux libvirtd[11665]: internal error: QEMU unexpectedly closed the monitor (vm='win11'): 2025-11-26T15:35:53.324495Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:01:00.0","id":"hostdev0","bus":"pci.4","addr":"0x0"}: vfio 0000:01:00.0: group 14 is not viable
Nov 26 11:45:25 archlinux libvirtd[11665]: internal error: QEMU unexpectedly closed the monitor (vm='win11'): 2025-11-26T16:45:25.699737Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:01:00.0","id":"hostdev0","bus":"pci.5","addr":"0x0"}: vfio 0000:01:00.0: group 14 is not viable
Dec 03 10:36:59 archlinux libvirtd[716]: internal error: Unknown PCI header type '127' for device '0000:01:00.0'
Dec 03 10:36:59 archlinux libvirtd[716]: internal error: Unknown PCI header type '127' for device '0000:01:00.1'
Dec 04 09:38:59 archlinux libvirtd[4932]: internal error: Unknown PCI header type '127' for device '0000:01:00.0'
Dec 04 09:38:59 archlinux libvirtd[4932]: internal error: Unknown PCI header type '127' for device '0000:01:00.1'As you can see the times of these logs are a bit different but I have been experiencing these issues for a few days and the only thing I changed in hopes of solving my issue was putting the line in vfio.conf around 10am today so after the latest lines in the immediately above logs and the logs that include the below line (full log above)
...
2025-12-04T18:19:56.179427Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3I am fairly new to this so if there is other information that is required, please let me know and I'll try to grab that for you. Thanks for taking the time to read my post.
Edit: Added details on what happens when I try to shutdown or destory the VM.
Last edited by Jukelyn (2025-12-04 19:40:57)
Offline