[SOLVED] NVIDIA GPU passthrough freezes on host startup

sick_mitch · 2023-07-04 07:26:16

need a kvm to run a windows only program for work so i followed in order those guides:

https://wiki.archlinux.org/title/KVM
https://wiki.archlinux.org/title/PCI_pa … h_via_OVMF
https://wiki.archlinux.org/title/Virt-Manager

all the kvm testing results compatible
using the iommu detecting script from the 2nd guide return me this

IOMMU Group 2:
	00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07)
	01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116M [GeForce GTX 1660 Ti Mobile] [10de:2191] (rev a1)
	01:00.1 Audio device [0403]: NVIDIA Corporation TU116 High Definition Audio Controller [10de:1aeb] (rev a1)
	01:00.2 USB controller [0c03]: NVIDIA Corporation TU116 USB 3.1 Host Controller [10de:1aec] (rev a1)
	01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 USB Type-C UCSI Controller [10de:1aed] (rev a1)

so i have done those system configurations

/etc/default/grub:
GRUB_CMDLINE_LINUX_DEFAULT='nowatchdog nvme_load=YES loglevel=3 intel_iommu=on iommu=pt vfio-pci.ids=8086:1901,10de:2191,10de:1aeb,10de:1aec,10de:1aed'

/etc/mkinitcpio.conf:
MODULES=(vfio_pci vfio vfio_iommu_type1)

/etc/dracut.conf.d/10-vfio.conf
force_drivers+=" vfio-pci vfio vfio_iommu_type1 "

on reboot the system is fine till it freeze, forever
lspci runs only on some boot and for a short period of time then result in a stuck terminal, got this out of lspci -k

00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 07)
	Subsystem: Dell 6th-10th Gen Core Processor PCIe Controller (x16)
	Kernel driver in use: pcieport
01:00.0 VGA compatible controller: NVIDIA Corporation TU116M [GeForce GTX 1660 Ti Mobile] (rev a1)
	Subsystem: Dell TU116M [GeForce GTX 1660 Ti Mobile]
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau, nvidia_drm, nvidia
01:00.1 Audio device: NVIDIA Corporation TU116 High Definition Audio Controller (rev a1)
	Subsystem: Dell TU116 High Definition Audio Controller
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel
01:00.2 USB controller: NVIDIA Corporation TU116 USB 3.1 Host Controller (rev a1)
	Subsystem: Dell TU116 USB 3.1 Host Controller
	Kernel driver in use: vfio-pci
	Kernel modules: xhci_pci
01:00.3 Serial bus controller: NVIDIA Corporation TU116 USB Type-C UCSI Controller (rev a1)
	Subsystem: Dell TU116 USB Type-C UCSI Controller
	Kernel driver in use: vfio-pci
	Kernel modules: i2c_nvidia_gpu

here a couple of freezing boots with journalctl

lug 04 07:00:53 mike kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=84025bdb-838c-48c8-a9ab-a8ba531d72b8 rw nowatchdog nvme_load=YES loglevel=3 intel_iommu=on iommu=pt vfio-pci.ids=18086:190,10de:2191,10de:1aeb,10de:1aec,10de:1aed
lug 04 07:00:53 mike kernel: Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=84025bdb-838c-48c8-a9ab-a8ba531d72b8 rw nowatchdog nvme_load=YES loglevel=3 intel_iommu=on iommu=pt vfio-pci.ids=18086:190,10de:2191,10de:1aeb,10de:1aec,10de:1aed
lug 04 07:00:53 mike dracut-cmdline[207]: Using kernel command line parameters:  rd.driver.pre=vfio_pci rd.driver.pre=vfio rd.driver.pre=vfio_iommu_type1   BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=84025bdb-838c-48c8-a9ab-a8ba531d72b8 rw nowatchdog nvme_load=YES loglevel=3 intel_iommu=on iommu=pt vfio-pci.ids=18086:190,10de:2191,10de:1aeb,10de:1aec,10de:1aed
lug 04 07:00:53 mike kernel: vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
lug 04 07:00:59 mike kernel: vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
lug 04 07:01:02 mike kernel: vfio-pci 0000:01:00.0: not ready 1023ms after resume; waiting
lug 04 07:01:04 mike kernel: vfio-pci 0000:01:00.0: not ready 2047ms after resume; waiting
lug 04 07:01:06 mike kernel: vfio-pci 0000:01:00.0: not ready 4095ms after resume; waiting
lug 04 07:01:10 mike kernel: vfio-pci 0000:01:00.0: not ready 8191ms after resume; waiting
lug 04 07:01:18 mike kernel: vfio-pci 0000:01:00.0: not ready 16383ms after resume; waiting
lug 04 07:01:36 mike kernel: vfio-pci 0000:01:00.0: not ready 32767ms after resume; waiting
lug 04 07:03:28 mike kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=84025bdb-838c-48c8-a9ab-a8ba531d72b8 rw nowatchdog nvme_load=YES loglevel=3 intel_iommu=on iommu=pt vfio-pci.ids=18086:190,10de:2191,10de:1aeb,10de:1aec,10de:1aed
lug 04 07:03:28 mike kernel: Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=84025bdb-838c-48c8-a9ab-a8ba531d72b8 rw nowatchdog nvme_load=YES loglevel=3 intel_iommu=on iommu=pt vfio-pci.ids=18086:190,10de:2191,10de:1aeb,10de:1aec,10de:1aed
lug 04 07:03:28 mike dracut-cmdline[208]: Using kernel command line parameters:  rd.driver.pre=vfio_pci rd.driver.pre=vfio rd.driver.pre=vfio_iommu_type1   BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=84025bdb-838c-48c8-a9ab-a8ba531d72b8 rw nowatchdog nvme_load=YES loglevel=3 intel_iommu=on iommu=pt vfio-pci.ids=18086:190,10de:2191,10de:1aeb,10de:1aec,10de:1aed
lug 04 07:03:28 mike kernel: vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
lug 04 07:03:34 mike kernel: vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none

output of dmesg | grep -i vfio-pci

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=84025bdb-838c-48c8-a9ab-a8ba531d72b8 rw nowatchdog nvme_load=YES loglevel=3 intel_iommu=on iommu=pt vfio-pci.ids=18086:190,10de:2191,10de:1aeb,10de:1aec,10de:1aed
[    0.043646] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=84025bdb-838c-48c8-a9ab-a8ba531d72b8 rw nowatchdog nvme_load=YES loglevel=3 intel_iommu=on iommu=pt vfio-pci.ids=18086:190,10de:2191,10de:1aeb,10de:1aec,10de:1aed
[    1.424587] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[    7.648474] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none

the only thing i am getting as a problem is the kernel driver in use from the Intel PCI bridge that doesn't change at all. I am noob and in all this journal waiting for the GPU must hide something I can't see
hoping the infos up there are enough, BIG THANKS for reading through all of this
ask me anything, ill try to get what is needed if freeze allows me

Last edited by sick_mitch (2023-07-09 12:52:21)

V1del · 2023-07-04 08:44:24

You need a secondary GPU for proper passthrough to really work. Otherwise you are basically in a glorified dual boot situation where you must start the VM immediately on boot, e.g.: https://github.com/QaidVoid/Complete-Si … assthrough

sick_mitch · 2023-07-04 08:52:47

V1del wrote:

You need a secondary GPU for proper passthrough to really work. Otherwise you are basically in a glorified dual boot situation where you must start the VM immediately on boot, e.g.: https://github.com/QaidVoid/Complete-Si … assthrough

I have this integrated intel wich should run out of the box, am i wrong?

00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-H GT2 [UHD Graphics 630] [8086:3e9b]
	DeviceName: Onboard - Video
	Subsystem: Dell CoffeeLake-H GT2 [UHD Graphics 630] [1028:0949]
	Kernel driver in use: i915
	Kernel modules: i915

Gonna study the github you linked and the troubleshooting at the end.
Update:
I read the github, isn't this guide missing the isolation of the GPU part? am I misundestanding this? https://wiki.archlinux.org/title/PCI_pa … ng_the_GPU

Last edited by sick_mitch (2023-07-04 09:19:52)

Lone_Wolf · 2023-07-04 12:14:29

The systems the github guide targets switch the gpu from the host to the guest. isolating the gpu would prevent the host to do that.

For your usecase that guide is not relevant.

The log snippets you posted don't hold enough info to say what went wrong .

Please post full dmesg / journal & lscpi -k from a boot.
(if it freezes to soon, journal from a previous boot will also help)

seth · 2023-07-04 12:58:47

"vfio-pci.ids=18086:190" is a typo, you're looking for "8086:1901"
"intel_iommu=on iommu=pt" should™ not be necessary but might cause trouble, so first try to skip that

the system is fine till it freeze, forever

When about does that happen and does it not happen if you don't pass the GPU through?
And what is then (w/o passing the GPU through) the output of "eglinfo -B"?

sick_mitch · 2023-07-04 14:10:17

Trying to experiment some exotic solutions i broke my boot since that was a sacrificial install for test of endevour i restart the test system fresh with only arch and a few more to get a system i feel confortable in.
Replicated the same issue following the same procedure and on boot the screen freezes almost instantly after boot on a full screen terminal with just a stationary bar not even blinking.

Lone_Wolf wrote:

The systems the github guide targets switch the gpu from the host to the guest. isolating the gpu would prevent the host to do that.
For your usecase that guide is not relevant.
The log snippets you posted don't hold enough info to say what went wrong .
Please post full dmesg / journal & lscpi -k from a boot.
(if it freezes to soon, journal from a previous boot will also help)

Here what you asked thanks!

dmesg
journal freeze
journal no passthrogh
lspci -k

seth wrote:

"vfio-pci.ids=18086:190" is a typo, you're looking for "8086:1901"
"intel_iommu=on iommu=pt" should™ not be necessary but might cause trouble, so first try to skip that
 the system is fine till it freeze, forever
When about does that happen and does it not happen if you don't pass the GPU through?
And what is then (w/o passing the GPU through) the output of "eglinfo -B"?

I found the typo a bit later but didn't change anything, the pcieport driver still loaded in before the vfio one sadly
I'm booted into the system w/o passthrough and to do that i removed "intel_iommu=on iommu=pt" from the grub menu on boot resulting me in the system w/o pass and no iommu running
I can't get you eglinfo -B since i'm on wayland and i use egl-wayland to run xwayland but i never touched him
Is this wayland related maybe? Thanks a lot tho

Last edited by sick_mitch (2023-07-04 14:48:14)

seth · 2023-07-04 14:19:03

resulting me in the system w/o pass and no iommu running

Wow, wait a second: you only get an IOMMU if you enforce it?
Please post a system journal w/o those parameters.
Edit: you did, the link's just bogus, https://pastebin.com/raw/BVzh49Mn

Last edited by seth (2023-07-04 14:20:19)

sick_mitch · 2023-07-04 14:54:07

seth wrote:

resulting me in the system w/o pass and no iommu running
Wow, wait a second: you only get an IOMMU if you enforce it?
Please post a system journal w/o those parameters.
Edit: you did, the link's just bogus, https://pastebin.com/raw/BVzh49Mn

Fixed the links sorry about it.
Yes i get iommu only if forced in kernel parameters, thought this was quite normal guess not

Got the eglinfo -B output.

Last edited by sick_mitch (2023-07-04 16:13:50)

seth · 2023-07-04 16:19:11

You're at that point running on the nvidia chip, you're running SDDM on X11 and unless you're really fast, probably auto-loginto a hyprland session?
See whether you can boot the multi-user.target when passing the GPU through (w/o freeze, 2nd link below) and post your Xorg log, https://wiki.archlinux.org/title/Xorg#General

sick_mitch · 2023-07-04 21:06:06

You got me! right on spot: sddm with auto-login on hypr!
So thinking a bit about some previous better attempt i changed to tlp the battery optimization. Removing it, i don't have any sign of the waiting on the GPU and no freeze.
Got you a Xorg log.
Tomorrow i'll try virtualizing and update. Thanks!

Last edited by sick_mitch (2023-07-05 04:14:35)

seth · 2023-07-04 21:17:09

It's actually the same log, but you're running on the intel chip w/ the modesetting driver and there's no odd config to direct you towards the nvidia chip - so that's fine.
If this wasn't just caused by too aggressive power management, I'd try to cut out the second display server (ie. disable SDDM and start hyprland from the console)

sick_mitch · 2023-07-05 05:50:58

So this morning on start i'm back on freeze, disabled sddm and launching from tty.
i have a journal and a Xorglog

[     6.227] (--) Log file renamed from "/var/log/Xorg.pid-470.log" to "/var/log/Xorg.0.log"
[     6.228] 
X.Org X Server 1.21.1.8
X Protocol Version 11, Revision 0
[     6.228] Current Operating System: Linux duoT 6.4.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 01 Jul 2023 16:17:21 +0000 x86_64
[     6.228] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=60390d62-5cec-4fc1-808f-97354f49b900 rw intel_iommu=on iommu=pt vfio-pci.ids=8086:1901,10de:2191,10de:1aeb,10de:1aec,10de:1aed
[     6.228]  
[     6.228] Current version of pixman: 0.42.2
[     6.228] 	Before reporting problems, check http://wiki.x.org
	to make sure that you have the latest version.
[     6.228] Markers: (--) probed, (**) from config file, (==) default setting,
	(++) from command line, (!!) notice, (II) informational,
	(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
[     6.228] (==) Log file: "/var/log/Xorg.0.log", Time: Wed Jul  5 06:34:15 2023
[     6.229] (==) Using system config directory "/usr/share/X11/xorg.conf.d"
[     6.229] (==) No Layout section.  Using the first Screen section.
[     6.229] (==) No screen section available. Using defaults.
[     6.229] (**) |-->Screen "Default Screen Section" (0)
[     6.229] (**) |   |-->Monitor "<default monitor>"
[     6.230] (==) No monitor specified for screen "Default Screen Section".
	Using a default monitor configuration.
[     6.230] (==) Automatically adding devices
[     6.230] (==) Automatically enabling devices
[     6.230] (==) Automatically adding GPU devices
[     6.230] (==) Automatically binding GPU devices
[     6.230] (==) Max clients allowed: 256, resource mask: 0x1fffff
[     6.230] (WW) The directory "/usr/share/fonts/misc" does not exist.
[     6.230] 	Entry deleted from font path.
[     6.231] (WW) `fonts.dir' not found (or not valid) in "/usr/share/fonts/TTF".
[     6.231] 	Entry deleted from font path.
[     6.231] 	(Run 'mkfontdir' on "/usr/share/fonts/TTF").
[     6.231] (WW) The directory "/usr/share/fonts/OTF" does not exist.
[     6.231] 	Entry deleted from font path.
[     6.231] (WW) The directory "/usr/share/fonts/Type1" does not exist.
[     6.231] 	Entry deleted from font path.
[     6.231] (WW) The directory "/usr/share/fonts/100dpi" does not exist.
[     6.231] 	Entry deleted from font path.
[     6.231] (WW) The directory "/usr/share/fonts/75dpi" does not exist.
[     6.231] 	Entry deleted from font path.
[     6.231] (==) FontPath set to:
	
[     6.231] (==) ModulePath set to "/usr/lib/xorg/modules"
[     6.231] (II) The server relies on udev to provide the list of input devices.
	If no devices become available, reconfigure udev or disable AutoAddDevices.
[     6.231] (II) Module ABI versions:
[     6.231] 	X.Org ANSI C Emulation: 0.4
[     6.231] 	X.Org Video Driver: 25.2
[     6.231] 	X.Org XInput driver : 24.4
[     6.231] 	X.Org Server Extension : 10.0
[     6.231] (++) using VT number 2

[     6.231] (II) systemd-logind: logind integration requires -keeptty and -keeptty was not provided, disabling logind integration
[     6.232] (II) xfree86: Adding drm device (/dev/dri/card0)
[     6.232] (II) Platform probe for /sys/devices/pci0000:00/0000:00:02.0/drm/card0

update:
Trying a reboot changing nothing is all fine, no waiting in journal lspci-k up and running and i'm installing windows while writing this no sign of stress or whatever.
I'll take a dmesg a journal and a log to hold as reference.
Inconsistently but looks like is working

Last edited by sick_mitch (2023-07-05 06:06:30)

seth · 2023-07-05 06:26:11

That's just part of the Xorg log and it smells like it freezes either on probing the GPU on 02.0 (intel) or right after (so maybe it tries to probe the nvidia GPU?)
=> try adding i915 to the initramfs, https://wiki.archlinux.org/title/Kernel … _KMS_start

sick_mitch · 2023-07-05 06:32:39

The log is as i found it in /var/log, can I get it more detailed? At the end the good session of last post froze too trying to include the Intel chip!

update:
Included the intel chip in KMS, got some good boot out of it. Tried to set up virt-manager but when i try to link the audio component of the GPU to the kvm it handle indefinetly till freeze.

lug 05 09:15:10 duoT libvirtd[435]: Client hit max requests limit 5. This may result in keep-alive timeouts. Consider tuning the max_client_requests server parameter
lug 05 09:14:52 duoT kernel: vfio-pci 0000:01:00.0: not ready 32767ms after resume; waiting
lug 05 09:14:47 duoT systemd[1]: blueman-mechanism.service: Deactivated successfully.
lug 05 09:14:35 duoT kernel: vfio-pci 0000:01:00.0: not ready 16383ms after resume; waiting
lug 05 09:14:27 duoT systemd[564]: Started User preferences database.
lug 05 09:14:27 duoT dbus-daemon[805]: [session uid=1000 pid=805] Successfully activated service 'ca.desrt.dconf'
lug 05 09:14:27 duoT systemd[564]: Starting User preferences database...
lug 05 09:14:27 duoT dbus-daemon[805]: [session uid=1000 pid=805] Activating via systemd: service name='ca.desrt.dconf' unit='dconf.service' requested by ':1.46' (uid=1000 pid=1833 comm="/usr/bin/python /usr/bin/virt-manager")
lug 05 09:14:26 duoT kernel: vfio-pci 0000:01:00.0: not ready 8191ms after resume; waiting
lug 05 09:14:25 duoT fuzzel[1824]: fuzzel: theme: hicolor
lug 05 09:14:25 duoT fuzzel[1824]: wayland: cursor theme: (null), size: 24, scale: 1
lug 05 09:14:25 duoT fuzzel[1824]: fcft: /usr/share/fonts/gnu-free/FreeMono.otf: size=12.00pt/24px, dpi=143.66
lug 05 09:14:25 duoT fuzzel[1824]: fcft: fontconfig: 2.14.2, freetype: 2.13.1, harfbuzz: 7.3.0, utf8proc: 2.8.0 (Unicode 15.0.0)
lug 05 09:14:25 duoT fuzzel[1824]: fcft: fcft: 3.1.5 +graphemes +runs +svg(nanosvg) -assertions
lug 05 09:14:25 duoT fuzzel[1824]: wayland: eDP-1: 1920x1080+0x0@60Hz 0x0819 15.33" scale=1 PPI=147x154 (physical) PPI=147x154 (logical), DPI=143.66
lug 05 09:14:25 duoT fuzzel[1824]: config: loading configuration from /etc/xdg/fuzzel/fuzzel.ini
lug 05 09:14:25 duoT fuzzel[1824]: fuzzel: version: 1.9.1 +cairo +png +svg(nanosvg) -assertions
lug 05 09:14:22 duoT kernel: vfio-pci 0000:01:00.0: not ready 4095ms after resume; waiting
lug 05 09:14:20 duoT kernel: vfio-pci 0000:01:00.0: not ready 2047ms after resume; waiting
lug 05 09:14:19 duoT kernel: vfio-pci 0000:01:00.0: not ready 1023ms after resume; waiting
lug 05 09:14:17 duoT rtkit-daemon[993]: Supervising 12 threads of 4 processes of 1 users.
lug 05 09:14:17 duoT rtkit-daemon[993]: Supervising 12 threads of 4 processes of 1 users.
lug 05 09:14:17 duoT rtkit-daemon[993]: Supervising 12 threads of 4 processes of 1 users.
lug 05 09:14:17 duoT rtkit-daemon[993]: Supervising 12 threads of 4 processes of 1 users.
lug 05 09:14:17 duoT rtkit-daemon[993]: Supervising 12 threads of 4 processes of 1 users.
lug 05 09:14:17 duoT rtkit-daemon[993]: Supervising 12 threads of 4 processes of 1 users.
lug 05 09:14:16 duoT rtkit-daemon[993]: Supervising 12 threads of 4 processes of 1 users.
lug 05 09:14:16 duoT rtkit-daemon[993]: Supervising 12 threads of 4 processes of 1 users.
lug 05 09:14:16 duoT rtkit-daemon[993]: Supervising 12 threads of 4 processes of 1 users.
lug 05 09:14:16 duoT rtkit-daemon[993]: Successfully made thread 1467 of process 852 owned by '1000' RT at priority 10.
lug 05 09:14:16 duoT rtkit-daemon[993]: Supervising 11 threads of 3 processes of 1 users.
lug 05 09:14:16 duoT rtkit-daemon[993]: Supervising 11 threads of 3 processes of 1 users.
lug 05 09:14:16 duoT rtkit-daemon[993]: Supervising 11 threads of 3 processes of 1 users.
lug 05 09:14:16 duoT rtkit-daemon[993]: Supervising 11 threads of 3 processes of 1 users.
lug 05 09:14:16 duoT rtkit-daemon[993]: Supervising 11 threads of 3 processes of 1 users.
lug 05 09:14:16 duoT rtkit-daemon[993]: Supervising 11 threads of 3 processes of 1 users.
lug 05 09:14:16 duoT kernel: warning: `ThreadPoolForeg' uses wireless extensions which will stop working for Wi-Fi 7 hardware; use nl80211

I'm probably gonna give up on passthrought and try to use a shared card method like linked before in this thread. I need this up and running for the end of this week and i'm not seeing the end here

Last edited by sick_mitch (2023-07-05 07:53:00)

sick_mitch · 2023-07-05 10:47:34

Tried one last procedure before refreshing and looks like it worked!!
I have all up and running using ACS override patch built-in in zen kernel to split up the iommu groups so i could pass complete groups. Gonna test in the next days and update with complete solution if that's the case.

lug 05 12:35:55 duoT wpa_supplicant[457]: wlp4s0: CTRL-EVENT-SIGNAL-CHANGE above=1 signal=-33 noise=9999 txrate=866700
lug 05 12:35:54 duoT systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
lug 05 12:35:45 duoT systemd-machined[396]: Machine qemu-1-win10 terminated.
lug 05 12:35:45 duoT systemd[1]: machine-qemu\x2d1\x2dwin10.scope: Consumed 6min 39.091s CPU time.
lug 05 12:35:45 duoT systemd[1]: machine-qemu\x2d1\x2dwin10.scope: Deactivated successfully.
lug 05 12:35:45 duoT systemd-machined[396]: Machine qemu-1-win10 terminated.
lug 05 12:35:45 duoT systemd[1]: machine-qemu\x2d1\x2dwin10.scope: Consumed 6min 39.091s CPU time.
lug 05 12:35:45 duoT systemd[1]: machine-qemu\x2d1\x2dwin10.scope: Deactivated successfully.
lug 05 12:35:44 duoT systemd[1]: Started Network Manager Script Dispatcher Service.
lug 05 12:35:44 duoT dbus-daemon[394]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
lug 05 12:35:44 duoT systemd[1]: Starting Network Manager Script Dispatcher Service...
lug 05 12:35:44 duoT dbus-daemon[394]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.5' (>
lug 05 12:35:44 duoT NetworkManager[398]: <info>  [1688553344.0374] device (vnet0): released from master device virbr0
lug 05 12:35:44 duoT NetworkManager[398]: <info>  [1688553344.0372] device (vnet0): state change: activated -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed')
lug 05 12:35:44 duoT kernel: virbr0: port 1(vnet0) entered disabled state
lug 05 12:35:44 duoT kernel: vnet0 (unregistering): left promiscuous mode
lug 05 12:35:44 duoT kernel: vnet0 (unregistering): left allmulticast mode
lug 05 12:35:44 duoT kernel: virbr0: port 1(vnet0) entered disabled state
lug 05 12:35:33 duoT rtkit-daemon[959]: Supervising 12 threads of 4 processes of 1 users.
lug 05 12:35:33 duoT rtkit-daemon[959]: Supervising 12 threads of 4 processes of 1 users.
lug 05 12:32:29 duoT dnsmasq-dhcp[536]: DHCPACK(virbr0) 192.168.122.246 52:54:00:93:ca:65 DESKTOP-FR148IE
lug 05 12:32:29 duoT dnsmasq-dhcp[536]: DHCPREQUEST(virbr0) 192.168.122.246 52:54:00:93:ca:65
lug 05 12:32:29 duoT dnsmasq-dhcp[536]: DHCPOFFER(virbr0) 192.168.122.246 52:54:00:93:ca:65
lug 05 12:32:29 duoT dnsmasq-dhcp[536]: DHCPDISCOVER(virbr0) 192.168.122.246 52:54:00:93:ca:65
lug 05 12:32:25 duoT systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
lug 05 12:32:19 duoT kernel: virbr0: topology change detected, propagating
lug 05 12:32:19 duoT kernel: virbr0: port 1(vnet0) entered forwarding state
lug 05 12:32:19 duoT NetworkManager[398]: <info>  [1688553139.1666] device (virbr0): carrier: link connected
lug 05 12:32:17 duoT kernel: vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)
lug 05 12:32:17 duoT kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
lug 05 12:32:17 duoT kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258
lug 05 12:32:17 duoT kernel: vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
lug 05 12:32:17 duoT kernel: virbr0: port 1(vnet0) entered learning state
lug 05 12:32:15 duoT systemd[1]: Started Network Manager Script Dispatcher Service.
lug 05 12:32:15 duoT systemd[1]: Started Virtual Machine qemu-1-win10.
lug 05 12:32:15 duoT NetworkManager[398]: <info>  [1688553135.1190] device (vnet0): Activation: successful, device activated.
lug 05 12:32:15 duoT NetworkManager[398]: <info>  [1688553135.1187] device (vnet0): state change: secondaries -> activated (reason 'none', sys-iface-state: 'external')
lug 05 12:32:15 duoT NetworkManager[398]: <info>  [1688553135.1185] device (vnet0): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'external')
lug 05 12:32:15 duoT dbus-daemon[394]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
lug 05 12:32:15 duoT systemd-machined[396]: New machine qemu-1-win10.
lug 05 12:32:15 duoT systemd[1]: Starting Network Manager Script Dispatcher Service...
lug 05 12:32:15 duoT dbus-daemon[394]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.5' (>
lug 05 12:32:15 duoT NetworkManager[398]: <info>  [1688553135.0936] device (vnet0): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'external')
lug 05 12:35:55 duoT wpa_supplicant[457]: wlp4s0: CTRL-EVENT-SIGNAL-CHANGE above=1 signal=-33 noise=9999 txrate=866700
lug 05 12:35:54 duoT systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
lug 05 12:35:45 duoT systemd-machined[396]: Machine qemu-1-win10 terminated.
lug 05 12:35:45 duoT systemd[1]: machine-qemu\x2d1\x2dwin10.scope: Consumed 6min 39.091s CPU time.
lug 05 12:35:45 duoT systemd[1]: machine-qemu\x2d1\x2dwin10.scope: Deactivated successfully.
lug 05 12:35:44 duoT systemd[1]: Started Network Manager Script Dispatcher Service.
lug 05 12:35:44 duoT dbus-daemon[394]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
lug 05 12:35:44 duoT systemd[1]: Starting Network Manager Script Dispatcher Service...
lug 05 12:35:44 duoT dbus-daemon[394]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.5' (>
lug 05 12:35:44 duoT NetworkManager[398]: <info>  [1688553344.0374] device (vnet0): released from master device virbr0
lug 05 12:35:44 duoT NetworkManager[398]: <info>  [1688553344.0372] device (vnet0): state change: activated -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed')
lug 05 12:35:44 duoT kernel: virbr0: port 1(vnet0) entered disabled state
lug 05 12:35:44 duoT kernel: vnet0 (unregistering): left promiscuous mode
lug 05 12:35:44 duoT kernel: vnet0 (unregistering): left allmulticast mode
lug 05 12:35:44 duoT kernel: virbr0: port 1(vnet0) entered disabled state
lug 05 12:35:33 duoT rtkit-daemon[959]: Supervising 12 threads of 4 processes of 1 users.
lug 05 12:35:33 duoT rtkit-daemon[959]: Supervising 12 threads of 4 processes of 1 users.
lug 05 12:32:29 duoT dnsmasq-dhcp[536]: DHCPACK(virbr0) 192.168.122.246 52:54:00:93:ca:65 DESKTOP-FR148IE
lug 05 12:32:29 duoT dnsmasq-dhcp[536]: DHCPREQUEST(virbr0) 192.168.122.246 52:54:00:93:ca:65
lug 05 12:32:29 duoT dnsmasq-dhcp[536]: DHCPOFFER(virbr0) 192.168.122.246 52:54:00:93:ca:65
lug 05 12:32:29 duoT dnsmasq-dhcp[536]: DHCPDISCOVER(virbr0) 192.168.122.246 52:54:00:93:ca:65
lug 05 12:32:25 duoT systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
lug 05 12:32:19 duoT kernel: virbr0: topology change detected, propagating
lug 05 12:32:19 duoT kernel: virbr0: port 1(vnet0) entered forwarding state
lug 05 12:32:19 duoT NetworkManager[398]: <info>  [1688553139.1666] device (virbr0): carrier: link connected
lug 05 12:32:17 duoT kernel: vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)
lug 05 12:32:17 duoT kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
lug 05 12:35:44 duoT NetworkManager[398]: <info>  [1688553344.0374] device (vnet0): released from master device virbr0
lug 05 12:35:44 duoT NetworkManager[398]: <info>  [1688553344.0372] device (vnet0): state change: activated -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed')
lug 05 12:35:44 duoT kernel: virbr0: port 1(vnet0) entered disabled state
lug 05 12:35:44 duoT kernel: vnet0 (unregistering): left promiscuous mode
lug 05 12:35:44 duoT kernel: vnet0 (unregistering): left allmulticast mode
lug 05 12:35:44 duoT kernel: virbr0: port 1(vnet0) entered disabled state
lug 05 12:35:33 duoT rtkit-daemon[959]: Supervising 12 threads of 4 processes of 1 users.
lug 05 12:35:33 duoT rtkit-daemon[959]: Supervising 12 threads of 4 processes of 1 users.
lug 05 12:32:29 duoT dnsmasq-dhcp[536]: DHCPACK(virbr0) 192.168.122.246 52:54:00:93:ca:65 DESKTOP-FR148IE
lug 05 12:32:29 duoT dnsmasq-dhcp[536]: DHCPREQUEST(virbr0) 192.168.122.246 52:54:00:93:ca:65
lug 05 12:32:29 duoT dnsmasq-dhcp[536]: DHCPOFFER(virbr0) 192.168.122.246 52:54:00:93:ca:65
lug 05 12:32:29 duoT dnsmasq-dhcp[536]: DHCPDISCOVER(virbr0) 192.168.122.246 52:54:00:93:ca:65
lug 05 12:32:25 duoT systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
lug 05 12:32:19 duoT kernel: virbr0: topology change detected, propagating
lug 05 12:32:19 duoT kernel: virbr0: port 1(vnet0) entered forwarding state
lug 05 12:32:19 duoT NetworkManager[398]: <info>  [1688553139.1666] device (virbr0): carrier: link connected
lug 05 12:32:17 duoT kernel: vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)
lug 05 12:32:17 duoT kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
lug 05 12:32:17 duoT kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258
lug 05 12:32:17 duoT kernel: vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
lug 05 12:32:17 duoT kernel: virbr0: port 1(vnet0) entered learning state
lug 05 12:32:15 duoT systemd[1]: Started Network Manager Script Dispatcher Service.
lug 05 12:32:15 duoT systemd[1]: Started Virtual Machine qemu-1-win10.
lug 05 12:32:15 duoT NetworkManager[398]: <info>  [1688553135.1190] device (vnet0): Activation: successful, device activated.
lug 05 12:32:15 duoT NetworkManager[398]: <info>  [1688553135.1187] device (vnet0): state change: secondaries -> activated (reason 'none', sys-iface-state: 'external')
lug 05 12:32:15 duoT NetworkManager[398]: <info>  [1688553135.1185] device (vnet0): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'external')
lug 05 12:32:15 duoT dbus-daemon[394]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
lug 05 12:32:15 duoT systemd-machined[396]: New machine qemu-1-win10.
lug 05 12:32:15 duoT systemd[1]: Starting Network Manager Script Dispatcher Service...
lug 05 12:32:15 duoT dbus-daemon[394]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.5' (>
lug 05 12:32:15 duoT NetworkManager[398]: <info>  [1688553135.0936] device (vnet0): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'external')
lug 05 12:32:15 duoT NetworkManager[398]: <info>  [1688553135.0935] device (vnet0): Activation: connection 'vnet0' enslaved, continuing activation
lug 05 12:32:15 duoT NetworkManager[398]: <info>  [1688553135.0935] device (virbr0): bridge port vnet0 was attached
lug 05 12:32:15 duoT NetworkManager[398]: <info>  [1688553135.0934] device (vnet0): state change: config -> ip-config (reason 'none', sys-iface-state: 'external')
lug 05 12:32:15 duoT NetworkManager[398]: <info>  [1688553135.0933] device (vnet0): state change: prepare -> config (reason 'none', sys-iface-state: 'external')
lug 05 12:32:15 duoT NetworkManager[398]: <info>  [1688553135.0931] device (vnet0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'external')
lug 05 12:32:15 duoT NetworkManager[398]: <info>  [1688553135.0930] device (vnet0): Activation: starting connection 'vnet0' (8057110b-78d8-4710-8883-9967de2e3d8d)
lug 05 12:32:15 duoT NetworkManager[398]: <info>  [1688553135.0927] device (vnet0): state change: unavailable -> disconnected (reason 'connection-assumed', sys-iface-state: 'external')
lug 05 12:32:15 duoT NetworkManager[398]: <info>  [1688553135.0922] device (vnet0): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external')
lug 05 12:32:15 duoT NetworkManager[398]: <info>  [1688553135.0845] manager: (vnet0): new Tun device (/org/freedesktop/NetworkManager/Devices/6)
lug 05 12:32:15 duoT kernel: virbr0: port 1(vnet0) entered listening state
lug 05 12:32:15 duoT kernel: virbr0: port 1(vnet0) entered blocking state
lug 05 12:32:15 duoT kernel: vnet0: entered promiscuous mode
lug 05 12:32:15 duoT kernel: vnet0: entered allmulticast mode
lug 05 12:32:15 duoT kernel: virbr0: port 1(vnet0) entered disabled state
lug 05 12:32:15 duoT kernel: virbr0: port 1(vnet0) entered blocking state
lug 05 12:32:15 duoT kernel: tun: Universal TUN/TAP device driver, 1.6
lug 05 12:32:15 duoT systemd[1]: Started Virtual machine log manager.
lug 05 12:32:15 duoT systemd[1]: Listening on Virtual machine log manager socket.

THANKS A LOT SETH for your patience and your time

Update
Splitting IOMMU groups did help but didn't solve the freeze it just reduced how often it occurs and added an output just one time before freezing

can't change power state from D3cold to D0

so i kept digging and found this in a post in proxmox forum:

pcie_port_pm=off

adding this to my grub config finally got the freeze out of the way
this is not a total win tho, my gpu fully pass to the kvm but looks like if the kvm couln't use it at full power but limit itself around 25% of the gpu usage. My guesses now are:

Is the GPU locked in D3cold state? How to verify/control the state?
Is this a good solution? I don't like cutting all PCI power management maybe i can cut only interested PCI
Is there a way to let the control of the state to the kvm?

For my use a fourth of the GPU is enough keeping the load light so in the meantime i use it i'll keep studying how to optimize it.

Last edited by sick_mitch (2023-07-09 13:12:56)

Arch Linux

#1 2023-07-04 07:26:16

[SOLVED] NVIDIA GPU passthrough freezes on host startup

#2 2023-07-04 08:44:24

Re: [SOLVED] NVIDIA GPU passthrough freezes on host startup

#3 2023-07-04 08:52:47

Re: [SOLVED] NVIDIA GPU passthrough freezes on host startup

#4 2023-07-04 12:14:29

Re: [SOLVED] NVIDIA GPU passthrough freezes on host startup

#5 2023-07-04 12:58:47

Re: [SOLVED] NVIDIA GPU passthrough freezes on host startup

#6 2023-07-04 14:10:17

Re: [SOLVED] NVIDIA GPU passthrough freezes on host startup

#7 2023-07-04 14:19:03

Re: [SOLVED] NVIDIA GPU passthrough freezes on host startup

#8 2023-07-04 14:54:07

Re: [SOLVED] NVIDIA GPU passthrough freezes on host startup

#9 2023-07-04 16:19:11

Re: [SOLVED] NVIDIA GPU passthrough freezes on host startup

#10 2023-07-04 21:06:06

Re: [SOLVED] NVIDIA GPU passthrough freezes on host startup

#11 2023-07-04 21:17:09

Re: [SOLVED] NVIDIA GPU passthrough freezes on host startup

#12 2023-07-05 05:50:58

Re: [SOLVED] NVIDIA GPU passthrough freezes on host startup

#13 2023-07-05 06:26:11

Re: [SOLVED] NVIDIA GPU passthrough freezes on host startup

#14 2023-07-05 06:32:39

Re: [SOLVED] NVIDIA GPU passthrough freezes on host startup

#15 2023-07-05 10:47:34

Re: [SOLVED] NVIDIA GPU passthrough freezes on host startup

Board footer