You are not logged in.
- Topics: Active | Unanswered
#1 2026-07-14 22:43:25
- MichaelMocha
- Member
- Registered: 2025-05-23
- Posts: 12
Virtual Machine crashing with VFIO. Had a working setup for 6 months.
My virtual machine has been crashing repeatedly and intermittently and I am not sure how to troubleshoot.
I followed these steps and had a working system for over 6 months before getting these issues. https://wiki.archlinux.org/title/PCI_pa … h_via_OVMF
The host OS is arch with KVM, and Virtmanager, the VM is Linux Mint. I have an NVIDIA 5070. There are no NVIDIA drivers on my host and nouveau is blacklisted which I verified with lsmod | grep nouveau.
The VM boots and works for a few minutes. I can start a game and inconsistently the VM will crash. I am playing Minecraft and have monitored GPU usage in NVIDIA-SMI and it is not that demanding on either the GPU util or the memory usage.
The screen goes black and audio shuts off, but the VM is still shown as running in virtmanager,
What makes me think this is a GPU driver issue is sometimes when this has happened the GPUs fans will go to max speed blowing cold air, and virtmanager won't restart the VM without a reboot and gives the Unknown PCI header type '127' error, which I have read is the GPU reset bug.
I will update the post if I get this error message again, it seems with the crash that happened right now the VM was able to restart without an error.
System info Arch Linux Host
/etc/modprobe.d/vfio.conf
blacklist nouveau
softdep drm pre: vfio-pci
options vfio-pci ids=10de:2c05,10de:22e9IOMMU Group 13:
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GB203 [GeForce RTX 5070 Ti] [10de:2c05] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GB203 High Definition Audio Controller [10de:22e9] (rev a1)
IOMMU Group 14:Lspci -v
`01:00.0 VGA compatible controller: NVIDIA Corporation GB203 [GeForce RTX 5070 Ti] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 5310
Flags: fast devsel, IRQ 51, IOMMU group 13
Memory at d8000000 (32-bit, non-prefetchable) [virtual] [size=64M]
Memory at f800000000 (64-bit, prefetchable) [disabled] [size=16G]
Memory at fc00000000 (64-bit, prefetchable) [disabled] [size=32M]
I/O ports at f000 [disabled] [size=128]
Expansion ROM at dc000000 [virtual] [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable- Count=1/16 Maskable+ 64bit+
Capabilities: [60] Express Legacy Endpoint, IntMsgNum 0
Capabilities: [9c] Vendor Specific Information: Len=14 <?>
Capabilities: [b0] MSI-X: Enable- Count=9 Masked-
Capabilities: [100] Secondary PCI Express
Capabilities: [12c] Latency Tolerance Reporting
Capabilities: [134] Physical Resizable BAR
Capabilities: [140] Virtual Resizable BAR
Capabilities: [14c] Data Link Feature <?>
Capabilities: [158] Physical Layer 16.0 GT/s
Capabilities: [188] Physical Layer 32.0 GT/s
Capabilities: [1b8] Advanced Error Reporting
Capabilities: [200] Lane Margining at the Receiver
Capabilities: [248] Alternative Routing-ID Interpretation (ARI)
Capabilities: [250] Single Root I/O Virtualization (SR-IOV)
Capabilities: [290] L1 PM Substates
Capabilities: [2a4] Vendor Specific Information: ID=0001 Rev=1 Len=014 <?>
Capabilities: [2bc] Power Budgeting <?>
Capabilities: [2f4] Device Serial Number bc-3c-13-54-23-2d-b0-48
Kernel driver in use: vfio-pci
Kernel modules: nouveau
01:00.1 Audio device: NVIDIA Corporation GB203 High Definition Audio Controller (rev a1) (prog-if 00 [HDA compatible])
Subsystem: NVIDIA Corporation Device 0000
Flags: fast devsel, IRQ 52, IOMMU group 13
Memory at dc080000 (32-bit, non-prefetchable) [virtual] [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [60] Express Endpoint, IntMsgNum 0
Capabilities: [9c] Vendor Specific Information: Len=14 <?>
Capabilities: [100] Data Link Feature <?>
Capabilities: [10c] Advanced Error Reporting
Capabilities: [154] Alternative Routing-ID Interpretation (ARI)
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel`Bios version from dmesg
[ 0.000000] DMI: SMBIOS 3.4.0 present.
[ 0.000000] DMI: ASRock X870 Steel Legend WiFi/X870 Steel Legend WiFi, BIOS 4.03 12/17/2025
System Info Linux Mint VM
Nvidia Driver 595-open from the linux mint driver manager.
Last edited by MichaelMocha (2026-07-14 22:58:32)
Offline
#2 2026-07-14 22:53:47
- MichaelMocha
- Member
- Registered: 2025-05-23
- Posts: 12
Re: Virtual Machine crashing with VFIO. Had a working setup for 6 months.
Virtmanager xml
<domain type="kvm">
<name>pepper_mint</name>
<uuid>66318c72-848f-49f0-8ee5-ca84d0bf4c8b</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://debian.org/debian/13"/>
</libosinfo:libosinfo>
</metadata>
<memory unit="KiB">31667200</memory>
<currentMemory unit="KiB">31667200</currentMemory>
<vcpu placement="static">24</vcpu>
<iothreads>1</iothreads>
<cputune>
<vcpupin vcpu="0" cpuset="4"/>
<vcpupin vcpu="1" cpuset="20"/>
<vcpupin vcpu="2" cpuset="5"/>
<vcpupin vcpu="3" cpuset="21"/>
<vcpupin vcpu="4" cpuset="6"/>
<vcpupin vcpu="5" cpuset="22"/>
<vcpupin vcpu="6" cpuset="7"/>
<vcpupin vcpu="7" cpuset="23"/>
<vcpupin vcpu="8" cpuset="8"/>
<vcpupin vcpu="9" cpuset="24"/>
<vcpupin vcpu="10" cpuset="9"/>
<vcpupin vcpu="11" cpuset="25"/>
<vcpupin vcpu="12" cpuset="10"/>
<vcpupin vcpu="13" cpuset="26"/>
<vcpupin vcpu="14" cpuset="11"/>
<vcpupin vcpu="15" cpuset="27"/>
<vcpupin vcpu="16" cpuset="12"/>
<vcpupin vcpu="17" cpuset="28"/>
<vcpupin vcpu="18" cpuset="13"/>
<vcpupin vcpu="19" cpuset="29"/>
<vcpupin vcpu="20" cpuset="14"/>
<vcpupin vcpu="21" cpuset="30"/>
<vcpupin vcpu="22" cpuset="15"/>
<vcpupin vcpu="23" cpuset="31"/>
<emulatorpin cpuset="0-1,16-17"/>
<iothreadpin iothread="1" cpuset="0-1,16-17"/>
</cputune>
<os firmware="efi">
<type arch="x86_64" machine="pc-q35-9.2">hvm</type>
<firmware>
<feature enabled="no" name="enrolled-keys"/>
<feature enabled="yes" name="secure-boot"/>
</firmware>
<loader readonly="yes" secure="yes" type="pflash" format="raw">/usr/share/edk2/x64/OVMF_CODE.secboot.4m.fd</loader>
<nvram template="/usr/share/edk2/x64/OVMF_VARS.4m.fd" templateFormat="raw" format="raw">/var/lib/libvirt/qemu/nvram/pepper_mint_VARS.fd</nvram>
</os>
<features>
<acpi/>
<apic/>
<kvm>
<hidden state="on"/>
</kvm>
<vmport state="off"/>
<smm state="on"/>
</features>
<cpu mode="host-passthrough" check="none" migratable="on">
<topology sockets="1" dies="1" clusters="1" cores="12" threads="2"/>
<feature policy="require" name="topoext"/>
</cpu>
<clock offset="utc">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled="yes"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2" io="threads" discard="unmap"/>
<source file="/home/arch/disks/primary_mint_gaming.qcow2"/>
<target dev="vda" bus="virtio"/>
<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
</disk>
<disk type="file" device="cdrom">
<driver name="qemu" type="raw"/>
<target dev="sda" bus="sata"/>
<readonly/>
<boot order="1"/>
<address type="drive" controller="0" bus="0" target="0" unit="0"/>
</disk>
<controller type="usb" index="0" model="qemu-xhci" ports="15">
<address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
</controller>
<controller type="pci" index="0" model="pcie-root"/>
<controller type="pci" index="1" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="1" port="0x10"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="2" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="2" port="0x11"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
</controller>
<controller type="pci" index="3" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="3" port="0x12"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
</controller>
<controller type="pci" index="4" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="4" port="0x13"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
</controller>
<controller type="pci" index="5" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="5" port="0x14"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
</controller>
<controller type="pci" index="6" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="6" port="0x15"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
</controller>
<controller type="pci" index="7" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="7" port="0x16"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
</controller>
<controller type="pci" index="8" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="8" port="0x17"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
</controller>
<controller type="pci" index="9" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="9" port="0x18"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="10" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="10" port="0x19"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
</controller>
<controller type="pci" index="11" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="11" port="0x1a"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
</controller>
<controller type="pci" index="12" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="12" port="0x1b"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
</controller>
<controller type="pci" index="13" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="13" port="0x1c"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
</controller>
<controller type="pci" index="14" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="14" port="0x1d"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
</controller>
<controller type="sata" index="0">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
</controller>
<controller type="virtio-serial" index="0">
<address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
</controller>
<interface type="network">
<mac address="52:54:00:d9:ba:32"/>
<source network="default"/>
<model type="virtio"/>
<address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
<serial type="pty">
<target type="isa-serial" port="0">
<model name="isa-serial"/>
</target>
</serial>
<console type="pty">
<target type="serial" port="0"/>
</console>
<channel type="unix">
<target type="virtio" name="org.qemu.guest_agent.0"/>
<address type="virtio-serial" controller="0" bus="0" port="1"/>
</channel>
<input type="mouse" bus="ps2"/>
<input type="keyboard" bus="ps2"/>
<sound model="ich9">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
</sound>
<audio id="1" type="none"/>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x09" slot="0x00" function="0x0"/>
</source>
<address type="pci" domain="0x0000" bus="0x09" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</source>
<address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x01" slot="0x00" function="0x1"/>
</source>
<address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
</hostdev>
<watchdog model="itco" action="reset"/>
<memballoon model="virtio">
<address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
</memballoon>
<rng model="virtio">
<backend model="random">/dev/urandom</backend>
<address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
</rng>
</devices>
</domain>Last edited by MichaelMocha (2026-07-14 22:56:26)
Offline
#3 2026-07-14 23:34:46
- MichaelMocha
- Member
- Registered: 2025-05-23
- Posts: 12
Re: Virtual Machine crashing with VFIO. Had a working setup for 6 months.
Playing a more demanding game with expedition 33 has not crashed at least yet. I don't think the problem is overstressing the GPU or thermals, but hard to say since it has been inconsistent.
Offline
#4 2026-07-15 06:49:02
- seth
- Member
- From: Won't reply 2 private help req
- Registered: 2012-09-03
- Posts: 76,888
Re: Virtual Machine crashing with VFIO. Had a working setup for 6 months.
The host OS is arch with KVM, and Virtmanager, the VM is Linux Mint … The VM boots and works for a few minutes. I can start a game and inconsistently the VM
Sanity check ahead: why are we using vfio to a linux guest for gaming instead of just playing on the host?
I can start a game and inconsistently the VM will crash … The screen goes black and audio shuts off, but the VM is still shown as running in virtmanager
Do you happen to have logs from the guest (journal from the crashing boot) to gauge the nature of the "crash"?
Ie. what makes you believe that this is related to the VM, VFIO or the host *at all*?
Edit: did you try some other graphical session on the guest? Openbox or so? Could just be the game crashing and the compositor there nosediving…
Last edited by seth (2026-07-15 06:50:03)
Online
#5 2026-07-16 00:27:33
- MichaelMocha
- Member
- Registered: 2025-05-23
- Posts: 12
Re: Virtual Machine crashing with VFIO. Had a working setup for 6 months.
Sanity check ahead: why are we using vfio to a linux guest for gaming instead of just playing on the host?
I am using this as a replacement for dual boot. I have a windows VM, a separate work VM both with GPU passthrough that run at different times and a VM that runs Truenas in the background.
Do you happen to have logs from the guest (journal from the crashing boot) to gauge the nature of the "crash"?
I don't. I am looking for what troubleshooting steps would be useful so that next time the crash occurs I can find these logs. I can't reproduce the crash, it just seems to happen randomly.
Ie. what makes you believe that this is related to the VM, VFIO or the host *at all*?
On previous crashes Virtmanager was very laggy when force shutting down the VM, and it would not restart the VM because of a GPU driver 127 error. Virtmanager was only able to restart the VM after rebooting the host OS.
This did not happen on my most recent crash so that one probably is different. I will get this exact error message if it happens again.
Edit: did you try some other graphical session on the guest? Openbox or so? Could just be the game crashing and the compositor there nosediving…
I haven't tried this. I could attempt to add spice to the VM next time.
Thanks
Offline
#6 2026-07-16 05:53:15
- seth
- Member
- From: Won't reply 2 private help req
- Registered: 2012-09-03
- Posts: 76,888
Re: Virtual Machine crashing with VFIO. Had a working setup for 6 months.
I am using this as a replacement for dual boot.
3rd link below. Mandatory.
Disable it (it's NOT the BIOS setting!) and reboot windows and linux twice for voodo reasons.
Since you're forwarding devices, this could still leave them in undefined states.
I am looking for what troubleshooting steps would be useful
journalctl should™ work on Mint as well, allowing you to access journals of previous boots as well, but everytime you push the (virtual) power button, you're prematurely terminating the journal.
Online
#7 2026-07-18 01:58:31
- MichaelMocha
- Member
- Registered: 2025-05-23
- Posts: 12
Re: Virtual Machine crashing with VFIO. Had a working setup for 6 months.
3rd link below. Mandatory.
Disable it (it's NOT the BIOS setting!) and reboot windows and linux twice for voodo reasons.Since you're forwarding devices, this could still leave them in undefined states.
I haven't booted the windows VM for a few months before the issue appeared, and I don't need to use it often. I can disable this setting, but I doubt its the issue. Was thinking that VMs should completely avoid this type of problem.
It just happened again but this time it has maxed the GPU fans with cold air. I can't add spice graphics while the vm is running.
Seeing what I can find in arch journalctl before I shut down the VM. Thanks.
Last edited by MichaelMocha (2026-07-18 02:24:00)
Offline
#8 2026-07-18 02:05:59
- MichaelMocha
- Member
- Registered: 2025-05-23
- Posts: 12
Re: Virtual Machine crashing with VFIO. Had a working setup for 6 months.
The VM name is pepper-mint
Jul 17 20:34:36 archlinux dnsmasq-dhcp[1003]: DHCPREQUEST(virbr0) 192.168.122.29 52:54:00:d9:ba:32
Jul 17 20:34:36 archlinux dnsmasq-dhcp[1003]: DHCPACK(virbr0) 192.168.122.29 52:54:00:d9:ba:32 pepper-mint
Jul 17 20:46:26 archlinux kernel: pci_bus 0000:0c: Allocating resources
Jul 17 20:46:28 archlinux kernel: xhci_hcd 0000:6d:00.0: bad transfer trb length 4 in event trb
Jul 17 20:47:09 archlinux libvirtd[926]: Operation not supported: live attach of device 'graphics' is not supportedI believe the last one is from when I attempted to add spice graphics to the crashed VM in virtmanager. Left out are usb connection messages.
I don't see anything other than my USB devices in dmesg -T from the time of the crash.
lspci | grep 6d
6d:00.0 USB controller: ASMedia Technology Inc. ASM4242 USB 3.2 xHCI Controller (rev 01)The VM does respond to ping pepper_mint
01:00.0 VGA compatible controller: NVIDIA Corporation GB203 [GeForce RTX 5070 Ti] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 5310
!!! Unknown header type 7f
Memory at d8000000 (32-bit, non-prefetchable) [size=64M]
Memory at f800000000 (64-bit, prefetchable) [size=16G]
Memory at fc00000000 (64-bit, prefetchable) [size=32M]
I/O ports at f000 [size=128]
Expansion ROM at dc000000 [disabled] [size=512K]
Kernel driver in use: vfio-pci
Kernel modules: nouveau
01:00.1 Audio device: NVIDIA Corporation GB203 High Definition Audio Controller (rev a1) (prog-if 00 [HDA compatible])
Subsystem: NVIDIA Corporation Device 0000
!!! Unknown header type 7f
Memory at dc080000 (32-bit, non-prefetchable) [size=16K]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intelAfter shutting down the VM when attempting to restart it.
Error starting domain: internal error: Unknown PCI header type '127' for device '0000:01:00.0'
Traceback (most recent call last):
File "/usr/share/virt-manager/virtManager/asyncjob.py", line 67, in cb_wrapper
callback(asyncjob, *args, **kwargs)
~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/virt-manager/virtManager/asyncjob.py", line 101, in tmpcb
callback(*args, **kwargs)
~~~~~~~~^^^^^^^^^^^^^^^^^
File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57, in newfn
ret = fn(self, *args, **kwargs)
File "/usr/share/virt-manager/virtManager/object/domain.py", line 1446, in startup
self._backend.create()
~~~~~~~~~~~~~~~~~~~~^^
File "/usr/lib/python3.14/site-packages/libvirt.py", line 1401, in create
raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: internal error: Unknown PCI header type '127' for device '0000:01:00.0'Last edited by MichaelMocha (2026-07-18 02:22:36)
Offline
#9 2026-07-18 03:03:15
- MichaelMocha
- Member
- Registered: 2025-05-23
- Posts: 12
Re: Virtual Machine crashing with VFIO. Had a working setup for 6 months.
Jorunalctl from pepper mint.
Jul 17 20:42:11 pepper-mint systemd[1]: flatpak-system-helper.service: Deactivated successfully.
Jul 17 20:42:11 pepper-mint systemd[1]: flatpak-system-helper.service: Consumed 1.754s CPU time.
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU at PCI:0000:07:00: GPU-edfbd6c3-61a5-9713-bed4-2678ff1a2780
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU Board Serial Number: 0
Jul 17 20:45:48 pepper-mint kernel: NVRM: Xid (PCI:0000:07:00): 79, GPU has fallen off the bus.
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU 0000:07:00.0: GPU has fallen off the bus.
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU 0000:07:00.0: GPU serial number is 0.
Jul 17 20:45:48 pepper-mint kernel: NVRM: krcRcAndNotifyAllChannels_IMPL: RC all channels for critical error 79.
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000ca7e:6:0:0x0000000f
Jul 17 20:45:48 pepper-mint kernel: nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000ca7e:4:0:0x0000000f
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 20:45:48 pepper-mint kernel: NVRM: prbEncStartAlloc: Can't allocate memory for protocol buffers.
Jul 17 20:45:48 pepper-mint kernel: NVRM: A GPU crash dump has been created. If possible, please run
NVRM: nvidia-bug-report.sh as root to collect this data before
NVRM: the NVIDIA kernel module is unloaded.
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5869!
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00030; hObject=0xbeef0403; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5870!
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00030; hObject=0xbeef4903; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 20:45:48 pepper-mint kernel: NVRM: Xid (PCI:0000:07:00): 154, GPU recovery action changed from 0x0 (None) to 0x1 (GPU Reset Required)
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5871!
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00030; hObject=0xbeef0402; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5872!
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00030; hObject=0xbeef4902; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5873!
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00030; hObject=0xbeef0401; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5874!
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00030; hObject=0xbeef4901; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5875!
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00030; hObject=0xbeef0400; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5876!
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00030; hObject=0xbeef4900; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5877!
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00030; hObject=0xbeefa0b5; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5878!
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00030; hObject=0xbeef9097; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5879!
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00030; hObject=0xbeef502d; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5880!
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00030; hObject=0xbeef3901; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5881!
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00030; hObject=0xbeef0100; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5882!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5883!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5884!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5885!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5886!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5887!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5888!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5889!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5890!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5891!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5892!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5893!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5894!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5895!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5896!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5897!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5898!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5899!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5900!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5901!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5902!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5903!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5904!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5905!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 5906!
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 20:45:48 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: status == NV_OK @ journal.c:2239
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Current device is not valid [NV_ERR_INVALID_DEVICE] (0x00000026) returned from rmDeviceGpuLocksAcquire(pGpu, GPUS_LOCK_FLAGS_NONE, RM_LOCK_MODULES_MEM_PMA) @ virtual_mem.c:115
Jul 17 20:45:48 pepper-mint kernel: NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Current device is not valid [NV_ERR_INVALID_DEVICE] (0x00000026) returned from rmDeviceGpuLocksAcquire(pGpu, GPUS_LOCK_FLAGS_NONE, RM_LOCK_MODULES_MEM_PMA) @ virtual_mem.c:115
Jul 17 20:45:48 pepper-mint systemd[1]: Started systemd-coredump@4-8655-0.service - Process Core Dump (PID 8655/UID 0).
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5907!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef0403; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5908!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef4903; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5909!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef0402; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5910!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef4902; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5911!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef0401; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5912!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef4901; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5913!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef0400; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5914!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef4900; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5915!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeefa0b5; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5916!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef9097; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5917!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef502d; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5918!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef3901; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5919!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef0100; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5920!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00010; hObject=0x00015070; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5921!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00010; hObject=0x00015071; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5922!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00010; hObject=0x00015072; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5923!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00010; hObject=0x00015073; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5924!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00010; hObject=0x00019800; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5925!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00010; hObject=0x0001b500; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5926!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00010; hObject=0x00019700; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5927!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00010; hObject=0x00012d00; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 20:45:49 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 5928!
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00010; hObject=0x01000013; paramsStatus=0x00000000; status=0x0000000f
Jul 17 20:45:49 pepper-mint kernel: NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Current device is not valid [NV_ERR_INVALID_DEVICE] (0x00000026) returned from rmDeviceGpuLocksAcquire(pGpu, GPUS_LOCK_FLAGS_NONE, RM_LOCK_MODULES_MEM_PMA) @ virtual_mem.c:115
Jul 17 20:45:49 pepper-mint kernel: nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000ca7e:0:0:0x0000000f
Jul 17 20:45:49 pepper-mint kernel: nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000ca7e:2:0:0x0000000f
Jul 17 20:45:49 pepper-mint kernel: nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000ca7e:4:0:0x0000000f
Jul 17 20:45:49 pepper-mint kernel: nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000ca7e:6:0:0x0000000f
Jul 17 20:45:49 pepper-mint kernel: nvidia-modeset: ERROR: GPU:0: DP-0: Failed to disable DisplayPort audio stream-3
Jul 17 20:46:06 pepper-mint systemd-coredump[8656]: [?] Process 3479 (java) of user 1000 dumped core.
Module /home/pepper/.minecraft/runtime/java-runtime-delta/linux/java-runtime-delta/bin/java without build-id.
Module /home/pepper/.minecraft/runtime/java-runtime-delta/linux/java-runtime-delta/lib/libfontmanager.so without build-id.
Module /home/pepper/.minecraft/runtime/java-runtime-delta/linux/java-runtime-delta/lib/libawt.so without build-id.
Module /home/pepper/.minecraft/runtime/java-runtime-delta/linux/java-runtime-delta/lib/libawt_xawt.so without build-id.
Module /home/pepper/.minecraft/runtime/java-runtime-delta/linux/java-runtime-delta/lib/libjsvml.so without build-id.
Module /home/pepper/.minecraft/runtime/java-runtime-delta/linux/java-runtime-delta/lib/libextnet.so without build-id.
Module /home/pepper/.minecraft/runtime/java-runtime-delta/linux/java-runtime-delta/lib/libverify.so without build-id.
Module /home/pepper/.minecraft/runtime/java-runtime-delta/linux/java-runtime-delta/lib/libnet.so without build-id.
Module /home/pepper/.minecraft/runtime/java-runtime-delta/linux/java-runtime-delta/lib/libnio.so without build-id.
Module /home/pepper/.minecraft/runtime/java-runtime-delta/linux/java-runtime-delta/lib/server/libjvm.so without build-id.
Module /home/pepper/.minecraft/runtime/java-runtime-delta/linux/java-runtime-delta/lib/libmanagement_ext.so without build-id.
Module /home/pepper/.minecraft/runtime/java-runtime-delta/linux/java-runtime-delta/lib/libmanagement.so without build-id.
Module /home/pepper/.minecraft/runtime/java-runtime-delta/linux/java-runtime-delta/lib/libjava.so without build-id.
Module /home/pepper/.minecraft/runtime/java-runtime-delta/linux/java-runtime-delta/lib/libjimage.so without build-id.
Module /home/pepper/.minecraft/runtime/java-runtime-delta/linux/java-runtime-delta/lib/libjli.so without build-id.
Module /home/pepper/.minecraft/runtime/java-runtime-delta/linux/java-runtime-delta/lib/libzip.so without build-id.
Module libgcc_s.so.1 from deb gcc-14-14.2.0-4ubuntu2~24.04.1.amd64
Module libstdc++.so.6 from deb gcc-14-14.2.0-4ubuntu2~24.04.1.amd64
Module libzstd.so.1 from deb libzstd-1.5.5+dfsg2-2build1.1.amd64
Module libsystemd.so.0 from deb systemd-255.4-1ubuntu8.16.amd64
Module libudev.so.1.7.8 from deb systemd-255.4-1ubuntu8.16.amd64
Stack trace of thread 3481:
#0 0x000074fb19a9eb2c __pthread_kill_implementation (libc.so.6 + 0x9eb2c)
#1 0x000074fb19a4527e __GI_raise (libc.so.6 + 0x4527e)
#2 0x000074fb19a288ff __GI_abort (libc.so.6 + 0x288ff)
#3 0x000074f94daeea1e n/a (libnvidia-glcore.so.595.71.05 + 0x2eea1e)
#4 0x000074f94e2df5f7 n/a (libnvidia-glcore.so.595.71.05 + 0xadf5f7)
#5 0x000074f94e1edfa9 n/a (libnvidia-glcore.so.595.71.05 + 0x9edfa9)
#6 0x000074f9adb7993c n/a (libGLX_nvidia.so.0 + 0xa493c)
#7 0x000074f9adb47bd0 n/a (libGLX_nvidia.so.0 + 0x72bd0)
#8 0x000074fb094d0c76 n/a (n/a + 0x0)
#9 0x000074fb0a8d5224 n/a (n/a + 0x0)
ELF object binary architecture: AMD x86-64
Jul 17 20:46:06 pepper-mint systemd[1]: systemd-coredump@4-8655-0.service: Deactivated successfully.
Jul 17 20:46:06 pepper-mint systemd[1]: systemd-coredump@4-8655-0.service: Consumed 13.301s CPU time.
Jul 17 20:46:11 pepper-mint kernel: snd_hda_codec_hdmi hdaudioC2D0: out of range cmd 0:4:707:ffffffbfInteresting that it says the java process crashed too. Not sure if thats just because it was using the GPU when the GPU crashed or if a java issue is the cause.
Also PCI devices from inside the VM.
lspci -v
07:00.0 VGA compatible controller: NVIDIA Corporation Device 2c05 (rev a1) (prog-if 00 [VGA controller])
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 5310
Physical Slot: 0-7
Flags: bus master, fast devsel, latency 0, IRQ 22
Memory at 80000000 (32-bit, non-prefetchable) [size=64M]
Memory at 383000000000 (64-bit, prefetchable) [size=16G]
Memory at 383400000000 (64-bit, prefetchable) [size=32M]
I/O ports at 6000 [size=128]
Capabilities: <access denied>
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
08:00.0 Audio device: NVIDIA Corporation Device 22e9 (rev a1)
Subsystem: NVIDIA Corporation Device 0000
Physical Slot: 0-8
Flags: bus master, fast devsel, latency 0, IRQ 23
Memory at 84c00000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intelLast edited by MichaelMocha (2026-07-18 03:04:52)
Offline
#10 2026-07-18 04:16:43
- MichaelMocha
- Member
- Registered: 2025-05-23
- Posts: 12
Re: Virtual Machine crashing with VFIO. Had a working setup for 6 months.
I reproduced the crash on expedition 33 with steam and proton, so its probably not java to blame.
Jul 17 22:45:00 pepper-mint steam[3627]: [2026-07-17 22:45:00] Nothing to do
Jul 17 22:46:01 pepper-mint systemd[1570]: launchpadlib-cache-clean.service - Clean up old files in the Launchpadlib cache was skipped because of an unmet condition check (ConditionPathExists=/home/pepper/.launchpadlib/api.launchpad.net/cache).
Jul 17 22:49:54 pepper-mint kernel: NVRM: gpuHandleSanityCheckRegReadError_GH100: Possible bad register read: addr: 0xb81024, regvalue: 0xbadf2100, error code: Unknown SYS_PRI_ERROR_CODE
Jul 17 22:49:55 pepper-mint kernel: sched: RT throttling activated
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _intrServiceStallExactList: Stuck interrupt detected for mcEngine 50
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: 0 @ intr.c:1524
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: 0 @ gpu_device_mapping.c:277
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 gpuSanityCheck_IMPL: Failed test flags: 0x1
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _intrServiceStallExactList: Interrupt is stuck. Bailing after 1000 iterations.
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU at PCI:0000:07:00: GPU-edfbd6c3-61a5-9713-bed4-2678ff1a2780
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU Board Serial Number: 0
Jul 17 22:49:57 pepper-mint kernel: NVRM: Xid (PCI:0000:07:00): 79, GPU has fallen off the bus.
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU 0000:07:00.0: GPU has fallen off the bus.
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU 0000:07:00.0: GPU serial number is 0.
Jul 17 22:49:57 pepper-mint kernel: NVRM: krcRcAndNotifyAllChannels_IMPL: RC all channels for critical error 79.
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _threadNodeCheckTimeout: API_GPU_ATTACHED_SANITY_CHECK failed!
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11340!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11341!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11342!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11343!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11344!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11345!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11346!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11347!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11348!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11349!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11350!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11351!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11352!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11353!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11354!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11355!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11356!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11357!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11358!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11359!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11360!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11361!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11362!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11363!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 78 sequence 11364!
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvCheckOkFailedNoLog: Check failed: GPU lost from the bus [NV_ERR_GPU_IS_LOST] (0x0000000F) returned from nvdEngineDumpCallbackHelper(pGpu, pPrbEnc, pNvDumpState, pEngineCallback) @ nv_debug_dump.c:273
Jul 17 22:49:57 pepper-mint kernel: NVRM: RmLogGpuCrash: RmLogGpuCrash: failed to save GPU crash data
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 76 sequence 11365!
Jul 17 22:49:57 pepper-mint kernel: nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000ca7e:6:0:0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: Xid (PCI:0000:07:00): 154, GPU recovery action changed from 0x0 (None) to 0x2 (Node Reboot Required)
Jul 17 22:49:57 pepper-mint kernel: nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000ca7e:4:0:0x0000000f
Jul 17 22:49:57 pepper-mint steam[3627]: err:winevulkan:signaller_worker error waiting for semaphores, vr -4.
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11366!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef0403; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11367!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef4903; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11368!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef0402; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11369!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef4902; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11370!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef0401; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11371!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef4901; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11372!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef0400; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11373!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef4900; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11374!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeefa0b5; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11375!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef9097; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11376!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef502d; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11377!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef3901; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11378!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef0100; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Current device is not valid [NV_ERR_INVALID_DEVICE] (0x00000026) returned from rmDeviceGpuLocksAcquire(pGpu, GPUS_LOCK_FLAGS_NONE, RM_LOCK_MODULES_MEM_PMA) @ virtual_mem.c:115
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Current device is not valid [NV_ERR_INVALID_DEVICE] (0x00000026) returned from rmDeviceGpuLocksAcquire(pGpu, GPUS_LOCK_FLAGS_NONE, RM_LOCK_MODULES_MEM_PMA) @ virtual_mem.c:115
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Current device is not valid [NV_ERR_INVALID_DEVICE] (0x00000026) returned from rmDeviceGpuLocksAcquire(pGpu, GPUS_LOCK_FLAGS_NONE, RM_LOCK_MODULES_MEM_PMA) @ virtual_mem.c:115
Jul 17 22:49:57 pepper-mint kernel: ------------[ cut here ]------------
Jul 17 22:49:57 pepper-mint kernel: WARNING: CPU: 19 PID: 2517 at /var/lib/dkms/nvidia/595.71.05/build/nvidia/nv.c:5384 nvidia_dev_put+0xbf/0xd0 [nvidia]
Jul 17 22:49:57 pepper-mint kernel: Modules linked in: snd_seq_dummy snd_hrtimer nvidia_uvm(OE) nvidia_drm(OE) nvidia_modeset(OE) qrtr intel_rapl_msr intel_rapl_common snd_hda_codec_hdmi snd_hda_codec_generic snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_usb_audio snd_hda_codec snd_usbmidi_lib snd_hda_core s>
Jul 17 22:49:57 pepper-mint kernel: xhci_pci_renesas aesni_intel crypto_simd cryptd
Jul 17 22:49:57 pepper-mint kernel: CPU: 19 PID: 2517 Comm: brave Tainted: G OE 6.8.0-134-generic #134-Ubuntu
Jul 17 22:49:57 pepper-mint kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 02/02/2022
Jul 17 22:49:57 pepper-mint kernel: RIP: 0010:nvidia_dev_put+0xbf/0xd0 [nvidia]
Jul 17 22:49:57 pepper-mint kernel: Code: 57 4b ec d6 5b 41 5c 41 5d 41 5e 5d 31 c0 31 d2 31 f6 31 ff c3 cc cc cc cc 31 d2 48 89 de 4c 89 ef e8 c5 96 14 00 85 c0 74 b1 <0f> 0b eb ad 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 90 90 90
Jul 17 22:49:57 pepper-mint kernel: RSP: 0018:ffffcfa089fbfab8 EFLAGS: 00010202
Jul 17 22:49:57 pepper-mint kernel: RAX: 0000000000000026 RBX: ffff8a121cf18000 RCX: 0000000000000000
Jul 17 22:49:57 pepper-mint kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffcfa089fbf9f8
Jul 17 22:49:57 pepper-mint kernel: RBP: ffffcfa089fbfad8 R08: 0000000000000000 R09: 0000000000000000
Jul 17 22:49:57 pepper-mint kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8a121cf18730
Jul 17 22:49:57 pepper-mint kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Jul 17 22:49:57 pepper-mint kernel: FS: 0000000000000000(0000) GS:ffff8a18ee580000(0000) knlGS:0000000000000000
Jul 17 22:49:57 pepper-mint kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 17 22:49:57 pepper-mint kernel: CR2: 0000733f12779000 CR3: 000000054cc3c000 CR4: 0000000000f50ef0
Jul 17 22:49:57 pepper-mint kernel: PKRU: 55555558
Jul 17 22:49:57 pepper-mint kernel: Call Trace:
Jul 17 22:49:57 pepper-mint kernel: <TASK>
Jul 17 22:49:57 pepper-mint kernel: nvidia_close+0x219/0x2f0 [nvidia]
Jul 17 22:49:57 pepper-mint kernel: __fput+0xa0/0x2e0
Jul 17 22:49:57 pepper-mint kernel: ____fput+0xe/0x20
Jul 17 22:49:57 pepper-mint kernel: task_work_run+0x5e/0xa0
Jul 17 22:49:57 pepper-mint kernel: do_exit+0x2be/0x530
Jul 17 22:49:57 pepper-mint kernel: do_group_exit+0x35/0x90
Jul 17 22:49:57 pepper-mint kernel: __x64_sys_exit_group+0x18/0x20
Jul 17 22:49:57 pepper-mint kernel: x64_sys_call+0x259b/0x25a0
Jul 17 22:49:57 pepper-mint kernel: do_syscall_64+0x7f/0x180
Jul 17 22:49:57 pepper-mint kernel: ? unix_seqpacket_recvmsg+0x43/0x70
Jul 17 22:49:57 pepper-mint kernel: ? sock_recvmsg+0xde/0xf0
Jul 17 22:49:57 pepper-mint kernel: ? ____sys_recvmsg+0x111/0x230
Jul 17 22:49:57 pepper-mint kernel: ? ___sys_recvmsg+0x9c/0xf0
Jul 17 22:49:57 pepper-mint kernel: ? rseq_get_rseq_cs+0x22/0x280
Jul 17 22:49:57 pepper-mint kernel: ? rseq_ip_fixup+0x90/0x170
Jul 17 22:49:57 pepper-mint kernel: ? restore_fpregs_from_fpstate+0x3d/0xd0
Jul 17 22:49:57 pepper-mint kernel: ? switch_fpu_return+0x55/0xf0
Jul 17 22:49:57 pepper-mint kernel: ? arch_exit_to_user_mode_prepare.isra.0+0x95/0xe0
Jul 17 22:49:57 pepper-mint kernel: ? syscall_exit_to_user_mode+0x43/0x1e0
Jul 17 22:49:57 pepper-mint kernel: ? do_syscall_64+0x8c/0x180
Jul 17 22:49:57 pepper-mint kernel: ? do_syscall_64+0x8c/0x180
Jul 17 22:49:57 pepper-mint kernel: ? irqentry_exit+0x43/0x50
Jul 17 22:49:57 pepper-mint kernel: ? exc_page_fault+0x94/0x1b0
Jul 17 22:49:57 pepper-mint kernel: entry_SYSCALL_64_after_hwframe+0x78/0x80
Jul 17 22:49:57 pepper-mint kernel: RIP: 0033:0x718d782ee21d
Jul 17 22:49:57 pepper-mint kernel: Code: Unable to access opcode bytes at 0x718d782ee1f3.
Jul 17 22:49:57 pepper-mint kernel: RSP: 002b:00007ffc0d9c5d68 EFLAGS: 00000202 ORIG_RAX: 00000000000000e7
Jul 17 22:49:57 pepper-mint kernel: RAX: ffffffffffffffda RBX: 00000bd400b7e680 RCX: 0000718d782ee21d
Jul 17 22:49:57 pepper-mint kernel: RDX: 00000000000000e7 RSI: fffffffffffff8f8 RDI: 0000000000000001
Jul 17 22:49:57 pepper-mint kernel: RBP: 00007ffc0d9c5df0 R08: 0000000000000000 R09: 0000002c00000000
Jul 17 22:49:57 pepper-mint kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffc0d9c5db0
Jul 17 22:49:57 pepper-mint kernel: R13: 00007ffc0d9c6101 R14: 0000000000000952 R15: 00007ffc0d9c5d78
Jul 17 22:49:57 pepper-mint kernel: </TASK>
Jul 17 22:49:57 pepper-mint kernel: ---[ end trace 0000000000000000 ]---
Jul 17 22:49:57 pepper-mint kernel: ------------[ cut here ]------------
Jul 17 22:49:57 pepper-mint kernel: WARNING: CPU: 19 PID: 2517 at /var/lib/dkms/nvidia/595.71.05/build/nvidia/nv.c:5384 nvidia_dev_put+0xbf/0xd0 [nvidia]
Jul 17 22:49:57 pepper-mint kernel: Modules linked in: snd_seq_dummy snd_hrtimer nvidia_uvm(OE) nvidia_drm(OE) nvidia_modeset(OE) qrtr intel_rapl_msr intel_rapl_common snd_hda_codec_hdmi snd_hda_codec_generic snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_usb_audio snd_hda_codec snd_usbmidi_lib snd_hda_core s>
Jul 17 22:49:57 pepper-mint kernel: xhci_pci_renesas aesni_intel crypto_simd cryptd
Jul 17 22:49:57 pepper-mint kernel: CPU: 19 PID: 2517 Comm: brave Tainted: G W OE 6.8.0-134-generic #134-Ubuntu
Jul 17 22:49:57 pepper-mint kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 02/02/2022
Jul 17 22:49:57 pepper-mint kernel: RIP: 0010:nvidia_dev_put+0xbf/0xd0 [nvidia]
Jul 17 22:49:57 pepper-mint kernel: Code: 57 4b ec d6 5b 41 5c 41 5d 41 5e 5d 31 c0 31 d2 31 f6 31 ff c3 cc cc cc cc 31 d2 48 89 de 4c 89 ef e8 c5 96 14 00 85 c0 74 b1 <0f> 0b eb ad 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 90 90 90
Jul 17 22:49:57 pepper-mint kernel: RSP: 0018:ffffcfa089fbfa50 EFLAGS: 00010202
Jul 17 22:49:57 pepper-mint kernel: RAX: 0000000000000026 RBX: ffff8a121cf18000 RCX: 0000000000000000
Jul 17 22:49:57 pepper-mint kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffcfa089fbf990
Jul 17 22:49:57 pepper-mint kernel: RBP: ffffcfa089fbfa70 R08: 0000000000000000 R09: 0000000000000000
Jul 17 22:49:57 pepper-mint kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8a121cf18730
Jul 17 22:49:57 pepper-mint kernel: R13: 0000000000000000 R14: 0000000000000000 R15: ffff8a1205089a40
Jul 17 22:49:57 pepper-mint kernel: FS: 0000000000000000(0000) GS:ffff8a18ee580000(0000) knlGS:0000000000000000
Jul 17 22:49:57 pepper-mint kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 17 22:49:57 pepper-mint kernel: CR2: 0000733f12779000 CR3: 000000054cc3c000 CR4: 0000000000f50ef0
Jul 17 22:49:57 pepper-mint kernel: PKRU: 55555558
Jul 17 22:49:57 pepper-mint kernel: Call Trace:
Jul 17 22:49:57 pepper-mint kernel: <TASK>
Jul 17 22:49:57 pepper-mint kernel: nvkms_close_gpu+0x53/0x90 [nvidia_modeset]
Jul 17 22:49:57 pepper-mint kernel: nvFreePerOpenDev+0xa7/0xc0 [nvidia_modeset]
Jul 17 22:49:57 pepper-mint kernel: WARNING: kernel stack frame pointer at 00000000761db7b1 in brave:2517 has bad value 000000003547c2bf
Jul 17 22:49:57 pepper-mint kernel: unwind stack type:0 next_sp:0000000000000000 mask:0x2 graph_idx:0
Jul 17 22:49:57 pepper-mint kernel: 000000003d22379c: ffffcfa089fbfaa0 (0xffffcfa089fbfaa0)
Jul 17 22:49:57 pepper-mint kernel: 000000002f0603df: ffffffffc1f28fc3 (nvkms_close_gpu+0x53/0x90 [nvidia_modeset])
Jul 17 22:49:57 pepper-mint kernel: 000000008c60d4e8: 0000000000000000 ...
Jul 17 22:49:57 pepper-mint kernel: 000000006104da28: e03157482ac83500 (0xe03157482ac83500)
Jul 17 22:49:57 pepper-mint kernel: 00000000c5c21644: ffff8a1215ca8588 (0xffff8a1215ca8588)
Jul 17 22:49:57 pepper-mint kernel: 000000009668397a: ffff8a1215ca85d8 (0xffff8a1215ca85d8)
Jul 17 22:49:57 pepper-mint kernel: 00000000761db7b1: ffff8a1207af0008 (0xffff8a1207af0008)
Jul 17 22:49:57 pepper-mint kernel: 0000000071eba776: ffffffffc1fbcbe7 (nvFreePerOpenDev+0xa7/0xc0 [nvidia_modeset])
Jul 17 22:49:57 pepper-mint kernel: 00000000242a5849: ffff8a1215a07840 (0xffff8a1215a07840)
Jul 17 22:49:57 pepper-mint kernel: 00000000791c2b70: ffff8a1215ca8588 (0xffff8a1215ca8588)
Jul 17 22:49:57 pepper-mint kernel: 00000000b9198397: ffff8a1215a07840 (0xffff8a1215a07840)
Jul 17 22:49:57 pepper-mint kernel: 000000004f5cfd44: ffff8a1215ca8588 (0xffff8a1215ca8588)
Jul 17 22:49:57 pepper-mint kernel: 000000000b38e951: ffff8a1215ca85d8 (0xffff8a1215ca85d8)
Jul 17 22:49:57 pepper-mint kernel: 000000004c6663da: ffffffffc1fbd808 (nvKmsClose+0x88/0x170 [nvidia_modeset])
Jul 17 22:49:57 pepper-mint kernel: 00000000cbec7b8e: ffffcfa089fbfb00 (0xffffcfa089fbfb00)
Jul 17 22:49:57 pepper-mint kernel: 000000002031a650: 0000000197c60ea6 (0x197c60ea6)
Jul 17 22:49:57 pepper-mint kernel: 00000000a3bc72f3: ffff8a1215a07840 (0xffff8a1215a07840)
Jul 17 22:49:57 pepper-mint kernel: 00000000e53993d5: ffffcfa089fbfb20 (0xffffcfa089fbfb20)
Jul 17 22:49:57 pepper-mint kernel: 000000009b1f8f69: 000000000008001b (0x8001b)
Jul 17 22:49:57 pepper-mint kernel: 00000000c07540bc: ffffffffc1f27920 (nvkms_close+0x40/0xc0 [nvidia_modeset])
Jul 17 22:49:57 pepper-mint kernel: 000000006efc62a9: ffff8a12189e1100 (0xffff8a12189e1100)
Jul 17 22:49:57 pepper-mint kernel: 00000000954a1139: 000000000008001b (0x8001b)
Jul 17 22:49:57 pepper-mint kernel: 00000000e74be438: ffffcfa089fbfb58 (0xffffcfa089fbfb58)
Jul 17 22:49:57 pepper-mint kernel: 00000000cd5fbf22: ffffffff96ef7a00 (__fput+0xa0/0x2e0)
Jul 17 22:49:57 pepper-mint kernel: 0000000082a505bc: ffff8a12189e1000 (0xffff8a12189e1000)
Jul 17 22:49:57 pepper-mint kernel: 00000000ca70a215: ffff8a121c062f40 (0xffff8a121c062f40)
Jul 17 22:49:57 pepper-mint kernel: 00000000b5526193: ffff8a121c063c2c (0xffff8a121c063c2c)
Jul 17 22:49:57 pepper-mint kernel: 000000007e3aa127: 0000000000000001 (0x1)
Jul 17 22:49:57 pepper-mint kernel: 000000002caa3eb8: 0000000000000000 ...
Jul 17 22:49:57 pepper-mint kernel: 000000002dbb94bb: ffffcfa089fbfb68 (0xffffcfa089fbfb68)
Jul 17 22:49:57 pepper-mint kernel: 0000000062e35e06: ffffffff96ef7cae (____fput+0xe/0x20)
Jul 17 22:49:57 pepper-mint kernel: 00000000dc03865a: ffffcfa089fbfb90 (0xffffcfa089fbfb90)
Jul 17 22:49:57 pepper-mint kernel: 00000000c558a0a7: ffffffff96b3a2be (task_work_run+0x5e/0xa0)
Jul 17 22:49:57 pepper-mint kernel: 00000000a4b8d04e: ffff8a121c062f40 (0xffff8a121c062f40)
Jul 17 22:49:57 pepper-mint kernel: 000000009c51a918: 0000000000000001 (0x1)
Jul 17 22:49:57 pepper-mint kernel: 00000000e18b8065: 0000000000000100 (0x100)
Jul 17 22:49:57 pepper-mint kernel: 000000009224cf2c: ffffcfa089fbfbe0 (0xffffcfa089fbfbe0)
Jul 17 22:49:57 pepper-mint kernel: 00000000f2412f9d: ffffffff96b0e7ee (do_exit+0x2be/0x530)
Jul 17 22:49:57 pepper-mint kernel: 00000000f7584d12: 00000000000000e7 (0xe7)
Jul 17 22:49:57 pepper-mint kernel: 000000001939bdd8: 0000000000000000 ...
Jul 17 22:49:57 pepper-mint kernel: 000000004faaaee9: e03157482ac83500 (0xe03157482ac83500)
Jul 17 22:49:57 pepper-mint kernel: 00000000237f5673: 0000000000000100 (0x100)
Jul 17 22:49:57 pepper-mint kernel: 0000000020c7873c: ffff8a1208c43a80 (0xffff8a1208c43a80)
Jul 17 22:49:57 pepper-mint kernel: 0000000078df3789: ffff8a121c062f40 (0xffff8a121c062f40)
Jul 17 22:49:57 pepper-mint kernel: 00000000b4b20f04: ffff8a1207baeb40 (0xffff8a1207baeb40)
Jul 17 22:49:57 pepper-mint kernel: 00000000bfc91cdb: ffffcfa089fbfc10 (0xffffcfa089fbfc10)
Jul 17 22:49:57 pepper-mint kernel: 0000000011d757dc: ffffffff96b0ec55 (do_group_exit+0x35/0x90)
Jul 17 22:49:57 pepper-mint kernel: 000000008d92c991: ffffffffffffffff (0xffffffffffffffff)
Jul 17 22:49:57 pepper-mint kernel: 0000000080abc970: ffffcfa089fbff58 (0xffffcfa089fbff58)
Jul 17 22:49:57 pepper-mint kernel: 0000000085d65c85: 00000000000000e7 (0xe7)
Jul 17 22:49:57 pepper-mint kernel: 00000000e60e5d31: 0000000000000000 ...
Jul 17 22:49:57 pepper-mint kernel: 00000000cdbc9374: ffffcfa089fbfc20 (0xffffcfa089fbfc20)
Jul 17 22:49:57 pepper-mint kernel: 00000000c9999b01: ffffffff96b0ed08 (__x64_sys_exit_group+0x18/0x20)
Jul 17 22:49:57 pepper-mint kernel: 00000000cbc459ec: ffffcfa089fbfc30 (0xffffcfa089fbfc30)
Jul 17 22:49:57 pepper-mint kernel: 000000002acdf9f9: ffffffff96a07ecb (x64_sys_call+0x259b/0x25a0)
Jul 17 22:49:57 pepper-mint kernel: 000000009c49a65e: ffffcfa089fbff48 (0xffffcfa089fbff48)
Jul 17 22:49:57 pepper-mint kernel: 0000000023238571: ffffffff97c4e9af (do_syscall_64+0x7f/0x180)
Jul 17 22:49:57 pepper-mint kernel: 0000000012092dc6: ffffffff97aab163 (unix_seqpacket_recvmsg+0x43/0x70)
Jul 17 22:49:57 pepper-mint kernel: 00000000ec5936c3: ffffcfa089fbfc78 (0xffffcfa089fbfc78)
Jul 17 22:49:57 pepper-mint kernel: 000000002fedf8ec: ffffffff978e6d6e (sock_recvmsg+0xde/0xf0)
Jul 17 22:49:57 pepper-mint kernel: 00000000af858d0b: 0000000000000000 ...
Jul 17 22:49:57 pepper-mint kernel: 00000000fb28a2df: ffffcfa089fbfe48 (0xffffcfa089fbfe48)
Jul 17 22:49:57 pepper-mint kernel: 00000000b4851412: 0000000000000000 ...
Jul 17 22:49:57 pepper-mint kernel: 000000008b434eae: ffffcfa089fbfd50 (0xffffcfa089fbfd50)
Jul 17 22:49:57 pepper-mint kernel: 0000000057544271: ffffffff978e6fb1 (____sys_recvmsg+0x111/0x230)
Jul 17 22:49:57 pepper-mint kernel: 00000000d111135c: 0000000000010000 (0x10000)
Jul 17 22:49:57 pepper-mint kernel: 000000002945d443: 00007ffc0d9c3ad8 (0x7ffc0d9c3ad8)
Jul 17 22:49:57 pepper-mint kernel: 00000000fb818729: 00007ffc0d9c3a50 (0x7ffc0d9c3a50)
Jul 17 22:49:57 pepper-mint kernel: 000000005d4e52fe: 0000000000000000 ...
Jul 17 22:49:57 pepper-mint kernel: 00000000967afa29: e03157482ac83500 (0xe03157482ac83500)
Jul 17 22:49:57 pepper-mint kernel: 000000001f44c685: e03157482ac83500 (0xe03157482ac83500)
Jul 17 22:49:57 pepper-mint kernel: 00000000fece43f0: 0000000000000000 ...
Jul 17 22:49:57 pepper-mint kernel: 000000004afcc22d: ffffcfa089fbfe48 (0xffffcfa089fbfe48)
Jul 17 22:49:57 pepper-mint kernel: 000000005f7cd780: 0000000000000000 ...
Jul 17 22:49:57 pepper-mint kernel: 00000000ee26b13b: 00007ffc0d9c3ad0 (0x7ffc0d9c3ad0)
Jul 17 22:49:57 pepper-mint kernel: 0000000069737a26: ffffcfa089fbfe30 (0xffffcfa089fbfe30)
Jul 17 22:49:57 pepper-mint kernel: 00000000cd64bc32: ffffffff978ea1ec (___sys_recvmsg+0x9c/0xf0)
Jul 17 22:49:57 pepper-mint kernel: 00000000f8f53b47: 0000000000000000 ...
Jul 17 22:49:57 pepper-mint kernel: 0000000087faca55: 00007ffc0d9c4d20 (0x7ffc0d9c4d20)
Jul 17 22:49:57 pepper-mint kernel: 00000000362dda80: 0000000000001000 (0x1000)
Jul 17 22:49:57 pepper-mint kernel: 00000000389d24d2: 0000000000000000 ...
Jul 17 22:49:57 pepper-mint kernel: 00000000e3405f1f: ffffcfa089fbfe30 (0xffffcfa089fbfe30)
Jul 17 22:49:57 pepper-mint kernel: 000000008c1e9b2d: e03157482ac83500 (0xe03157482ac83500)
Jul 17 22:49:57 pepper-mint kernel: 00000000f21ecd25: ffffcfa089fbfe40 (0xffffcfa089fbfe40)
Jul 17 22:49:57 pepper-mint kernel: 000000005798c877: ffffffff96dbfae2 (rseq_get_rseq_cs+0x22/0x280)
Jul 17 22:49:57 pepper-mint kernel: 00000000ff2a7352: ffff8a12007964c0 (0xffff8a12007964c0)
Jul 17 22:49:57 pepper-mint kernel: 0000000078d4fc38: ffff8a121c062f40 (0xffff8a121c062f40)
Jul 17 22:49:57 pepper-mint kernel: 000000000d4e42c1: ffffcfa089fbff58 (0xffffcfa089fbff58)
Jul 17 22:49:57 pepper-mint kernel: 0000000051a73c56: 0000718d7832be6b (0x718d7832be6b)
Jul 17 22:49:57 pepper-mint kernel: 00000000de7f192b: ffffcfa089fbfeb0 (0xffffcfa089fbfeb0)
Jul 17 22:49:57 pepper-mint kernel: 00000000e8035992: ffffffff96dbfde0 (rseq_ip_fixup+0x90/0x170)
Jul 17 22:49:57 pepper-mint kernel: 000000002a2d0e2b: 0000000000000000 ...
Jul 17 22:49:57 pepper-mint kernel: 00000000485a6e0f: ffffcfa089fbfe90 (0xffffcfa089fbfe90)
Jul 17 22:49:57 pepper-mint kernel: 000000005b578300: ffffffff96a6d02d (restore_fpregs_from_fpstate+0x3d/0xd0)
Jul 17 22:49:57 pepper-mint kernel: 00000000ca66f11e: ffff8a121c062f40 (0xffff8a121c062f40)
Jul 17 22:49:57 pepper-mint kernel: 00000000d95410b5: ffff8a121c065440 (0xffff8a121c065440)
Jul 17 22:49:57 pepper-mint kernel: 0000000095b3c76d: 0000000000000013 (0x13)
Jul 17 22:49:57 pepper-mint kernel: 000000008ef2828b: ffffcfa089fbfeb8 (0xffffcfa089fbfeb8)
Jul 17 22:49:57 pepper-mint kernel: 00000000db4e35e6: ffffffff96a6d125 (switch_fpu_return+0x55/0xf0)
Jul 17 22:49:57 pepper-mint kernel: 00000000bc4d4ef9: 0000000000004000 (0x4000)
Jul 17 22:49:57 pepper-mint kernel: 000000005f06c7e4: ffffcfa089fbff58 (0xffffcfa089fbff58)
Jul 17 22:49:57 pepper-mint kernel: 000000006a674fa2: ffff8a121c062f40 (0xffff8a121c062f40)
Jul 17 22:49:57 pepper-mint kernel: 00000000b0b4e794: ffffcfa089fbfed0 (0xffffcfa089fbfed0)
Jul 17 22:49:57 pepper-mint kernel: 00000000620aa6df: ffffffff96bf8455 (arch_exit_to_user_mode_prepare.isra.0+0x95/0xe0)
Jul 17 22:49:57 pepper-mint kernel: 0000000053a34845: 0000000000004200 (0x4200)
Jul 17 22:49:57 pepper-mint kernel: 000000007afc0d8d: ffffcfa089fbfef8 (0xffffcfa089fbfef8)
Jul 17 22:49:57 pepper-mint kernel: 0000000066d9c521: ffffffff97c55173 (syscall_exit_to_user_mode+0x43/0x1e0)
Jul 17 22:49:57 pepper-mint kernel: 000000006b95299c: ffffffffffffffff (0xffffffffffffffff)
Jul 17 22:49:57 pepper-mint kernel: 00000000248c6085: ffffcfa089fbff58 (0xffffcfa089fbff58)
Jul 17 22:49:57 pepper-mint kernel: 00000000762dbd88: 000000000000002f (0x2f)
Jul 17 22:49:57 pepper-mint kernel: 00000000bfe69357: ffffcfa089fbff48 (0xffffcfa089fbff48)
Jul 17 22:49:57 pepper-mint kernel: 000000009c1e9504: ffffffff97c4e9bc (do_syscall_64+0x8c/0x180)
Jul 17 22:49:57 pepper-mint kernel: 00000000255cc2e3: ffffffff97c4e9bc (do_syscall_64+0x8c/0x180)
Jul 17 22:49:57 pepper-mint kernel: 00000000cbcfcbc4: ffffffff97c55613 (irqentry_exit+0x43/0x50)
Jul 17 22:49:57 pepper-mint kernel: 00000000cdd2d817: ffffcfa089fbff48 (0xffffcfa089fbff48)
Jul 17 22:49:57 pepper-mint kernel: 00000000c6b9bcd1: ffffffff97c54d34 (exc_page_fault+0x94/0x1b0)
Jul 17 22:49:57 pepper-mint kernel: 00000000629c0fe7: 0000000000000000 ...
Jul 17 22:49:57 pepper-mint kernel: 00000000f8f9df22: ffffffff97e00130 (entry_SYSCALL_64_after_hwframe+0x78/0x80)
Jul 17 22:49:57 pepper-mint kernel: 00000000ea91bdca: 00007ffc0d9c5d78 (0x7ffc0d9c5d78)
Jul 17 22:49:57 pepper-mint kernel: 00000000aa92449a: 0000000000000952 (0x952)
Jul 17 22:49:57 pepper-mint kernel: 000000003beda934: 00007ffc0d9c6101 (0x7ffc0d9c6101)
Jul 17 22:49:57 pepper-mint kernel: 00000000cde835bd: 00007ffc0d9c5db0 (0x7ffc0d9c5db0)
Jul 17 22:49:57 pepper-mint kernel: 00000000b0187c2c: 00007ffc0d9c5df0 (0x7ffc0d9c5df0)
Jul 17 22:49:57 pepper-mint kernel: 00000000073c8892: 00000bd400b7e680 (0xbd400b7e680)
Jul 17 22:49:57 pepper-mint kernel: 000000001171b822: 0000000000000202 (0x202)
Jul 17 22:49:57 pepper-mint kernel: 0000000052f4d7df: 0000000000000000 ...
Jul 17 22:49:57 pepper-mint kernel: 00000000c8bf92ea: 0000002c00000000 (0x2c00000000)
Jul 17 22:49:57 pepper-mint kernel: 0000000050025fe6: 0000000000000000 ...
Jul 17 22:49:57 pepper-mint kernel: 000000000c18ee26: ffffffffffffffda (0xffffffffffffffda)
Jul 17 22:49:57 pepper-mint kernel: 0000000039c11ab4: 0000718d782ee21d (0x718d782ee21d)
Jul 17 22:49:57 pepper-mint kernel: 00000000b215b6c5: 00000000000000e7 (0xe7)
Jul 17 22:49:57 pepper-mint kernel: 00000000ae1585d9: fffffffffffff8f8 (0xfffffffffffff8f8)
Jul 17 22:49:57 pepper-mint kernel: 00000000d3b299c3: 0000000000000001 (0x1)
Jul 17 22:49:57 pepper-mint kernel: 00000000b4fa845f: 00000000000000e7 (0xe7)
Jul 17 22:49:57 pepper-mint kernel: 00000000351ae318: 0000718d782ee21d (0x718d782ee21d)
Jul 17 22:49:57 pepper-mint kernel: 000000003650053f: 0000000000000033 (0x33)
Jul 17 22:49:57 pepper-mint kernel: 000000005981945b: 0000000000000202 (0x202)
Jul 17 22:49:57 pepper-mint kernel: 000000004ffb0c7b: 00007ffc0d9c5d68 (0x7ffc0d9c5d68)
Jul 17 22:49:57 pepper-mint kernel: 00000000a73f5d32: 000000000000002b (0x2b)
Jul 17 22:49:57 pepper-mint kernel: ? nvKmsClose+0x88/0x170 [nvidia_modeset]
Jul 17 22:49:57 pepper-mint kernel: ? nvkms_close+0x40/0xc0 [nvidia_modeset]
Jul 17 22:49:57 pepper-mint kernel: ? __fput+0xa0/0x2e0
Jul 17 22:49:57 pepper-mint kernel: ? ____fput+0xe/0x20
Jul 17 22:49:57 pepper-mint kernel: ? task_work_run+0x5e/0xa0
Jul 17 22:49:57 pepper-mint kernel: ? do_exit+0x2be/0x530
Jul 17 22:49:57 pepper-mint kernel: ? do_group_exit+0x35/0x90
Jul 17 22:49:57 pepper-mint kernel: ? __x64_sys_exit_group+0x18/0x20
Jul 17 22:49:57 pepper-mint kernel: ? x64_sys_call+0x259b/0x25a0
Jul 17 22:49:57 pepper-mint kernel: ? do_syscall_64+0x7f/0x180
Jul 17 22:49:57 pepper-mint kernel: ? unix_seqpacket_recvmsg+0x43/0x70
Jul 17 22:49:57 pepper-mint kernel: ? sock_recvmsg+0xde/0xf0
Jul 17 22:49:57 pepper-mint kernel: ? ____sys_recvmsg+0x111/0x230
Jul 17 22:49:57 pepper-mint kernel: ? ___sys_recvmsg+0x9c/0xf0
Jul 17 22:49:57 pepper-mint kernel: ? rseq_get_rseq_cs+0x22/0x280
Jul 17 22:49:57 pepper-mint kernel: ? rseq_ip_fixup+0x90/0x170
Jul 17 22:49:57 pepper-mint kernel: ? restore_fpregs_from_fpstate+0x3d/0xd0
Jul 17 22:49:57 pepper-mint kernel: ? switch_fpu_return+0x55/0xf0
Jul 17 22:49:57 pepper-mint kernel: ? arch_exit_to_user_mode_prepare.isra.0+0x95/0xe0
Jul 17 22:49:57 pepper-mint kernel: ? syscall_exit_to_user_mode+0x43/0x1e0
Jul 17 22:49:57 pepper-mint kernel: ? do_syscall_64+0x8c/0x180
Jul 17 22:49:57 pepper-mint kernel: ? do_syscall_64+0x8c/0x180
Jul 17 22:49:57 pepper-mint kernel: ? irqentry_exit+0x43/0x50
Jul 17 22:49:57 pepper-mint kernel: ? exc_page_fault+0x94/0x1b0
Jul 17 22:49:57 pepper-mint kernel: ? entry_SYSCALL_64_after_hwframe+0x78/0x80
Jul 17 22:49:57 pepper-mint kernel: </TASK>
Jul 17 22:49:57 pepper-mint kernel: ---[ end trace 0000000000000000 ]---
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11379!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef8500; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11380!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef0101; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11381!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef8501; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11382!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef0102; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11383!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xcef00000; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11384!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xcef80000; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11385!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xcef80001; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11386!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef0202; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ mem.c:178
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11387!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef0300; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ vaspace_api.c:573
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11388!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef0013; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ mem.c:178
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11389!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1e00004; hObject=0xcaf0000c; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11390!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1e00004; hObject=0xcaf0000d; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11391!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1e00004; hObject=0xcaf0000e; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11392!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef0404; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11393!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef9960; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11394!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef0004; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11395!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef0073; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11396!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00022; hObject=0xbeef0003; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11397!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef0403; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11398!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef4903; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11399!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef0402; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11400!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef4902; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11401!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef0401; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11402!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef4901; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11403!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef0400; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11404!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef4900; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11405!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeefa0b5; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11406!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef9097; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11407!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef502d; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11408!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef3901; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:49:57 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11409!
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d0001b; hObject=0xbeef0100; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:49:57 pepper-mint kernel: NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Current device is not valid [NV_ERR_INVALID_DEVICE] (0x00000026) returned from rmDeviceGpuLocksAcquire(pGpu, GPUS_LOCK_FLAGS_NONE, RM_LOCK_MODULES_MEM_PMA) @ virtual_mem.c:115
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11410!
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00010; hObject=0x00015070; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:50:02 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11411!
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00010; hObject=0x00015071; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:50:02 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11412!
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00010; hObject=0x00015072; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:50:02 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11413!
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00010; hObject=0x00015073; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:50:02 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11414!
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00010; hObject=0x00019800; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11415!
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00010; hObject=0x0001b500; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:50:02 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:50:02 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11416!
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00010; hObject=0x00019700; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:50:02 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:50:02 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11417!
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00010; hObject=0x00012d00; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
Jul 17 22:50:02 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
Jul 17 22:50:02 pepper-mint kernel: NVRM: nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:1375
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 _issueRpcAndWait: rpcSendMessage failed with status 0x0000000f for fn 10 sequence 11418!
Jul 17 22:50:02 pepper-mint kernel: NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1d00010; hObject=0x01000013; paramsStatus=0x00000000; status=0x0000000f
Jul 17 22:50:02 pepper-mint kernel: nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000ca7e:0:0:0x0000000f
Jul 17 22:50:02 pepper-mint kernel: nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000ca7e:2:0:0x0000000f
Jul 17 22:50:02 pepper-mint kernel: nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000ca7e:4:0:0x0000000f
Jul 17 22:50:02 pepper-mint kernel: nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000ca7e:6:0:0x0000000f
Jul 17 22:50:02 pepper-mint kernel: nvidia-modeset: ERROR: GPU:0: DP-0: Failed to disable DisplayPort audio stream-3
Jul 17 22:50:48 pepper-mint kernel: usb 3-9: USB disconnect, device number 13
Jul 17 22:52:01 pepper-mint kernel: snd_hda_codec_hdmi hdaudioC2D0: out of range cmd 0:4:707:ffffffbf
Jul 17 22:54:41 pepper-mint systemd[1]: Starting systemd-tmpfiles-clean.service - Cleanup of Temporary Directories...
Jul 17 22:54:41 pepper-mint systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Jul 17 22:54:41 pepper-mint systemd[1]: Finished systemd-tmpfiles-clean.service - Cleanup of Temporary Directories.
Jul 17 22:55:24 pepper-mint kernel: perf: interrupt took too long (2588 > 2500), lowering kernel.perf_event_max_sample_rate to 77000
Jul 17 22:55:36 pepper-mint kernel: INFO: task OnlineAsyncTask:4998 blocked for more than 122 seconds.
Jul 17 22:55:36 pepper-mint kernel: Tainted: G W OE 6.8.0-134-generic #134-Ubuntu
Jul 17 22:55:36 pepper-mint kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message
Jul 17 22:55:36 pepper-mint kernel: task:OnlineAsyncTask state:D stack:0 pid:4998 tgid:4640 ppid:4520 flags:0x00004002
Jul 17 22:55:36 pepper-mint kernel: Call Trace:
Jul 17 22:55:36 pepper-mint kernel: <TASK>
Jul 17 22:55:36 pepper-mint kernel: __schedule+0x27c/0x6b0
Jul 17 22:55:36 pepper-mint kernel: schedule+0x33/0x110
Jul 17 22:55:36 pepper-mint kernel: schedule_timeout+0x157/0x170
Jul 17 22:55:36 pepper-mint kernel: ___down_common+0xfd/0x160
Jul 17 22:55:36 pepper-mint kernel: __down_common+0x22/0xd0
Jul 17 22:55:36 pepper-mint kernel: __down+0x1d/0x30
Jul 17 22:55:36 pepper-mint kernel: down+0x54/0x80
Jul 17 22:55:36 pepper-mint kernel: nvkms_close+0x38/0xc0 [nvidia_modeset]
Jul 17 22:55:36 pepper-mint kernel: __fput+0xa0/0x2e0
Jul 17 22:55:36 pepper-mint kernel: ____fput+0xe/0x20
Jul 17 22:55:36 pepper-mint kernel: task_work_run+0x5e/0xa0
Jul 17 22:55:36 pepper-mint kernel: do_exit+0x2be/0x530
Jul 17 22:55:36 pepper-mint kernel: do_group_exit+0x35/0x90
Jul 17 22:55:36 pepper-mint kernel: get_signal+0x96e/0x9b0
Jul 17 22:55:36 pepper-mint kernel: arch_do_signal_or_restart+0x42/0x210
Jul 17 22:55:36 pepper-mint kernel: syscall_exit_to_user_mode+0x15f/0x1e0
Jul 17 22:55:36 pepper-mint kernel: do_syscall_64+0x8c/0x180
Jul 17 22:55:36 pepper-mint kernel: ? syscall_exit_to_user_mode+0x43/0x1e0
Jul 17 22:55:36 pepper-mint kernel: ? do_syscall_64+0x8c/0x180
Jul 17 22:55:36 pepper-mint kernel: ? do_syscall_64+0x8c/0x180
Jul 17 22:55:36 pepper-mint kernel: ? do_syscall_64+0x8c/0x180
Jul 17 22:55:36 pepper-mint kernel: ? do_syscall_64+0x8c/0x180
Jul 17 22:55:36 pepper-mint kernel: ? do_syscall_64+0x8c/0x180
Jul 17 22:55:36 pepper-mint kernel: ? do_syscall_64+0x8c/0x180
Jul 17 22:55:36 pepper-mint kernel: ? do_syscall_64+0x8c/0x180
Jul 17 22:55:36 pepper-mint kernel: entry_SYSCALL_64_after_hwframe+0x78/0x80
Jul 17 22:55:36 pepper-mint kernel: RIP: 0033:0x79ea8652a072
Jul 17 22:55:36 pepper-mint kernel: RSP: 002b:000079e71a7f2870 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
Jul 17 22:55:36 pepper-mint kernel: RAX: fffffffffffffffc RBX: 000079e819f19bab RCX: 000079ea8652a072
Jul 17 22:55:36 pepper-mint kernel: RDX: 0000000000000020 RSI: 000079e71a7f2940 RDI: 000000000000002c
Jul 17 22:55:36 pepper-mint kernel: RBP: 000079e71a7f2890 R08: 0000000000000000 R09: 0000000000000000
Jul 17 22:55:36 pepper-mint kernel: R10: 0000000000000064 R11: 0000000000000293 R12: 0000000000000064
Jul 17 22:55:36 pepper-mint kernel: R13: 0000000000000000 R14: 000079e819f7fad0 R15: 000079e7980a4360
Jul 17 22:55:36 pepper-mint kernel: </TASK>Trying a downgrade to the nvidia-driver-580-open.
Last edited by MichaelMocha (2026-07-18 04:18:56)
Offline
#11 2026-07-18 07:53:52
- seth
- Member
- From: Won't reply 2 private help req
- Registered: 2012-09-03
- Posts: 76,888
Re: Virtual Machine crashing with VFIO. Had a working setup for 6 months.
Jul 17 20:45:48 pepper-mint kernel: NVRM: Xid (PCI:0000:07:00): 79, GPU has fallen off the bus.
If the GPU isn't legit underpowered/badly seated¹ try to add "pcie_aspm=off" to the https://wiki.archlinux.org/title/Kernel_parameters (of both systems, host and client)
Do you also lose the GPU when not forwarding it and using it on the host system, https://aur.archlinux.org/packages?O=0&K=unigine
¹ This would eg. likely happen if the GPU isn't placed in the PEG but some random underdimensioned PCIe slot of if you forgot to attach the dedicated power supply cable
Online
#12 2026-07-18 21:46:35
- Akusari
- Member
- Registered: 2019-02-26
- Posts: 32
Re: Virtual Machine crashing with VFIO. Had a working setup for 6 months.
From a distance, it's difficult to provide an accurate diagnosis. There could be many possible causes.
However, if the setup has really been stable for six months, and you have not:
a) updated, reset, or modified the BIOS, and
b) changed any hardware (such as moving devices between PCIe slots),
then I would recommend removing the affected virtual hardware from the libvirt configuration and adding it back again.
Testing with an older kernel would also be worthwhile. I'd also suggest reviewing the changelogs for libvirt and related components to see if anything relevant has changed.
In addition, it certainly won't hurt to run the following commands inside the Windows VM:
-> chkdsk C: /F
-> sfc /scannow
-> DISM /Online /Cleanup-Image /RestoreHealth
Last edited by Akusari (2026-07-18 21:47:23)
black listed users: seth WorMzy
Please stay away from this thread - Thanks. 
Offline
#13 2026-07-18 21:57:55
- jonno2002
- Member
- Registered: 2016-11-21
- Posts: 882
Re: Virtual Machine crashing with VFIO. Had a working setup for 6 months.
as seth asked before.. why are you using a linux vm to play games on a linux pc ? doesnt make any sense, why not just play the games on arch ? im very confused and interested at the same time
Offline
#14 2026-07-18 22:20:47
- MichaelMocha
- Member
- Registered: 2025-05-23
- Posts: 12
Re: Virtual Machine crashing with VFIO. Had a working setup for 6 months.
Do you also lose the GPU when not forwarding it and using it on the host system, https://aur.archlinux.org/packages?O=0&K=unigine
The GPU has never been used on the host system. The host does not have any NVIDIA drivers and Nouveau is blacklisted at boot.
jonno2002 wrote:
as seth asked before.. why are you using a linux vm to play games on a linux pc ? doesnt make any sense, why not just play the games on arch ? im very confused and interested at the same time
This is why I don't game on the host. It simplified the Libvirt setup quite a bit to never have the host OS touch the GPU. This is also where most of the problems I read online come from.
This would eg. likely happen if the GPU isn't placed in the PEG but some random underdimensioned PCIe slot of if you forgot to attach the dedicated power supply cable
Reconnecting everything is worth a try. Thanks.
Just added the setting to both the VM and Host. The VM grub file line is GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pcie_aspm=off". The host is also grub.
Last edited by MichaelMocha (2026-07-18 22:39:03)
Offline
#15 2026-07-18 22:50:33
- seth
- Member
- From: Won't reply 2 private help req
- Registered: 2012-09-03
- Posts: 76,888
Re: Virtual Machine crashing with VFIO. Had a working setup for 6 months.
The GPU has never been used on the host system.
The plan was then to just do that 
It simplified the Libvirt setup quite a bit to never have the host OS touch the GPU.
Fwwi see the script in https://wiki.archlinux.org/title/PCI_pa … _device_ID
Online
#16 2026-07-19 03:54:22
- MichaelMocha
- Member
- Registered: 2025-05-23
- Posts: 12
Re: Virtual Machine crashing with VFIO. Had a working setup for 6 months.
Unplugging and replugging the GPU cables might have done it. Will see how long it stays stable. Thanks for the help.
Last edited by MichaelMocha (2026-07-19 04:18:17)
Offline