linux-lts and linux 7.1.2 with nvidia-open - GSP Timeout

Whoracle · Yesterday 07:48:09

So, a few days ago I upgraded to linux 7.1.2. Immediately got sluggish performance once I launched my browser. Didn't have time to debug (work machine), so I downgraded to 7.0.14. Due to unrelated reasons I had to switch to linux-lts on another machine, and I wanted to synchronize the two, so I switched my desktop to linux-lts, too, yesterday evening. First boot into the system this morning, and the sluggishness issue was back. Basically, as soon as something uses the GPU on linux-lts 6.18.38 and linux 7.1.2, the GPU chokes.

"Choking" here is - overall sluggishness, mouse jumps around (because the GPU doesn't draw all frames it needs to, I'd assume) and about a minute or two later the whole session locks up - no more keyboard, no more spawning new windows, only mouse cursor still moves. Had to hard power down the system in that last boot.

Spec: RTX 5070 TI, on X11 AwesomeWM

[ ~ ] pacman -Qs nvidia
local/egl-gbm 1.1.3-1
    The GBM EGL external platform library
local/egl-wayland 4:1.1.21-1
    EGLStream-based Wayland external platform
local/egl-wayland2 1.0.1-1
    EGLStream-based Wayland external platform (2)
local/egl-x11 1.0.5-1
    NVIDIA XLib and XCB EGL Platform Library
local/lib32-nvidia-utils 610.43.02-1
    NVIDIA drivers utilities (32-bit)
local/lib32-opencl-nvidia 610.43.02-1
    OpenCL implemention for NVIDIA (32-bit)
local/libnvidia-container 1.19.1-1
    NVIDIA container runtime library
local/libva-nvidia-driver 0.0.17-1
    VA-API implementation that uses NVDEC as a backend
local/libvdpau 1.5-4
    Nvidia VDPAU library
local/libxnvctrl 610.43.02-1
    NVIDIA NV-CONTROL X extension
local/linux-firmware-nvidia 20260622-1
    Firmware files for Linux - Firmware for NVIDIA GPUs and SoCs
local/nvfancontrol 0.5.1-2
    NVidia dynamic fan control
local/nvidia-container-toolkit 1.19.1-1
    NVIDIA container toolkit
local/nvidia-open-dkms 610.43.02-3
    NVIDIA open kernel modules - module sources
local/nvidia-settings 610.43.02-1
    Tool for configuring the NVIDIA graphics driver
local/nvidia-utils 610.43.02-3
    NVIDIA drivers utilities
local/opencl-nvidia 610.43.02-3
    OpenCL implemention for NVIDIA

Currently on nvidia-open-dkms because I needed to downgrade, but happens with plain nvidia-open, too.

Full journal of choking boot: https://paste.c-net.org/StupiderSpraying

Pertinent bit from the end:

Jul 09 08:02:15 lynxcore kernel: NVRM: GPU at PCI:0000:01:00: GPU-4e391288-897c-ba5d-6944-dfbea55ebcfa
Jul 09 08:02:15 lynxcore kernel: NVRM: GPU Board Serial Number: 0
Jul 09 08:02:15 lynxcore kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception:  MISSING_INLINE_DATA
Jul 09 08:02:15 lynxcore kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x404600=0x80000002
Jul 09 08:02:15 lynxcore kernel: NVRM: Xid (PCI:0000:01:00): 13, pid=3719, name=vivaldi-bin, Graphics Exception: channel 0x00000008, Class 0000ce97, Offset 00000000, Data 00000000
Jul 09 08:02:34 lynxcore kernel: NVRM: Xid (PCI:0000:01:00): 31, pid=5270, name=vivaldi-bin, channel 0x00000008, intr 00000000. MMU Fault: ENGINE GR_HOST0 HUBCLIENT_ESC0 faulted @ 0x100_00000000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
Jul 09 08:04:29 lynxcore kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: Class 0x1 Subchannel 0x0 Mismatch
Jul 09 08:04:29 lynxcore kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x4041b0=0x200001
Jul 09 08:04:29 lynxcore kernel: NVRM: Xid (PCI:0000:01:00): 13, Graphics Exception: ESR 0x404000=0x80000002
Jul 09 08:04:29 lynxcore kernel: NVRM: Xid (PCI:0000:01:00): 13, pid=7563, name=vivaldi-bin, Graphics Exception: channel 0x00000008, Class 0000ce97, Offset 00000000, Data 00000000
Jul 09 08:04:29 lynxcore kernel: NVRM: Xid (PCI:0000:01:00): 32, pid=7563, name=vivaldi-bin, channel 0x00000008 intr 00200000, method0 80000008, data0 301d0e02, method1 00000000, data1 00000000, method2 00000000, data2 00000000, method3 00000000, data3 00000000
Jul 09 08:05:12 lynxcore kernel: NVRM: _kgspIsHeartbeatTimedOut: Heartbeat timed out, currentTimeMs 1172886152 heartbeat 1716193 heartbeatWithOffsetMs 1172879541 diff 6611 timeout 5200
Jul 09 08:05:12 lynxcore kernel: NVRM: _kgspIsHeartbeatTimedOut: Heartbeat timed out, currentTimeMs 1172886152 heartbeat 1716193 heartbeatWithOffsetMs 1172879541 diff 6611 timeout 5200
Jul 09 08:05:16 lynxcore kernel: NVRM: kgmmuInvalidateTlb_GM107: TLB invalidation failed waiting for completion (status=0x00000065) for PDB 0x00000000, vaspaceFlags 0x0, scope 0x0, GFID 0
Jul 09 08:05:16 lynxcore kernel: NVRM: kgmmuCheckMemSubsysError_IMPL: Memory Subsystem Error detected. kgmmuInvalidateTlb failed.
Jul 09 08:05:16 lynxcore kernel: NVRM: _kgspCheckIfMemsysError: Unexpected error when checking for memory subsystem error: 0x72 
Jul 09 08:05:16 lynxcore kernel: NVRM: _kgspLogRpcTimeout: ********************************* GSP Timeout **********************************
Jul 09 08:05:16 lynxcore kernel: NVRM: _kgspLogRpcTimeout: Note: Please also check logs above.
Jul 09 08:05:16 lynxcore kernel: NVRM: nvAssertFailedNoLog: Assertion failed: !pKernelGsp->bPollingForRpcResponse @ kernel_gsp.c:2893
Jul 09 08:05:16 lynxcore kernel: NVRM: _issueRpcAndWait: rpcRecvPoll failed with status 0x00000040 for fn 76 sequence 15929!
Jul 09 08:05:16 lynxcore kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Generic Error: Invalid state [NV_ERR_INVALID_STATE] (0x00000040) returned from pRmApi->Control(pRmApi, pGpu->hInternalClient, pGpu->hInternalSubdevice, NV2080_CTRL_CMD_INTERNAL_LOG_OOB_XID, &params, sizeof(params)) @ gpu.c:7695
Jul 09 08:05:16 lynxcore kernel: NVRM: Xid (PCI:0000:01:00): 175, Timeout after 10s of waiting for RPC response from GPU0 GSP! Expected function 76 (GSP_RM_CONTROL) sequence 15928 (0x2080853b 0x37f0).
Jul 09 08:05:16 lynxcore kernel: NVRM: _kgspDumpRmState: Failed to get RM state, timeout
Jul 09 08:05:16 lynxcore kernel: NVRM: _kgspLogGspTraceCrashBuffer: Gsp Trace Crash Buffer

[ ~ ] cat /proc/cmdline
initrd=\amd-ucode.img initrd=\initramfs-linux.img root=/dev/lynxcore/root rw nvidia_drm.fbdev=1 acpi_enforce_resources=lax

Anyone got an idea?

Last edited by Whoracle (Yesterday 07:49:10)

seth · Yesterday 15:40:04

1. what happened to the raphael chip (it's there but not operated - are you blacklisting amdgpu?)
2. what is "coolercontrold"? https://aur.archlinux.org/packages/coolercontrold ?
What if you disable that?

Whoracle · Yesterday 15:43:47

1. Yes, I blacklisted amdgpu - all my Monitors hang on the RTX.
2. Exactly that. Will disable it. Doesn't make a whole lot of sense to me, but eh, worth a shot. My AIO is configured anyways, I can even completely uninstall that. Will get back to you once I got around to it.

seth · Yesterday 15:53:58

Can you disable the APU in the firmware (UEFI)?

Whoracle · Today 07:08:18

Disabled the iGPU and coolercontrold, uninstalled coolercontrold. Issue is gone on both linux-lts and current 7.1.3. Will reenable the iGPU to see which one it was (and because I want that on ideally in case the main GPU breaks), then I'll mark this a s solved.

cryptearth · Today 08:43:22

if you blacklist amdgpu having it enabled in the firmware is pointless

Whoracle · Today 10:21:01

Not for the UEFI. If the main GPU goes boom then I don't care for the OS at that moment. OS is just one live iso boot away from being fixed.

But resetting the UEFI in my tiny SFF case is a pain.

seth · Today 12:04:15

@cryptearth, it's still on the bus
@Whoracle, disabling the APU might be beneficial no matter what because w/o driver it's power state is going to be undefined (ie. it could draw battery no matter what)

Whoracle · Today 12:20:49

Since this is a desktop, I don't care about battery ?

I have since reenabled the iGPU and have had no adverse effects, but I didn't run the thing long. Will see on Monday how it behaves after a full day.

Arch Linux

#1 Yesterday 07:48:09

linux-lts and linux 7.1.2 with nvidia-open - GSP Timeout

#2 Yesterday 15:40:04

Re: linux-lts and linux 7.1.2 with nvidia-open - GSP Timeout

#3 Yesterday 15:43:47

Re: linux-lts and linux 7.1.2 with nvidia-open - GSP Timeout

#4 Yesterday 15:53:58

Re: linux-lts and linux 7.1.2 with nvidia-open - GSP Timeout

#5 Today 07:08:18

Re: linux-lts and linux 7.1.2 with nvidia-open - GSP Timeout

#6 Today 08:43:22

Re: linux-lts and linux 7.1.2 with nvidia-open - GSP Timeout

#7 Today 10:21:01

Re: linux-lts and linux 7.1.2 with nvidia-open - GSP Timeout

#8 Today 12:04:15

Re: linux-lts and linux 7.1.2 with nvidia-open - GSP Timeout

#9 Today 12:20:49

Re: linux-lts and linux 7.1.2 with nvidia-open - GSP Timeout

Board footer