Nvidia GPU suddenly undetected after update.

AungS8430 · 2024-05-07 23:40:02

I got Nvidia GPU working last week with envycontrol, not it's not detected anymore. The only thing I did before it broke was full system upgrade.

journalctl -b | grep nvidia

May 08 06:14:15 archlinux kernel: nvidia: loading out-of-tree module taints kernel.
May 08 06:14:15 archlinux kernel: nvidia: module license 'NVIDIA' taints kernel.
May 08 06:14:15 archlinux kernel: nvidia: module verification failed: signature and/or required key missing - tainting kernel
May 08 06:14:15 archlinux kernel: nvidia: module license taints kernel.
May 08 06:14:15 archlinux kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 235
May 08 06:14:15 archlinux kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 235
May 08 06:14:16 archlinux systemd-modules-load[284]: Failed to insert module 'nvidia_uvm': No such device
May 08 06:14:18 archlinux kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 234
May 08 06:14:18 archlinux kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 234
May 08 06:14:19 archlinux kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 234
May 08 06:14:19 archlinux kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 234
May 08 06:14:19 archlinux systemd-modules-load[882]: Failed to insert module 'nvidia_uvm': No such device
May 08 06:14:19 archlinux systemd[1]: Started nvidia-powerd service.
May 08 06:14:19 archlinux nvidia-persistenced[1312]: Started (1312)
May 08 06:14:19 archlinux /usr/bin/nvidia-powerd[1310]: nvidia-powerd version:1.0(build 1)
May 08 06:14:19 archlinux nvidia-persistenced[1312]: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 143 has read and write permissions for those files.
May 08 06:14:19 archlinux nvidia-persistenced[1312]: Shutdown (1312)
May 08 06:14:19 archlinux nvidia-persistenced[1309]: nvidia-persistenced failed to initialize. Check syslog for more details.
May 08 06:14:19 archlinux /usr/bin/nvidia-powerd[1310]: Allocate client failed 89
May 08 06:14:19 archlinux /usr/bin/nvidia-powerd[1310]: Failed to initialize RM Client
May 08 06:14:19 archlinux systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
May 08 06:14:19 archlinux systemd[1]: nvidia-powerd.service: Main process exited, code=exited, status=1/FAILURE
May 08 06:14:19 archlinux systemd[1]: nvidia-powerd.service: Failed with result 'exit-code'.
May 08 06:14:19 archlinux systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.

lspci -k | grep -A 2 -E "(VGA|3D)"

00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 620 (rev 07)
	Subsystem: Lenovo ThinkPad T480
	Kernel driver in use: i915

dGPU was 00:01.0 but now it's undetected, so I think that it might be disabled after an upgrade but I'm not able to find a solution.

seth · 2024-05-08 07:48:10

Please don't grep - post the complete system journal for the boot (a lot of relevant information will be under NVRM and any kind of bus error not covered by that either)

AungS8430 · 2024-05-08 08:08:36

Here you go!
https://0x0.st/X8mO.txt

seth · 2024-05-08 14:08:30

May 08 15:06:40 archlinux kernel: pci 0000:01:00.0: [10de:1d10] type 00 class 0x030200 PCIe Endpoint
May 08 15:06:40 archlinux kernel: pci 0000:01:00.0: BAR 0 [mem 0xec000000-0xecffffff]
May 08 15:06:40 archlinux kernel: pci 0000:01:00.0: BAR 1 [mem 0x90000000-0x9fffffff 64bit pref]
May 08 15:06:40 archlinux kernel: pci 0000:01:00.0: BAR 3 [mem 0xa0000000-0xa1ffffff 64bit pref]
May 08 15:06:40 archlinux kernel: pci 0000:01:00.0: BAR 5 [io  0xd000-0xd07f]
May 08 15:06:40 archlinux kernel: pci 0000:01:00.0: ROM [mem 0xfff80000-0xffffffff pref]
May 08 15:06:40 archlinux kernel: pci 0000:01:00.0: Enabling HDA controller
May 08 15:06:40 archlinux kernel: pci 0000:01:00.0: ROM [mem 0xfff80000-0xffffffff pref]: can't claim; no compatible bridge window
May 08 15:06:40 archlinux kernel: pci 0000:01:00.0: ROM [mem size 0x00080000 pref]: can't assign; no space
May 08 15:06:40 archlinux kernel: pci 0000:01:00.0: ROM [mem size 0x00080000 pref]: failed to assign

Did the update bring 6.8.9?
Does the GPU still show up w/ the LTS kernel?

AungS8430 · 2024-05-08 14:31:11

I'm not sure what you meant in the first question. For the second one, it doesn't show up with LTS kernel in lspci as well.

seth · 2024-05-08 14:50:45

The only thing I did before it broke was full system upgrade.

What packages got updated?
Does downgrading the kernel to the previous version bring back the GPU?
What if you add "pci=realloc" to the https://wiki.archlinux.org/title/Kernel_parameters ?

AungS8430 · 2024-05-08 15:15:43

It was a full system upgrade, so everything should probably be latest version, I'm not sure which one of them got updated.
I tried downgrading to 6.8.7, same results.
Also adding "pci=realloc" to kernel parameter doesn't fix it.

seth · 2024-05-08 15:35:27

Did you run a BIOS/UEFI update or change any of its settings or alter the hardware (add a PCI card or so)?

firekage · 2024-05-08 17:43:57

Sorry for bumping, but after upgrade i have similar issue:

==> ERROR: module not found: 'nvidia'
==> ERROR: module not found: 'nvidia_modeset'
==> ERROR: module not found: 'nvidia_uvm'
==> ERROR: module not found: 'nvidia_drm'

So it is something with the kernel.

V1del · 2024-05-08 17:46:13

That's on it's own not at all similar and this just suggests you either had a partial update or haven't rebuilt the nvidia drivers against the new kernel/gcc combination. If you're using an AUR package here make sure to rebuild them.

firekage · 2024-05-08 18:01:33

I always do this. This time i did this several times and always with the same error when nvidia-dkms is prepared.

BTW - full update.

==> ERROR: module not found: 'nvidia'
==> ERROR: module not found: 'nvidia_modeset'
==> ERROR: module not found: 'nvidia_uvm'
==> ERROR: module not found: 'nvidia_drm'
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux-fallback.img'
==> WARNING: errors were encountered during the build. The image may not be complete.
error: command failed to execute correctly
[firekage@7500 6.8.9-arch1-2]$ yay -Ss nvidia-470xx | grep Installed
aur/nvidia-470xx-utils 470.239.06-1 (+92 1.04) (Installed)
aur/nvidia-470xx-dkms 470.239.06-1 (+92 1.04) (Installed)
[firekage@7500 6.8.9-arch1-2]$

The same error is reported when nvidia is uninstalled and only kernel is installed. Looks like there is no nvidia modules in kernel?

==> ERROR: module not found: 'nvidia'
==> ERROR: module not found: 'nvidia_modeset'
==> ERROR: module not found: 'nvidia_uvm'
==> ERROR: module not found: 'nvidia_drm'
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux-fallback.img'
==> WARNING: errors were encountered during the build. The image may not be complete.
error: command failed to execute correctly
[firekage@7500 ~]$

I deleted nvidia. Updated again kernel. Rebooted. Installed again nvidia and the same. After reboot lsmod | grep nvidia shows nothing.

Edit: i did another thing. Removed nvidia, dkms. Rebooted. Installed nvidia not from the aur but from the pacman -S nvidia. The same. After rebot lsmod | grep nvidia shows nothing. It just does not work with the newest kernel.

Last edited by firekage (2024-05-08 19:06:07)

seth · 2024-05-08 20:04:14

dkms status

and check the dkms build logs, but as V1del mentioned, this is a completely different situation from the OP

Edit: https://bbs.archlinux.org/viewtopic.php?id=295600

Last edited by seth (2024-05-08 20:06:40)

AungS8430 · 2024-05-08 21:52:20

seth wrote:

Did you run a BIOS/UEFI update or change any of its settings or alter the hardware (add a PCI card or so)?

I didn't, I did check if dGPU is disabled in BIOS, but it is not possible to do that on my specific laptop model.

seth · 2024-05-08 22:12:26

Does

echo 1 | sudo tee /sys/bus/pci/devices/0000\:01\:00.0/remove
echo 1 | sudo tee /sys/bus/pci/rescan

bring up the device?
Also please post a journal for a pci=realloc" boot.

Possibly addressed by https://patchwork.kernel.org/project/li … edhat.com/

AungS8430 · 2024-05-08 22:23:56

I just realized that after rebooting /proc/cmdline return to default, I followed section 1.9 in Kernel parameters of arch wiki. Before rebooting, I did check if the change was made, but it reverted back after reboot. I'm not sure if it's applied or not, if it was then here is the log.
https://0x0.st/X8QR.txt

seth · 2024-05-08 22:27:03

pci=realloc is not in that journal.
This https://wiki.archlinux.org/title/Kernel … ng_cmdline is absolutely not what you want - follow the instructions for your bootloader.
If you're using UKI, see https://wiki.archlinux.org/title/Unifie … mmand_line

AungS8430 · 2024-05-08 22:31:43

https://0x0.st/X8QF.txt
Is this correct?

Edit: Also,

echo 1 | sudo tee /sys/bus/pci/devices/0000\:01\:00.0/remove
echo 1 | sudo tee /sys/bus/pci/rescan

returns

tee: '/sys/bus/pci/devices/0000:01:00.0/remove': No such file or directory
1
1

Last edited by AungS8430 (2024-05-08 23:27:02)

seth · 2024-05-09 06:16:53

May 09 05:29:44 archlinux kernel: Command line: initrd=\intel-ucode.img initrd=\initramfs-linux.img root=PARTUUID=6a5d5e3b-e959-4093-9cd5-cb6184e4613a zswap.enabled=0 rw rootfstype=ext4 pci=realloc

Yes, is.

But we stay at

May 09 05:29:44 archlinux kernel: pci 0000:01:00.0: ROM [mem size 0x00080000 pref]: can't assign; no space
May 09 05:29:44 archlinux kernel: pci 0000:01:00.0: ROM [mem size 0x00080000 pref]: failed to assign

Ok, you can't remove the device because it's not added itfp, but does the rescan do anything (in doubt give you more errors?

The only thing I did before it broke was full system upgrade.

Please post your pacman log.

AungS8430 · 2024-05-09 12:34:30

Rescan doesn't return any errors.
Here's the pacman log.
https://0x0.st/X8Lm.log

seth · 2024-05-09 14:38:08

Here're some excerpts reflectign your struggle to get the nvidia GPU working (I assume)

[2024-04-07T12:01:03+0700] [ALPM-SCRIPTLET] ==> ERROR: Missing dev kernel headers for module nvidia/550.67.
[2024-04-07T12:29:28+0700] [ALPM] removed nvidia-dkms (550.67-1)
[2024-04-07T12:29:28+0700] [ALPM] installed nvidia (550.67-5)
[2024-04-07T21:17:55+0700] [ALPM] installed nvidia-exec (0.1.2-1)
[2024-04-10T10:57:43+0700] [ALPM] removed nvidia-exec (0.1.2-1)
[2024-04-10T11:43:28+0700] [ALPM] installed nvidia-prime (1.0-4)
[2024-04-10T11:43:45+0700] [ALPM] installed nvidia-prime-rtd3pm (1.1-1)

At which point/date did the nvidia GPU no longer show up in lspci?

Also

[2024-04-01T07:03:54+0000] [ALPM] installed linux (6.8.2.arch2-1)
[2024-04-01T07:03:54+0000] [ALPM] installed linux-lts (6.6.23-1)
[2024-04-07T12:24:29+0700] [ALPM] upgraded linux (6.8.2.arch2-1 -> 6.8.4.arch1-1)
[2024-04-07T12:24:30+0700] [ALPM] upgraded linux-lts (6.6.23-1 -> 6.6.25-1)

are the only kernel installs/updates - how did you end up booting 6.8.7 ?

The rescan isn't supposed to generate errors but some output in the dmesg (about all the fancy PCI devices found) and - ideally - add the nvidia GPU?

Do you still have a journal from when the nvidia device showed up?
("sudo journalctl -b δ" allows you to go back in time the smaller the -negative- δ the older the boot, eg. try "-b -10")

cloverskull · 2024-05-09 20:00:58

Interesting - I have a work computer that uses Ubuntu 24.04 and this just started happening there. I can still use the GPU in hybrid mode when fully booted into the OS, but nvidia-persistenced is failing. Very strange. But maybe this isn't limited to Arch.

seth · 2024-05-09 20:05:15

Do you have a journal from the ubuntu system?
Just persistenced failing isn't close to the situation the OP is encountering - the device cannot be addressed for them (ie. it doesn't even show up in lspci *at all*)

AungS8430 · 2024-05-09 21:32:34

I don't have a system backup of 6.8.7 that has dGPU working. However, I have a backup of 6.8.2, which was when I did fresh arch installation. It was working on nonveau driver, will the log from 6.8.2 works?

Note: I tried to make PRIME, nvidia-exec, bumblebeed, some others in the wiki works but it doesn't or I don't like it so I ended up using envycontrol before the upgrade. I think that is potentially the issue, I messed up something.

Last edited by AungS8430 (2024-05-09 21:40:15)

seth · 2024-05-10 06:05:42

You don't need backup of the system - "sudo journalctl -b δ # δ < 0" allows you to go back in time, eg. "sudo journalctl -b -10" will show the journal for 10 boots ago.
Since this the problem happens before any driver gets loaded, I'd be surprised if using nouveau would make any difference - but likewise if the older kernel works.

What if you just downgrade the kernel to 6.8.7 (you'll need nvidia-dkms or downgrade the nvidia and nvidia-utils packages along the kernel)
https://wiki.archlinux.org/title/Arch_Linux_Archive

AungS8430 · 2024-05-10 06:48:56

I just went through boot journal saved, none of them have Nvidia GPU working. I did try to downgrade to 6.8.7 but didn't downgrade nvidia package or install nvidia-dkms, I think at least it should show up in lspci which it doesn't, should I try again but also install nvidia-dkms, will that make any difference?

Arch Linux

#1 2024-05-07 23:40:02

Nvidia GPU suddenly undetected after update.

#2 2024-05-08 07:48:10

Re: Nvidia GPU suddenly undetected after update.

#3 2024-05-08 08:08:36

Re: Nvidia GPU suddenly undetected after update.

#4 2024-05-08 14:08:30

Re: Nvidia GPU suddenly undetected after update.

#5 2024-05-08 14:31:11

Re: Nvidia GPU suddenly undetected after update.

#6 2024-05-08 14:50:45

Re: Nvidia GPU suddenly undetected after update.

#7 2024-05-08 15:15:43

Re: Nvidia GPU suddenly undetected after update.

#8 2024-05-08 15:35:27

Re: Nvidia GPU suddenly undetected after update.

#9 2024-05-08 17:43:57

Re: Nvidia GPU suddenly undetected after update.

#10 2024-05-08 17:46:13

Re: Nvidia GPU suddenly undetected after update.

#11 2024-05-08 18:01:33

Re: Nvidia GPU suddenly undetected after update.

#12 2024-05-08 20:04:14

Re: Nvidia GPU suddenly undetected after update.

#13 2024-05-08 21:52:20

Re: Nvidia GPU suddenly undetected after update.

#14 2024-05-08 22:12:26

Re: Nvidia GPU suddenly undetected after update.

#15 2024-05-08 22:23:56

Re: Nvidia GPU suddenly undetected after update.

#16 2024-05-08 22:27:03

Re: Nvidia GPU suddenly undetected after update.

#17 2024-05-08 22:31:43

Re: Nvidia GPU suddenly undetected after update.

#18 2024-05-09 06:16:53

Re: Nvidia GPU suddenly undetected after update.

#19 2024-05-09 12:34:30

Re: Nvidia GPU suddenly undetected after update.

#20 2024-05-09 14:38:08

Re: Nvidia GPU suddenly undetected after update.

#21 2024-05-09 20:00:58

Re: Nvidia GPU suddenly undetected after update.

#22 2024-05-09 20:05:15

Re: Nvidia GPU suddenly undetected after update.

#23 2024-05-09 21:32:34

Re: Nvidia GPU suddenly undetected after update.

#24 2024-05-10 06:05:42

Re: Nvidia GPU suddenly undetected after update.

#25 2024-05-10 06:48:56

Re: Nvidia GPU suddenly undetected after update.

Board footer