I think the new nvidia driver is broken

HoseinLavaei · 2023-11-04 17:49:00

today i updated my system (kernels and nvidia driver and other apps) and when i wanted to play some games (dota 2 on steam) it send me an error about vulkan problem.
i checked every things about vulkan and it seemed fine but when i checked my NVIDIA card i found out it doesn't work.
i am using wayland protocol with gnome desktop environment
here are things I discovered :
when i open nvidia settings app it shows application profiles and configurations only and it I think it thinks i dont have a nvidia card.
output of nvidia-smi :

Failed to initialize NVML: Driver/library version mismatch
NVML library version: 545.29

output of vulkaninfo :

ERROR: [Loader Message] Code 0 : loader_scanned_icd_add: Could not get 'vkCreateInstance' via 'vk_icdGetInstanceProcAddr' for ICD libGLX_nvidia.so.0
WARNING: [Loader Message] Code 0 : terminator_CreateInstance: Failed to CreateInstance in ICD 0. Skipping ICD.
ERROR: [Loader Message] Code 0 : terminator_CreateInstance: Found no drivers!
Cannot create Vulkan instance.
This problem is often caused by a faulty installation of the Vulkan driver or attempting to use a GPU that does not support Vulkan.
ERROR at /usr/src/debug/vulkan-tools/Vulkan-Tools-1.3.269/vulkaninfo/./vulkaninfo.h:688:vkCreateInstance failed with ERROR_INCOMPATIBLE_DRIVER

and nvtop only founded gpu0 (integrated one)

neofetch :

OS: Arch Linux x86_64
Kernel: 6.5.9-zen2-1-zen
Uptime: 37 mins
Packages: 1338 (pacman)
Shell: fish 3.6.1
Resolution: 1920x1080
DE: GNOME 45.1
WM: Mutter
WM Theme: default-pure
Theme: Adwaita [GTK2/3]
cons: Adwaita [GTK2/3]
Terminal: kgx
CPU: Intel i5-8400 (6) @ 4.000GHz
GPU: NVIDIA GeForce GTX 1650
GPU: Intel CoffeeLake-S GT2 [UHD Grap
Memory: 4723MiB / 7811MiB

output of pacman -Qi nvidia-dkms :

Name : nvidia-dkms
Version : 545.29.02-2
Description : NVIDIA drivers - module sources
Architecture : x86_64
URL : http://www.nvidia.com/
Licenses : custom
Groups : None
Provides : NVIDIA-MODULE nvidia
Depends On : dkms nvidia-utils=545.29.02 libglvnd
Optional Deps : None
Required By : None
Optional For : None
Conflicts With : NVIDIA-MODULE nvidia
Replaces : None
Installed Size : 69.83 MiB
Packager : Sven-Hendrik Haase <svenstaro@archlinux.org>
Build Date : Thu 02 Nov 2023 03:59:08 AM +0330
Install Date : Sat 04 Nov 2023 08:22:04 PM +0330
Install Reason : Explicitly installed
Install Script : No
Validated By : Signature

i think its good to tell that i have multiple kernels and two monitors too.my default kernel is zen

any suggestion to fix problem ?

Last edited by HoseinLavaei (2023-11-04 17:49:30)

Roken · 2023-11-04 17:53:07

Downgrade nvidia packages. There are several threads reporting problems with 545

fabrixx · 2023-11-04 20:09:47

Same problem with 545.29.02 & wayland

Black screen after update&reboot

API mismatch error
https://ibb.co/XW7LWkR

"Fixed" with:

$ sudo downgrade nvidia-dkms nvidia nvidia-utils

Downgrade all to
535.113.01

Last edited by fabrixx (2023-11-04 20:13:46)

sl1pkn07 · 2023-11-04 20:17:14

install new version of nvidia drivers required reboot. is mandatory

seth · 2023-11-04 20:52:58

@fabrix, you were obviously running the 535 kernel module against the 545 userspace - update again, reboot and if the issue remains post the outputs of

uname -a
pacman -Qs nvidia
dkms status

@HoseinLavaei, if you did reboot after the update, post the same outputs.

fabrixx · 2023-11-04 21:18:59

rebooted and same problem:
https://imageupload.io/FdnO5jIyE6Pjnld

seth · 2023-11-04 21:27:47

Please post text, not pictures thereof - see the 1st link below.

It looks like no precompiled nvidia kernel module (the "nvidia" package) is installed and dkms also hasn't built anything.
EIther you don't have the linux-headers installed or there was another dkms issue.

Reuild the modules, https://wiki.archlinux.org/title/Dynami … ld_modules and post the log
Alternatively install the precompiled nvidia kernel module.

Polarnova · 2023-11-05 11:25:52

Reloading the corresponding kernel modules temporarily works for me.

 lsmod | grep nvidia

Update:
I regenerate the kernel image to avoid having to reload kernel modules of nvidia every time I reboot.

After upgrading to driver version 545.29.02, I’ve noticed that Xwayland (Version 23.2.2 (12302002)) often crashes when I use Chrome (Version 119.0.6045.105).

crash log: https://pastebin.com/2VX00UKn

some system info:

OS: Arch Linux x86_64 
Kernel: 6.5.9-arch2-1 
DE: Plasma 5.27.9 
WM: kwin 
CPU: AMD Ryzen 9 5900HX with Radeon Graphics (16) @ 4.890GHz 
GPU: NVIDIA GeForce RTX 3060 Mobile / Max-Q 
GPU: AMD ATI Radeon Vega Series / Radeon Vega Mobile Series 
Memory: 6115MiB / 63721MiB

Last edited by Polarnova (2023-11-05 14:07:52)

HoseinLavaei · 2023-11-05 15:33:31

thank you all for answering my question quickly
I'd reboot several times and it didn't work
i am downgrading now i post again the result
i thank you for being active again

HoseinLavaei · 2023-11-05 15:47:06

downgrading fixed the issue. thanks
hope someday i could help someone like you too

seth · 2023-11-05 15:55:28

Downgrading doesn't "fix" anything - you're just shifting the brokeness some days to the right.
Another dkms-related thread just was down to the module not making it into the initramfs.
Post the dkms status and the system journal.

@Polarnova, there's absolutely no trace of any nvidia device in the journal excerpt you posted - kwin_wayland/XWayland instead crashes in /usr/lib/dri/radeonsi_dri.so

pezz · 2023-11-06 01:48:48

There's at least a noticeable performance regression in 545.29.02-2 that I've experienced.

Menu jerkiness and playback choppiness in Kodi and scrolling issues in my browser.

I've gone back to 535.113.01-8 in the meantime.

Cr0c0diLe · 2023-11-06 05:38:33

I got the same problem, and now i fixed. My solution looks like very close to the solution provided by Polarnova

My conjecture since i'm not familiar with the linux:
>The problem is that the `initramfs` still keeps(while the nvidia driver updated) the old version infomation or something of nvidia driver. When you boot the computer, initramfs works to initialize the os module. Since the initramfs records the old version of nvidia driver, dismatch occured and that lead the `nvidia-smi` failed to match library and driver. If the nvidia-smi output is `Failed to initialize NVML: Driver/library version mismatch`, find all module by ' lsmod | grep nvidia' and remove them temporary by 'rmmod <module name>', then try 'nvidia-smi' again, it should be normal.

My solution:
1. update the nvidia driver
`pacman -S nvidia-dkms nvidia-utils`
2.be sure that you installed the linux-headers
`pacman -S <kernal name>-headers`
since i used a special kernal, for me <kernal name> is linux-g14
3.reinstall the linux kernal, this will generate the initramfs automatically.
`pacman -S <kernal name>`

waitnsea · 2023-11-06 06:31:45

I had problem with nvidia 545.29.02-2 , nvtop returning "No GPU to monitor.", but only when booting on new linux 6.5.9.arch2-1
All is fine, without any problem booting on linux-lts 6.1.61-1

seth · 2023-11-06 08:22:37

PSA: https://wiki.archlinux.org/title/NVIDIA#pacman_hook

@pezz, did you check eg. "glxinfo -B" to make sure you were actually using the nvidia driver and not vesa/llvm because of a module/userspace mismatch?

pezz · 2023-11-06 08:51:14

seth wrote:

@pezz, did you check eg. "glxinfo -B" to make sure you were actually using the nvidia driver and not vesa/llvm because of a module/userspace mismatch?

Yes, and I've tested it a couple of times now to make sure I'm not just seeing things, there's a visible performance hit with 545 on my 1660Ti.

seth · 2023-11-06 09:08:49

If this doesn't go away you could check nvidia-smi for the performance state w/ the old and new drivers, maybe the GPU was clocking higher before 545 with kodi active?

waitnsea · 2023-11-06 09:55:14

seth wrote:

PSA: https://wiki.archlinux.org/title/NVIDIA#pacman_hook

Thank you seth - it works well with linux-linux now

tabre · 2023-11-09 02:47:50

I'm having issues getting my Nvidia kernel module version to match my package version when upgrading to nvidia-dkms and nvidia-utils 545.29.02-2. I uninstalled the upgrades and installed 535 from cache to be able to boot my window manager... temporarily added these to IgnorePkg in my pacman config just to get by for a bit. That was about 6 days ago, so I tried upgrading again just a bit ago after seeing the new kernel update... but got the same result. Not sure what I'm doing wrong. The following output is after installing the new Nvidia packages and rebooting. Any tips would be appreciated.

uname -a

Linux liquilite 6.6.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 08 Nov 2023 16:05:38 +0000 x86_64 GNU/Linux

cat /proc/driver/nvidia/version

NVRM version: NVIDIA UNIX x86_64 Kernel Module 535.113.01 Tue Sep 12 19:41:24 UTC 2023
GCC version: gcc version 13.2.1 20230801 (GCC)

sudo pacman -Q | grep nvidia

hyprland-nvidia-git 0.32.0.r0.g751d2851-1
nvidia-dkms 545.29.02-2
nvidia-utils 545.29.02-2

lsmod | grep nvidia

nvidia_drm 94208 0
nvidia_uvm 3477504 0
nvidia_modeset 1556480 1 nvidia_drm
video 77824 1 nvidia_modeset
nvidia 62734336 2 nvidia_uvm,nvidia_modeset

journalctl

Nov 08 20:03:25 liquilite kernel: usbcore: registered new interface driver usbhid
Nov 08 20:03:25 liquilite kernel: usbhid: USB HID core driver
Nov 08 20:03:25 liquilite kernel: nvidia: loading out-of-tree module taints kernel.
Nov 08 20:03:25 liquilite kernel: nvidia: module license 'NVIDIA' taints kernel.
Nov 08 20:03:25 liquilite kernel: Disabling lock debugging due to kernel taint
Nov 08 20:03:25 liquilite kernel: nvidia: module verification failed: signature and/or required key missing - tainting kernel
Nov 08 20:03:25 liquilite kernel: nvidia: module license taints kernel.
Nov 08 20:03:25 liquilite kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 238
Nov 08 20:03:25 liquilite kernel:
Nov 08 20:03:25 liquilite kernel: nvidia 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
Nov 08 20:03:25 liquilite kernel: NVRM: loading NVIDIA UNIX x86_64 Kernel Module 535.113.01 Tue Sep 12 19:41:24 UTC 2023
Nov 08 20:03:25 liquilite kernel: nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 535.113.01 Tue Sep 12 19:45:42 UTC 2023
Nov 08 20:03:25 liquilite kernel: nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint.
Nov 08 20:03:25 liquilite kernel: nvidia-uvm: Loaded the UVM driver, major device number 236.
...
Nov 08 20:03:25 liquilite (udev-worker)[347]: nvidia: Process '/usr/bin/bash -c '/usr/bin/mknod -Z -m 666 /dev/nvidiactl c $(grep nvidia-frontend /proc/devices | cut -d \ -f 1) 255'' failed with exit code 1.
Nov 08 20:03:25 liquilite (udev-worker)[347]: nvidia: Process '/usr/bin/bash -c 'for i in $(cat /proc/driver/nvidia/gpus/*/information | grep Minor | cut -d \ -f 4); do /usr/bin/mknod -Z -m 666 /dev/nvidia${i} c $(grep nvidia-frontend /proc/devices | cut -d \ -f 1) ${i}; done'' failed with exit code 1.

Last edited by tabre (2023-11-09 02:49:17)

Scimmia · 2023-11-09 03:15:22

So, yet again, rebuild the initramfs if you put the module in there.

tabre · 2023-11-09 03:40:10

Scimmia wrote:

So, yet again, rebuild the initramfs if you put the module in there.

smh... yes, I'm not sure how I missed it.

I put in the rebuild hook as suggested by seth. All is good. Thank you.

❯ cat /proc/driver/nvidia/version

NVRM version: NVIDIA UNIX x86_64 Kernel Module 545.29.02 Thu Oct 26 21:21:38 UTC 2023
GCC version: gcc version 13.2.1 20230801 (GCC)

❯ pacman -Q | grep nvidia

hyprland-nvidia-git 0.32.0.r0.g751d2851-1
nvidia-dkms 545.29.02-2
nvidia-utils 545.29.02-2

HoseinLavaei · 2023-11-09 17:40:49

Seth was right.
downgrading doesn't fix the problem
i finally fixed it !
just run these 2 commands :

nvidia-modprobe

systemctl enable --now nvidia-resume.service

Arch Linux

#1 2023-11-04 17:49:00

I think the new nvidia driver is broken

#2 2023-11-04 17:53:07

Re: I think the new nvidia driver is broken

#3 2023-11-04 20:09:47

Re: I think the new nvidia driver is broken

#4 2023-11-04 20:17:14

Re: I think the new nvidia driver is broken

#5 2023-11-04 20:52:58

Re: I think the new nvidia driver is broken

#6 2023-11-04 21:18:59

Re: I think the new nvidia driver is broken

#7 2023-11-04 21:27:47

Re: I think the new nvidia driver is broken

#8 2023-11-05 11:25:52

Re: I think the new nvidia driver is broken

#9 2023-11-05 15:33:31

Re: I think the new nvidia driver is broken

#10 2023-11-05 15:47:06

Re: I think the new nvidia driver is broken

#11 2023-11-05 15:55:28

Re: I think the new nvidia driver is broken

#12 2023-11-06 01:48:48

Re: I think the new nvidia driver is broken

#13 2023-11-06 05:38:33

Re: I think the new nvidia driver is broken

#14 2023-11-06 06:31:45

Re: I think the new nvidia driver is broken

#15 2023-11-06 08:22:37

Re: I think the new nvidia driver is broken

#16 2023-11-06 08:51:14

Re: I think the new nvidia driver is broken

#17 2023-11-06 09:08:49

Re: I think the new nvidia driver is broken

#18 2023-11-06 09:55:14

Re: I think the new nvidia driver is broken

#19 2023-11-09 02:47:50

Re: I think the new nvidia driver is broken

#20 2023-11-09 03:15:22

Re: I think the new nvidia driver is broken

#21 2023-11-09 03:40:10

Re: I think the new nvidia driver is broken

#22 2023-11-09 17:40:49

Re: I think the new nvidia driver is broken

Board footer