Arch Linux

mapa17

Member

Registered: 2023-03-30

Posts: 10

Broken nvidia-tools after Nvidia Driver upgrade

Hi,

I previously had Cuda 12.1 installed with nvidia-utils (using nvidia-smi) but needed to upgrade for ML development use cases.

I have a Nvidia GTX 3060 that is not attached to any monitor but is used for ML use-cases only.

After upgrading I get the following "standard" driver miss-match error calling

> nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Following outher forum posts I have removed any nvidia and cuda packages and re-installed the "newest" version using

sudo pacman -S cuda nvidia cuda-tools
...
> sudo pacman -Qs nvidia
local/cuda 12.3.0-6
    NVIDIA's GPU programming toolkit
local/cuda-tools 12.3.0-6
    NVIDIA's GPU programming toolkit (extra tools: nvvp, nsight)
local/egl-wayland 2:1.1.11-3
    EGLStream-based Wayland external platform
local/libvdpau 1.5-2
    Nvidia VDPAU library
local/nvidia 545.29.06-3
    NVIDIA drivers for linux
local/nvidia-utils 545.29.06-1
    NVIDIA drivers utilities
local/opencl-nvidia 545.29.06-1
    OpenCL implemention for NVIDIA

I checked the driver compatibility of cuda 12.3 at https://docs.nvidia.com/cuda/cuda-toolk … index.html
and nvidia >=525.60.13 should be fine (which is the case).

Still I am not able to use nvidia-smi.

Anyone has run into similar issues and has some recommendations?

thx,
Manuel

Scimmia

Fellow

Registered: 2012-09-01

Posts: 13,729

Re: Broken nvidia-tools after Nvidia Driver upgrade

Did you reboot the load the new module?

mapa17

Member

Registered: 2023-03-30

Posts: 10

Re: Broken nvidia-tools after Nvidia Driver upgrade

Yes, i rebooted several times to make sure nothing is out of sync.
At the moment my "workaround" is to downgrade getting older versions of the relevant packages from Arch linux archive.

cd /var/cache/pacman/pkg
sudo pacman -U cuda-12.2.0-1-x86_64.pkg.tar.zst cuda-tools-12.2.0-1-x86_64.pkg.tar.zst  nvidia-utils-530.41.03-1-x86_64.pkg.tar.zst nvidia-530.41.03-1-x86_64.pkg.tar.zst opencl-nvidia-530.41.03-1-x86_64.pkg.tar.zst

This older version seems to work. Although i dont understand why nvidia-smi shows me

NVIDIA-SMI 530.41.03              Driver Version: 530.41.03    CUDA Version: 12.1

Being stuck at Cuda 12.1 and not 12.2 as was installed.

seth

Member

From: Won't reply 2 private help req

Registered: 2012-09-03

Posts: 76,631

Re: Broken nvidia-tools after Nvidia Driver upgrade

Please post your complete system journal for a boot w/ the updated versions:

sudo journalctl -b | curl -F 'file=@-' 0x0.st

Chances are you're loading an older kernel module version from the initramfs?

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc