Docker Arch issue?

Histole · 2025-08-15 03:21:14

Sorry, not sure if this is the right place for this topic, but I am unsure if this is an Arch related issue with docker/NVIDIA since I know it's not officially supported.

I ran

docker run --rm --gpus=all nvidia/cuda:12.1.1-base-ubuntu22.04 nvidia-smi

to test, and the output gave me a signal 9 error:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'

nvidia-container-cli: ldcache error: process /sbin/ldconfig terminated with signal 9

Tried reinstalling the nvidia-dkms drivers, as well as the nvidia-container-toolkit but to no avail

Linux Zen Kernel: 6.16.0

Basic Hello World docker works.

Docker Info shows the nvidia runtime is installed.

Tried:

 sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

but got the same error.

Any help is appreciated. Thanks.

Last edited by Histole (2025-08-15 03:22:37)

skye · 2025-08-15 03:47:40

i'm having the same issue after updating today. im on linux mainline kernel. getting the same error as you. recreated the container and it has same issue. my non-nvidia containers are working normally as well.

Linux Kernel: 6.16.0-arch2-1
Nvidia suite (open-dkms driver): 580.76.05-1

Histole · 2025-08-15 03:53:36

skye wrote:

i'm having the same issue

So is this a rolling release issue? Too new versions? New to Arch, and Linux in general.

Skooj · 2025-08-15 04:03:15

I'm also having this issue. Only my containers using the GPU are not starting. Others are fine.

pepperjj31 · 2025-08-15 04:05:12

Also having this issue after updating today.

Histole · 2025-08-15 04:12:28

Is this an issue with the new Kernel? Or the 580.XX driver?

Histole · 2025-08-15 04:45:08

I changed my mirrorlist to August 10th, ran pacman -Syyuu and it's working now.

Skooj · 2025-08-15 05:15:56

Histole wrote:

I changed my mirrorlist to August 10th, ran pacman -Syyuu and it's working now.

I have also rolled back my system to then. It appears to be some issue with nvidia somewhere but I could be wrong. Not sure if it's the drivers, or the container runtime itself, thought it seems telling that it happened shortly after a driver update.
Earlier, I edited /etc/nvidia-container-runtime/config.toml to output some logging, and, I'm not sure if it's related, but there appears to be a lot of missing libraries:

-- WARNING, the following logs are for debugging purposes only --

I0814 19:17:37.815815 212285 nvc.c:396] initializing library context (version=1.17.8, build=1.17.8)
I0814 19:17:37.815872 212285 nvc.c:367] using root /
I0814 19:17:37.815879 212285 nvc.c:368] using ldcache /etc/ld.so.cache
I0814 19:17:37.815884 212285 nvc.c:369] using unprivileged user 65534:65534
I0814 19:17:37.815907 212285 nvc.c:413] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL)
I0814 19:17:37.815966 212285 nvc.c:415] dxcore initialization failed, continuing assuming a non-WSL environment
I0814 19:17:37.816602 212292 nvc.c:278] loading kernel module nvidia
I0814 19:17:37.816774 212292 nvc.c:282] running mknod for /dev/nvidiactl
I0814 19:17:37.816830 212292 nvc.c:286] running mknod for /dev/nvidia0
I0814 19:17:37.816864 212292 nvc.c:290] running mknod for all nvcaps in /dev/nvidia-caps
I0814 19:17:37.821382 212292 nvc.c:218] running mknod for /dev/nvidia-caps/nvidia-cap1 from /proc/driver/nvidia/capabilities/mig/config
I0814 19:17:37.821530 212292 nvc.c:218] running mknod for /dev/nvidia-caps/nvidia-cap2 from /proc/driver/nvidia/capabilities/mig/monitor
I0814 19:17:37.823395 212292 nvc.c:304] loading kernel module nvidia_uvm
I0814 19:17:37.823565 212292 nvc.c:308] running mknod for /dev/nvidia-uvm
I0814 19:17:37.823650 212292 nvc.c:313] loading kernel module nvidia_modeset
I0814 19:17:37.823779 212292 nvc.c:317] running mknod for /dev/nvidia-modeset
I0814 19:17:37.824231 212293 rpc.c:71] starting driver rpc service
I0814 19:17:37.830146 212295 rpc.c:71] starting nvcgo rpc service
I0814 19:17:37.830783 212285 nvc_container.c:244] configuring container with 'cuda-compat-mode=ldconfig compute utility video supervised' (flags: 0x00009961)
I0814 19:17:37.831958 212285 nvc_container.c:267] setting pid to 212273
I0814 19:17:37.831974 212285 nvc_container.c:268] setting rootfs to /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged
I0814 19:17:37.831978 212285 nvc_container.c:269] setting owner to 0:0
I0814 19:17:37.831981 212285 nvc_container.c:270] setting bins directory to /usr/bin
I0814 19:17:37.831985 212285 nvc_container.c:271] setting libs directory to /usr/lib/x86_64-linux-gnu
I0814 19:17:37.831988 212285 nvc_container.c:272] setting libs32 directory to /usr/lib/i386-linux-gnu
I0814 19:17:37.831991 212285 nvc_container.c:273] setting cudart directory to /usr/local/cuda
I0814 19:17:37.831995 212285 nvc_container.c:274] setting ldconfig to @/sbin/ldconfig (host relative)
I0814 19:17:37.831998 212285 nvc_container.c:275] setting mount namespace to /proc/212273/ns/mnt
I0814 19:17:37.832001 212285 nvc_container.c:277] detected cgroupv2
I0814 19:17:37.832004 212285 nvc_container.c:278] setting devices cgroup to /sys/fs/cgroup/system.slice/docker-285dcf3fdc09222370f00cd0e491e7e1518d6d27dd770840ca74016e403bd433.scope
I0814 19:17:37.832013 212285 nvc_info.c:807] requesting driver information with ''
I0814 19:17:37.832739 212285 nvc_info.c:175] selecting /usr/lib/libnvoptix.so.580.76.05
I0814 19:17:37.832787 212285 nvc_info.c:175] selecting /usr/lib/libnvidia-tls.so.580.76.05
I0814 19:17:37.832820 212285 nvc_info.c:175] selecting /usr/lib/libnvidia-rtcore.so.580.76.05
I0814 19:17:37.832851 212285 nvc_info.c:175] selecting /usr/lib/libnvidia-ptxjitcompiler.so.580.76.05
I0814 19:17:37.832883 212285 nvc_info.c:175] selecting /usr/lib/libnvidia-pkcs11.so.580.76.05
I0814 19:17:37.832913 212285 nvc_info.c:175] selecting /usr/lib/libnvidia-pkcs11-openssl3.so.580.76.05
I0814 19:17:37.832979 212285 nvc_info.c:175] selecting /usr/lib/libnvidia-opticalflow.so.580.76.05
I0814 19:17:37.833016 212285 nvc_info.c:175] selecting /usr/lib/libnvidia-opencl.so.580.76.05
I0814 19:17:37.833050 212285 nvc_info.c:175] selecting /usr/lib/libnvidia-nvvm.so.580.76.05
I0814 19:17:37.833081 212285 nvc_info.c:175] selecting /usr/lib/libnvidia-ngx.so.580.76.05
I0814 19:17:37.833112 212285 nvc_info.c:175] selecting /usr/lib/libnvidia-ml.so.580.76.05
I0814 19:17:37.833143 212285 nvc_info.c:175] selecting /usr/lib/libnvidia-gpucomp.so.580.76.05
I0814 19:17:37.833172 212285 nvc_info.c:175] selecting /usr/lib/libnvidia-glvkspirv.so.580.76.05
I0814 19:17:37.833202 212285 nvc_info.c:175] selecting /usr/lib/libnvidia-glsi.so.580.76.05
I0814 19:17:37.833231 212285 nvc_info.c:175] selecting /usr/lib/libnvidia-glcore.so.580.76.05
I0814 19:17:37.833263 212285 nvc_info.c:175] selecting /usr/lib/libnvidia-fbc.so.580.76.05
I0814 19:17:37.833294 212285 nvc_info.c:175] selecting /usr/lib/libnvidia-encode.so.580.76.05
I0814 19:17:37.833324 212285 nvc_info.c:175] selecting /usr/lib/libnvidia-eglcore.so.580.76.05
I0814 19:17:37.833360 212285 nvc_info.c:175] selecting /usr/lib/libnvidia-cfg.so.580.76.05
I0814 19:17:37.833392 212285 nvc_info.c:175] selecting /usr/lib/libnvidia-allocator.so.580.76.05
I0814 19:17:37.833425 212285 nvc_info.c:175] selecting /usr/lib/libnvcuvid.so.580.76.05
I0814 19:17:37.833828 212285 nvc_info.c:175] selecting /usr/lib/libcudadebugger.so.580.76.05
I0814 19:17:37.833860 212285 nvc_info.c:175] selecting /usr/lib/libcuda.so.580.76.05
I0814 19:17:37.834185 212285 nvc_info.c:175] selecting /usr/lib/libGLX_nvidia.so.580.76.05
I0814 19:17:37.834222 212285 nvc_info.c:175] selecting /usr/lib/libGLESv2_nvidia.so.580.76.05
I0814 19:17:37.834254 212285 nvc_info.c:175] selecting /usr/lib/libGLESv1_CM_nvidia.so.580.76.05
I0814 19:17:37.834290 212285 nvc_info.c:175] selecting /usr/lib/libEGL_nvidia.so.580.76.05
I0814 19:17:37.834376 212285 nvc_info.c:175] selecting /usr/lib32/libnvidia-tls.so.580.76.05
I0814 19:17:37.834407 212285 nvc_info.c:175] selecting /usr/lib32/libnvidia-ptxjitcompiler.so.580.76.05
I0814 19:17:37.834438 212285 nvc_info.c:175] selecting /usr/lib32/libnvidia-opticalflow.so.580.76.05
I0814 19:17:37.834468 212285 nvc_info.c:175] selecting /usr/lib32/libnvidia-ml.so.580.76.05
I0814 19:17:37.834498 212285 nvc_info.c:175] selecting /usr/lib32/libnvidia-gpucomp.so.580.76.05
I0814 19:17:37.834527 212285 nvc_info.c:175] selecting /usr/lib32/libnvidia-glvkspirv.so.580.76.05
I0814 19:17:37.834555 212285 nvc_info.c:175] selecting /usr/lib32/libnvidia-glsi.so.580.76.05
I0814 19:17:37.834584 212285 nvc_info.c:175] selecting /usr/lib32/libnvidia-glcore.so.580.76.05
I0814 19:17:37.834615 212285 nvc_info.c:175] selecting /usr/lib32/libnvidia-fbc.so.580.76.05
I0814 19:17:37.834646 212285 nvc_info.c:175] selecting /usr/lib32/libnvidia-encode.so.580.76.05
I0814 19:17:37.834676 212285 nvc_info.c:175] selecting /usr/lib32/libnvidia-eglcore.so.580.76.05
I0814 19:17:37.834706 212285 nvc_info.c:175] selecting /usr/lib32/libnvidia-allocator.so.580.76.05
I0814 19:17:37.834737 212285 nvc_info.c:175] selecting /usr/lib32/libnvcuvid.so.580.76.05
I0814 19:17:37.834817 212285 nvc_info.c:175] selecting /usr/lib32/libcuda.so.580.76.05
I0814 19:17:37.834873 212285 nvc_info.c:175] selecting /usr/lib32/libGLX_nvidia.so.580.76.05
I0814 19:17:37.834905 212285 nvc_info.c:175] selecting /usr/lib32/libGLESv2_nvidia.so.580.76.05
I0814 19:17:37.834963 212285 nvc_info.c:175] selecting /usr/lib32/libGLESv1_CM_nvidia.so.580.76.05
I0814 19:17:37.834998 212285 nvc_info.c:175] selecting /usr/lib32/libEGL_nvidia.so.580.76.05
W0814 19:17:37.835027 212285 nvc_info.c:411] missing library libnvidia-nscq.so
W0814 19:17:37.835032 212285 nvc_info.c:411] missing library libnvidia-fatbinaryloader.so
W0814 19:17:37.835036 212285 nvc_info.c:411] missing library libnvidia-compiler.so
W0814 19:17:37.835039 212285 nvc_info.c:411] missing library libvdpau_nvidia.so
W0814 19:17:37.835042 212285 nvc_info.c:411] missing library libnvidia-ifr.so
W0814 19:17:37.835049 212285 nvc_info.c:411] missing library libnvidia-cbl.so
W0814 19:17:37.835052 212285 nvc_info.c:415] missing compat32 library libnvidia-cfg.so
W0814 19:17:37.835056 212285 nvc_info.c:415] missing compat32 library libnvidia-nscq.so
W0814 19:17:37.835059 212285 nvc_info.c:415] missing compat32 library libcudadebugger.so
W0814 19:17:37.835062 212285 nvc_info.c:415] missing compat32 library libnvidia-opencl.so
W0814 19:17:37.835065 212285 nvc_info.c:415] missing compat32 library libnvidia-fatbinaryloader.so
W0814 19:17:37.835068 212285 nvc_info.c:415] missing compat32 library libnvidia-compiler.so
W0814 19:17:37.835071 212285 nvc_info.c:415] missing compat32 library libnvidia-pkcs11.so
W0814 19:17:37.835074 212285 nvc_info.c:415] missing compat32 library libnvidia-pkcs11-openssl3.so
W0814 19:17:37.835078 212285 nvc_info.c:415] missing compat32 library libnvidia-nvvm.so
W0814 19:17:37.835081 212285 nvc_info.c:415] missing compat32 library libnvidia-ngx.so
W0814 19:17:37.835084 212285 nvc_info.c:415] missing compat32 library libvdpau_nvidia.so
W0814 19:17:37.835087 212285 nvc_info.c:415] missing compat32 library libnvidia-ifr.so
W0814 19:17:37.835090 212285 nvc_info.c:415] missing compat32 library libnvidia-rtcore.so
W0814 19:17:37.835093 212285 nvc_info.c:415] missing compat32 library libnvoptix.so
W0814 19:17:37.835096 212285 nvc_info.c:415] missing compat32 library libnvidia-cbl.so
I0814 19:17:37.835263 212285 nvc_info.c:301] selecting /usr/bin/nvidia-smi
I0814 19:17:37.835279 212285 nvc_info.c:301] selecting /usr/bin/nvidia-debugdump
I0814 19:17:37.835296 212285 nvc_info.c:301] selecting /usr/bin/nvidia-persistenced
I0814 19:17:37.835321 212285 nvc_info.c:301] selecting /usr/bin/nvidia-cuda-mps-control
I0814 19:17:37.835337 212285 nvc_info.c:301] selecting /usr/bin/nvidia-cuda-mps-server
W0814 19:17:37.835392 212285 nvc_info.c:437] missing binary nv-fabricmanager
I0814 19:17:37.835437 212285 nvc_info.c:497] listing firmware path /lib/firmware/nvidia/580.76.05/gsp_ga10x.bin
I0814 19:17:37.835442 212285 nvc_info.c:497] listing firmware path /lib/firmware/nvidia/580.76.05/gsp_tu10x.bin
I0814 19:17:37.835464 212285 nvc_info.c:570] listing device /dev/nvidiactl
I0814 19:17:37.835468 212285 nvc_info.c:570] listing device /dev/nvidia-uvm
I0814 19:17:37.835471 212285 nvc_info.c:570] listing device /dev/nvidia-uvm-tools
I0814 19:17:37.835474 212285 nvc_info.c:570] listing device /dev/nvidia-modeset
W0814 19:17:37.835495 212285 nvc_info.c:359] missing ipc path /var/run/nvidia-persistenced/socket
W0814 19:17:37.835513 212285 nvc_info.c:359] missing ipc path /var/run/nvidia-fabricmanager/socket
W0814 19:17:37.835527 212285 nvc_info.c:359] missing ipc path /tmp/nvidia-mps
I0814 19:17:37.835531 212285 nvc_info.c:863] requesting device information with ''
I0814 19:17:37.845468 212285 nvc_info.c:754] listing device /dev/nvidia0 (GPU-b13bf7f5-0f62-9b95-9b0e-eae168c7a448 at 00000000:01:00.0)
I0814 19:17:37.845561 212285 nvc_mount.c:369] mounting tmpfs at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/proc/driver/nvidia
W0814 19:17:37.845792 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/bin already exists with the required mode; skipping create
W0814 19:17:37.845828 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/bin/nvidia-smi already exists with the required mode; skipping create
I0814 19:17:37.845834 212285 nvc_mount.c:89] mounting /usr/bin/nvidia-smi at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/bin/nvidia-smi with flags 0x7
W0814 19:17:37.845871 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/bin/nvidia-debugdump already exists with the required mode; skipping create
I0814 19:17:37.845877 212285 nvc_mount.c:89] mounting /usr/bin/nvidia-debugdump at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/bin/nvidia-debugdump with flags 0x7
W0814 19:17:37.845918 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/bin/nvidia-persistenced already exists with the required mode; skipping create
I0814 19:17:37.845924 212285 nvc_mount.c:89] mounting /usr/bin/nvidia-persistenced at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/bin/nvidia-persistenced with flags 0x7
W0814 19:17:37.845962 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/bin/nvidia-cuda-mps-control already exists with the required mode; skipping create
I0814 19:17:37.845968 212285 nvc_mount.c:89] mounting /usr/bin/nvidia-cuda-mps-control at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/bin/nvidia-cuda-mps-control with flags 0x7
W0814 19:17:37.845998 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/bin/nvidia-cuda-mps-server already exists with the required mode; skipping create
I0814 19:17:37.846003 212285 nvc_mount.c:89] mounting /usr/bin/nvidia-cuda-mps-server at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/bin/nvidia-cuda-mps-server with flags 0x7
W0814 19:17:37.846051 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu already exists with the required mode; skipping create
W0814 19:17:37.846079 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.580.76.05 already exists with the required mode; skipping create
I0814 19:17:37.846084 212285 nvc_mount.c:89] mounting /usr/lib/libnvidia-ml.so.580.76.05 at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.580.76.05 with flags 0x7
W0814 19:17:37.846118 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.580.76.05 already exists with the required mode; skipping create
I0814 19:17:37.846123 212285 nvc_mount.c:89] mounting /usr/lib/libnvidia-cfg.so.580.76.05 at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.580.76.05 with flags 0x7
W0814 19:17:37.846157 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libcuda.so.580.76.05 already exists with the required mode; skipping create
I0814 19:17:37.846162 212285 nvc_mount.c:89] mounting /usr/lib/libcuda.so.580.76.05 at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libcuda.so.580.76.05 with flags 0x7
W0814 19:17:37.846194 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libcudadebugger.so.580.76.05 already exists with the required mode; skipping create
I0814 19:17:37.846200 212285 nvc_mount.c:89] mounting /usr/lib/libcudadebugger.so.580.76.05 at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libcudadebugger.so.580.76.05 with flags 0x7
W0814 19:17:37.846232 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.580.76.05 already exists with the required mode; skipping create
I0814 19:17:37.846237 212285 nvc_mount.c:89] mounting /usr/lib/libnvidia-opencl.so.580.76.05 at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.580.76.05 with flags 0x7
W0814 19:17:37.846275 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-gpucomp.so.580.76.05 already exists with the required mode; skipping create
I0814 19:17:37.846280 212285 nvc_mount.c:89] mounting /usr/lib/libnvidia-gpucomp.so.580.76.05 at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-gpucomp.so.580.76.05 with flags 0x7
W0814 19:17:37.846314 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.580.76.05 already exists with the required mode; skipping create
I0814 19:17:37.846319 212285 nvc_mount.c:89] mounting /usr/lib/libnvidia-ptxjitcompiler.so.580.76.05 at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.580.76.05 with flags 0x7
W0814 19:17:37.846352 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.580.76.05 already exists with the required mode; skipping create
I0814 19:17:37.846357 212285 nvc_mount.c:89] mounting /usr/lib/libnvidia-allocator.so.580.76.05 at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.580.76.05 with flags 0x7
W0814 19:17:37.846390 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-pkcs11.so.580.76.05 already exists with the required mode; skipping create
I0814 19:17:37.846395 212285 nvc_mount.c:89] mounting /usr/lib/libnvidia-pkcs11.so.580.76.05 at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-pkcs11.so.580.76.05 with flags 0x7
W0814 19:17:37.846428 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-pkcs11-openssl3.so.580.76.05 already exists with the required mode; skipping create
I0814 19:17:37.846433 212285 nvc_mount.c:89] mounting /usr/lib/libnvidia-pkcs11-openssl3.so.580.76.05 at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-pkcs11-openssl3.so.580.76.05 with flags 0x7
W0814 19:17:37.846466 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.580.76.05 already exists with the required mode; skipping create
I0814 19:17:37.846472 212285 nvc_mount.c:89] mounting /usr/lib/libnvidia-nvvm.so.580.76.05 at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.580.76.05 with flags 0x7
W0814 19:17:37.846504 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-encode.so.580.76.05 already exists with the required mode; skipping create
I0814 19:17:37.846510 212285 nvc_mount.c:89] mounting /usr/lib/libnvidia-encode.so.580.76.05 at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-encode.so.580.76.05 with flags 0x7
W0814 19:17:37.846543 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.580.76.05 already exists with the required mode; skipping create
I0814 19:17:37.846551 212285 nvc_mount.c:89] mounting /usr/lib/libnvidia-opticalflow.so.580.76.05 at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.580.76.05 with flags 0x7
W0814 19:17:37.846591 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.76.05 already exists with the required mode; skipping create
I0814 19:17:37.846596 212285 nvc_mount.c:89] mounting /usr/lib/libnvcuvid.so.580.76.05 at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvcuvid.so.580.76.05 with flags 0x7
I0814 19:17:37.846615 212285 nvc_mount.c:530] creating symlink /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libcuda.so -> libcuda.so.1
I0814 19:17:37.846640 212285 nvc_mount.c:530] creating symlink /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so -> libnvidia-opticalflow.so.1
W0814 19:17:37.846727 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/lib/firmware/nvidia/580.76.05/gsp_ga10x.bin already exists with the required mode; skipping create
I0814 19:17:37.846732 212285 nvc_mount.c:89] mounting /usr/lib/firmware/nvidia/580.76.05/gsp_ga10x.bin at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/lib/firmware/nvidia/580.76.05/gsp_ga10x.bin with flags 0x7
W0814 19:17:37.846786 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/lib/firmware/nvidia/580.76.05/gsp_tu10x.bin already exists with the required mode; skipping create
I0814 19:17:37.846791 212285 nvc_mount.c:89] mounting /usr/lib/firmware/nvidia/580.76.05/gsp_tu10x.bin at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/lib/firmware/nvidia/580.76.05/gsp_tu10x.bin with flags 0x7
W0814 19:17:37.846817 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/dev/nvidiactl already exists with the required mode; skipping create
I0814 19:17:37.846822 212285 nvc_mount.c:233] mounting /dev/nvidiactl at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/dev/nvidiactl
W0814 19:17:37.847324 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/dev/nvidia-uvm already exists with the required mode; skipping create
I0814 19:17:37.847337 212285 nvc_mount.c:233] mounting /dev/nvidia-uvm at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/dev/nvidia-uvm
W0814 19:17:37.847504 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/dev/nvidia-uvm-tools already exists with the required mode; skipping create
I0814 19:17:37.847511 212285 nvc_mount.c:233] mounting /dev/nvidia-uvm-tools at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/dev/nvidia-uvm-tools
W0814 19:17:37.847689 212285 utils.c:547] The path /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/dev/nvidia0 already exists with the required mode; skipping create
I0814 19:17:37.847696 212285 nvc_mount.c:233] mounting /dev/nvidia0 at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/dev/nvidia0
I0814 19:17:37.847775 212285 nvc_mount.c:443] mounting /proc/driver/nvidia/gpus/0000:01:00.0 at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged/proc/driver/nvidia/gpus/0000:01:00.0
I0814 19:17:37.848441 212285 nvc_ldcache.c:424] creating a virtual copy of the ldconfig binary
I0814 19:17:37.848996 212285 nvc_ldcache.c:513] executing /sbin/ldconfig from host at /var/lib/docker/overlay2/eaf1df05d2a9b9ea5c6c72716c4b8bcfbeb87185ece0a46435e7ccea5b0f4eb6/merged
I0814 19:17:47.903415 212285 nvc.c:455] shutting down library context
I0814 19:17:47.903503 212295 rpc.c:95] terminating nvcgo rpc service
I0814 19:17:47.903950 212285 rpc.c:132] nvcgo rpc service terminated successfully
I0814 19:17:47.907170 212293 rpc.c:95] terminating driver rpc service
I0814 19:17:47.907554 212285 rpc.c:132] driver rpc service terminated successfully

Histole · 2025-08-15 05:33:38

Skooj wrote:

I have also rolled back my system to then.

If I had to guess, I think it’s the 580.XX driver. Could be wrong though.

fpadula · 2025-08-15 09:19:10

I am experiencing the same issue after updating today. Using nvidia-open 580.76.05-3, nvidia-container-toolkit 1.17.8-1 and docker 1:28.3.3-1

Liangkun · 2025-08-15 14:51:47

It might be a container-toolkit issue. I found no error in the NVIDIA log, but see the runtime terminate itself automatically after the initialization.

GaaH2 · 2025-08-15 16:36:55

Histole wrote:

I changed my mirrorlist to August 10th, ran pacman -Syyuu and it's working now.

Thank you, it works now.

For those that have no clue what "changing mirrorlist to August 10th", just replace the content of /etc/pacman.d/mirrorlist with the line below

Server=https://archive.archlinux.org/repos/2025/08/10/$repo/os/$arch

MrSnrub · 2025-08-15 17:17:53

Seems to be a conflict with the new kernel and the NVIDIA driver. My system is on Linux kernel 6.16.0-arch2-1 and the NVIDIA driver is 580.76.05-1.

The error I get is a persistent ldcache error that kills the container startup process with signal 9.

Error Message:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: ldcache error: process /sbin/ldconfig terminated with signal 9

Things I've already tried:

Reinstalled the toolkit: I did a full pacman -Rns followed by a fresh yay -S nvidia-container-toolkit and a sudo systemctl restart docker. No luck.
Reconfigured the runtime: Ran sudo nvidia-ctk runtime configure --runtime=docker, which completed successfully, but the issue persists.
Tried different CUDA images: I tested with both nvidia/cuda:11.8.0-base-ubuntu22.04 and nvidia/cuda:12.3.1-base-ubuntu22.04. The error is the same with both images, so it doesn't seem to be an issue with a specific container image.
Confirmed nvidia-smi works: The driver is working fine on my host system; I can run nvidia-smi without any problems, so the base driver installation isn't the issue. My nvidia-smi shows the driver version 580.76.05 and CUDA version 13.0.

Last edited by MrSnrub (2025-08-15 17:18:14)

slip · 2025-08-15 18:29:35

As long as whatever is in the new kernel was also thrown into the LTS kernel. I get the exact same error as you with the LTS kernel and actually went through all of the steps you did. The only thing I didn't do was specifically target downgrading the LTS kernel. However, using the above mirror list rollback worked, so I left it at that for now.

edit - link to the git issue that brought me here https://github.com/NVIDIA/nvidia-contai … ssues/1246

Last edited by slip (2025-08-15 18:33:22)

Liangkun · 2025-08-15 18:46:46

slip wrote:

As long as whatever is in the new kernel was also thrown into the LTS kernel. I get the exact same error as you with the LTS kernel and actually went through all of the steps you did. The only thing I didn't do was specifically target downgrading the LTS kernel. However, using the above mirror list rollback worked, so I left it at that for now.
edit - link to the git issue that brought me here https://github.com/NVIDIA/nvidia-contai … ssues/1246

The kernel might not be the problem. I guess it's because of some memory sharing issue with the new 580 driver

Last edited by Liangkun (2025-08-15 18:47:08)

skye · 2025-08-16 06:04:38

i rolled back only my kernel and my nvidia driver packages just out of curiosity and i still had the same issue, however i didn't dig deeper as compared to the full system rollback which did solve the issue, it was just a lot of packages. i hadn't rolled back amd-ucode so if that had something to do with it i can't say. unhelpful comment but i thought i'd throw it out there that it could maybe be something else

editing to say that it is also possible i just missed an essential driver or firmware module.

Last edited by skye (2025-08-16 06:05:25)

tyfon · 2025-08-16 19:02:21

Finally registered to the forum after using arch for 15+ years
I'm having the same issue with docker, but with podman it works fine and it (ollama) is using the GPU.

So I suspect the issue is with docker and not the nvidia drivers etc.

ChiefHardi · 2025-08-16 19:53:21

tyfon wrote:

Finally registered to the forum after using arch for 15+ years
I'm having the same issue with docker, but with podman it works fine and it (ollama) is using the GPU.
So I suspect the issue is with docker and not the nvidia drivers etc.

Same here, finally signed up because of this lol. Thanks for the tip about using Podman, I was able to get back up and running.

Liangkun · 2025-08-17 08:09:21

Since podman does work. I also tried Docker with CDI, and it also works. I guess it is a problem with the legacy hook.

Refer to this link: https://github.com/NVIDIA/nvidia-contai … 3194219487

Last edited by Liangkun (2025-08-17 08:16:55)

unkwn1 · 2025-08-17 08:27:09

Liangkun wrote:

Since podman does work. I also tried Docker with CDI, and it also works. I guess it is a problem with the legacy hook.
Refer to this link: https://github.com/NVIDIA/nvidia-contai … 3194219487

This worked for me, as the poster mentioned the output referencing `legacy`. Running this below and restarting my ollama container worked - no kernel downgrades or drivers, just a config tweak.

`sudo nvidia-ctk config --in-place --set nvidia-container-runtime.mode=cdi`

vostok4 · 2025-08-17 19:48:34

If you switch to CDI mode (which is more modern, so shouldn't be a downfall), you do need to adjust your compose files, as described here: https://github.com/NVIDIA/nvidia-contai … 3194227600

pepperjj31 · 2025-08-18 14:59:12

For anyone else running on a headless server and looking to switch to CDI mode, I ran into an issue where /dev/nvidiactl was not being created.
You can either run
nvidia-smi

which will manually initialize the file or what I did was run
sudo systemctl enable nvidia-persistenced

which caused everything to be properly initialized on boot.

Got info from here: https://forums.developer.nvidia.com/t/w … ctl/108272

Last edited by pepperjj31 (2025-08-18 14:59:24)

GruffyPuffy · 2025-08-19 12:25:10

I got the problem when updating with some incus containers (GPU passthrough). However I could not understand why so I tried to boil down when it broke. For me, it was the last glibc update that broke it. I know it is not recommended to put this in IgnorePkg but I did it for testing purpose. So when updating today I have this:

-> glibc: ignoring package upgrade (2.42+r3+gbc13db739377-1 => 2.42+r17+gd7274d718e6f-1)
-> lib32-glibc: ignoring package upgrade (2.42+r3+gbc13db739377-1 => 2.42+r17+gd7274d718e6f-1)

Other than that, I have all packages updated and my containers work as before (nvidia-smi works etc). Do not know what causes this, but might help others tracking it down I guess.

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+

Maybe some more experienced persons can get some clue from this at least.

Last edited by GruffyPuffy (2025-08-19 12:25:36)

dmsh_public_contact · 2025-08-25 07:59:02

Hi All,

I have all packages updated, no downgrades.
What I did to get docker work with nvidia drivers 580.76.05:

sudo nvidia-ctk runtime configure --runtime=docker
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
sudo nvidia-ctk config --in-place --set nvidia-container-runtime.mode=cdi && sudo systemctl restart docker

cat /etc/docker/daemon.json

{
    "default-address-pools": [
        {
            "base": "172.17.0.0/16",
            "size": 24
        }
    ],
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }
}

In compose files I have:

runtime: nvidia
deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: [ 'nvidia.com/gpu=all' ]
              capabilities:
                - gpu

Added delay for docker startup:

sudo systemctl edit docker

[Service]
ExecStartPre=/bin/sleep 30

Last edited by dmsh_public_contact (2025-08-25 09:22:29)

tj123 · 2025-08-25 11:47:11

I just tried what dmsh_public_contact posted, and my container is running again.

Arch Linux

#1 2025-08-15 03:21:14

Docker Arch issue?

#2 2025-08-15 03:47:40

Re: Docker Arch issue?

#3 2025-08-15 03:53:36

Re: Docker Arch issue?

#4 2025-08-15 04:03:15

Re: Docker Arch issue?

#5 2025-08-15 04:05:12

Re: Docker Arch issue?

#6 2025-08-15 04:12:28

Re: Docker Arch issue?

#7 2025-08-15 04:45:08

Re: Docker Arch issue?

#8 2025-08-15 05:15:56

Re: Docker Arch issue?

#9 2025-08-15 05:33:38

Re: Docker Arch issue?

#10 2025-08-15 09:19:10

Re: Docker Arch issue?

#11 2025-08-15 14:51:47

Re: Docker Arch issue?

#12 2025-08-15 16:36:55

Re: Docker Arch issue?

#13 2025-08-15 17:17:53

Re: Docker Arch issue?

#14 2025-08-15 18:29:35

Re: Docker Arch issue?

#15 2025-08-15 18:46:46

Re: Docker Arch issue?

#16 2025-08-16 06:04:38

Re: Docker Arch issue?

#17 2025-08-16 19:02:21

Re: Docker Arch issue?

#18 2025-08-16 19:53:21

Re: Docker Arch issue?

#19 2025-08-17 08:09:21

Re: Docker Arch issue?

#20 2025-08-17 08:27:09

Re: Docker Arch issue?

#21 2025-08-17 19:48:34

Re: Docker Arch issue?

#22 2025-08-18 14:59:12

Re: Docker Arch issue?

#23 2025-08-19 12:25:10

Re: Docker Arch issue?

#24 2025-08-25 07:59:02

Re: Docker Arch issue?

#25 2025-08-25 11:47:11

Re: Docker Arch issue?

Board footer