can't get nvidia to work within a systemd-nspawn container

gacki · 2025-06-27 23:53:28

so in the past couple days i was trying to get a nspawn container up and running to have a graphical environment i can remote into via rdp, through windows or whatever machines when i'm not at home.
while that was successful (mostly), i couldn't get the nvidia userspace components to talk with the host gpu at all. as i am writing this, the DE present on that thing has to run on software rendering mode.

the container was set up using this thing by running

nspawn -i archlinux/archlinux/tar

as root, and after that, i've done the usual procedures that i do on fresh arch installs (put on pacman mirrors, update the system, install base-devel and set up a non-root user with sudo priviliges). while that could be the root cause of my frustration - by doing that instead of setting this up using plain pacstrap directly @ /var/lib/machines, i'm not certain if this has anything to do with nvidia-smi exhibiting this behavior, even after doing all the necessary work to have the device nodes accessible from inside the container.

[root@juno ~]# nvidia-smi
Failed to initialize NVML: GPU access blocked by the operating system

same thing happens when trying to use it as a non-root user from within the container

[neiceng@juno ~]$ nvidia-smi
Failed to initialize NVML: GPU access blocked by the operating system

unsurprisingly, if nvidia-smi can't access the gpu, i can't expect vulkaninfo or glxinfo to do anything useful instead of yelling at me that there's no drivers or something like that. and, well, that happened:

[neiceng@juno ~]$ vulkaninfo
ERROR: [Loader Message] Code 0 : loader_scanned_icd_add: Could not get 'vkCreateInstance' via 'vk_icdGetInstanceProcAddr' for ICD libGLX_nvidia.so.0
ERROR: [Loader Message] Code 0 : vkCreateInstance: Found no drivers!
Cannot create Vulkan instance.
This problem is often caused by a faulty installation of the Vulkan driver or attempting to use a GPU that does not support Vulkan.
ERROR at /usr/src/debug/vulkan-tools/Vulkan-Tools/vulkaninfo/./vulkaninfo.h:573:vkCreateInstance failed with ERROR_INCOMPATIBLE_DRIVER

the nvidia-utils package version is the same both on the host and the container. both are running the same arch distribution with (at the time of writing) up to date packages. strangely, btop won't run on this container (this one named juno - btop on there spits "Operation not permitted" at me for some reason), but in another arch nspawn that i did set up for doing other things, btop works just fine.

full output of machinectl cat juno and systemctl cat systemd-nspawn@juno are at http://0x0.st/8lkY.txt.

funnily enough, FUSE inside systemd-nspawn containers requires very similar prep work, and doing the same process with it's device node (/dev/fuse) worked just fine (by binding the device and allowing it's access with issuing a systemctl set-property DeviceAllow or something like that). i can sshfs mount the host filesystem or other containers in the same veth just fine. can't say the same for nvidia being able to access my gpu in (i guess) the same way.

i have tried turning off PrivateUsers and it doesn't work. i have tried binding the nvidia device paths with noidmap, rootidmap and doing that makes the container not start, whether the respective DeviceAllow= properties are present or not.

now i am not sure what to try next, and searching the web for people that went through the same situation were mostly working with Docker on WSL. had found someone asking about a similar issue on a gentoo mailing list and the conversation ended with "i still think your container is misconfigured" or something, but haven't been able to find it again since. this forum had like 3 people asking about stuff like that, 2 of them are not really in line of what happened on my end and one that is very close to it had no replies.

here's what i have referenced so far
https://wiki.archlinux.org/title/System … vidia_GPUs
https://www.freedesktop.org/software/sy … viceAllow=
https://www.freedesktop.org/software/sy … ml#--bind=
https://www.freedesktop.org/software/sy … trol.html#
https://liolok.com/containerize-steam-w … phic-cards
https://www.reddit.com/r/archlinux/comm … emdnspawn/

unsure if relevant, but my system runs the nvidia drivers from the nvidia-open package.

[gacki@anubi ~]$ ssh neiceng@juno 'head -n1 /etc/os-release'
NAME="Arch Linux"
[gacki@anubi ~]$ uname -a
Linux anubi 6.15.3-zen1-1-zen #1 ZEN SMP PREEMPT_DYNAMIC Thu, 19 Jun 2025 14:41:01 +0000 x86_64 GNU/Linux
[gacki@anubi ~]$ ssh neiceng@juno 'pacman -Qs | grep -A1 nvidia-utils'
local/nvidia-utils 575.64-1
    NVIDIA drivers utilities
[gacki@anubi ~]$ pacman -Qs | grep -A1 nvidia-utils
local/lib32-nvidia-utils 575.64-2
    NVIDIA drivers utilities (32-bit)
--
local/nvidia-utils 575.64-1
    NVIDIA drivers utilities
[gacki@anubi ~]$ uname -a
Linux anubi 6.15.3-zen1-1-zen #1 ZEN SMP PREEMPT_DYNAMIC Thu, 19 Jun 2025 14:41:01 +0000 x86_64 GNU/Linux

just what i am doing wrong?

Last edited by gacki (2025-06-30 02:02:10)

seth · 2025-06-28 08:35:05

[root@juno ~]# nvidia-smi
Failed to initialize NVML: GPU access blocked by the operating system

From https://wiki.archlinux.org/title/System … VIDIA_GPUs

…
DeviceAllow=/dev/dri rw
DeviceAllow=/dev/shm rw
DeviceAllow=/dev/nvidia0 rw
DeviceAllow=/dev/nvidiactl rw
DeviceAllow=/dev/nvidia-modeset rw

In doubt "strace -f nvidia-smi" to see where it fails exactly.
Do you?

gacki · 2025-06-28 16:55:34

seth wrote:

[root@juno ~]# nvidia-smi
Failed to initialize NVML: GPU access blocked by the operating system
From https://wiki.archlinux.org/title/System … VIDIA_GPUs

these DeviceAllow lines are present on the machine unit file (both as char-nvidia (+ some other things nvidia at /proc/devices) and the literal device paths).

seth wrote:

In doubt "strace -f nvidia-smi" to see where it fails exactly.
Do you?

i tried to do that yesterday or so but I thought this utility was called "ptrace" and not "strace". so I stopped trying at the time due to frustration. silly me.

apart from that, this thing is failing some mknod call at /dev/nvidia-caps. trying to make sense of that led me to check the container capabilites, and giving it CAP_MKNOD didn't work. setting PrivateUsers to no didn't work. interestingly, an attempt to access /sys/module/nvidia/initstate returned ENOENT and indeed, this file isn't there in the container. trying to bind /sys from the host at /sys within the container makes this thing not start.

not sure if I should be expecting that but vulkaninfo attempted to access the same file.

below follows the full strace outputs of both programs
http://0x0.st/8lCA.txt (vulkaninfo)
http://0x0.st/8lCa.txt (nvidia-smi)

Last edited by gacki (2025-06-28 18:07:02)

seth · 2025-06-29 07:51:19

Are you https://bugs.gentoo.org/958945 as well?
https://jocke.no/2022/02/23/plex-gpu-tr … n-proxmox/ binds them, did you try

…
DeviceAllow=/dev/nvidia-caps/nvidia-cap1 rw
DeviceAllow=/dev/nvidia-caps/nvidia-cap2 rw

For vulkan , "export VK_DRIVER_FILES=/usr/share/vulkan/icd.d/nvidia_icd.json" and test vkgears or vkcube or so (the scanning stuff like eglinfo, vulkaninfo, glxinfo might introduce more obstacles than necessary for plain use)

gacki · 2025-06-29 21:39:42

no, that's not me.

I've allowed and bound both device nodes, and they register on the container with r--r--r-- permissions and now nvidia-smi can open them! except that it also says that GPU access is blocked by the OS and further inspection in strace has shown a few failed chmod calls and near the end of it's execution, an ioctl returned E_BADF.

out of frustration I've set DevicePolicy to auto. i am aware that this has glaring implications (most likely not just security) but have done so for the sake of testing.

completely ignoring everything, pretending that this could be somehow working for plain use, led me nowhere useful. installing xorgxrdp-nvidia had the effect of Xorg not starting at all. I looked at the logs (the .xorgxrdp.something.log files this thing spits out) and Xorg can't put it's hands on /dev/dri/card1. explicitly adding

DeviceAllow=/dev/dri/card1 rw

to the nspawn unit file didn't do anything.

so then I tried to get vkcube running. I bound /tmp/.X11-unix/X100 from the host and got a Xephyr window on :100 (X100 on .X11-unix as I have guessed) and xterm opens there. good! so I ran vkcube like you mentioned

[neiceng@juno ~]$ VK_DRIVER_FILES=/usr/share/vulkan/icd.d/nvidia_icd.json vkcube
Selected WSI platform: xcb
vkEnumeratePhysicalDevices reported zero accessible devices.

Do you have a compatible Vulkan installable client driver (ICD) installed?
Please look at the Getting Started guide for additional information.

and have also tried doing so under xterm

below follows some more logs...
journal: http://0x0.st/8U_M.txt
related xorg logs from xrdp: http://0x0.st/8U_T.txt
nvidia-smi strace output: http://0x0.st/8U_f.txt
juno container config (.nspawn): http://0x0.st/8U_4.txt
juno unit config (systemd-nspawn@.service): http://0x0.st/8U_y.txt

don't know what else to do.

i've read again the comments on the bug you're linked and this shouldn't be a systemd issue at all. i've done the same with an AMD card before, back when I still had one, and it was just as painless as allowing FUSE to work within the container. there's no indication that DeviceAllow= is in fact not allowing anything. if this is a restriction somewhere, I don't know if there's a way of knowing where these restrictions are being administered.

as a last resort I adapted an excerpt from the wiki that binds the driver files into the containers into this: http://0x0.st/8U_l.txt
and neither nvidia-smi and vulkaninfo (or vkcube) do anything different.

Last edited by gacki (2025-06-30 02:16:46)

gacki · 2025-06-30 05:37:40

what's weirder is that the nvidia-cap1 and nvidia-cap2 files are created only when nvidia-smi has been run at least once on the host. so if I reboot my machine, the container will not start until I run nvidia-smi again.

out of curiosity I've bound my input devices to the container, so I could test if trying to access them would also result in "Operation not permitted" errors or anything like them. given the DeviceAllow= lines are present. however their permissions behave weirdly in the container: they're set as rw-rw---, however the root user in the container can't access device nodes that aren't readable by regular users on the host. setting PrivateUsers to off on the nspawn file removes this limitation.

but seemingly, none of this is relevant to nvidia. everything /dev/nvidia* on the host is rw-rw-rw- (nvidia-cap1 and cap2 are r--r--r--).
vulkaninfo fails. nvidia-smi fails. vkcube tells me that the ICDs are incompatible or something, whether VK_DRIVER_FILES is set or not. straces led me to the same place. this should not be happening with or without user namespacing enabled.

I feel like I'm paraphrasing on this post more than I should, so I recorded some of the terminal session. I don't know how helpful is that.
https://asciinema.org/a/0YiRAzTcIHdSdhCB7fjHs8DiF

Last edited by gacki (2025-06-30 14:40:20)

seth · 2025-06-30 18:11:33

https://www.reddit.com/r/archlinux/comm … emdnspawn/
https://unix.stackexchange.com/question … -and-gpgpu

If nothing else goes to show that it's not a "you-problem"

gacki · 2025-06-30 23:29:52

added these lines to the unit file

[Service]
DevicePolicy=auto
DeviceAllow=char-input rw
DeviceAllow=char-drm rw
DeviceAllow=char-nvidia rw
DeviceAllow=char-nvidiactl rw
DeviceAllow=char-nvidia-modeset rw
DeviceAllow=char-nvidia-caps rw
DeviceAllow=char-nvidia-caps-imex-channels rw
DeviceAllow=char-nvidia-uvm rw
DeviceAllow=char-nvidia-nvlink rw
# why do i have to do this
DeviceAllow=/dev/dri/card1 rw
DeviceAllow=/dev/dri/renderD128 rw
DeviceAllow=/dev/nvidia0 rw
DeviceAllow=/dev/nvidiactl rw
DeviceAllow=/dev/nvidia-uvm rw
DeviceAllow=/dev/nvidia-uvm-tools rw
DeviceAllow=/dev/nvidia-modeset rw
DeviceAllow=/dev/nvidia-caps* rw

and this is the whole nspawn file

[Exec]
Hostname=juno
ResolvConf=off
PrivateUsers=off

[Network]
Zone=embryo

[Files]
Bind=/dev/fuse
Bind=/dev/dri
Bind=/dev/shm
Bind=/dev/nvidia0
Bind=/dev/nvidiactl
Bind=/dev/nvidia-modeset
Bind=/dev/nvidia-caps
Bind=/dev/nvidia-uvm
Bind=/dev/nvidia-uvm-tools

and not to my surprise, it didn't have any effect.

[root@juno ~]# nvidia-smi
Failed to initialize NVML: GPU access blocked by the operating system
[root@juno ~]# vulkaninfo
ERROR: [Loader Message] Code 0 : loader_scanned_icd_add: Could not get 'vkCreateInstance' via 'vk_icdGetInstanceProcAddr' for ICD libGLX_nvidia.so.0
ERROR: [Loader Message] Code 0 : vkCreateInstance: Found no drivers!
Cannot create Vulkan instance.
This problem is often caused by a faulty installation of the Vulkan driver or attempting to use a GPU that does not support Vulkan.
ERROR at /usr/src/debug/vulkan-tools/Vulkan-Tools/vulkaninfo/./vulkaninfo.h:573:vkCreateInstance failed with ERROR_INCOMPATIBLE_DRIVER
[root@juno ~]#

at least it's good to know this is not a me-problem

Arch Linux

#1 2025-06-27 23:53:28

can't get nvidia to work within a systemd-nspawn container

#2 2025-06-28 08:35:05

Re: can't get nvidia to work within a systemd-nspawn container

#3 2025-06-28 16:55:34

Re: can't get nvidia to work within a systemd-nspawn container

#4 2025-06-29 07:51:19

Re: can't get nvidia to work within a systemd-nspawn container

#5 2025-06-29 21:39:42

Re: can't get nvidia to work within a systemd-nspawn container

#6 2025-06-30 05:37:40

Re: can't get nvidia to work within a systemd-nspawn container

#7 2025-06-30 18:11:33

Re: can't get nvidia to work within a systemd-nspawn container

#8 2025-06-30 23:29:52

Re: can't get nvidia to work within a systemd-nspawn container

Board footer