[SOLVED] Steam: Can't initialize vulkan in systemd-nspawn container

skyfoosooc · 2018-08-27 16:18:53

I was trying to run Windows games with the Steam Proton Beta. While things run fine on the host, Vulkan can't initialize in the systemd-nspawn container. My guess is that I haven't allowed access to some host hardware that Vulkan needs to work. I bind "/dev/dri" and "/dev/shm" as rw to the container and this gets most games working.

The container could be missing a package but vulkan-icd-loader and vulkan-radeon are both installed as well as updated packages for mesa.

Thanks for any help you can offer and let my know what relavent info I may have missed.

System info:

# uname -a
Linux flavanego 4.18.5-arch1-1-ARCH #1 SMP PREEMPT Fri Aug 24 12:48:58 UTC 2018 x86_64 GNU/Linux
# lspci | grep VGA
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT / Grenada XT [Radeon R9 290X/390X]

Error:

Fatal : VkResult is "ERROR_INITIALIZATION_FAILED" in /home/pgriffais/src/Vulkan/base/vulkanexamplebase.cpp at line 823
vulkandriverquery: /home/pgriffais/src/Vulkan/base/vulkanexamplebase.cpp:823: void VulkanExampleBase::initVulkan(): Assertion `res == VK_SUCCESS' failed.

Full Steam log [container] : https://pastebin.com/3vp1ihBr
Full Steam log [host]: https://pastebin.com/8FNDkWsJ

/etc/systemd/system/systemd-nspawn@.service.d/override.conf

[Service]
DeviceAllow=/dev/dri rw
DeviceAllow=/dev/shm rw
DeviceAllow=char-usb_device rwm 
DeviceAllow=char-input rwm 
DeviceAllow=char-alsa rwm 
ExecStart=
ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --machine=%I --directory=/var/lib/machinez/%I

/etc/systemd/nspawn/gaming.nspawn

[Exec]
Boot=1

[Files]
# video
Bind=/tmp/.X11-unix
Bind=/dev/dri
Bind=/dev/shm

# input
Bind=/dev/input

# pulseaudio
Bind=/run/user/1001/pulse:/run/user/host/pulse

# sound
Bind=/dev/snd

# files
Bind=/opt/steam
Bind=/opt/steam-win
Bind=/var/cache/pacman/pkg
Bind=/var/cache/aur

Last edited by skyfoosooc (2018-12-10 23:55:06)

lamarpavel · 2018-08-29 19:55:43

Same issue, slightly different hardware:

# lspci | grep VGA
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti PRO [Radeon HD 7950/8950 OEM / R9 280]

# vulkaninfo
==========
VULKANINFO
==========

Vulkan Instance Version: 1.1.82

/build/vulkan-tools/src/Vulkan-Tools/vulkaninfo/vulkaninfo.c:3339: failed with VK_ERROR_INITIALIZATION_FAILED

On the host system I get the proper output for the supported extensions with vulkaninfo. And to verify that it actually works, I've tested some Vulkan demos [1] on the host and can verify that they play correctly.

When starting the nspawn container I am binding /dev/dri as well.
I suspected it could be something about the environment not being exported. I only export "DISPLAY=:0" via --setenv and could imagine that some other environment is being required by the init routines.
Not sure where best to ask about this issue though, from my limited insight it looks like this could be related to any of the vulkan tools, the AMD driver, systemd or X11.

[1] https://github.com/SaschaWillems/Vulkan

Last edited by lamarpavel (2018-08-29 19:56:06)

lamarpavel · 2018-09-14 09:45:57

Update: I was partly working based on an assumption that turned out to be wrong:
Steam proton does work inside the container and it does not require Vulkan to run. What was missing was python2, which is a dependency for proton, but isn't listed as dependency or optional for the steam package.

I've confirmed proton working now with one of the games from Steams officially supported list.

That said, I still haven't managed to get Vulkan running inside the nspawn container, so the original problem of this thread is not solved.

sincomil · 2018-09-14 10:46:14

Proton is not required thing for running windows games. You can install winetricks, create new WINEPREFIX, install required dlls, including "dxvk" and freely run almost any game requiring DX10/11 you will get exactly same performance as with Steam/Proton

Last edited by sincomil (2018-09-14 10:46:48)

lamarpavel · 2018-09-14 10:54:24

sincomil wrote:

Proton is not required thing for running windows games. You can install winetricks, create new WINEPREFIX, install required dlls, including "dxvk" and freely run almost any game requiring DX10/11 you will get exactly same performance as with Steam/Proton

Yes, indeed. But the core of the issue is that no Vulkan context can be acquired from the installed drivers while in a systemd-nspawn container even though this works on the host system. Regardless of proton.
I have another container for wine (separate from the container for steam) and there the same problem exists: VK_ERROR_INITIALIZATION_FAILED

progandy · 2018-09-14 11:02:43

I think you may have to enable read/write permissions for the graphics card in /sys/devices and /sys/class/drm.

lamarpavel · 2018-09-14 11:28:51

progandy wrote:

I think you may have to enable read/write permissions for the graphics card in /sys/devices and /sys/class/drm.

I've bound /dev/dri, which shows up in the container with the same permissions for card0, card1, renderD128 and renderD129. According to /dev/dri/by-path these are pci-0000:00:02.0-card etc, which are available in the container under /sys/devices/pci0000:00/0000:00:02.0/ with the same permissions (and path) as on the host. A link to /sys/class/drm is also available with the same permissions as on the host.

The only difference I could find was that /dev/dri/renderD128 (and renderD129) are associated with group 993, which is render on the host but doesn't exist in the container. I'll investigate this further as soon as I have the time.
Thanks for your help so far.

lamarpavel · 2018-09-15 08:11:01

Alright, I've fixed the discrepancy regarding group render (gid 993) between host and container. To verify I've diff'ed `tree -ug /sys/devices` (same with /dev/dri and /sys/class/drm) and the ouput is identical.

Vulkaninfo still reports the following in the container.

[gaming@gaming ~]$ vulkaninfo
==========
VULKANINFO
==========

Vulkan Instance Version: 1.1.82

/build/vulkan-tools/src/Vulkan-Tools/vulkaninfo/vulkaninfo.c:3339: failed with VK_ERROR_INITIALIZATION_FAILED

lamarpavel · 2018-09-15 09:51:40

Looking at the code of vulkaninfo.c I will make a careful estimate that it has little to do with drivers. The problem occurs during instance creation of a Vulkan context, somewhere around the parts where a drawing surface or window is requested from the display server.
Based on this (again, careful estimate) the problem lies with the communication to an X11 component such as XCB or XLIB. I know that Vulkan has a more intimate relationship with the memory of the GPU and I can imagine this extends to mechanisms for taking control of the drawing surface. Maybe the X server in it's current state does not allow this kind of access to more than one client and that slot is already filled by the host system such that the requests coming from the nspawn container can't acquire any further instances of drawing surfaces.

I don't know much about xorg, though. Is it possible to start a new instance of X11 from within a systemd-container and then use that rather than relaying the $DISPLAY of the host? Or can I somehow tweak how an existing X server treats new clients?

tinywrkbee · 2018-11-27 16:44:06

https://github.com/systemd/systemd/issues/10960

progandy · 2018-11-27 17:40:43

You could try to run all applications inside your nspawn container on the render-node instead of the full graphics card interface.

DRI_PRIME=/dev/dri/renderD128 ...
DRI_PRIME=/dev/dri/renderD129 ...

tinywrkbee · 2018-11-27 18:21:13

You could try to run all applications inside your nspawn container on the render-node instead of the full graphics card interface.

The problem is that an application which uses Vulkan (I only tried vulkaninfo and vkcube) is trying to open the render node with O_RDWR|O_CLOEXEC flags when calling vkEnumeratePhysicalDevices, but the default behavior for systemd-nspawn is to limit access to devices by employing cgroup.

By adding the property DeviceAllow with the render node device as its value the cgroup limit is being bypassed.

I reported this upstream as I thought maybe it's not an intended behavior, but I guess I was wrong.

Last edited by tinywrkbee (2018-11-27 22:03:58)

randomguy · 2018-11-28 02:53:29

tinywrkbee wrote:

https://github.com/systemd/systemd/issues/10960

Interesting and "fair enough" indeed, when thinking about the actual implications when allowing this access to hardware.

lamarpavel · 2018-11-28 13:57:29

randomguy wrote:

tinywrkbee wrote:
https://github.com/systemd/systemd/issues/10960
Interesting and "fair enough" indeed, when thinking about the actual implications when allowing this access to hardware.

A process running directly on the host has this access anyway, so it would be nice to be able to use a container to limit as much access as possible and bite the bullet with access that has to be given. You shouldn't consider steam properly castrated just because it runs in an nspawn container anyway; the access it has to your xorg server is already a security concern.

So what's the way forward? Wait for the virtualization to be implemented in the kernel as poettering commented, or change how the vulkan lib (maybe the driver itself?) accesses hardware, or not use an nspawn contrainer at all?

I'd dread to go back to running steam under a different user, the way I did before I started to use nspawn...

tinywrkbee · 2018-11-28 14:15:17

lamarpavel wrote:

You shouldn't consider steam properly castrated just because it runs in an nspawn container anyway; the access it has to your xorg server is already a security concern.

The host can run a Wayland session, I tested vkcuke which default to XWayland, it works, no idea about Steam.

lamarpavel wrote:

or change how the vulkan lib (maybe the driver itself?) accesses hardware,

It seems like this is the driver (mesa).
Worth noting that I notice similar issue when running QT5 Wayland apps is systemd-container, they try to get rw access to /dev/dri/card0, and I believe these are not Vulkan (Calibe, qutebrowser with qt5-webkit backend). No such issue when those QT5 apps are using X/XWayland and I also didn't noticed this with the GTK apps I tried nor with Vulkan apps running as native Wayland clients (see).

lamarpavel · 2018-11-30 07:38:50

tinywrkbee wrote:

lamarpavel wrote:
You shouldn't consider steam properly castrated just because it runs in an nspawn container anyway; the access it has to your xorg server is already a security concern.
The host can run a Wayland session, I tested vkcuke which default to XWayland, it works, no idea about Steam.

I don't really know much about wayland so I can't comment on that with confidence. I would guess, though, that passing on access to the wayland server(?) to create windows also poses a security risk in terms of information bleed - just like with xorg.

tinywrkbee wrote:

lamarpavel wrote:
or change how the vulkan lib (maybe the driver itself?) accesses hardware,
It seems like this is the driver (mesa).
Worth noting that I notice similar issue when running QT5 Wayland apps is systemd-container, they try to get rw access to /dev/dri/card0, and I believe these are not Vulkan (Calibe, qutebrowser with qt5-webkit backend). No such issue when those QT5 apps are using X/XWayland and I also didn't noticed this with the GTK apps I tried nor with Vulkan apps running as native Wayland clients (see).

Interesting, I'll need to try this. I've got Sasha Willems Vulkan examples running on my host but never tried them in the nspawn container because both vulkaninfo and steam already tell me they can't access Vulkan.

tinywrkbee · 2018-12-05 10:42:09

I can now confirm that Steam works correctly with Vulkan in a systemd-nspawn container.
I tried Dota 2 with the Vulkan DLC, added -vulkan to the launch options and the game's video settings menu shows that Vulkan API is being used.
I also tested Einar, that's a DX11 game so DXVK is used, it runs fine though terribly slow with Intel iGPU.

skyfoosooc · 2018-12-10 23:55:25

Adding the following switch works for me as well.

--property=DeviceAllow="/dev/dri/renderD128"

Though not exactly pretty; it solves the problem in the first post. Thanks for all the info and solution.

Arch Linux

#1 2018-08-27 16:18:53

[SOLVED] Steam: Can't initialize vulkan in systemd-nspawn container

#2 2018-08-29 19:55:43

Re: [SOLVED] Steam: Can't initialize vulkan in systemd-nspawn container

#3 2018-09-14 09:45:57

Re: [SOLVED] Steam: Can't initialize vulkan in systemd-nspawn container

#4 2018-09-14 10:46:14

Re: [SOLVED] Steam: Can't initialize vulkan in systemd-nspawn container

#5 2018-09-14 10:54:24

Re: [SOLVED] Steam: Can't initialize vulkan in systemd-nspawn container

#6 2018-09-14 11:02:43

Re: [SOLVED] Steam: Can't initialize vulkan in systemd-nspawn container

#7 2018-09-14 11:28:51

Re: [SOLVED] Steam: Can't initialize vulkan in systemd-nspawn container

#8 2018-09-15 08:11:01

Re: [SOLVED] Steam: Can't initialize vulkan in systemd-nspawn container

#9 2018-09-15 09:51:40

Re: [SOLVED] Steam: Can't initialize vulkan in systemd-nspawn container

#10 2018-11-27 16:44:06

Re: [SOLVED] Steam: Can't initialize vulkan in systemd-nspawn container

#11 2018-11-27 17:40:43

Re: [SOLVED] Steam: Can't initialize vulkan in systemd-nspawn container

#12 2018-11-27 18:21:13

Re: [SOLVED] Steam: Can't initialize vulkan in systemd-nspawn container

#13 2018-11-28 02:53:29

Re: [SOLVED] Steam: Can't initialize vulkan in systemd-nspawn container

#14 2018-11-28 13:57:29

Re: [SOLVED] Steam: Can't initialize vulkan in systemd-nspawn container

#15 2018-11-28 14:15:17

Re: [SOLVED] Steam: Can't initialize vulkan in systemd-nspawn container

#16 2018-11-30 07:38:50

Re: [SOLVED] Steam: Can't initialize vulkan in systemd-nspawn container

#17 2018-12-05 10:42:09

Re: [SOLVED] Steam: Can't initialize vulkan in systemd-nspawn container

#18 2018-12-10 23:55:25

Re: [SOLVED] Steam: Can't initialize vulkan in systemd-nspawn container

Board footer