VA-API SIGSEGV with nvidia

gehngis · 2020-06-18 16:51:29

Hi,

I am trying to get Chiaki running with VA-API video acceleration, but it seems that when a program tries to use VA-API it crashes.

My GPU is an RTX 2070, I am using nvidia proprietary drivers and I have tried with both linux and linux-zen kernels.

I have installed:

nvidia-dkms 440.82-2
nvidia-utils 440.82-2
libva 2.7.1-1
libva-vdpau-driver 0.7.4-4

I have then tried to run:

VDPAU_DRIVER=nvidia LIBVA_DRIVER_NAME=vdpau chiaki

But it crashes with SIGSEGV. The stack trace looks like this:

#0  0x00007ffff1efa110 in XDisplayString () at /usr/lib/libX11.so.6
#1  0x00007fffe41138a7 in __vaDriverInit_1_4 () at /usr/lib/dri/vdpau_drv_video.so
#2  0x00007ffff203d028 in  () at /usr/lib/libva.so.2
#3  0x00007ffff20413af in vaInitialize () at /usr/lib/libva.so.2
#4  0x00007ffff631517e in  () at /usr/lib/libavutil.so.56
#5  0x00007ffff630d8bf in av_hwdevice_ctx_create () at /usr/lib/libavutil.so.56

Frame #6 is in Chiaki code.

The problem seems to be not specific to Chiaki as even VLC crashes by selecting "VA-API video decoder" in the "Codecs" preferences of VLC.

$ VDPAU_DRIVER=nvidia LIBVA_DRIVER_NAME=vdpau vlc -v video.mkv
VLC media player 3.0.11 Vetinari (revision 3.0.11-0-gdc0c5ced72)
[0000557098ae3670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/codec/libzvbi_plugin.so' (libzvbi.so.0: cannot open shared object file: No such file or directory)
[0000557098ae3670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/codec/libsdl_image_plugin.so' (libSDL_image-1.2.so.0: cannot open shared object file: No such file or directory)
[0000557098ae3670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/codec/libmpg123_plugin.so' (libmpg123.so.0: cannot open shared object file: No such file or directory)
[0000557098ae3670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/codec/libkate_plugin.so' (libtiger.so.5: cannot open shared object file: No such file or directory)
[0000557098ae3670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/codec/libtwolame_plugin.so' (libtwolame.so.0: cannot open shared object file: No such file or directory)
[0000557098ae3670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/video_output/libaa_plugin.so' (libaa.so.1: cannot open shared object file: No such file or directory)
[0000557098ae3670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/control/liblirc_plugin.so' (liblirc_client.so.0: cannot open shared object file: No such file or directory)
[0000557098ae3670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/access_output/libaccess_output_shout_plugin.so' (libshout.so.3: cannot open shared object file: No such file or directory)
[0000557098ae3670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/visualization/libgoom_plugin.so' (libgoom2.so.0: cannot open shared object file: No such file or directory)
[0000557098ae3670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/visualization/libprojectm_plugin.so' (libprojectM.so.3: cannot open shared object file: No such file or directory)
[0000557098ae3670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/services_discovery/libmicrodns_plugin.so' (libmicrodns.so.0: cannot open shared object file: No such file or directory)
[0000557098ae3670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/stream_out/libstream_out_chromaprint_plugin.so' (libchromaprint.so.1: cannot open shared object file: No such file or directory)
[0000557098ae3670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/access/libv4l2_plugin.so' (libzvbi.so.0: cannot open shared object file: No such file or directory)
[0000557098ae3670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/access/libnfs_plugin.so' (libnfs.so.13: cannot open shared object file: No such file or directory)
[0000557098ae3670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/access/liblinsys_sdi_plugin.so' (libzvbi.so.0: cannot open shared object file: No such file or directory)
[0000557098ae3670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/demux/libgme_plugin.so' (libgme.so.0: cannot open shared object file: No such file or directory)
[0000557098ae3670] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
[00007fa1b0d16c10] faad decoder warning: decoded zero sample
[00007fa1a409c7b0] vaapi_filters filter warning: Using SW chroma filter for 1920x1088 VAOP -> I420
fish: “VDPAU_DRIVER=nvidia LIBVA_DRIVE…” terminated by signal SIGSEGV (Address boundary error)

However, if I set VLC "Codecs" preferences to "Automatic" or "VDPAU video decoder", it works fine and I have the following output:

VLC media player 3.0.11 Vetinari (revision 3.0.11-0-gdc0c5ced72)
[0000559494da2670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/codec/libzvbi_plugin.so' (libzvbi.so.0: cannot open shared object file: No such file or directory)
[0000559494da2670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/codec/libsdl_image_plugin.so' (libSDL_image-1.2.so.0: cannot open shared object file: No such file or directory)
[0000559494da2670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/codec/libmpg123_plugin.so' (libmpg123.so.0: cannot open shared object file: No such file or directory)
[0000559494da2670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/codec/libkate_plugin.so' (libtiger.so.5: cannot open shared object file: No such file or directory)
[0000559494da2670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/codec/libtwolame_plugin.so' (libtwolame.so.0: cannot open shared object file: No such file or directory)
[0000559494da2670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/video_output/libaa_plugin.so' (libaa.so.1: cannot open shared object file: No such file or directory)
[0000559494da2670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/control/liblirc_plugin.so' (liblirc_client.so.0: cannot open shared object file: No such file or directory)
[0000559494da2670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/access_output/libaccess_output_shout_plugin.so' (libshout.so.3: cannot open shared object file: No such file or directory)
[0000559494da2670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/visualization/libgoom_plugin.so' (libgoom2.so.0: cannot open shared object file: No such file or directory)
[0000559494da2670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/visualization/libprojectm_plugin.so' (libprojectM.so.3: cannot open shared object file: No such file or directory)
[0000559494da2670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/services_discovery/libmicrodns_plugin.so' (libmicrodns.so.0: cannot open shared object file: No such file or directory)
[0000559494da2670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/stream_out/libstream_out_chromaprint_plugin.so' (libchromaprint.so.1: cannot open shared object file: No such file or directory)
[0000559494da2670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/access/libv4l2_plugin.so' (libzvbi.so.0: cannot open shared object file: No such file or directory)
[0000559494da2670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/access/libnfs_plugin.so' (libnfs.so.13: cannot open shared object file: No such file or directory)
[0000559494da2670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/access/liblinsys_sdi_plugin.so' (libzvbi.so.0: cannot open shared object file: No such file or directory)
[0000559494da2670] main libvlc warning: cannot load module `/usr/lib/vlc/plugins/demux/libgme_plugin.so' (libgme.so.0: cannot open shared object file: No such file or directory)
[0000559494da2670] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
[00007f5b40d16c10] faad decoder warning: decoded zero sample
[00007f5b2009c7b0] vaapi_filters filter warning: Using SW chroma filter for 1920x1088 VAOP -> I420
[00007f5b40c31fa0] avcodec decoder: Using NVIDIA VDPAU Driver Shared Library  440.82  Wed Apr  1 19:39:06 UTC 2020 for hardware decoding
[0000559494e760b0] pulse audio output warning: starting late (-13509 us)
[00007f5b24061840] main video output warning: picture is too late to be displayed (missing 26 ms)

Also here is the output of vainfo:

$ VDPAU_DRIVER=nvidia LIBVA_DRIVER_NAME=vdpau vainfo
vainfo: VA-API version: 1.7 (libva 2.7.1)
vainfo: Driver version: Splitted-Desktop Systems VDPAU backend for VA-API - 0.7.4
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileMPEG4Simple            : VAEntrypointVLD
      VAProfileMPEG4AdvancedSimple    : VAEntrypointVLD
      <unknown profile>               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD

I have been searching the Arch Wiki for hours, but now I do not not where to look to find a solution.

Lone_Wolf · 2020-06-19 11:14:06

Those outputs suggest you're missing many files that vlc needs.

please post the output of pacman -Qkk vlc .

V1del · 2020-06-19 11:22:39

Those look like optional packages and they should be irrelevant to the question at hand.

What I'm wondering as why you want to do this in the first place. From a quick googler chiaki should support VDPAU natively, as does VLC. Just use VDPAU instead of the VAAPI-over-VDPAU bridge that often runs into issues like this.

Lone_Wolf · 2020-06-19 11:30:57

Some may be optional, but atleast two of them are owned by the vlc package so should be present and appear to be missing.

$ pacman -Qo /usr/lib/vlc/plugins/codec/libmpg123_plugin.so
/usr/lib/vlc/plugins/codec/libmpg123_plugin.so is owned by vlc 3.0.11-1
$ pacman -Qo /usr/lib/vlc/plugins/codec/libkate_plugin.so
/usr/lib/vlc/plugins/codec/libkate_plugin.so is owned by vlc 3.0.11-1
$

gehngis · 2020-06-19 13:09:07

Lone_Wolf wrote:

Some may be optional, but atleast two of them are owned by the vlc package so should be present and appear to be missing.

libmpg123_plugin.so is owned by VLC, but I do have this file.
VLC does not complain that libmpg123_plugin.so does not exist, but complains that it cannot load libmpg123_plugin.so because libmpg123.so.0 (which is not owned by VLC) does not exist.

These are optional plugins and should not be an issue.

Anyway to satisfy your curiosity:

$ pacman -Qkk vlc
warning: vlc: /usr/lib/vlc/plugins/plugins.dat (Modification time mismatch)
warning: vlc: /usr/lib/vlc/plugins/plugins.dat (Size mismatch)
vlc: 1056 total files, 1 altered file

V1del wrote:

What I'm wondering as why you want to do this in the first place. From a quick googler chiaki should support VDPAU natively, as does VLC. Just use VDPAU instead of the VAAPI-over-VDPAU bridge that often runs into issues like this.

Yes VLC can use VDPAU directly, and has not issue unless I force VA-API. And the only reason I am doing that is to check whether it is a VA-API issue or a Chiaki issue.

To my knowledge Chiaki only supports VA-API.
Chiaki Settings dialog, the "Hardware decode method" can be set not "none" or "vaapi".
Looking into issues about HW acceleration, it seems only VA-API is done at this time and VDPAU might be done in the future.
https://github.com/thestr4ng3r/chiaki/i … -639521041

gehngis · 2020-06-20 09:53:40

I've continue to explore the issue.

Building libva-vdpau-driver in debug I was able to see that the issue is at the following line of code:

const char * const x11_dpy_name = XDisplayString(driver_data->x11_dpy);

The issue is that x11_dpy is NULL.
x11_dpy is a copy of the value set in ctx->native_dpy provided by libva to the driver.

So I built a debug version on libva, however with this new compiled version I had the following error when starting chiaki:

chiaki: symbol lookup error: /usr/lib/libavutil.so.56: undefined symbol: vaSetErrorCallback

libavutil.so.56 being owned by ffmeg, I rebuilt ffmpeg.

And now Chiaki does not crash anymore, but it still does not work.
Before it was crashing inside the call to av_hwdevice_ctx_create().
Now it fails at avcodec_get_hw_config() which is called before av_hwdevice_ctx_create(), i.e. av_hwdevice_ctx_create() is not even called now.

Gilis95 · 2020-10-10 06:52:23

I am running in mostly the same issue. Here it's the environment I am using:

GPU: GTX 1070 TI
driver: nvidia v455.28
kernel: ZEN 5.8.13
libva-vdpau-driver 0.7.4-4
libva-git 2.9.0.pre2.r3.g6220d0d-1

I succeed to reproduce it using xfreerdp:

 xfreerdp /u:${any_username} /d:${any_domain} /bpp:8 /gdi:sw /gfx /f -max-loop-time:1000 /v:${any_address}

This is using H264 codec, instead of VLC. That's the first difference, second is that I succeed to reproduce it with rebuilded 'ffmeg'. Here is the crash log:

                                                 Stack trace of thread 3873:
                                                 #0  0x00007fac22bd2615 raise (libc.so.6 + 0x3d615)
                                                 #1  0x00007fac2300323e n/a (libfreerdp2.so.2 + 0x2d23e)
                                                 #2  0x00007fac22bd26a0 __restore_rt (libc.so.6 + 0x3d6a0)
                                                 #3  0x00007fac2328f170 XDisplayString (libX11.so.6 + 0x2f170)
                                                 #4  0x00007fac1c17b8a7 __vaDriverInit_1_4 (vdpau_drv_video.so + 0x68a7)
                                                 #5  0x00007fac1dfd2c39 n/a (libva.so.2 + 0x3c39)
                                                 #6  0x00007fac1dfd8c4f vaInitialize (libva.so.2 + 0x9c4f)
                                                 #7  0x00007fac20c9d2de n/a (libavutil.so.56 + 0x2f2de)
                                                 #8  0x00007fac20c95baf av_hwdevice_ctx_create (libavutil.so.56 + 0x27baf)
                                                 #9  0x00007fac230d8fd6 n/a (libfreerdp2.so.2 + 0x102fd6)
                                                 #10 0x00007fac230d3906 h264_context_new (libfreerdp2.so.2 + 0xfd906)
                                                 #11 0x00007fac23066068 freerdp_client_codecs_prepare (libfreerdp2.so.2 + 0x90068)
                                                 #12 0x00007fac2306d63a n/a (libfreerdp2.so.2 + 0x9763a)
                                                 #13 0x00007fac23060aa4 freerdp_connect (libfreerdp2.so.2 + 0x8aaa4)
                                                 #14 0x000055e98002781d n/a (xfreerdp + 0x1d81d)
                                                 #15 0x00007fac22dd9732 n/a (libwinpr2.so.2 + 0x7b732)
                                                 #16 0x00007fac22b4c3e9 start_thread (libpthread.so.0 + 0x93e9)
                                                 #17 0x00007fac22c95293 __clone (libc.so.6 + 0x100293)

The only workaround for me is not exporting `LIBVA_DRIVER_NAME` and `VDPAU_DRIVER` and so not using hardware acceleration. Did you find any long term solution for the problem?

vredez · 2020-12-03 17:10:33

I ran into the same issue using xfreerdp. This seems to be a libva bug, a handle to the X11 display is not initialized correctly.
As a quickfix, I managed to patch libva and get xfreerdp running again:

diff --unified --recursive --text libva/va/va.c libva_patched/va/va.c
--- libva/va/va.c	2020-12-03 18:06:39.480335569 +0100
+++ libva_patched/va/va.c	2020-12-03 17:38:59.001928000 +0100
@@ -422,6 +422,8 @@
 static VAStatus va_openDriver(VADisplay dpy, char *driver_name)
 {
     VADriverContextP ctx = CTX(dpy);
+    ctx->native_dpy = dpy;
+
     VAStatus vaStatus = VA_STATUS_ERROR_UNKNOWN;
     char *search_path = NULL;
     char *saveptr;

I will file a bug to the libva project as soon as I've analyzed this further.

/edit: https://github.com/intel/libva/issues/479

/edit #2:
Further analysis revealed that this is actually a problem with the libva-vdpau-driver, which doesn't support DRM mode.
The problem could be addressed by freerdp if they offered a way to specify the VAAPI connection type: https://github.com/FreeRDP/FreeRDP/issues/6666

Last edited by vredez (2020-12-14 17:19:08)

Arch Linux

#1 2020-06-18 16:51:29

VA-API SIGSEGV with nvidia

#2 2020-06-19 11:14:06

Re: VA-API SIGSEGV with nvidia

#3 2020-06-19 11:22:39

Re: VA-API SIGSEGV with nvidia

#4 2020-06-19 11:30:57

Re: VA-API SIGSEGV with nvidia

#5 2020-06-19 13:09:07

Re: VA-API SIGSEGV with nvidia

#6 2020-06-20 09:53:40

Re: VA-API SIGSEGV with nvidia

#7 2020-10-10 06:52:23

Re: VA-API SIGSEGV with nvidia

#8 2020-12-03 17:10:33

Re: VA-API SIGSEGV with nvidia

Board footer