CUDA is not working in Nextcloud but works in any other program.

Woozy · 2023-01-11 04:17:38

I hope I picked the correct forum for this question, long time user, first time posting. I recently setup Nextcloud and I am trying to get hardware transcoding with nvenc working in "Memories" and CUDA accelerated Tensorflow working in "Recognize". However despite the fact that CUDA seems to be working just fine, my GPU seems to be invisible to Nextcloud or any processes spawned from Nextcloud.

Server Info:
- KVM Hypervisor
- Minimal Arch Linux
- 6 CPU Cores
- 8GB RAM
- Nvidia Quadro P400
- Nextcloud 25.0.2 (From Arch Repo)
- Apache 2.4.54
- PHP 8.1
- PostgreSQL 14 (Docker on Host)
- Redis 7.0.7 (Docker on Host)
- Nvidia Driver 525.78.01
- CUDA Version 12.0
- ffmpeg-cuda (From AUR)
- python-tensorflow-cuda (From AUR)

NVENC:

The Memories app uses go-vod to transcode video files to HLS streams using ffmpeg.

I have a video I recorded at work that I used to test nvenc with in the terminal and it worked perfectly. I was monitoring

nvidia-smi

and during the test ffmpeg was in fact using my GPU. Below is the command & output from the test run.

ffmpeg -i TEST.MOV -vcodec hevc_nvenc -acodec copy TEST-OUT.mkv

ffmpeg version n5.1.2 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 12.2.0 (GCC)
  configuration: --prefix=/usr --extra-cflags=-I/opt/cuda/include --extra-ldflags=-L/opt/cuda/lib64 --nvccflags='-gencode arch=compute_52,code=sm_52 -O2' --disable-debug --disa
ble-static --disable-stripping --enable-amf --enable-avisynth --enable-nonfree --enable-cuda-nvcc --enable-libnpp --enable-lto --enable-fontconfig --enable-gmp --enable-gnutls 
--enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --en
able-libgsm --enable-libiec61883 --enable-libjack --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopen
jpeg --enable-libopus --enable-libpulse --enable-librav1e --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libthe
ora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2
 --enable-libxvid --enable-libzimg --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-version3 --enable-vulkan
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'TEST.MOV':
  Metadata:
    major_brand     : qt  
    minor_version   : 0
    compatible_brands: qt  
    creation_time   : 2022-12-31T04:06:25.000000Z
    com.apple.quicktime.location.accuracy.horizontal: 16.487849
    com.apple.quicktime.location.ISO6709: +29.3197-081.1113+008.685/
    com.apple.quicktime.make: Apple
    com.apple.quicktime.model: iPhone SE (2nd generation)
    com.apple.quicktime.software: 16.1.1
    com.apple.quicktime.creationdate: 2022-12-30T23:06:25-0500
  Duration: 00:01:20.59, start: 0.000000, bitrate: 59708 kb/s
  Stream #0:0[0x1](und): Video: hevc (Main) (hvc1 / 0x31637668), yuv420p(tv, bt709), 3840x2160, 59423 kb/s, 59.99 fps, 60 tbr, 600 tbn (default)
    Metadata:
      creation_time   : 2022-12-31T04:06:25.000000Z
      handler_name    : Core Media Video
      vendor_id       : [0][0][0][0]
      encoder         : HEVC
    Side data:
      displaymatrix: rotation of -90.00 degrees
  Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 195 kb/s (default)
    Metadata:
      creation_time   : 2022-12-31T04:06:25.000000Z
      handler_name    : Core Media Audio
      vendor_id       : [0][0][0][0]
  Stream #0:2[0x3](und): Data: none (mebx / 0x7862656D) (default)
    Metadata:
      creation_time   : 2022-12-31T04:06:25.000000Z
      handler_name    : Core Media Metadata
  Stream #0:3[0x4](und): Data: none (mebx / 0x7862656D) (default)
    Metadata:
      creation_time   : 2022-12-31T04:06:25.000000Z
      handler_name    : Core Media Metadata
  Stream #0:4[0x5](und): Data: none (mebx / 0x7862656D), 69 kb/s (default)
    Metadata:
      creation_time   : 2022-12-31T04:06:25.000000Z
      handler_name    : Core Media Metadata
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> hevc (hevc_nvenc))
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
Output #0, matroska, to 'TEST-OUT.mkv':
  Metadata:
    major_brand     : qt  
    minor_version   : 0
    compatible_brands: qt  
    com.apple.quicktime.creationdate: 2022-12-30T23:06:25-0500
    com.apple.quicktime.location.accuracy.horizontal: 16.487849
    com.apple.quicktime.location.ISO6709: +29.3197-081.1113+008.685/
    com.apple.quicktime.make: Apple
    com.apple.quicktime.model: iPhone SE (2nd generation)
    com.apple.quicktime.software: 16.1.1
    encoder         : Lavf59.27.100
  Stream #0:0(und): Video: hevc (Main), yuv420p(tv, bt709, progressive), 2160x3840, q=2-31, 2000 kb/s, 60 fps, 1k tbn (default)
    Metadata:
      creation_time   : 2022-12-31T04:06:25.000000Z
      handler_name    : Core Media Video
      vendor_id       : [0][0][0][0]
      encoder         : Lavc59.37.100 hevc_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 4000000 vbv_delay: N/A
      displaymatrix: rotation of -0.00 degrees
  Stream #0:1(und): Audio: aac (LC) ([255][0][0][0] / 0x00FF), 44100 Hz, stereo, fltp, 195 kb/s (default)
    Metadata:
      creation_time   : 2022-12-31T04:06:25.000000Z
      handler_name    : Core Media Audio
      vendor_id       : [0][0][0][0]
frame= 4835 fps= 25 q=42.0 Lsize=   28321kB time=00:01:20.59 bitrate=2878.6kbits/s speed=0.413x    
video:26340kB audio:1922kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.208392%

However any time I try to play the exact same file in Memeories with nvenc enabled all I get is a black window and the errors below in the go-vod log file.

2023/01/09 19:29:39 ffmpeg-error: [AVHWDeviceContext @ 0x56423212bcc0] cu->cuInit(0) failed -> CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2023/01/09 19:29:39 ffmpeg-error: Device creation failed: -542398533.
2023/01/09 19:29:39 ffmpeg-error: [hevc @ 0x56423212b480] No device available for decoder: device type cuda needed for codec hevc.
2023/01/09 19:29:39 ffmpeg-error: Device setup failed for decoder on input stream #0:0 : Generic error in an external library

Tensorflow:

This is used by the Recognize app for facial recognition. The issue with Tensorflow is oddly similar to the issue with nvenc which is why I'm posting them together. Recognize has a script in the apps directory of Nextcloud that can be ran to check if your GPU is accessible, if I manually run this script in the terminal it would appear - at least to me - that it is in fact detected and working. Command & output below;

node /var/lib/nextcloud/apps/recognize/src/test_gputensorflow.js

2023-01-09 21:43:42.929972: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the
 following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-01-09 21:43:43.724276: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least
 one NUMA node, so returning NUMA node zero
2023-01-09 21:43:43.842613: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least
 one NUMA node, so returning NUMA node zero
2023-01-09 21:43:43.842889: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least
 one NUMA node, so returning NUMA node zero
2023-01-09 21:43:44.947881: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least
 one NUMA node, so returning NUMA node zero
2023-01-09 21:43:44.948107: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least
 one NUMA node, so returning NUMA node zero
2023-01-09 21:43:44.948282: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least
 one NUMA node, so returning NUMA node zero
2023-01-09 21:43:44.948607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1621 MB memory:  -> device
: 0, name: Quadro P400, pci bus id: 0000:07:00.0, compute capability: 6.1

It appears to me as if my card is detected. That is the correct GPU info at the bottom, and I see the word "successful" an awful lot. However when I go into the Admin settings for Recognize and enable GPU support I get the following error on the page.

Successfully loaded libtensorflow in Node.js, but couldn't load GPU. Make sure CUDA Toolkit and cuDNN are installed and accessible, or turn off GPU mode.

I did check to make sure that the cuda libraries are located in;

 /etc/ld.so.conf.d

inside is a cuda.conf file with the following contents;

/opt/cuda/lib64
/opt/cuda/nvvm/lib64
/opt/cuda/extras/CUPTI/lib64

Permissions

Since this almost seems like a permissions issue I thought I would go ahead and add the relevant info.

Nextcloud itself runs as user "nextcloud", as does "php-fpm". it would also appear that processes spawned by Nextcloud are ran as this user.

id -nG nextcloud
nextcloud

Below is the go-vod bin taken from "htop"

 PID.    USER     PRI  NI  VIRT   RES   SHR  S CPU% MEM%  TIME+   Command  
1990  nextcloud   20   0   698M 14436  3992  S  0.0  0.1  0:00.09   /var/lib/nextcloud/apps/memories/exiftool-bin/go-vod-amd64

Apache itself runs as user "http".

id -nG http
http

Neither users are part of any groups other than their own.

Below is any directory permissions I thought might be relevant.

Directories:

GPU:

ls -lh /dev/dri/card1
crwxrwxrwx+ 1 root video 226, 1 Jan 10 07:50 /dev/dri/card1

CUDA lib:

ls -lh /opt/cuda
total 164K
drwxr-xr-x 3 root root 4.0K Jan  8 23:25 bin
-rw-r--r-- 1 root root   25 Oct  5 14:19 cublas_version.txt
-rw-r--r-- 1 root root  79K Oct  5 14:19 CUDA_Toolkit_Release_Notes.txt
-rw-r--r-- 1 root root  160 Oct  5 14:19 DOCS
-rw-r--r-- 1 root root  61K Oct  5 14:19 EULA.txt
drwxr-xr-x 6 root root  113 Jan  8 23:25 extras
drwxr-xr-x 8 root root  113 Jan  8 23:25 gds
drwxr-xr-x 2 root root   40 Jan  8 23:25 gds-11.8
drwxr-xr-x 3 root root   17 Jan  8 23:25 gds-tools
lrwxrwxrwx 1 root root   28 Oct  5 14:19 include -> targets/x86_64-linux/include
drwxr-xr-x 5 root root   67 Jan  8 23:25 integration
lrwxrwxrwx 1 root root   24 Oct  5 14:19 lib64 -> targets/x86_64-linux/lib
drwxr-xr-x 3 root root   21 Jan  8 23:25 nvml
drwxr-xr-x 7 root root   85 Jan  8 23:25 nvvm
-rw-r--r-- 1 root root  524 Oct  5 14:19 README
drwxr-xr-x 3 root root   17 Jan  8 23:25 share
drwxr-xr-x 2 root root 4.0K Jan  8 23:25 src
drwxr-xr-x 3 root root   26 Jan  8 23:25 targets
drwxr-xr-x 2 root root   43 Jan  8 23:25 tools
drwxr-xr-x 4 root root   30 Jan  8 23:25 usr

GO-VOD tmp dir:

ls -lh /nc-mem-tmp/
total 8.0K
drwxr-xr-x 3 nextcloud nextcloud   36 Jan 10 18:51 go-vod
-rw-r--r-- 1 nextcloud nextcloud 7.5K Jan 10 18:55 go-vod.log

GO-VOD executable:

ls -lh /var/lib/nextcloud/apps/memories/exiftool-bin
total 41M
drwxr-xr-x 6 nextcloud http 4.0K Jan  9 08:18 exiftool
-rw-r--r-- 1 nextcloud http 6.9M Jan  9 08:18 exiftool-aarch64-glibc
-rw-r--r-- 1 nextcloud http 9.2M Jan  9 08:18 exiftool-aarch64-musl
-rw-r--r-- 1 nextcloud http 7.2M Jan  9 08:18 exiftool-amd64-glibc
-rw-r--r-- 1 nextcloud http 8.7M Jan  9 08:18 exiftool-amd64-musl
-rw-r--r-- 1 nextcloud http 4.5M Jan  9 08:18 go-vod-aarch64
-rwxr-xr-x 1 nextcloud http 4.7M Jan  9 08:18 go-vod-amd64

FFMPEG:

ls -lh /usr/bin/ffmpeg
-rwxr-xr-x 1 root root 275K Jan  8 23:37 /usr/bin/ffmpeg

ls -lh /usr/bin/ffprobe
-rwxr-xr-x 1 root root 167K Jan  8 23:37 /usr/bin/ffprobe

I'm not sure if there is much more to add, I'm really hoping I can get this working and any help would be appreciated.

pig2014 · 2023-01-14 14:37:51

The systemd service "php-fpm.service" (or so) limits the access to graphic card(/dev/nvidia*).
To solve the issue, edit php-fpm.service and add one line to the [Service] unit:

PrivateDevices=false

And everything about CUDA would be fine.
I don't know if it works with NVENC, but I think they are the same.

Arch Linux

#1 2023-01-11 04:17:38