Can't get hardware encoding to work on new amd gpu

agurkmeister · 2024-03-03 14:38:06

So I bought an RX7800xt a few days ago and I eventually wanted to try hardware AV1 encoding as I've never been able to before. Then I notice I'm not able to do any kind of hardware encoding at all on neither OBS nor handbrake.
The issue isn't that I get some error every time I try to encode it. Basically I just don't get the options for hardware encoding at all only options for software encoding like SVT-AV1, x264/5 and stuff.
I looked at the AMDGPU article on the arch wiki and did everything it told me and I also had a long conversation with ChatGPT-4 on the subject but I still don't seem to be able to get it working no matter what.
So actually directly asking the arch community is now my last resort before I just give up.

Here are some outputs of some commands ChatGPT asked me to type that seem relevant:

┌─[~][]
└─▪ glxinfo | grep "OpenGL renderer"
OpenGL renderer string: AMD Radeon RX 7800 XT (radeonsi, navi32, LLVM 16.0.6, DRM 3.57, 6.7.7-arch1-1)

┌─[~][]
└─▪ vainfo
Trying display: wayland
Trying display: x11
vainfo: VA-API version: 1.20 (libva 2.20.1)
vainfo: Driver version: Mesa Gallium driver 24.0.2-arch1.1 for AMD Radeon RX 7800 XT (radeonsi, navi32, LLVM 16.0.6, DRM 3.57, 6.7.7-arch1-1)
vainfo: Supported profile and entrypoints
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointEncSlice
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileVP9Profile0 : VAEntrypointVLD
VAProfileVP9Profile2 : VAEntrypointVLD
VAProfileAV1Profile0 : VAEntrypointVLD
VAProfileAV1Profile0 : VAEntrypointEncSlice
VAProfileNone : VAEntrypointVideoProc

┌─[~][]
└─▪ vdpauinfo
display: :0 screen: 0
API version: 1
Information string: G3DVL VDPAU Driver Shared Library version 1.0

Video surface:

name width height types
-------------------------------------------
420 16384 16384 NV12 YV12
422 16384 16384 UYVY YUYV
444 16384 16384 Y8U8V8A8 V8U8Y8A8
420_16 16384 16384 P010 P016
422_16 16384 16384
444_16 16384 16384

Decoder capabilities:

name level macbs width height
----------------------------------------------------
MPEG1 --- not supported ---
MPEG2_SIMPLE --- not supported ---
MPEG2_MAIN --- not supported ---
H264_BASELINE 52 65536 4096 4096
H264_MAIN 52 65536 4096 4096
H264_HIGH 52 65536 4096 4096
VC1_SIMPLE --- not supported ---
VC1_MAIN --- not supported ---
VC1_ADVANCED --- not supported ---
MPEG4_PART2_SP --- not supported ---
MPEG4_PART2_ASP --- not supported ---
DIVX4_QMOBILE --- not supported ---
DIVX4_MOBILE --- not supported ---
DIVX4_HOME_THEATER --- not supported ---
DIVX4_HD_1080P --- not supported ---
DIVX5_QMOBILE --- not supported ---
DIVX5_MOBILE --- not supported ---
DIVX5_HOME_THEATER --- not supported ---
DIVX5_HD_1080P --- not supported ---
H264_CONSTRAINED_BASELINE 0 65536 4096 4096
H264_EXTENDED --- not supported ---
H264_PROGRESSIVE_HIGH --- not supported ---
H264_CONSTRAINED_HIGH --- not supported ---
H264_HIGH_444_PREDICTIVE --- not supported ---
VP9_PROFILE_0 --- not supported ---
VP9_PROFILE_1 --- not supported ---
VP9_PROFILE_2 --- not supported ---
VP9_PROFILE_3 --- not supported ---
HEVC_MAIN 186 139264 8192 4352
HEVC_MAIN_10 186 139264 8192 4352
HEVC_MAIN_STILL --- not supported ---
HEVC_MAIN_12 --- not supported ---
HEVC_MAIN_444 --- not supported ---
HEVC_MAIN_444_10 --- not supported ---
HEVC_MAIN_444_12 --- not supported ---
AV1_MAIN --- not supported ---
AV1_HIGH --- not supported ---
AV1_PROFESSIONAL --- not supported ---

Output surface:

name width height nat types
----------------------------------------------------
B8G8R8A8 16384 16384 y NV12 YV12 UYVY YUYV Y8U8V8A8 V8U8Y8A8 P010 P016 A8I8 I8A8
R8G8B8A8 16384 16384 y NV12 YV12 UYVY YUYV Y8U8V8A8 V8U8Y8A8 P010 P016 A8I8 I8A8
R10G10B10A2 16384 16384 y NV12 YV12 UYVY YUYV Y8U8V8A8 V8U8Y8A8 P010 P016 A8I8 I8A8
B10G10R10A2 16384 16384 y NV12 YV12 UYVY YUYV Y8U8V8A8 V8U8Y8A8 P010 P016 A8I8 I8A8

Bitmap surface:

name width height
------------------------------
B8G8R8A8 16384 16384
R8G8B8A8 16384 16384
R10G10B10A2 16384 16384
B10G10R10A2 16384 16384
A8 16384 16384

Video mixer:

feature name sup
------------------------------------
DEINTERLACE_TEMPORAL y
DEINTERLACE_TEMPORAL_SPATIAL -
INVERSE_TELECINE -
NOISE_REDUCTION y
SHARPNESS y
LUMA_KEY y
HIGH QUALITY SCALING - L1 y
HIGH QUALITY SCALING - L2 -
HIGH QUALITY SCALING - L3 -
HIGH QUALITY SCALING - L4 -
HIGH QUALITY SCALING - L5 -
HIGH QUALITY SCALING - L6 -
HIGH QUALITY SCALING - L7 -
HIGH QUALITY SCALING - L8 -
HIGH QUALITY SCALING - L9 -

parameter name sup min max
-----------------------------------------------------
VIDEO_SURFACE_WIDTH y 48 4096
VIDEO_SURFACE_HEIGHT y 48 4096
CHROMA_TYPE y
LAYERS y 0 4

attribute name sup min max
-----------------------------------------------------
BACKGROUND_COLOR y
CSC_MATRIX y
NOISE_REDUCTION_LEVEL y 0.00 1.00
SHARPNESS_LEVEL y -1.00 1.00
LUMA_KEY_MIN_LUMA y
LUMA_KEY_MAX_LUMA y

agurkmeister · 2024-03-03 19:33:36

For anyone coming here with the same issue this is what I did to solve the problem:

1 > Installed "amdgpu-pro-installer" from the AUR: https://aur.archlinux.org/pkgbase/amdgpu-pro-installer (Didn't work when using yay so had to install it via makepkg)
2 > For OBS support I was told to install this package from the AUR: https://aur.archlinux.org/packages/obs-studio-amf
3 > Rebooted
4 > Removed Handbrake and OBS that I had installed as Flatpaks
5 > Installed handbrake-git and obs-studio-git from the AUR: Handbrake: https://aur.archlinux.org/packages/handbrake-git OBS: https://aur.archlinux.org/packages/obs-studio-git
6 > Magically it just decided to work.

Note that hardware encoding AV1 with handbrake just crashes my desktop. Doesn't necessarily mean it'll happen to you but I'd say be ready to hold the power button down for 5 seconds. Also I'm on gnome, which might be related to the problem. AV1 encoding seems to work fine in OBS however and I can also do it by just using ffmpeg directly from the terminal.

kokoko3k · 2024-03-03 19:40:10

> hold the power button down for 5 seconds

Really? Weird!
Hardware reset switch did not work either?

agurkmeister · 2024-03-04 08:20:24

kokoko3k wrote:

> hold the power button down for 5 seconds
Really? Weird!
Hardware reset switch did not work either?

Well basically it'd get stuck on 0.17% completion and suddenly my screen would just go black like my GPU just crashed or something. Turning the monitor on and off didn't work. Pressing Ctrl+Alt+F4 to get to the tty also didn't work. So basically I just resorted to holding down the power button until my PC turned off and decided welp not gonna do that again.

Nikolai5 · 2024-03-04 11:47:22

I have an AMD 7800XT and I'm running OBS with AV1 and HEVC hardware accelerated encoding just fine via VAAPI.

I'm using the main repo version of OBS and not using the pro drivers. I don't know who told you to install the AUR version but I've found that to not be required, just change the the output options menu to "Advanced" and you get additional options for encoding which include vaapi AMD via Ffmpeg.

I can't speak for handbrake, but I assume it just uses preset ffmpeg profiles. I do know that running

ffmpeg -vaapi_device /dev/dri/renderD128 -i inputfile.mp4 -vf 'format=nv12,hwupload' -vcodec av1_vaapi av1.mp4

works fine for me.

I'm using mesa and friends packages alongside "libva-mesa-driver" for hardware acceleration. AMD Pro might be necessary for some things, but for OBS and Firefox decoding it's not necessary.

agurkmeister · 2024-03-04 12:50:18

Nikolai5 wrote:

I have an AMD 7800XT and I'm running OBS with AV1 and HEVC hardware accelerated encoding just fine via VAAPI.
I'm using the main repo version of OBS and not using the pro drivers. I don't know who told you to install the AUR version but I've found that to not be required, just change the the output options menu to "Advanced" and you get additional options for encoding which include vaapi AMD via Ffmpeg.
I can't speak for handbrake, but I assume it just uses preset ffmpeg profiles. I do know that running
ffmpeg -vaapi_device /dev/dri/renderD128 -i inputfile.mp4 -vf 'format=nv12,hwupload' -vcodec av1_vaapi av1.mp4
works fine for me.
I'm using mesa and friends packages alongside "libva-mesa-driver" for hardware acceleration. AMD Pro might be necessary for some things, but for OBS and Firefox decoding it's not necessary.

I don't know what to tell you. The steps I wrote above were just what lead me to having a functional setup. I don't know exactly what I did to solve the problem but something within the steps above worked.
The amdgpu-pro driver shouldn't be necessary nor should the git versions of both programs but it didn't work when I just installed handbrake and obs through the official repo nor via flatpak. Blender also immediately crashes when rendering with lighting using HIP on the official repo package. Installing the steam version of it works fine though.

Also a few things worth noting:
- I did try to switch OBS to the Advanced output option and I did get the options for vaapi but for some reason when I started recording I just immediately got an error that ChatGPT said was somehow related to drivers.
- I installed OBS and Handbrake on a clean Arch install with Gnome on it. I'm a newbie so I just installed it via the "archinstall" script and I selected AMD / ATI as the drivers for my card. idk if that method of installation somehow caused any issues. I had the option to pick "all open-source".
- I also tried installing Manjaro and Linux Mint as I wasn't sure if I maybe just had something configured wrong that a more "pre set up" distro would have configured by default. Manjaro didn't solve the problem and Mint was just a complete failure not worth going into detail about.

Nikolai5 · 2024-03-04 13:17:14

agurkmeister wrote:

Also a few things worth noting:
- I did try to switch OBS to the Advanced output option and I did get the options for vaapi but for some reason when I started recording I just immediately got an error that ChatGPT said was somehow related to drivers.
- I installed OBS and Handbrake on a clean Arch install with Gnome on it. I'm a newbie so I just installed it via the "archinstall" script and I selected AMD / ATI as the drivers for my card. idk if that method of installation somehow caused any issues. I had the option to pick "all open-source".
- I also tried installing Manjaro and Linux Mint as I wasn't sure if I maybe just had something configured wrong that a more "pre set up" distro would have configured by default. Manjaro didn't solve the problem and Mint was just a complete failure not worth going into detail about.

I also used archinstall but I selected AMD open source as the driver option and then manually checked that everything was installed from the Arch Wiki AMDGPU and Hardware Acceleration pages.

Manjaro should have AV1 encoding but Manjaro builds mesa without the non-free codecs, so you don't get any other encoding. They do offer a solution via nonfree.eu but they don't "support" it, and you have to be on Manjaro Unstable at which point you might as well just be running Arch Linux, so why not use Arch in that scenario and just cut out the middle man.
Linux Mint should be OK on the mesa front in terms of codecs, but you have a new GPU and Mint may be too LTS at the moment, especially with recent kernel and mesa updates containing things for AMD 7 series.

I know its a pain when these things don't work, and I'm not saying you've done anything wrong, I just wanted to add my two cents to the post to show that I have it working on an AMD 7800XT, in case people see your unfortunate experience and assume it just doesn't work.

If the pro drivers are working nicely for you then stick with them.

Arch Linux

#1 2024-03-03 14:38:06

Can't get hardware encoding to work on new amd gpu

#2 2024-03-03 19:33:36

Re: Can't get hardware encoding to work on new amd gpu

#3 2024-03-03 19:40:10

Re: Can't get hardware encoding to work on new amd gpu

#4 2024-03-04 08:20:24

Re: Can't get hardware encoding to work on new amd gpu

#5 2024-03-04 11:47:22

Re: Can't get hardware encoding to work on new amd gpu

#6 2024-03-04 12:50:18

Re: Can't get hardware encoding to work on new amd gpu

#7 2024-03-04 13:17:14

Re: Can't get hardware encoding to work on new amd gpu

Board footer