AMD AV1 Hardware encoding not working

Nope. Only the 6000 series (mobile) and 7000 series (desktop) CPUs/APUs have integrated GPU HW AV1 support.

From AMD GPU Decoder Device Information
and [Reddit r/AV1] List of CPU SOC with AV1 support
The AMD APUs (CPU with integrated GPU) with HW AV1 support are:

  • Radeon Graphics (Rembrandt/Raphael/Mendocino)
  • Radeon 610M
  • Radeon 780M

This is how AMD defines AV1 support from the GPU graphics technology viewpoint:

  • RDNA2 = AV1 decode (RX 6800 GPUs)
  • RDNA3 = AV1 decode/encode (RX 7900 GPUs)

Here is how that maps from the GPU to the AMD APU Radeon iGPU features:
Source: AMD Reveals Mendocino Zen 2 APUs With RDNA 2 Graphics | Tom's Hardware

  • Rembrandt = Ryzen 6000 “premium” mobile CPUs = Zen3+ with RDNA2 = AV1 decode only

  • Mendocino = Ryzen 6020 “budget” mobile CPUs = Zen2 with RDNA2 = AV1 decode only

  • Raphael = Ryzen 7000 desktop CPUs = Zen4 with RDNA3 = AV1 decode and encode

Your APU, the Ryzen 7 5800H is a mobile CPU (Cezanne) with a Zen3 core with the Vega 8 iGPU technology. Notably, this is not RDNA-based, and as a result, all of your AV1 encoding and decoding are being done by the CPU, and not by the native iGPU. In short, no HW AV1 support. :cry:

Sorry about the news you probably didn’t want to hear.
And yes, AMD’s CPU numbering system is just a hot mess.
Here is the Anandtech secret decoder ring.