Unable to use Vaapi hardware decoding for all videos with resolution larger than 4096*4096.

[strayedstar]

I want to know that vaapi support hardware decoding h264/HEVC video which larger than 4096*4096 resolution?
I had installed all of libva and mesa drivers but VDPAU backend with VAAPI.
My platform:
OS:Arch Linux
WM:Kwin_wayland
Kernel:6.4.2-zen1-1-zen
vainfo:
Trying display: wayland
vainfo: VA-API version: 1.18 (libva 2.18.2)
vainfo: Driver version: Mesa Gallium driver 23.1.3 for AMD Radeon RX 6800 XT (navi21, LLVM 15.0.7, DRM 3.52, 6.4.2-zen1-1-zen)
vainfo: Supported profile and entrypoints
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileVC1Simple : VAEntrypointVLD
VAProfileVC1Main : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointEncSlice
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileVP9Profile0 : VAEntrypointVLD
VAProfileVP9Profile2 : VAEntrypointVLD
VAProfileAV1Profile0 : VAEntrypointVLD
VAProfileNone : VAEntrypointVideoProc
My graphics card is capable of supporting 8K video decoding.
I try to play 2K HEVC video with vlc:
[00005651eb1a8f30] main playlist: playlist is empty
uint DBusMenuExporterDBus::GetLayout(int, int, const QStringList&, DBusMenuLayoutItem&): Condition failed: menu
uint DBusMenuExporterDBus::GetLayout(int, int, const QStringList&, DBusMenuLayoutItem&): Condition failed: menu
[00007f8f70007260] gl gl: Initialized libplacebo v5.229.2 (API v229)
[00007f8f84c19910] avcodec decoder: Using Mesa Gallium driver 23.1.3 for AMD Radeon RX 6800 XT (navi21, LLVM 15.0.7, DRM 3.52, 6.4.2-zen1-1-zen) for hardware decoding

try to play 4K H264 video:
[00007fc5401c66b0] vaapi generic error: vaCreateContext: resolution not supported
[00007f12b4007260] gl gl: Initialized libplacebo v5.229.2 (API v229)
[00007f12dc203080] vaapi generic error: vaCreateContext: resolution not supported
[00007f12b423ed60] gl gl: Initialized libplacebo v5.229.2 (API v229)
Failed to open VDPAU backend libvdpau_nvidia.so: 无法打开共享对象文件: 没有那个文件或目录[00007f12b40eda00] gl gl: Initialized libplacebo v5.229.2 (API v229)

vaCreateContext: resolution not supported!
Is the support for VAAPI on AMD GPU much lower than AMF on Windows, or is it an issue with my environment?

[Dmitry Rogozhkin]

VAAPI (API itself) does support resolutions more than 4K, but actual limitation for max supported resolution is dictated by:

1. Capabilities of underlying GPU
2. Limitations queried by or specified by the driver handling this GPU

In you are using AMD Radeon RX 6800 XT you need to check capabilities of this GPU (I am not aware of them - sorry) and how these caps were handled by the driver, i.e. Mesa Gallium. If you believe AMD GPU does support codec/resolution you have in mind, then most likely that's some issue with Mesa Gallium driver in which case you need to report issue in their DB - https://gitlab.freedesktop.org/mesa/mesa/-/issues.

For the reference, you can check capabilities exposed by driver/GPU with vainfo -a. For example, see a snapshot (I posted couple related dumps for decoders) on Intel Skylake. You can see that different codecs support different max resolutions, some exceeding 4K:

$ vainfo -a
<...>
VAProfileMPEG2Main/VAEntrypointVLD
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV422
                                             VA_RT_FORMAT_RGB32
    VAConfigAttribDecSliceMode             : VA_DEC_SLICE_MODE_NORMAL
    VAConfigAttribDecProcessing            : VA_DEC_PROCESSING_NONE
    VAConfigAttribMaxPictureWidth          : 2048
    VAConfigAttribMaxPictureHeight         : 2048
    VAConfigAttribProcessingRate           : VA_PROCESSING_RATE_DECODE
<...>
VAProfileH264Main/VAEntrypointVLD
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV422
                                             VA_RT_FORMAT_RGB32
    VAConfigAttribDecSliceMode             : VA_DEC_SLICE_MODE_NORMAL
                                             VA_DEC_SLICE_MODE_BASE
    VAConfigAttribDecProcessing            : VA_DEC_PROCESSING
    VAConfigAttribMaxPictureWidth          : 4096
    VAConfigAttribMaxPictureHeight         : 4096
    VAConfigAttribProcessingRate           : VA_PROCESSING_RATE_DECODE
<...>
VAProfileHEVCMain/VAEntrypointVLD
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV422
                                             VA_RT_FORMAT_RGB32
    VAConfigAttribDecSliceMode             : VA_DEC_SLICE_MODE_NORMAL
    VAConfigAttribDecProcessing            : VA_DEC_PROCESSING
    VAConfigAttribMaxPictureWidth          : 8192
    VAConfigAttribMaxPictureHeight         : 8192
    VAConfigAttribProcessingRate           : VA_PROCESSING_RATE_DECODE