Support enabling VAAPI at runtime

[shmerl]

Enabling VAAPI at runtime with /gdi:hw.

See #9791.

[freerdp-bot]

Can one of the admins verify this patch?

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[akallabeth]

@shmerl oh, and no need to close the pr.
new commits to your branch will show up here and I suggest you do a cleanup of commit history if you are ready and force push before we merge.

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[shmerl]

Some mistake, I'll redo it.

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[akallabeth]

@shmerl there might still be a bug in the current vaapi implementation, so you need to debug that. (you only did do runtime on/off in this pr so far)

I did find this: https://intel.github.io/libva/group__api__core.html#gga2c3be94ce142fb92a4bf93e9b1b4fa01aa88d632c8c071d0899b09197087e4906

that we might need to query if the resolution can be handled by the hardware accelerator.
but bad thing: on my debian 12 these attributes are currently not supported :/

`vainfo -a`

vainfo -a
libva info: VA-API version 1.17.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_17
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.17 (libva 2.12.0)
vainfo: Driver version: Mesa Gallium driver 22.3.6 for AMD Radeon Pro WX 3200 Series (polaris12, LLVM 15.0.6, DRM 3.54, 6.6.13+bpo-amd64)
vainfo: Supported config attributes per profile/entrypoint pair
VAProfileMPEG2Simple/VAEntrypointVLD
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV422

VAProfileMPEG2Main/VAEntrypointVLD
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV422

VAProfileVC1Simple/VAEntrypointVLD
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV422

VAProfileVC1Main/VAEntrypointVLD
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV422

VAProfileVC1Advanced/VAEntrypointVLD
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV422

VAProfileH264ConstrainedBaseline/VAEntrypointVLD
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV422

VAProfileH264ConstrainedBaseline/VAEntrypointEncSlice
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
    VAConfigAttribRateControl              : VA_RC_CBR
                                             VA_RC_VBR
                                             VA_RC_CQP
    VAConfigAttribEncPackedHeaders         : VA_ENC_PACKED_HEADER_NONE
    VAConfigAttribEncMaxRefFrames          : l0=1
                                             l1=0
    VAConfigAttribEncMaxSlices             : 1

VAProfileH264Main/VAEntrypointVLD
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV422

VAProfileH264Main/VAEntrypointEncSlice
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
    VAConfigAttribRateControl              : VA_RC_CBR
                                             VA_RC_VBR
                                             VA_RC_CQP
    VAConfigAttribEncPackedHeaders         : VA_ENC_PACKED_HEADER_NONE
    VAConfigAttribEncMaxRefFrames          : l0=1
                                             l1=0
    VAConfigAttribEncMaxSlices             : 1

VAProfileH264High/VAEntrypointVLD
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV422

VAProfileH264High/VAEntrypointEncSlice
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
    VAConfigAttribRateControl              : VA_RC_CBR
                                             VA_RC_VBR
                                             VA_RC_CQP
    VAConfigAttribEncPackedHeaders         : VA_ENC_PACKED_HEADER_NONE
    VAConfigAttribEncMaxRefFrames          : l0=1
                                             l1=0
    VAConfigAttribEncMaxSlices             : 1

VAProfileHEVCMain/VAEntrypointVLD
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV422

VAProfileHEVCMain/VAEntrypointEncSlice
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
    VAConfigAttribRateControl              : VA_RC_CBR
                                             VA_RC_VBR
                                             VA_RC_CQP
    VAConfigAttribEncPackedHeaders         : VA_ENC_PACKED_HEADER_SEQUENCE
    VAConfigAttribEncMaxRefFrames          : l0=1
                                             l1=0
    VAConfigAttribEncMaxSlices             : 1

VAProfileHEVCMain10/VAEntrypointVLD
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV422
                                             VA_RT_FORMAT_YUV420_10
                                             VA_RT_FORMAT_YUV420_10BPP

VAProfileJPEGBaseline/VAEntrypointVLD
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV422
                                             VA_RT_FORMAT_YUV400

VAProfileNone/VAEntrypointVideoProc
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV420_10
                                             VA_RT_FORMAT_RGB32
                                             VA_RT_FORMAT_YUV420_10BPP

[shmerl]

The weird thing also is that with freerdp2, it worked a while ago on my set up (just when adding -DWITH_VAAPI=ON). So something must have changed between versions? Or may be it's something with underlying VAAPI / ffmpeg itself that changed.

[shmerl]

In my case, VAConfigAttribMaxPictureWidth gives quite a lot (4096), so I doubt that's the problem in my case (my window size is smaller test). But I can try checking what's going on when I'll have time.

[shmerl]

Does it also fail for you with a small resolution?

[akallabeth]

@shmerl don´t have time to test, working on something else currently, sorry.
but the h264 module should not have changed much since 2.x so maybe you can do a diff?

[hardening]

@akallabeth on my host, vainfo also doesn't report limits for VAAPI context, but IIRC ffmpeg was able to query these limits by other means (at least last time I've touched that). IIRC it was even possible to try HW decoding and fallback to software if beyond the limits...