You can query ffmpeg for the list of available options for each encoder, here’s what shows for h264_nvenc
As you can see there are 2 options -cq (which is what we want, constant quality) and -qp (which is NOT what we want as that’s a constant quantizer which is pretty useless).
You may need to set -rc vbr also but it you can try it out with and without it and let me know how it goes.
-rc E…V… Override the preset rate-control (from -1 to INT_MAX) (default -1)
constqp E…V… Constant QP mode
vbr E…V… Variable bitrate mode
cbr E…V… Constant bitrate mode
vbr_minqp E…V… Variable bitrate mode with MinQP
ll_2pass_quality E…V… Multi-pass optimized for image quality (only for low-latency presets)
ll_2pass_size E…V… Multi-pass optimized for constant frame size (only for low-latency presets)
vbr_2pass E…V… Multi-pass variable bitrate mode
-rc-lookahead E…V… Number of frames to look ahead for rate-control (from -1 to INT_MAX) (default -1)
-surfaces E…V… Number of concurrent surfaces (from 0 to 64) (default 32)
-cbr E…V… Use cbr encoding mode (default false)
-2pass E…V… Use 2pass encoding mode (default auto)
-gpu E…V… Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on. (from -2 to INT_MAX) (default any)
any E…V… Pick the first device available
list E…V… List the available devices
-delay E…V… Delay frame output by the given amount of frames (from 0 to INT_MAX) (default INT_MAX)
-no-scenecut E…V… When lookahead is enabled, set this to 1 to disable adaptive I-frame insertion at scene cuts (default false)
-forced-idr E…V… If forcing keyframes, force them as IDR frames. (default false)
-b_adapt E…V… When lookahead is enabled, set this to 0 to disable adaptive B-frame decision (default true)
-spatial-aq E…V… set to 1 to enable Spatial AQ (default false)
-temporal-aq E…V… set to 1 to enable Temporal AQ (default false)
-zerolatency E…V… Set 1 to indicate zero latency operation (no reordering delay) (default false)
-nonref_p E…V… Set this to 1 to enable automatic insertion of non-reference P-frames (default false)
-strict_gop E…V… Set 1 to minimize GOP-to-GOP rate fluctuations (default false)
-aq-strength E…V… When Spatial AQ is enabled, this field is used to specify AQ strength. AQ strength scale is from 1 (low) - 15 (aggressive) (from 1 to 15) (default 8)
-cq E…V… Set target quality level (0 to 51, 0 means automatic) for constant quality mode in VBR rate control (from 0 to 51) (default 0)
-aud E…V… Use access unit delimiters (default false)
-bluray-compat E…V… Bluray compatibility workarounds (default false)
-init_qpP E…V… Initial QP value for P frame (from -1 to 51) (default -1)
-init_qpB E…V… Initial QP value for B frame (from -1 to 51) (default -1)
-init_qpI E…V… Initial QP value for I frame (from -1 to 51) (default -1)
-qp E…V… Constant quantization parameter rate control method (from -1 to 51) (default -1)