

The rate control mode for older release builds of FFmpeg configured with older release versions of ffnvcodec header packages only need the rate control option modified to constant bitrate via -rc:v vbr_hq. Depending on the specific version of FFmpeg built & the version of the ffnvcodec header package in use, ie either from a release version OR from git master, the following paramerers will need to change: See this answer on why these parameters were selected. However, the maximum bitrate and buffer size(s) remain set to match your settings in libx264. The bitrate is explicitly unset via -b:v 0 to comply with the constraints of the selected rate control mode in NVENC. For the rate control in use, I've selected variable bitrate mode (set via the private codec option -rc:v vbr) with constant quality parameter rate control set to a value of 21. Note that not all video formats are supported by NVDEC, and as such, a fallback to software-based decoding will be provided.

These can be verified with Philip Langdale's nv-video-info project's nvdecinfo's program, as documented in this answer, if you're on Linux. Hardware acceleration support for specific codecs are platform and driver-dependent.

For their usability, please see ffmpeg -h filter=scale_npp and ffmpeg -h filter=scale_cuda respectively. The availability of these filters depends on how FFmpeg was configured, as explained below. Scaling operations are done with either scale_npp OR scale_cuda, as these run purely on the GPU. On a multi-GPU system, select a valid GPU as listed by nvidia-smi. GPU selection with NVDEC hardware acceleration is toggled via the global option -hwaccel_device 0 and for the encoder, via h264_nvenc's private codec option -gpu 0. See this for more information on NVENC capabilities, including the hardware acceleration infrastructure available to FFmpeg on capable NVIDIA hardware. The default GPU selected for NVENC is 0, and that the GPU is NVENC capable. Try these commands with the following assumptions:
