You might try altering the profile so that ffmpeg is first. When I was working with my profiles during the .8 beta, I couldn’t get handbrake to do HW acceleration in any meaningful capacity. FWIW, I’ve got the same CPU and RAM and am using a NVidia GTX 1060 - I’m getting H265 (HEVC) encodes of about 20 minutes for an hour of HD programming.
To change the order of the encoding engines, go into the config folder and pull up profiles.config. Look for this line in the MP4 Normal profile:
order=handbrake,ffmpeg,mencoder
Change it so ffmepg is first.
Good luck! Will.