Check out the documentation on how to write custom profiles:
You can also refer some of the existing profiles in profiles.conf and customize it to use handbrake presets. Feel free to post back here with your custom profile and if there’s any issue you’re facing.
There are profiles for H.265, H.264 and lots of other types as well in profiles.conf