as you all know, obs out of the box doesn't support recording with nvenc hevc (h265) so i've tried some stuff and finally got it working.

first of all you need latest ffmpeg binaries
  • (32bit)
  • (64bit)

next, you need to paste the contents of ffmpeg's bin/ folder to obs' 32bit/64bit bin/. load up your obs, or restart it if you had it open while you were replacing libs. go to recording tab, select container mp4 as it's the only container that supports x265 and select nvenc_hevc (libx265) from the list and there you have it. you can record with nvenc hevc.

you can check out my recordings below.

my sys specs:
> 8gb ddr4 @2666mhz
> nvidia gtx 960 (4gb ram)
> i5-6600 (skylake, nonclocked)

youtube preview

> QL
> 1:08 length, 126mb file size
> 49000 bitrate

youtube preview

> 3:45 legth, 994mb file size
> 49000 bitrate

if you want to stream with nvenc hevc, only thing you can do is record gameplay and stream the recording. twitch (or any other service that i know of) doesn't support x265 streaming anyway.