Try just using hevc-10bit for now at 130mbps bitrate. Also, check your win10/11 graphics settings and disable hardware acceleration graphics (HAGS). Probably also best to turn game mode off. Also best to run msfs2020 in the stock default dx11 mode, not experimental dx12 because this can cause stability issues.
Nvidia control panel 3d global settings all default except power = prefer max power and change the Quality default below this to High Performance.
Sorry, all I can think of right now. You may also want to go to the VD discord channel for more help. Good luck mate and cheers.