As a workaround for me, I leave Dev mode on when exiting MSFS and after starting it again, I can turn off Dev mode without the FPS overlay coming back.
But I agree, pretty annoying issue. I’d actually love to have a hotkey or a separate on/off option for the FPS display since I don’t want to have Dev mode running to see the FPS overlay, but right now they are bound to each other AND have this annoying bug on top of it.
Having separate options for them would make both better.